Approximately 300 million individuals, which represents around 4% of the global population, suffer from different forms of depression. However, identifying this mental health issue can be challenging, especially when affected individuals either choose not to share their feelings with friends, family, or healthcare professionals.
Currently, Professor Sang Won Bae from Stevens Institute is engaged in developing various applications powered by artificial intelligence that could warn individuals about the possibility of developing depression in a non-invasive manner.
Bae remarks, “Depression presents a significant challenge, and our goal is to assist in addressing it.”
“Given that most people regularly use smartphones nowadays, this could serve as an effective detection tool that is readily accessible.”
Capturing mood through eye images
One project that Bae is working on, along with Stevens doctoral student Rahul Islam, is called PupilSense. This system continuously captures images and measures the pupils of users while they interact with their smartphones.
Bae explains, “Over the last thirty years, research has consistently shown a link between pupil responses and episodes of depression.”
The system can accurately measure the diameter of pupils by comparing them to the surrounding irises during 10-second “burst” photo sessions taken while users unlock their phones or open various social media and other applications.
In an initial test conducted over four weeks with 25 volunteers, the embedded system collected approximately 16,000 phone interactions based on pupil image data. After training an AI to distinguish between typical pupil responses and those indicating potential issues, Bae and Islam analyzed the data alongside the volunteers’ self-reported moods.
The most successful version of PupilSense, referred to as TSF and utilizing only select high-quality data, demonstrated a 76% accuracy rate in identifying periods when participants felt depressed. This performance outmatches the leading smartphone-based depression detection system, known as AWARE.
Bae adds, “We plan to further advance this technology now that we have validated the concept,” noting her prior work on smartphone systems to predict binge drinking and cannabis consumption.
The technology was first introduced at the International Conference on Activity and Behavior Computing in Japan in the late spring and is currently available as open-source software on GitHub.
Facial expressions reveal signs of depression
In addition, Bae and Islam are creating another system named FacePsy that analyzes facial expressions to gauge emotions.
“An increasing number of psychological research findings indicate that depression is marked by nonverbal cues like facial movements and head tilts,” Bae highlights.
FacePsy operates in the background of a smartphone, capturing facial images whenever the device is unlocked or common apps are accessed. Importantly, the system deletes these images shortly after they are analyzed to ensure user privacy.
Bae explains, “Initially, we were uncertain which facial expressions or eye movements would relate to reported depression. While some correlations were anticipated, others were unexpected.”
For example, the study found that increased smiling did not align with feelings of happiness, but instead indicated possible signs of a depressive mood.
“This could be a way of coping, where individuals put on a ‘brave face’ for themselves and others despite struggling internally,” Bae suggests. “Alternatively, this could stem from the specific nature of the study, indicating a need for more research.”
Other signals identified early on included reduced facial movement during the morning and specific patterns of eye and head movements. For instance, yawning or side-to-side motions during morning hours appeared strongly linked to heightened depressive symptoms.
Interestingly, a greater likelihood of eyes being wide open during morning and evening also correlated with signs of depression, indicating that expressions of alertness or happiness might sometimes conceal underlying depressive feelings.
Bae concludes, “Unlike other AI-driven systems for detecting depression that require wearing one or multiple devices, we believe this FacePsy pilot study marks a promising initial step toward a simple, affordable, and user-friendly diagnostic solution.”
The FacePsy pilot study’s results will be shared at the ACM International Conference on Mobile Human-Computer Interaction (MobileHCI) in Australia in early October.
