Artificial Intelligence (AI) systems have reshaped modern life in several remarkable ways. From customer service chatbots to address our grievances to virtual assistants like Alexa that remind us about a pending task on our to-do lists, AI is firmly established in our daily lives, transforming the way we work, communicate, or access information. Conversational AI, such as Alexa or Google Assistant, has made voice commands and natural-language queries commonplace, providing to millions of users the convenience of speaking instead of typing out queries.
Current AI systems, however, are not trained to recognize the emotional and nonverbal aspects of human communication, such as voice tone, facial expressions, and body language, which are key to grasping the user’s full emotional profile. Imagine having a voice assistant that plays music according to your mood, or an AI tutor that adapts study lessons to suit your interest and willingness to learn! Sounds fascinating, right? To manifest this idea in the real world, scientists have been developing sentient or emotionally intelligent AI—often explored under the domain of affective computing— with the intention of creating systems that can interpret and respond to human emotions.
Professor Shogo Okada from Japan Advanced Institute of Science and Technology (JAIST), who leads the Social Signal and Multimodal Interaction Laboratory there, is working on this crucial aspect of human-AI interactions. Professor Okada’s lab uses multimodal communication signals, including language presentations, speech signals, body language, and physiological signals like heart rate variability, sweat gland activity, and nerve stimulations, to understand human-AI and human-human social interaction patterns. Prof. Okada’s lab uses these multimodal signals from real-time experiments involving social interactions to collect data and train computational models that can assess human emotions accurately.
When AI reads between the lines (on our face)
Prof. Okada joined JAIST in 2017, after completing his MS and PhD from the Tokyo Institute of Technology, Japan. At JAIST, he started working on human-centric AI systems. He has published several research papers on different aspects of human relations, including human group interactions. For example, in 2014-2016, Prof. Okada collaborated with Prof. Daniel Gatica-Perez from EPFL, Switzerland, to study how AI systems can be trained to predict personality traits. By analyzing humans’ nonverbal behavior in both one-on-one interviews (dyadic) as well as group conversations, the team studied traits like leadership or BigFive personality traits. In this work, they used pattern recognition sensors to understand emotional states in individuals through different types of signals – voice, body posture, and facial expressions – occurring together. Using group interaction experiments, this study also analyzed how one person’s nonverbal communication patterns align with those of another person in the group. Explaining this further, Prof. Okada says, “In our experiments, we noted that when a person with high leadership skills started speaking, others in the group started gathering their gaze/attention towards him/her and stopped speaking. So, we used such paired behavioral patterns in the group to evaluate a person’s level of influence over the group, or their interpersonal relations. Training AI systems with this kind of multimodaldata may help us understand specific personal and interpersonal traits of people.”
In recent years, research has focused on AI systems that sense human emotions. However, most studies focus on the tone of voice and facial expressions. But humans may hide their true emotions by controlling such observable features. Physiological signals, such as heart rate variability, EDA: Electro Dermal Activity, nerve stimulation, etc., are ‘unobservable’ signs of emotional states and cannot be controlled, reflecting the users’ true emotions. Exploring this ‘multimodal’ sentiment, Prof. Okada’s 2023 study published in IEEE Transactions on Affective Computing revealed that a combination of ‘observable signals’ and ‘unobservable’ physiological signals best predicted the emotional states of users. This study may pave the way for emotionally intelligent AI systems, allowing for a more natural and satisfying human-AI interaction. Prof. Okada also adds that this technology has potential applications in the fields of education as well as for monitoring mental illness. By assessing a student’s state of excitement or boredom, AI may adapt its teaching routines for better educational outcomes. Similarly, by continuously interacting with the user, AI may assess variations in emotional states in patients with mental illnesses, helping them access timely therapeutic interventions.
Yet another interesting work conducted at Prof. Okada’s lab features using social signal pattern processing to develop an adaptive interview strategy. “We can all agree that answering questions during interviews is not always easy! Sometimes, we do not know enough to speak at length on a certain subject, or perhaps the interviewer expects us to explain more. In 2024, we published our work on using social signal recognition techniques to sense a speaker’s/interviewee’s willingness to speak, allowing conversational robots to adapt and change their interaction strategies,” explains Prof. Okada. This enables selection of appropriate interview questions based on the estimated willingness of the interviewee, resulting in effective interview strategies. Prof. Okada’s research on multimodal signal processing technology also extends to developing better educational AI that helps students improve their spoken English skills. This study demonstrated that utilizing multiple communication signals together resulted in a more accurate assessment of speaking skills. The technology may help tutors and students understand what specific behaviors may lead to improved clarity scores. Instead of relying on traditional assessment metrics, this framework identifies the specific aspects of speaking skills that require students’/tutors’ attention, for instance, the use of filler words (like hmm/uh) or insufficient eye contact– that needs to be improved for better spoken English.
‘Sen Tan’ Edge of Innovation: How does JAIST facilitate this?
One thing’s for sure – AI is going to play an indispensable role in all walks of human life. But Prof. Okada believes that our current understanding of AI systems in still ‘insufficient.’ The ‘sen tan’ (cutting-edge aspect) of his research is to address this by conducting interdisciplinary research on AI systems. By combining information science, psychology, linguistics, and social science, Prof. Okada intends to add a design of empathy onto existing AI systems. He believes this is the only way we can make AI a powerful medium to enhance innate human capabilities.
JAIST plays a critical role in facilitating such innovations. With a campus that is nestled along the mountainside, far away from the bustling cities of Japan, JAIST provides the ideal environment for creative thinking and research. According to Prof. Okada, this idyllic environment not only gives students and faculty a peaceful space but also cuts out distractions. Also, the number of students at JAIST is relatively low, giving them better personal access to facilities like supercomputers. The support structure at JAIST is commendable, converging freedom to collaborate with researchers from all over the world and exceptionally supportive administrative staff who help you manage research budgets – allowing scientists and students more time and energy to focus on research.
Inspiration for a futuristic vision of the world
Prof. Okada looks up to visionaries like Professor Geoffrey Hinton, who is also known as the “Godfather of AI” for his work on artificial neural networks. Prof. Okada acknowledges that when it comes to sentient AI systems, there are differences of opinion among AI researchers globally. While some scientists believe that AI systems that can feel and respond to humane emotions might take over the world, Prof. Okada believes that it is important to reflect whether AI can ever cultivate the intrinsic motivation that human beings have. Moreover, contemplating on how, why, and for what we use AI is critical. For example, if AI systems can positively influence human behavior by encouraging interpersonal interactions rather than social isolation, it could enhance the quality of life for a large portion of the aging population, as well as for young adults who often find themselves isolated in the maze of social media.
Concluding his thoughts with a fictional analogy that he draws inspiration from, Prof. Okada says, “The way I think of sentient AI systems is inspired by the famous Japanese manga series “Doraemon”! The companionship of the lazy but kind-hearted 10-year-old – Nobita Nobi – and the blue robotic cat from the 22nd century – Doraemon – is how I envision human-AI relationships to evolve in the future. Just as Doraemon helps Nobita overcome his laziness to unlock his potential, I hope our relationship with AI systems will reflect the same kind of collaboration and personal growth.”
***
About Japan Advanced Institute of Science and Technology, Japan
Founded in 1990 in Ishikawa prefecture, the Japan Advanced Institute of Science and Technology (JAIST) was the first independent national graduate university that has its own campus in Japan to carry out research-based graduate education in advanced science and technology. The term “Advanced” in JAIST’s name reflects the Japanese term “Sen Tan,” meaning “cutting-edge,” representing the university’s focus on being at the forefront of innovative research and education. Now, after 30 years of steady progress, JAIST has become one of Japan’s top-ranking universities. JAIST aims to foster capable leaders through its advanced education and research curricula. About 40% of JAIST’s alumni are international students. The university has a unique style of graduate education to ensure that students have a thorough foundation to build cutting-edge research and technology in the future. JAIST also works closely with both local and overseas academic and industrial communities, promoting industry–academia collaborative research.
Website: https://www.jaist.ac.jp/english/
