What is affective computing / emotion AI?
Marco van Hurne
Partnering with the most innovative AI and RPA platforms to optimize back office processes, automate manual tasks, improve customer service, save money, and grow profits.
“Computing is not about computers anymore. It is about living.” — Nicholas Negroponte
What did you feel while seeing the price tag of a product in a supermarket? You may not realise it, but machines can read these incredibly nuanced subtilities of human expressions, and retailers can use it to their advantage. A device loaded with a specific software program can predict whether a person is smiling out of frustration or joy. Human-computer interaction has gone beyond the realms of sci-fi fantasies and turned into reality. The rise of emotionally aware machines has blurred the human-machine divide, and it is now redefining the way people experience technology.
In this article I'll be talking about Affective Computing, also called Emotion AI. It's all about how AI can decode the emotional status of a human being by analysing their facial expressions, such as head motion, facial distortions, movement of jaws, and speech pattern etc. It detects, recognises and emulates human emotions through a programmed AI neural network. Without a doubt, humans can analyse and interpret complex emotional signals better. However, the gap is narrowing faster than you can imagine, thanks to advancements in big data capability and powerful algorithms.
I believe that the only way that intelligent machines can exist is for machines to first be able to recognize, understand, and even be able to express human-like emotions. The reasoning behind this is that emotions play an important role in decision-making, perception, learning, and much more. It has been stated that affective computing or Emotion AI is the interdisciplinary field that entwines computer science, psychology, and cognitive science.
And, you could be thinking: Why we want computers to be able to do that? How computers could be able to do it? Two very important questions. Let’s elaborate on them.
Subscribe to the TechTonic Shifts newsletter
Why would we even want this in the first place
Because emotions signal what matters to us and what we care about; further it has been argue that emotions impacts our rational decision-making and action selection. Think about your decision-making when buying a new phone, a new laptop, a car, or selecting a Univerity, a new job in a new Company, etc. Are these emotion-driven or completely rational. By providing computers with the ability of understanding their user’s emotions a computer could show empathy –sense other’s emotions and react in consequence in a proper way. Imagine scenarios, such as: (1) a video game aware of your levels of engagement or excitement, it could adapt the difficulty level; or (2) a tutoring system or online course aware of your interest or boredom, it could decide which material to present to you next. Hope you get the idea –a lot of potential to improve human-computer interaction. Yes, we are aware of the possible issues including security. But, that is another story.
What is affective computing about anyway
Ok, we are talking about “affective” computing and we mention “emotions”. Let me clarify some important concepts. Disclaimer: I am going to oversimplify things — this is a computer science perspective on the topic. I am not an antropologist, nor a psychologist, nor a comedian although some people would disagree on the latter.
Affect is used as an encompassing term to describe emotions and moods. Affect refers to the underlaying experience of feeling. It is a construct of mental activity and physiological reactions.
Therefore, we talk about affective computing, affective states, affective signals, affect measurement, or affect recognition. Affect ranges from unpleasant to pleasant (valence), from agitated to calm (arousal), and from easy to hard to control (dominance).
Emotions are states of mind resulting of experiencing affect, i.e., experiencing chemicals released throughout our body and brain (physiological reactions) in response to our interpretation of a specific contextual stimulus (mental activity).
Computers do not care about the name or whether a name exists or not. Computer wants numbers, right? Think about using a camera as an input device (either using photos or video) and ask the computer to identify colors. We will work with RGB values. We can label some RGB vectors to make our live easier. The same apply to affetive states (emotions).
Psychological frameworks
Though technology and AI are essential aspects of Affective Computing, the psychological framework is critical to the overall understanding. A series of researchers, such as Paul Ekman, Lavenson and James Russel, proposed credible theories to firmly establish that human emotions can be separated with distinct physical signatures. Later, these theories became the cornerstone of Emotion AI technology. Among these theories, Ekman’s Discreet Model and the dimensional model of James Russel provided the critical foundation for the development of Affective Computing:
Ekman Discrete Model: Paul Ekman came with the concept of creation of seven basic emotions –anger, disgust, fear, happiness, sadness, and surprise, and a neutral expression which are created involuntarily by changing facial contours. Ekman’s model firmly established a correlation between facial expressions and emotional status.
Russel’s Dimensional Model For Emotion Classification: If Ekman’s model provided the qualitative foundation for Emotion AI, Russel’s model of emotions laid the quantitative foundation for Affective Computing technology. This model proposed that emotions, arousal and valence are distributed in a two-dimensional space. The dimensional distribution of emotions made it possible to gauge the intensity level of positive engagement (or positive arousal) or degree of negative disengagement (or negative arousal) and the degree of pleasantness (positive valence) or unpleasantness (negative valence).
How emotion AI tech works
The human face is a canvas for communicating different emotional expressions. Whether it’s love, anger, joy, surprise, sadness or fear, every human emotion sends facial signals whenever they occur. With the help of affective computing technology, these emotional signals can be deciphered precisely and quickly.
Here is how the emotion AI technology works. A simple explanation:
Webcam captures micro-level facial expressions data, and this is placed in the RAM of the device.
The computing device gathers cues about the emotional status by analysing facial expressions, such as head postures, speech patterns, and eye movements.
These images are analysed using computer vision, big data and AI to decipher nine major traits based on Ekman’s model, such as age, gender, pose, face detector, emotions, arousal, valence, attention, wish and other features.
These traits are designed in the form of a well-designed SDK. You can load the specific API based on your requirement. Though the architecture may vary in different Emotion AI products, the core concept remains the same.
Deep learning technology constantly works in the backend to provide necessary predictive output.
A specialist labels the emotions of a large number of images (and frame by frame in the case of videos), and the algorithm is trained to interpret this data accurately. The result obtained is compared with the manual labelling while taking errors into account.
Subscribe to the TechTonic Shifts newsletter
Which devices could do this
There are a lot of options! For example:
Brain-Computer interfaces, such as the Emotiv Headset. Brain-Computer interfaces capture the electromagnetic activity of the brain. Affective states (Pleasure, Arousal, Dominance) and therefore emotions can be related with particular values of electromagnetic activity in specific regions of our brain.
A camera, yes, a simple camera. We can use a camera to run facial recognition. And, in a face we can recognize gestures. Long story make short: there is a limited number of muscles in your face, and a limited number of movements they can do. Thus, there is a finite number of gestures common to any human face. And, some of these gestures correspond with a particular mental activity or physiological reactions, i.e., with experiencing an affective state (emotion). Similar claims can be done using a camera to recognize body postures.
Biometric sensors slike skin conductance sensors. These measure the electrical conductance of the skin, which changes with moisture level as an emotional response or heart rate sensors. These measure the heart rate and variability to infer stress levels and other emotional states.
Voice analysis tools like microphones which can capture voice fluctuations, tone, pace, and volume, which are processed to analyze emotional states. This technology is often used in call centers to assess customer satisfaction and agent performance. Or think of smart speakers and voice assistants: Devices like Amazon Echo or Google Home can use voice analysis to infer emotional states from the user’s vocal cues.
Or think of eye tracking and gaze analysis devices. Eye trackers analyze where a person is looking and the movement of the eyes, which can be indicators of cognitive and emotional states.
Computers can infer emotions from the data that we gather from one or more sources.
All these sensors will study the cues (body language, facial expressions, gestures, tonality of voice, physiological signals, and even brain activity) and infer an affective state.
The figure below depicts this process. One sensor gather data from one source, machine learning algorithms are applied to the collected data to infer an affect measurement (perception process). But, multiple monomodal measurements can be synchronized and fused (multimodal affect recognition).
How accurate are these inferences? Well, the goal is to make them as reliable as those infered by a human. Can you look to a person in the street and guess their emotional state? What about a friend or a person that you see everyday? It will be dificult or maybe impossible to have a 100% certainty of your inference, but the precision increase when you have more information or more time in contact with the person. The same is true for the computer!
What are the specifics of the machine learning algorithms applied to recognize affect? Well, that is the field of study for Emotion AI — Artificial Intelligence applied to the Affective Computing field. But, that is a future story.
What are leading companies in emotion detection?
The leading companies in this field and their number of employees to give a sense of their market presence and their specialization areas are detailed below. While larger companies are specializing towards specific solutions like marketing optimization, almost all companies in the space offer APIs for other companies to integrate affective computing into their solutions.
Some applications of Emotion AI
Marketing
Every marketer, at some stage, hears from some marketing guru that marketing should appeal to emotions. Until now, that was a vague, hard-to-measure concept. Now marketers have the ability to put numbers on perceived emotions as well:
Marketing communications: Businesses can analyze what makes their customers engaged and organize their communication strategies accordingly. For example, they can measure customer reactions to their campaigns, products, and services to optimize their marketing strategies.
Market research: Emotion AI can measure consumer reactions to new products and help companies understand what other products do well and what they should do to satisfy customers when they enter a new market.
Content optimization: Affective computing can also help businesses generate contents that resonate well with their customers.
Customer Service
Intelligent call routing: Businesses can detect angry customers from the beginning of the call, and such calls can be routed to more experienced and well-trained call agents.
Recommendations during calls: Emotion AI can also provide suggestions about handling customer calls based on similar speech patterns during the conversation.
Continuous improvement: Reviews are time-consuming and completed by only a small share of customers. Amazon sellers share that only 3-5% of their buyers leave product reviews. Like analyzing written reviews, emotion AI can also measure how effective the calls are and if the customer is satisfied at the end of the call by leveraging voice analysis. This data can be used to improve customer service even in cases where customers do not leave reviews.
Human Resources
Recruitment: Businesses can observe how stressful candidates are and how they communicate emotions during interviews to make better recruitment decisions. Unilever is one of the companies that is currently using emotion AI during job interviews. However, this requires interviewee approval for recording the interview, and HR teams shouldn’t rely too much on the accuracy of affective computing as people can express themselves differently.
Employee training: Affective computing can be used to train employees who interact directly with customers. Employees work with intelligent customer interaction simulations that evolve based on the employees’ responses and emotions, helping them improve their empathy and customer service skills.
Tracking employee satisfaction: HR teams can track employees’ stress and anxiety levels during the job and observe if they are satisfied with their current tasks and workload. However, it also brings an ethical issue of monitoring all employees during work hours and might require their consent to monitor their emotions continuously.
Healthcare
Patient care: A bot can be used not only to remind patients to take their medications but also to monitor their physical and emotional well-being daily to observe any problematic issues.
Medical diagnosis: Affective computing can leverage voice analysis to help doctors diagnose diseases like depression and dementia.
Counseling: Emotion AI can be used in counseling sessions to better track and understand mental states and help doctors support counselees more effectively.
Etc. Etc.
Subscribe to the TechTonic Shifts newsletter
Well, that's it for now. If you like my article, subscribe to my newsletter or connect with me. LinkedIn appreciates your likes by making my articles available to more readers.
Signing off - Marco