Enhancing intelligence in multimodal emotion assessments

摘要

Computer systems are a part of everyday life, since they influence human behavior and stimulate changes in the emotional states of the users. The assessment of users’ emotions during their interaction with computer systems can help to provide tailorable website interfaces and better recommendations systems. However, emotions are complex and difficult to identify or assess. Previous studies have shown that, in a real-world scenario, the use of single sensors do not provide an accurate emotional assessment. Hence, in this study, we propose a framework that takes into account multiple sensors so that conclusions can be drawn about the emotional state of the user at the time of interaction. The proposed multi-sensing approach includes several inputs from users (such as speech, facial movements, and everyday activities), and uses an artificial intelligent strategy to map these different responses into one or more emotional states. The Componential Emotion Theory and Scherer’s Emotional Semantic Space are used to underpin the theoretical framework. The experimental results show that the combination of outputs generated by multiple sensors provides a more accurate assessment of emotional states than when the sensors are treated individually.