Social Signals in the Wild

Check out: Medium Blog Post, Full Paper, Code

Nonverbal communication is essential for seamless human robot interaction (HRI). Emotional expressions in particular have been highly studied in Human Robot Interaction (HRI), and emotion taxonomies from the psychology domain are typically used to represent the nonverbal affective cues expressed by a human [1].

Humans express their emotions through a variety of social signals. For example, anger can be expressed through a scowl, narrowed eyes, a long stare, or many other expressions. This complexity is problematic when attempting to recognize a human’s expression in a human-robot interaction: categorical emotion models used in HRI typically use only a few prototypical classes, and do not cover the wide array of expressions in the wild.

In this project, we use a data-driven method towards increasing the number of known emotion classes present in human-robot interactions, to 28 classes or more.

We also showcase our initial results using a large in-the-wild HRI dataset (UE-HRI) [2], with 61 clips randomly sampled from the dataset, labeled with 28 different emojis. In particular, our results showed that the "skeptical" emoji was a common expression in our dataset, which is not often considered in typical emotion taxonomies.

This is the first step in developing a rich taxonomy of emotional expressions and social signals that can be used in the future as labels for training machine learning models, towards more accurate perception of humans by robots.

[1] Admoni, Henny. "Nonverbal Communication for Human-Robot Interaction." ACM Transactions on Embedded Computing Systems.2014

[2] Ben-Youssef, Atef, et al. "UE-HRI: a new dataset for the study of user engagement in spontaneous human-robot interactions." Proceedings of the 19th ACM international conference on multimodal interaction. 2017.

[3] Saheb Jam, Ghazal, Jimin Rhim, and Angelica Lim. "Developing a Data-Driven Categorical Taxonomy of Emotional Expressions in Real World Human Robot Interactions." Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction. 2021.