Multimodal Interaction and Communication

Principal Investigators:

Rasha Abdel Rahman
Verena Hafner
John-Dylan Haynes

Team members:

Anna Eiserbeck (Doctoral researcher)
Olga Wudarczyk (Postdoctoral researcher)
Antje Lorenz (Postdoctoral researcher)
Anna Kuhlen (external collaborator)
Murat Kirtay (external collaborator)
Doris Pischedda (external collaborator)
 

Creating a robot that can integrate information from different sources and modalities

Research Unit 1, SCIoI Project 09

The overall goal of this project is to create a robot that can represent and integrate information from different sources and modalities for successful, task-oriented interactions with other agents. To fully understand the mechanisms of social interaction and communication in humans and to replicate this complex human skill in technological artifacts, we must provide effective means of knowledge transfer between agents. The first step of this project is therefore to describe core components and determinants of communicative behavior including joint attention, partner co-representation, information processing from different modalities and the role of motivation and personal relevance (Kaplan, and Hafner, 2006; Kuhlen & Abdel Rahman, 2017; Kuhlen et al., 2017). We will compare these functions in human-human, human-robot, and robot-robot interactions to identify commonalities and differences. This comparison will also consider the role of different presumed partner attributes (e.g., a robot described as “social” or “intelligent”). We will conduct behavioral, electrophysiological, and fMRI experiments to describe the microstructure of communicative behavior.

The second step of the project is to create predictive models for multimodal communication that can account for these psychological findings in humans. Both the prerequisites and factors acting as priors will be identified, and suitable computational models will be developed that can represent multimodal sensory features in an abstract but biologically inspired way (suitable for extracting principles of intelligence; Schillaci et al., 2013). In perspective, the third step of this project is to use these models to generate novel predictions of social behavior in humans.

Throughout the project we will focus on the processing of complex multimodal information, a central characteristic of social interactions, that have nevertheless thus far been investigated mostly within modalities. We assume that multimodal information, e.g. from auditory (speech) and visual (face, eye gaze) or tactile (touch) information, will augment the partner co-representation and will therefore improve communicative behavior.

 

Related Publications

Yun, H. S., Taliaronak, V., Kirtay, M., Chevelère, J., Hübert, H., Hafner, V. V., Pinkwart, N., & Lazarides, R. (2022). Challenges in Designing Teacher Robots with Motivation Based Gestures. 17th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI 2022). https://drive.google.com/file/d/1IUxuJMiReGpGnYvaXa918lWF_t11aRLN/view
Yun, H. S., Hübert, H., Taliaronak, V., Mayet, R., Kirtay, M., Hafner, V. V., & Pinkwart, N. (2022). AI-based Open-Source Gesture Retargeting to a Humanoid Teaching Robot. AIED 2022: The 23rd International Conference on Artificial Intelligence in Education. https://link.springer.com/chapter/10.1007/978-3-031-11647-6_51
Wudarczyk, O. A., Kirtay, M., Pischedda, D., Hafner, V. V., Haynes, J.-D., Kuhlen, A. K., & Abdel Rahman, R. (2021). Robots facilitate human language production. Scientific Reports, 11(1), 16737. https://doi.org/10.1038/s41598-021-95645-9
Wudarczyk, O. A., Kirtay, M., Kuhlen, A. K., Abdel Rahman, R., Haynes, J.-D., Hafner, V. V., & Pischedda, D. (2021). Bringing Together Robotics, Neuroscience, and Psychology: Lessons Learned From an Interdisciplinary Project. Frontiers in Human Neuroscience, 15. https://doi.org/10.3389/fnhum.2021.630789
Spatola, N., & Wudarczyk, O. A. (2020). Implicit Attitudes Towards Robots Predict Explicit Attitudes, Semantic Distance Between Robots and Humans, Anthropomorphism, and Prosocial Behavior: From Attitudes to Human–Robot Interaction. International Journal of Social Robotics. https://doi.org/10.1007/s12369-020-00701-5
Pischedda, D., Lange, A., Kirtay, M., Wudarczyk, O. A., Abdel Rahman, R., Hafner, V. V., Kuhlen, A. K., & Haynes, J.-D. (2021). Who is my interlocutor? Partner-specific neural representations during communicative interactions with human or artificial task partners. 5th Virtual Social Interactions (VSI) Conference.
Pischedda, D., Lange, A., Kirtay, M., Wudarczyk, O. A., Abdel Rahman, R., Hafner, V. V., Kuhlen, A. K., & Haynes, J.-D. (2021). Am I speaking to a human, a robot, or a computer? Neural representations of task partners in communicative interactions with humans or artificial agents. Neuroscience 2021.
Kirtay, M., Oztop, E., Kuhlen, A. K., Asada, M., & Hafner, V. V. (2022). Forming robot trust in heterogeneous agents during a multimodal interactive game. 2022 IEEE International Conference on Development and Learning (ICDL), 307–313. https://doi.org/10.1109/ICDL53763.2022.9962212
Kirtay, M., Oztop, E., Asada, M., & Hafner, V. V. (2021). Trust me! I am a robot: an affective computational account of scaffolding in robot-robot interaction. 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), 189–196. https://doi.org/10.1109/RO-MAN50785.2021.9515494
Kirtay, M., Oztop, E., Asada, M., & Hafner, V. V. (2021). Modeling robot trust based on emergent emotion in an interactive task. 2021 IEEE International Conference on Development and Learning (ICDL), 1–8. https://doi.org/10.1109/ICDL49984.2021.9515645
Kirtay, M., Hafner, V. V., Asada, M., & Oztop, E. (2023). Trust in robot-robot scaffolding. IEEE Transactions on Cognitive and Developmental Systems. https://doi.org/10.1109/TCDS.2023.3235974
Kirtay, M., Oztop, E., Kuhlen, A. K., Asada, M., & Hafner, V. V. (2022). Trustworthiness assessment in multimodal human-robot interaction based on cognitive load. 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 469–476. https://doi.org/10.1109/RO-MAN53752.2022.9900730
Kirtay, M., Chevalère, J., Lazarides, R., & Hafner, V. V. (2021). Learning in Social Interaction: Perspectives from Psychology and Robotics. 2021 IEEE International Conference on Development and Learning (ICDL), 1–8. https://doi.org/10.1109/ICDL49984.2021.9515648
Kirtay, M., Wudarczyk, O. A., Pischedda, D., Kuhlen, A. K., Abdel Rahman, R., Haynes, J.-D., & Hafner, V. V. (2020). Modeling robot co-representation: state-of-the-art, open issues, and predictive learning as a possible framework. 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 1–8. https://doi.org/10.1109/ICDL-EpiRob48136.2020.9278031
Chevalère, J., Kirtay, M., Hafner, V., & Lazarides, R. (2022). Who to Observe and Imitate in Humans and Robots: The Importance of Motivational Factors. International Journal of Social Robotics. https://doi.org/10.1007/s12369-022-00923-9
Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) im Rahmen der Exzellenzstrategie des Bundes und der Länder – EXC-2002/1 – Projektnummer 390523135 .
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany ́s Excellence Strategy – EXC-2002/1 – Projectnumber 390523135.

Imprint     Contact     Press

Follow us
Our Newsletter