Social interactions are supported by the integration of multimodal signals. Successful social interaction and communication relies on the integration of multimodal information, social signals and the orchestration of different cognitive processes. In project 9, the team studied partner co-representation (i.e., the representation of the partner’s actions alongside one’s own actions), emotion processing, theory of mind (i.e., the ability to consider mental states – such as beliefs, desires, intentions – to predict people’s behaviour) and trust. They studied processes underlying social communication in humans and assessed potential changes in these processes when the interaction partner is an artificial agent. The team used this knowledge to implement similar mechanisms in their robots and assess how this affects their performance along other dimensions, such as trust or scaffolding with the ultimate goal to create robots with higher social intelligence that can interact successfully with humans and other agents.

On the analytic side, the team was able to characterize core components of communicative behaviour, such as partner co-representation, and proposed a promising framework to implement such modules in robots. They were also able to compare cognitive processes underlying communicative social interactions between human-human, human-robot, and human-computer, and to identify brain structures that encode information about the nature of the task partner. They could also show a top-down modulation of the effect as this was more pronounced in people who believed they were actually interacting with the partners as compared to those who didn’t. Moreover, they found that the attribution of social attributes to humanoid robots was significantly higher than to computers. Overall, these results suggest that people mentalize about the robot to some extent (e.g., by simulating/predicting their behavior) but not about the computer. Focusing on emotional speech processing, they found that visual cues of communicative intent (open eyes, direct gaze, and mouth movements) enhance emotional word processing with human speakers, but not with robot speakers, regardless of their degree of human-like appearance. Instead, differences were attributable to the degree of perceived animacy of the agent.

In addition, the team introduced the Brain-Based Turing Test, a novel test to discriminate between humans and artificial agents based on information implicitly encoded in the brain. They also investigated which features people use to distinguish between human and artificial agents in written interactions between humans or humans and chatbots and found high variability among participants in the characteristics that prompt them to decide for one or the other.

On the synthetic side, the team computationally modeled the selected core components of multimodal interaction and communication, including audiovisual associative memories for perceiving the environment, internal reward for generating actions, and a simple theory of mind model for determining partner specifics. The models were employed for playing interactive games, partner reliability assessment (i.e., selecting a trustworthy partner to perform a task), and partner-specific adaptation (i.e., differentiating the partner’s skills to complete a task). They noted that these tasks were carried out with the following interactive settings: 1) human-robot interaction, 2) robot-robot interaction, and 3) robot-multi-robot interaction.

What has the team learned about intelligence? How should the scientific output of this project affect future projects within SCIoI?

Social intelligence can be described as an orchestration of many different cognitive or mental functions and their interplay that support flexible navigations in the social world. Socially intelligent behavior can be best understood (and synthesized) by adopting such a holistic perspective.

The team also learned that there is high variability in the features that trigger the attribution of intelligence to artificial systems (reflecting the multifaceted nature of the concept of intelligence) and this should be taken into account when synthesizing intelligent agents to make sure everyone has pleasant experiences when interacting with them.

The computational models developed within the synthetic part of the project are well aligned with the definition of SCIoI’s intelligent behavior[1]. To be concrete, for example, these models enable synthetic agents (humanoid robots) to minimize the cost of perceptual processing (i.e., cost-effective) to select a reliable interaction partner that yields minimal cognitive load (i.e., goal-directed) on the robot. The team conducted experiments in a real-world setting where environmental noise was non-negligible in achieving the task.

[1] From SCIoI white paper: “Behavior is intelligent if it is adaptable, general, cost-effective, goaldirected, and can be performed in the real world.