Analytic side:

One of the results observed was that higher-achieving students who worked with an ITS that adapted to both their in-system performance and emotional experience experienced less boredom than lower achieving students and students working with an ITS that adapted to their in-system performance only. These findings suggest possible benefits of using emotional information to increase the contingency of the hint delivery strategy and improve students’ learning experience. Thus, learners use ITS differently and therefore effectiveness of these systems varies between students. The team identified distinct behavioral patterns of students in an ITS which were differentially advantageous for motivational-affective characteristics and that were related to students’ characteristics (performance level) and the kind of adaptivity of the artificial learning companion (emotionally and cognitively adaptive vs. cognitively adaptive only). These findings emphasize the relevance of identifying interindividual differences in learners’ behavior when interacting with artificially intelligent learning companions. Theoretical concepts of adaptivity thus should not only consider adapting instruction to learners’ needs, but also supporting students in the learning process effectively as not everyone might benefit from such settings equally.

Synthetic and Analytic side:

Learners’ perception of robotic social agents was associated with emotional experience during learning as well as with task performance. More specifically, learners who perceived the robotic tutor agent as less disturbing during social interaction in a learning situation experienced more initial on-task enjoyment while learning and students who perceived the robotic tutor agent as less sociable performed better in the task. On-task enjoyment was associated with higher task performance. Thus, although robotic tutor agents may increase initial on-task enjoyment, certain human-like characteristics may hinder learning through emotional experience.

Learners who were more interested in the topic taught in the ITS experienced more enjoyment while learning and were more engaged in the task. In turn, they also performed better in the ITS. Further, unlike enjoyment, boredom had only minimal negative effects on engagement and task performance. These findings identify key psychological processes involved in the contribution of affective and cognitive processes to learning in the ITS context. Students in the experimental condition perceived Cozmo (non-humanoid robot) as more sociable than their counterparts in the control condition and students perceiving Cozmo as more sociable were associated with larger gains in topic-related interest, utility and importance. These findings are surprising since Pepper’s (not Cozmo’s) embodied presence and enhanced human-like attributes in the experimental condition were expected to increase students’ perceptions of Human Agent Interaction (HAI) dimensions. Further analysis presented that higher perceptions of Cozmo’s sociability conveyed through its embodiment in the experimental condition contributed to increase the extrinsic motivational values (utility and importance) that learners ascribe to the learning content after the intervention.

The team created an open-source framework, Sobotify, which enables multiple robots (e.g. Pepper, NAO, Cozmo and MyKeepOn) to interact with learners as learning companions. In the framework, the team integrated open-source AI-based solutions (e.g. DeepFace for facial expression recognition, and VOSK for speech recognition, MediaPipe for gestures) to animate robots similar to humans. The developed framework aims to provide an environment for less to non-technical users to utilize robots as learning companions in their choice of scenarios.

By using the features in Sobotify, such as DeepFace, VOSK, and Mediapipe, Pepper was designed as an empathetic learning companion to support learners to converse more competently in a sales conversation. Even though there was in general increase in learning gain, most high performing students did not perceive that the interactive learning experience increased their skills in a sales conversation. An unexpected movement that Pepper performed during the experiment resulted in learners’ perception of the robot as robotic.

An HRI experiment for the language learning scenario was created and conducted at a vocational school. In this experiment, 4 different robots (Pepper, NAO, Cozmo and MyKeepOn) were controlled using Sobotify and emotion detection and emotionally adaptive feedback was provided to the participants. Similar to the previous study finding, Cozmo was found to be the most effective learning companion in the variables of agent value and facilitation in Pedagogical Agents as Learning companions (PAL) and Pepper was found to be credible and encouraging learners to reflect on what they are learning. Reviewing the design of the robots, all robots were attending to learners by maintaining eye contact, speaking, gesturing and responding to the detected emotions from learners by providing encouraging feedback. Two features that were different in Cozmo from other robots were, 1) “navigation” where Cozmo was not only gesturing with its arms (e.g., lift) but also moving closer to the learner and also retracting back and 2) “facial expression” where Cozmo displayed different facial expressions (e.g., smiling eyes to express happy, blinking eyes). The overall effectiveness of Cozmo as a learning companion may have been rated higher than other robots by learners due to Cozmo not only being co-present with learners but also due to dynamically sharing the physical space with learners along with expressing emotions in multiple ways (e.g., gesture, verbal and facial expression).

Through workshops with teachers, the team observed that Sobotify enabled teachers to easily adopt robots as learning companions in their teaching and learning scenarios. As it is an ongoing study, future studies will investigate deeper into how human users with less to non-technical background utilize synthetic systems such as robots and how the synthetic systems can reflect to humans. The team now better understands the key principles of social intelligence as their experimental studies showed that learning in social interaction requires adaptive interactions on both sides, the learner and the teacher – and that this concept is relevant for both the synthetic and the analytic side.