Principles of Intelligence
The principle of multimodality states that processing information through multiple sensory channels is advantageous when operating in an uncertain and non-stationary world.
Intelligent systems act more optimally in an uncertain, non-stationary world by processing information through multiple sensory channels as each modality captures overlapping yet non-redundant information. The simultaneous use of these different sensory channels, for example visual and tactile, is called multimodality. In biological systems, multimodal sensory integration is the common way for organisms to perceive the world. We were able to demonstrate that the use of multimodal information allows animals to make better decisions.
For example, bird predation poses a strong selection pressure on fish. Since birds must enter the water to catch fish, a combination of visual and mechano-acoustic cues characterize an immediate attack, while single cues (unimodal) may represent less dangerous disturbances, as shown in a study by Juliane Lukas et al.
Sense-specific features blend together to create a phenomenon’s unfied representation
The benefit of multimodality is that each modality (or sensory channel) captures overlapping, non-redundant, synergistic information about a particular phenomenon in the world.
In multimodal information processing, modality-specific (or sense-specific) features are fused together to create a unified representation. This synergistic information helps compensate incomplete or noisy information in a single modality. Multimodal systems exploit contingent interaction of information processing in different modalities. Seeing the location of a loudspeaker, for example, can alter the spatial localization of a sound source. Which modality-specific features enter this multimodal representation depends on a selection process that takes into account the reliability and relevance of modality-specific features. The multimodal representation in turn affects modality-specific information processing, leading to an interactive processing loop. Multimodality can be understood as a special case of adaptive representations and is the result of active interconnections.
Related Projects
Project 21 – Developing exploration behavior
