Boosting behavior science with video datasets
In recent decades, the use of video datasets such as wildlife cameras or lab videos has become one of the most important tools for the study of animal behavior. These datasets help researchers make behavioral observations by capturing detailed and real-time data that can be analyzed repeatedly, allowing for quantitative analyses of movements, interactions and patterns, tracking, and behavioral classifications. When it comes to mice, however, most of the many existing video datasets only show simple or social actions that barely allow for the study of more complex, goal-oriented behaviors, such as problem-solving. Researchers from the Cluster of Excellence Science of Intelligence in Berlin have created a new, large-scale collection of video data involving mice working to solve “lockbox” puzzles in order to get a reward. This dataset, which includes multiple camera angles, is one of the largest mouse datasets ever obtained and it provides a benchmark for researchers to develop and test new tools for the study of complex animal behavior.
Large datasets help better understand behavior
Computational ethology is an interdisciplinary field that uses modern advances in machine learning and computer vision for measuring, describing and analyzing natural behavior in freely moving animals. Although this field is becoming increasingly important, most existing datasets of freely moving animals are only meant for the analysis of trivial and social behaviors (for example eating, attacking, sniffing, or walking), and lack the well-defined database labels and annotations that allow the study of complex behaviors. This means that they cannot be fully applied to goal-directed actions, problem-solving, and other behaviors critical to understanding cognitive processes in natural and, if applicable, artificial intelligence.
One of the biggest mouse datasets out there
In order to create a sufficiently large database, SCIoI scientists filmed twelve mice attempting to repeatedly solve mechanical lockboxes ––consisting of either one mechanism or a series of consecutive mechanisms that lead to an oat flake as a final reward. “This is the first large-scale labeled, single-agent, multi-perspective video dataset of mice showing complex behavior as they solve mechanical puzzles,” said Mark Boon, one of the main authors of the study. “Every lockbox is baited with a food reward and consists of a single or a combination of four different mechanisms.”
The researchers obtained more than 100 hours of footage, and then labeled about 15 selected hours with information on observed behaviors, i.e., interactions between different parts of the mouse and the individual mechanisms of the lockboxes. How far are the mouse’s body parts from the puzzle? Is the mouse touching something with its front paws? Biting? Unlocking? Reaching the reward? This resulted in a dataset that includes multiple camera angles and is larger than most of any previous mouse datasets.
Agreement between annotators was a very important aspect in the creation of the dataset. “To provide high-quality label data, each labeled video was annotated by two skilled human raters who have been instructed prior to annotating, and their agreement was thoroughly assessed,” explained the main first author, Patrik Reiske. “Consistency between (human) raters is a crucial aspect, as any behavior analysis relies on high-quality labels as a baseline. For instance, if one human annotator labeled ‘biting into the lever mechanism’ as one of the behaviors observed, we must establish whether the other human annotator agreed, and also whether our automated pipeline (computer) delivered the same label as well. In doing so, we rely on a well-established qualitative metric for assessing the so-called inter-rater reliability to allow for a fair comparison of the human-human agreement—the gold standard—against the human-computer agreement.”
Why this matters
Understanding complex animal behavior is essential for understanding how intelligence works—both biological and artificial. While most existing datasets only capture simple actions, thus limiting our ability to study cognitive processes like planning, decision-making, and problem-solving, this new dataset fills that gap by providing high-quality, multi-angle, human-labeled data of animals engaging in goal-directed, sequential tasks.
“The primary importance of these datasets is that they serve as a benchmark against which new algorithms that analyze animal behavior can be compared. For instance, if researchers develop new pose estimation, action classification, and representation learning algorithms and want to assess their accuracy, they will find our dataset useful,” explained co-author Sole Traverso.
In brief
- Most animal behavior datasets capture only simple actions, limiting research on advanced cognitive processes like planning and problem-solving.
- A new dataset developed by Berlin researchers allows benchmarking algorithms that other scientists can use for comparison.
- This new dataset offers high-quality, multi-angle footage of mice solving complex tasks, with detailed human annotations.
It enables the more rapid development of algorithms that analyze the complex mechanisms underlying animal cognition, supporting the development of AI systems inspired by natural intelligence.
