Research

Active tracking using bioinspired event-based vision – Towards improving interpretability of individual and collective behavior

Using a combination of video cameras and event-based cameras to extract meaningful motion information on complex animal behavior

Research Unit: 3

Project Number: 36

Disciplines:
Computer Vision

 

Principal Investigators:
Guillermo Gallego

Doctoral Researchers:
Friedhelm Hamann

 

Expected Project Duration
2021 - 2024

← Projects Overview

Active tracking using bioinspired event-based vision – Towards improving interpretability of individual and collective behavior

Using a combination of video cameras and event-based cameras to extract meaningful motion information on complex animal behavior

Complex behavior of animals is often analyzed from video recordings because cameras provide an economical and non-invasive way to acquire abundant data. Hence, developing computer vision tools to extract relevant information from such a rich yet raw data source is essential to support behavioral analysis. We propose to develop computer vision algorithms using a combination of video (i.e., frame-based) cameras and event-based cameras in order to extract meaningful motion information in individuals (in isolation or as part of groups). Both sensor types are complementary: event-based cameras excel at capturing high-frequency temporal content, while traditional cameras are better at acquiring slowly-varying content.

Event-based cameras are novel, biologically-inspired sensors that mimic the transient pathway of the human visual system. These cameras respond to motion in the form of brightness changes (called “events”) at any pixel in time. They can capture the dynamics of a scene with high dynamic range and temporal resolution, w​ithout suffering from motion blur, as opposed to traditional cameras. Additionally, these cameras allow us to record only motion information, which we will exploit for long-term tracking and better segmentation of behaviors.

We consider an ​active vision ​approach, where the viewpoint of the camera can vary to improve the tracking performance. This system will enable robust detection of individuals regardless of their 3D location and avoid target disappearance during long-term tracking. In an analysis phase, motion tracks will help categorize relevant behavior of the interactions of individuals.

Project Results

– Event camera technology can reveal information in situations where traditional video cameras struggle. At SCIoI event cameras have been used to quantify animal behavior in uncontrolled conditions (outdoors) and low light, with low power and data consumption, advantages that allow long-term recordings compared to traditional ways of monitoring animal behavior.

– Adaptation of data processing algorithms to the characteristics of event cameras makes the best use of the advantages of these novel sensors. Much technology is still to be developed in this field.

– We develop a general-purpose point tracker and also particular detectors (temporal action detection, behavior detector) that can support the efforts of SCIoI across multiple projects, lifting the “computer vision bottleneck” often encountered in the first stages of behavior quantification.

Zhou, Y., Gallego, G., & Shen, S. (2021). Event-based Stereo Visual Odometry. IEEE Transactions on Robotics, 37(5), 1433–1450. https://doi.org/10.1109/TRO.2021.3062252
Shiba, S., Aoki, Y., & Gallego, G. (2025). Simultaneous Motion And Noise Estimation with Event Cameras. IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.48550/arXiv.2504.04029
Shiba, S., Hamann, F., Aoki, Y., & Gallego, G. (2023). Event-based Background-Oriented Schlieren. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–16. https://doi.org/10.1109/TPAMI.2023.3328188
Shiba, S., Aoki, Y., & Gallego, G. (2022). Event Collapse in Contrast Maximization Frameworks. Sensors, 22(14), 1–20. https://doi.org/10.3390/s22145190
Shiba, S., Klose, Y., Aoki, Y., & Gallego, G. (2024). Secrets of Event-based Optical Flow, Depth and Ego-motion Estimation by Contrast Maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7742–7759. https://doi.org/10.1109/TPAMI.2024.3396116
Reinold, T., Ghosh, S., & Gallego, G. (2025). Combined Physics and Event Camera Simulator for Slip Detection. IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), 872–880. https://doi.org/10.1109/WACVW65960.2025.00104
Hitzges, D., Ghosh, S., & Gallego, G. (2025). DERD-Net: Learning Depth from Event-based Ray Densities. Advances in Neural Information Processing Systems (NeurIPS). https://doi.org/10.48550/arXiv.2504.15863
Hidalgo-Carrió, J., Gallego, G., & Scaramuzza, D. (2022). Event-aided Direct Sparse Odometry. IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 5771–5780. https://doi.org/10.1109/CVPR52688.2022.00569
Hamann, F., Mededovic, E., Gülhan, F., Wu, Y., Stegmaier, J., He, J., Wang, Y., Zhang, K., Li, L., Jiao, L., Ma, M., Huang, H., Yan, Y., Ren, H., Lin, X., Huang, Y., Cheng, B., Lee, S. H., Ham, G. S., … Gallego, G. (2025). SIS-Challenge: Event-based Spatio-temporal Instance Segmentation Challenge at the CVPR 2025 Event-based Vision Workshop. IEEE International Conference on Computer Vision (ICCV) Workshops. https://doi.org/10.48550/arXiv.2508.12813
Hamann, F., Gehrig, D., Febryanto, F., Daniilidis, K., & Gallego, G. (2025). ETAP: Event-based Tracking of Any Point. IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 17. https://doi.org/10.48550/arXiv.2412.00133
Hamann, F., Li, H., Mieske, P., Lewejohann, L., & Gallego, G. (2024). MouseSIS: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice. European Conference on Computer Vision (ECCV) Workshops, 156–173. https://doi.org/10.1007/978-3-031-92460-6_10
Hamann, F., Ghosh, S., Martínez, I. J., Hart, T., Kacelnik, A., & Gallego, G. (2024). Fourier-based Action Recognition for Wildlife Behavior Quantification with Event Cameras. Advanced Intelligent Systems, 2400353. https://doi.org/10.1002/aisy.202400353
Hamann, F., Ghosh, S., Martínez, I. J., Hart, T., Kacelnik, A., & Gallego, G. (2024). Low-power, Continuous Remote Behavioral Localization with Event Cameras. IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 18612–18621. https://doi.org/10.1109/CVPR52733.2024.01761
Hamann, F., Wang, Z., Asmanis, I., Chaney, K., Gallego, G., & Daniilidis, K. (2024). Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation. European Conference on Computer Vision (ECCV), 18–37. https://doi.org/10.1007/978-3-031-72646-0_2
Hamann, F., & Gallego, G. (2022). Stereo Co-capture System for Recording and Tracking Fish with Frame- and Event Cameras. International Conference on Pattern Recognition (ICPR), Workshop on Visual observation and analysis of Vertebrate And Insect Behavior. https://arxiv.org/abs/2207.07332
Guo, S., Hamann, F., & Gallego, G. (2025). Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras. IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.48550/arXiv.2503.17262
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A., Conradt, J., Daniilidis, K., & Scaramuzza, D. (2022). Event-based Vision: A Survey. IEEE Transactions on Pattern Analysis and Machine Inteliigence, 44(1), 154–180. https://doi.org/10.1109/TPAMI.2020.3008413

Research

An overview of our scientific work

See our Research Projects