Friedhelm Hamann
Photo: SCIoI
Friedhelm Hamann joined SCIoI as a PhD researcher in October 2021. His background is in electrical engineering, computer vision, visual perception, event-based vision, and robotics. He received his B.Sc. from the University of Rostock and his M.Sc. in Electrical Engineering and Information Technology from RWTH Aachen University.
At SCIoI, under the supervision of Prof. Guillermo Gallego, he worked on event-based vision for robotic perception. His research focused on using bioinspired event cameras to analyze dynamic scenes and support the behavioral analysis of animals.
His doctoral thesis, Dynamic Scene Understanding Using Event Cameras (2025), examines how event-based vision can be used to extract and interpret information from rapidly changing environments.
Projects
Friedhelm Hamann is member of:
Publications +
6984777
Friedhelm Hamann
1
apa
50
date
desc
year
19864
https://www.scienceofintelligence.de/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%222994TT4M%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222025-10-19%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Mededovic%2C%20E.%2C%20G%26%23xFC%3Blhan%2C%20F.%2C%20Wu%2C%20Y.%2C%20Stegmaier%2C%20J.%2C%20He%2C%20J.%2C%20Wang%2C%20Y.%2C%20Zhang%2C%20K.%2C%20Li%2C%20L.%2C%20Jiao%2C%20L.%2C%20Ma%2C%20M.%2C%20Huang%2C%20H.%2C%20Yan%2C%20Y.%2C%20Ren%2C%20H.%2C%20Lin%2C%20X.%2C%20Huang%2C%20Y.%2C%20Cheng%2C%20B.%2C%20Lee%2C%20S.%20H.%2C%20Ham%2C%20G.%20S.%2C%20%26%23x2026%3B%20Gallego%2C%20G.%20%282025%29.%20SIS-Challenge%3A%20Event-Based%20Spatio-Temporal%20Instance%20Segmentation%20Challenge%20at%20the%20CVPR%202025%20Event-Based%20Vision%20Workshop.%20%26lt%3Bi%26gt%3B2025%20IEEE%5C%2FCVF%20International%20Conference%20on%20Computer%20Vision%20Workshops%20%28ICCVW%29%26lt%3B%5C%2Fi%26gt%3B%2C%204734%26%23x2013%3B4742.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FICCVW69036.2025.00491%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FICCVW69036.2025.00491%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22SIS-Challenge%3A%20Event-Based%20Spatio-Temporal%20Instance%20Segmentation%20Challenge%20at%20the%20CVPR%202025%20Event-Based%20Vision%20Workshop%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Emil%22%2C%22lastName%22%3A%22Mededovic%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22G%5Cu00fclhan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yuli%22%2C%22lastName%22%3A%22Wu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Johannes%22%2C%22lastName%22%3A%22Stegmaier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jing%22%2C%22lastName%22%3A%22He%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yiqing%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kexin%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lingling%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Licheng%22%2C%22lastName%22%3A%22Jiao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mengru%22%2C%22lastName%22%3A%22Ma%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hongxiang%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yuhao%22%2C%22lastName%22%3A%22Yan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hongwei%22%2C%22lastName%22%3A%22Ren%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xiaopeng%22%2C%22lastName%22%3A%22Lin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yulong%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bojun%22%2C%22lastName%22%3A%22Cheng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Se%20Hyun%22%2C%22lastName%22%3A%22Lee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gyu%20Sung%22%2C%22lastName%22%3A%22Ham%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kanghan%22%2C%22lastName%22%3A%22Oh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gi%20Hyun%22%2C%22lastName%22%3A%22Lim%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Boxuan%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bowen%22%2C%22lastName%22%3A%22Du%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%222025%20IEEE%5C%2FCVF%20International%20Conference%20on%20Computer%20Vision%20Workshops%20%28ICCVW%29%22%2C%22conferenceName%22%3A%222025%20IEEE%5C%2FCVF%20International%20Conference%20on%20Computer%20Vision%20Workshops%20%28ICCVW%29%22%2C%22date%22%3A%222025-10-19%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FICCVW69036.2025.00491%22%2C%22ISBN%22%3A%22979-8-3315-8988-2%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F11374462%5C%2F%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-20T07%3A02%3A13Z%22%7D%7D%2C%7B%22key%22%3A%22MYBFPA7Q%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Guo%20et%20al.%22%2C%22parsedDate%22%3A%222025-10%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGuo%2C%20S.%2C%20Hamann%2C%20F.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282025%29.%20Unsupervised%20Joint%20Learning%20of%20Optical%20Flow%20and%20Intensity%20with%20Event%20Cameras.%20%26lt%3Bi%26gt%3B2025%20IEEE%5C%2FCVF%20International%20Conference%20on%20Computer%20Vision%20%28ICCV%29%26lt%3B%5C%2Fi%26gt%3B%2C%207980%26%23x2013%3B7989.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FICCV51701.2025.00748%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FICCV51701.2025.00748%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Unsupervised%20Joint%20Learning%20of%20Optical%20Flow%20and%20Intensity%20with%20Event%20Cameras%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shuang%22%2C%22lastName%22%3A%22Guo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22Event%20cameras%20rely%20on%20motion%20to%20obtain%20information%20about%20scene%20appearance.%20This%20means%20that%20appearance%20and%20motion%20are%20inherently%20linked%3A%20either%20both%20are%20present%20and%20recorded%20in%20the%20event%20data%2C%20or%20neither%20is%20captured.%20Previous%20works%20treat%20the%20recovery%20of%20these%20two%20visual%20quantities%20as%20separate%20tasks%2C%20which%20does%20not%20fit%20with%20the%20above-mentioned%20nature%20of%20event%20cameras%20and%20overlooks%20the%20inherent%20relations%20between%20them.%20We%20propose%20an%20unsupervised%20learning%20framework%20that%20jointly%20estimates%20optical%20flow%20%28motion%29%20and%20image%20intensity%20%28appearance%29%20using%20a%20single%20network.%20From%20the%20data%20generation%20model%2C%20we%20newly%20derive%20the%20event-based%20photometric%20error%20as%20a%20function%20of%20optical%20flow%20and%20image%20intensity.%20This%20error%20is%20further%20combined%20with%20the%20contrast%20maximization%20framework%20to%20form%20a%20comprehensive%20loss%20function%20that%20provides%20proper%20constraints%20for%20both%20flow%20and%20intensity%20estimation.%20Exhaustive%20experiments%20show%20our%20method%26%23039%3Bs%20state-of-the-art%20performance%3A%20in%20optical%20flow%20estimation%2C%20it%20reduces%20EPE%20by%2020%25%20and%20A%20E%20by%2025%25%20compared%20to%20unsupervised%20approaches%2C%20while%20delivering%20competitive%20intensity%20estimation%20results%2C%20particularly%20in%20high%20dynamic%20range%20scenarios.%20Our%20method%20also%20achieves%20shorter%20inference%20time%20than%20all%20other%20optical%20flow%20methods%20and%20many%20of%20the%20image%20reconstruction%20methods%2C%20while%20they%20output%20only%20one%20quantity.%20Project%20page%3A%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Ftub-rip%5C%2FE2FAI%22%2C%22proceedingsTitle%22%3A%222025%20IEEE%5C%2FCVF%20International%20Conference%20on%20Computer%20Vision%20%28ICCV%29%22%2C%22conferenceName%22%3A%222025%20IEEE%5C%2FCVF%20International%20Conference%20on%20Computer%20Vision%20%28ICCV%29%22%2C%22date%22%3A%222025-10%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FICCV51701.2025.00748%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F11444738%22%2C%22ISSN%22%3A%222380-7504%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A25Z%22%7D%7D%2C%7B%22key%22%3A%22QRHVCW7L%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222025-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Gehrig%2C%20D.%2C%20Febryanto%2C%20F.%2C%20Daniilidis%2C%20K.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282025%29.%20ETAP%3A%20Event-based%20Tracking%20of%20Any%20Point.%20%26lt%3Bi%26gt%3B2025%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%26lt%3B%5C%2Fi%26gt%3B%2C%2027186%26%23x2013%3B27196.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPR52734.2025.02532%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPR52734.2025.02532%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22ETAP%3A%20Event-based%20Tracking%20of%20Any%20Point%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Gehrig%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Filbert%22%2C%22lastName%22%3A%22Febryanto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kostas%22%2C%22lastName%22%3A%22Daniilidis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22Tracking%20any%20point%20%28TAP%29%20recently%20shifted%20the%20motion%20estimation%20paradigm%20from%20focusing%20on%20individual%20salient%20points%20with%20local%20templates%20to%20tracking%20arbitrary%20points%20with%20global%20image%20contexts.%20However%2C%20while%20research%20has%20mostly%20focused%20on%20driving%20the%20accuracy%20of%20models%20in%20nominal%20settings%2C%20addressing%20scenarios%20with%20difficult%20lighting%20conditions%20and%20high-speed%20motions%20remains%20out%20of%20reach%20due%20to%20the%20limitations%20of%20the%20sensor.%20This%20work%20addresses%20this%20challenge%20with%20the%20first%20event%20camera-based%20TAP%20method.%20It%20leverages%20the%20high%20temporal%20resolution%20and%20high%20dynamic%20range%20of%20event%20cameras%20for%20robust%20high-speed%20tracking%2C%20and%20the%20global%20contexts%20in%20TAP%20methods%20to%20handle%20asynchronous%20and%20sparse%20event%20measurements.%20We%20further%20extend%20the%20TAP%20framework%20to%20handle%20event%20feature%20variations%20induced%20by%20motion%20%5Cu2014%20thereby%20addressing%20an%20open%20challenge%20in%20purely%20event-based%20tracking%20%5Cu2014%20with%20a%20novel%20feature%20alignment-loss%20which%20ensures%20the%20learning%20of%20motion-robust%20features.%20Our%20method%20is%20trained%20with%20data%20from%20a%20new%20data%20generation%20pipeline%20and%20systematically%20ablated%20across%20all%20design%20decisions.%20Our%20method%20shows%20strong%20cross-dataset%20generalization%20and%20performs%20136%25%20better%20on%20the%20average%20Jaccard%20metric%20than%20the%20baselines.%20Moreover%2C%20on%20an%20established%20feature%20tracking%20benchmark%2C%20it%20achieves%20a%2020%25%20improvement%20over%20the%20previous%20best%20event-only%20method%20and%20even%20surpasses%20the%20previous%20best%20events-and-frames%20method%20by%204.1%25.%20Our%20code%20is%20available%20at%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Ftub-rip%5C%2FETAP.%22%2C%22proceedingsTitle%22%3A%222025%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%22%2C%22conferenceName%22%3A%222025%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%22%2C%22date%22%3A%222025-06%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FCVPR52734.2025.02532%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F11094223%22%2C%22ISSN%22%3A%222575-7075%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-20T06%3A55%3A03Z%22%7D%7D%2C%7B%22key%22%3A%22W7B6NDYG%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Wang%20et%20al.%22%2C%22parsedDate%22%3A%222025-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BWang%2C%20Z.%2C%20Hamann%2C%20F.%2C%20Chaney%2C%20K.%2C%20Jiang%2C%20W.%2C%20Gallego%2C%20G.%2C%20%26amp%3B%20Daniilidis%2C%20K.%20%282025%29.%20Event-Based%20Continuous%20Color%20Video%20Decompression%20from%20Single%20Frames.%20%26lt%3Bi%26gt%3B2025%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20Workshops%20%28CVPRW%29%26lt%3B%5C%2Fi%26gt%3B%2C%204968%26%23x2013%3B4978.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPRW67362.2025.00490%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPRW67362.2025.00490%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Event-Based%20Continuous%20Color%20Video%20Decompression%20from%20Single%20Frames%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziyun%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kenneth%22%2C%22lastName%22%3A%22Chaney%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wen%22%2C%22lastName%22%3A%22Jiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kostas%22%2C%22lastName%22%3A%22Daniilidis%22%7D%5D%2C%22abstractNote%22%3A%22We%20present%20ContinuityCam%2C%20a%20novel%20approach%20to%20generate%20a%20continuous%20video%20from%20a%20single%20static%20RGB%20image%20and%20an%20event%20camera%20stream.%20Conventional%20cameras%20struggle%20with%20high-speed%20motion%20capture%20due%20to%20bandwidth%20and%20dynamic%20range%20limitations.%20Event%20cameras%20are%20ideal%20sensors%20to%20solve%20this%20problem%20because%20they%20encode%20compressed%20change%20information%20at%20high%20temporal%20resolution.%20In%20this%20work%2C%20we%20tackle%20the%20problem%20of%20event-based%20continuous%20color%20video%20decompression%2C%20pairing%20single%20static%20color%20frames%20and%20event%20data%20to%20reconstruct%20temporally%20continuous%20videos.%20Our%20approach%20combines%20continuous%20long-range%20motion%20modeling%20with%20a%20neural%20synthesis%20model%2C%20enabling%20frame%20prediction%20at%20arbitrary%20times%20within%20the%20events.%20Our%20method%20only%20requires%20an%20initial%20image%2C%20thus%20increasing%20the%20robustness%20to%20sudden%20motions%2C%20light%20changes%2C%20minimizing%20the%20prediction%20latency%2C%20and%20decreasing%20bandwidth%20usage.%20We%20also%20introduce%20a%20novel%20single-lens%20beamsplitter%20setup%20that%20acquires%20aligned%20images%20and%20events%2C%20and%20a%20novel%20and%20challenging%20Event%20Extreme%20Decompression%20Dataset%20%28E2D2%29%20that%20tests%20the%20method%20in%20various%20lighting%20and%20motion%20profiles.%20We%20thoroughly%20evaluate%20our%20method%20by%20benchmarking%20color%20frame%20reconstruction%2C%20outperforming%20the%20baseline%20methods%20by%203.61%20dB%20in%20PSNR%20and%20by%2033%25%20decrease%20in%20LPIPS%2C%20as%20well%20as%20showing%20superior%20results%20on%20two%20downstream%20tasks.%20Please%20see%20our%20project%20website%20for%20details%3A%20https%3A%5C%2F%5C%2Fwww.cis.upenn.edu%5C%2F%20ziyunw%5C%2Fcontinuity_cam%5C%2F.%22%2C%22proceedingsTitle%22%3A%222025%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20Workshops%20%28CVPRW%29%22%2C%22conferenceName%22%3A%222025%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20Workshops%20%28CVPRW%29%22%2C%22date%22%3A%222025-06%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FCVPRW67362.2025.00490%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F11147662%22%2C%22ISSN%22%3A%222160-7516%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T12%3A01%3A40Z%22%7D%7D%2C%7B%22key%22%3A%22SNCG6RHQ%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Ghosh%2C%20S.%2C%20Ju%26%23xE1%3Brez%20Mart%26%23xED%3Bnez%2C%20I.%2C%20Hart%2C%20T.%2C%20Kacelnik%2C%20A.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282025%29.%20Fourier%26%23x2010%3BBased%20Action%20Recognition%20for%20Wildlife%20Behavior%20Quantification%20with%20Event%20Cameras.%20%26lt%3Bi%26gt%3BAdvanced%20Intelligent%20Systems%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B7%26lt%3B%5C%2Fi%26gt%3B%282%29%2C%202400353.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1002%5C%2Faisy.202400353%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1002%5C%2Faisy.202400353%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Fourier%5Cu2010Based%20Action%20Recognition%20for%20Wildlife%20Behavior%20Quantification%20with%20Event%20Cameras%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Suman%22%2C%22lastName%22%3A%22Ghosh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ignacio%22%2C%22lastName%22%3A%22Ju%5Cu00e1rez%20Mart%5Cu00ednez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tom%22%2C%22lastName%22%3A%22Hart%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alex%22%2C%22lastName%22%3A%22Kacelnik%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22Event%20cameras%20are%20novel%20bioinspired%20vision%20sensors%20that%20measure%20pixel%5Cu2010wise%20brightness%20changes%20asynchronously%20instead%20of%20images%20at%20a%20given%20frame%20rate.%20They%20offer%20promising%20advantages%2C%20namely%2C%20a%20high%20dynamic%20range%2C%20low%20latency%2C%20and%20minimal%20motion%20blur.%20Modern%20computer%20vision%20algorithms%20often%20rely%20on%20artificial%20neural%20network%20approaches%2C%20which%20require%20image%5Cu2010like%20representations%20of%20the%20data%20and%20cannot%20fully%20exploit%20the%20characteristics%20of%20event%20data.%20Herein%2C%20approaches%20to%20action%20recognition%20based%20on%20the%20Fourier%20transform%20are%20proposed.%20The%20approaches%20are%20intended%20to%20recognize%20oscillating%20motion%20patterns%20commonly%20present%20in%20nature.%20In%20particular%2C%20the%20approaches%20are%20applied%20to%20a%20recent%20dataset%20of%20breeding%20penguins%20annotated%20for%20%5Cu201cecstatic%20display%2C%5Cu201d%20a%20behavior%20where%20the%20observed%20penguins%20flap%20their%20wings%20at%20a%20certain%20frequency.%20It%20is%20found%20that%20the%20approaches%20are%20both%20simple%20and%20effective%2C%20producing%20slightly%20lower%20results%20than%20a%20deep%20neural%20network%20%28DNN%29%20while%20relying%20just%20on%20a%20tiny%20fraction%20of%20the%20parameters%20compared%20to%20the%20DNN%20%28five%20orders%20of%20magnitude%20fewer%20parameters%29.%20They%20work%20well%20despite%20the%20uncontrolled%2C%20diverse%20data%20present%20in%20the%20dataset.%20It%20is%20hoped%20that%20this%20work%20opens%20a%20new%20perspective%20on%20event%5Cu2010based%20processing%20and%20action%20recognition.%22%2C%22date%22%3A%2202%5C%2F2025%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1002%5C%2Faisy.202400353%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fadvanced.onlinelibrary.wiley.com%5C%2Fdoi%5C%2F10.1002%5C%2Faisy.202400353%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%222640-4567%2C%202640-4567%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T11%3A57%3A38Z%22%7D%7D%2C%7B%22key%22%3A%22HCHUT8IC%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Guo%20et%20al.%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGuo%2C%20S.%2C%20Hamann%2C%20F.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282025%29.%20%26lt%3Bi%26gt%3BUnsupervised%20Joint%20Learning%20of%20Optical%20Flow%20and%20Intensity%20with%20Event%20Cameras%26lt%3B%5C%2Fi%26gt%3B.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2503.17262%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2503.17262%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Unsupervised%20Joint%20Learning%20of%20Optical%20Flow%20and%20Intensity%20with%20Event%20Cameras%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shuang%22%2C%22lastName%22%3A%22Guo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22Event%20cameras%20rely%20on%20motion%20to%20obtain%20information%20about%20scene%20appearance.%20This%20means%20that%20appearance%20and%20motion%20are%20inherently%20linked%3A%20either%20both%20are%20present%20and%20recorded%20in%20the%20event%20data%2C%20or%20neither%20is%20captured.%20Previous%20works%20treat%20the%20recovery%20of%20these%20two%20visual%20quantities%20as%20separate%20tasks%2C%20which%20does%20not%20fit%20with%20the%20above-mentioned%20nature%20of%20event%20cameras%20and%20overlooks%20the%20inherent%20relations%20between%20them.%20We%20propose%20an%20unsupervised%20learning%20framework%20that%20jointly%20estimates%20optical%20flow%20%28motion%29%20and%20image%20intensity%20%28appearance%29%20using%20a%20single%20network.%20From%20the%20data%20generation%20model%2C%20we%20newly%20derive%20the%20event-based%20photometric%20error%20as%20a%20function%20of%20optical%20flow%20and%20image%20intensity.%20This%20error%20is%20further%20combined%20with%20the%20contrast%20maximization%20framework%20to%20form%20a%20comprehensive%20loss%20function%20that%20provides%20proper%20constraints%20for%20both%20flow%20and%20intensity%20estimation.%20Exhaustive%20experiments%20show%20our%20method%26%23039%3Bs%20state-of-the-art%20performance%3A%20in%20optical%20flow%20estimation%2C%20it%20reduces%20EPE%20by%2020%25%20and%20AE%20by%2025%25%20compared%20to%20unsupervised%20approaches%2C%20while%20delivering%20competitive%20intensity%20estimation%20results%2C%20particularly%20in%20high%20dynamic%20range%20scenarios.%20Our%20method%20also%20achieves%20shorter%20inference%20time%20than%20all%20other%20optical%20flow%20methods%20and%20many%20of%20the%20image%20reconstruction%20methods%2C%20while%20they%20output%20only%20one%20quantity.%20Project%20page%3A%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Ftub-rip%5C%2FE2FAI%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22%22%2C%22date%22%3A%222025%22%2C%22DOI%22%3A%2210.48550%5C%2FARXIV.2503.17262%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2503.17262%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-12T08%3A00%3A04Z%22%7D%7D%2C%7B%22key%22%3A%22WJTBHM4B%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Wang%2C%20Z.%2C%20Asmanis%2C%20I.%2C%20Chaney%2C%20K.%2C%20Gallego%2C%20G.%2C%20%26amp%3B%20Daniilidis%2C%20K.%20%282025%29.%20Motion-Prior%20Contrast%20Maximization%20for%26%23xA0%3BDense%20Continuous-Time%20Motion%20Estimation.%20In%20A.%20Leonardis%2C%20E.%20Ricci%2C%20S.%20Roth%2C%20O.%20Russakovsky%2C%20T.%20Sattler%2C%20%26amp%3B%20G.%20Varol%20%28Eds.%29%2C%20%26lt%3Bi%26gt%3BComputer%20Vision%20%26%23x2013%3B%20ECCV%202024%26lt%3B%5C%2Fi%26gt%3B%20%28pp.%2018%26%23x2013%3B37%29.%20Springer%20Nature%20Switzerland.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2F978-3-031-72646-0_2%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2F978-3-031-72646-0_2%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Motion-Prior%20Contrast%20Maximization%20for%5Cu00a0Dense%20Continuous-Time%20Motion%20Estimation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziyun%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ioannis%22%2C%22lastName%22%3A%22Asmanis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kenneth%22%2C%22lastName%22%3A%22Chaney%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kostas%22%2C%22lastName%22%3A%22Daniilidis%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Ale%5Cu0161%22%2C%22lastName%22%3A%22Leonardis%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Elisa%22%2C%22lastName%22%3A%22Ricci%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Stefan%22%2C%22lastName%22%3A%22Roth%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Olga%22%2C%22lastName%22%3A%22Russakovsky%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Torsten%22%2C%22lastName%22%3A%22Sattler%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22G%5Cu00fcl%22%2C%22lastName%22%3A%22Varol%22%7D%5D%2C%22abstractNote%22%3A%22Current%20optical%20flow%20and%20point-tracking%20methods%20rely%20heavily%20on%20synthetic%20datasets.%20Event%20cameras%20are%20novel%20vision%20sensors%20with%20advantages%20in%20challenging%20visual%20conditions%2C%20but%20state-of-the-art%20frame-based%20methods%20cannot%20be%20easily%20adapted%20to%20event%20data%20due%20to%20the%20limitations%20of%20current%20event%20simulators.%20We%20introduce%20a%20novel%20self-supervised%20loss%20combining%20the%20Contrast%20Maximization%20framework%20with%20a%20non-linear%20motion%20prior%20in%20the%20form%20of%20pixel-level%20trajectories%20and%20propose%20an%20efficient%20solution%20to%20solve%20the%20high-dimensional%20assignment%20problem%20between%20non-linear%20trajectories%20and%20events.%20Their%20effectiveness%20is%20demonstrated%20in%20two%20scenarios%3A%20In%20dense%20continuous-time%20motion%20estimation%2C%20our%20method%20improves%20the%20zero-shot%20performance%20of%20a%20synthetically%20trained%20model%20on%20the%20real-world%20dataset%20EVIMO2%20by%2029%25.%20In%20optical%20flow%20estimation%2C%20our%20method%20elevates%20a%20simple%20UNet%20to%20achieve%20state-of-the-art%20performance%20among%20self-supervised%20methods%20on%20the%20DSEC%20optical%20flow%20benchmark.%20Our%20code%20is%20available%20at%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Ftub-rip%5C%2FMotionPriorCMax.%22%2C%22proceedingsTitle%22%3A%22Computer%20Vision%20%5Cu2013%20ECCV%202024%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222025%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1007%5C%2F978-3-031-72646-0_2%22%2C%22ISBN%22%3A%22978-3-031-72646-0%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A16Z%22%7D%7D%2C%7B%22key%22%3A%22CZNXPLGP%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Li%2C%20H.%2C%20Mieske%2C%20P.%2C%20Lewejohann%2C%20L.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282025%29.%20MouseSIS%3A%20A%20Frames-and-Events%20Dataset%20for%26%23xA0%3BSpace-Time%20Instance%20Segmentation%20of%26%23xA0%3BMice.%20In%20A.%20Del%20Bue%2C%20C.%20Canton%2C%20J.%20Pont-Tuset%2C%20%26amp%3B%20T.%20Tommasi%20%28Eds.%29%2C%20%26lt%3Bi%26gt%3BComputer%20Vision%20%26%23x2013%3B%20ECCV%202024%20Workshops%26lt%3B%5C%2Fi%26gt%3B%20%28pp.%20156%26%23x2013%3B173%29.%20Springer%20Nature%20Switzerland.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2F978-3-031-92460-6_10%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2F978-3-031-92460-6_10%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22MouseSIS%3A%20A%20Frames-and-Events%20Dataset%20for%5Cu00a0Space-Time%20Instance%20Segmentation%20of%5Cu00a0Mice%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hanxiong%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22Mieske%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lars%22%2C%22lastName%22%3A%22Lewejohann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Alessio%22%2C%22lastName%22%3A%22Del%20Bue%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Cristian%22%2C%22lastName%22%3A%22Canton%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Jordi%22%2C%22lastName%22%3A%22Pont-Tuset%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Tatiana%22%2C%22lastName%22%3A%22Tommasi%22%7D%5D%2C%22abstractNote%22%3A%22Enabled%20by%20large%20annotated%20datasets%2C%20tracking%20and%20segmentation%20of%20objects%20in%20videos%20has%20made%20remarkable%20progress%20in%20recent%20years.%20Despite%20these%20advancements%2C%20algorithms%20still%20struggle%20under%20degraded%20conditions%20and%20during%20fast%20movements.%20Event%20cameras%20are%20novel%20sensors%20with%20high%20temporal%20resolution%20and%20high%20dynamic%20range%20that%20offer%20promising%20advantages%20to%20address%20these%20challenges.%20However%2C%20annotated%20data%20for%20developing%20learning-based%20mask-level%20tracking%20algorithms%20with%20events%20is%20not%20available.%20To%20this%20end%2C%20we%20introduce%3A%20%28i%29%20a%20new%20task%20termed%20space-time%20instance%20segmentation%2C%20similar%20to%20video%20instance%20segmentation%2C%20whose%20goal%20is%20to%20segment%20instances%20throughout%20the%20entire%20duration%20of%20the%20sensor%20input%20%28here%2C%20the%20input%20are%20quasi-continuous%20events%20and%20optionally%20aligned%20frames%29%3B%20and%20%28ii%29%20MouseSIS%2C%20a%20dataset%20for%20the%20new%20task%2C%20containing%20aligned%20grayscale%20frames%20and%20events.%20It%20includes%20annotated%20ground-truth%20labels%20%28pixel-level%20instance%20segmentation%20masks%29%20of%20a%20group%20of%20up%20to%20seven%20freely%20moving%20and%20interacting%20mice.%20We%20also%20provide%20two%20reference%20methods%2C%20which%20show%20that%20leveraging%20event%20data%20can%20consistently%20improve%20tracking%20performance%2C%20especially%20when%20used%20in%20combination%20with%20conventional%20cameras.%20The%20results%20highlight%20the%20potential%20of%20event-aided%20tracking%20in%20difficult%20scenarios.%20We%20hope%20our%20dataset%20opens%20the%20field%20of%20event-based%20video%20instance%20segmentation%20and%20enables%20the%20development%20of%20robust%20tracking%20algorithms%20for%20challenging%20conditions.%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Ftub-rip%5C%2FMouseSIS%22%2C%22proceedingsTitle%22%3A%22Computer%20Vision%20%5Cu2013%20ECCV%202024%20Workshops%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222025%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1007%5C%2F978-3-031-92460-6_10%22%2C%22ISBN%22%3A%22978-3-031-92460-6%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A16Z%22%7D%7D%2C%7B%22key%22%3A%22IABM7PYQ%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222024-06-16%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Ghosh%2C%20S.%2C%20Mart%26%23xED%3Bnez%2C%20I.%20J.%2C%20Hart%2C%20T.%2C%20Kacelnik%2C%20A.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282024%29.%20Low-power%2C%20Continuous%20Remote%20Behavioral%20Localization%20with%20Event%20Cameras.%20%26lt%3Bi%26gt%3B2024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%26lt%3B%5C%2Fi%26gt%3B%2C%2018612%26%23x2013%3B18621.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPR52733.2024.01761%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPR52733.2024.01761%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Low-power%2C%20Continuous%20Remote%20Behavioral%20Localization%20with%20Event%20Cameras%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Suman%22%2C%22lastName%22%3A%22Ghosh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ignacio%20Ju%5Cu00e1rez%22%2C%22lastName%22%3A%22Mart%5Cu00ednez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tom%22%2C%22lastName%22%3A%22Hart%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alex%22%2C%22lastName%22%3A%22Kacelnik%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%222024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%22%2C%22conferenceName%22%3A%222024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%22%2C%22date%22%3A%222024-6-16%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FCVPR52733.2024.01761%22%2C%22ISBN%22%3A%22979-8-3503-5300-6%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F10658483%5C%2F%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T11%3A55%3A27Z%22%7D%7D%2C%7B%22key%22%3A%229IFDXGX9%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Shiba%20et%20al.%22%2C%22parsedDate%22%3A%222024%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BShiba%2C%20S.%2C%20Hamann%2C%20F.%2C%20Aoki%2C%20Y.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282024%29.%20Event-Based%20Background-Oriented%20Schlieren.%20%26lt%3Bi%26gt%3BIEEE%20Transactions%20on%20Pattern%20Analysis%20and%20Machine%20Intelligence%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B46%26lt%3B%5C%2Fi%26gt%3B%284%29%2C%202011%26%23x2013%3B2026.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FTPAMI.2023.3328188%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FTPAMI.2023.3328188%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Event-Based%20Background-Oriented%20Schlieren%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shintaro%22%2C%22lastName%22%3A%22Shiba%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yoshimitsu%22%2C%22lastName%22%3A%22Aoki%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%224%5C%2F2024%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FTPAMI.2023.3328188%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F10301562%5C%2F%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%220162-8828%2C%202160-9292%2C%201939-3539%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-20T06%3A55%3A09Z%22%7D%7D%2C%7B%22key%22%3A%22C7QQFVJ4%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222024%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Gehrig%2C%20D.%2C%20Febryanto%2C%20F.%2C%20Daniilidis%2C%20K.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282024%29.%20%26lt%3Bi%26gt%3BETAP%3A%20Event-based%20Tracking%20of%20Any%20Point%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2412.00133%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2412.00133%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22ETAP%3A%20Event-based%20Tracking%20of%20Any%20Point%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Gehrig%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Filbert%22%2C%22lastName%22%3A%22Febryanto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kostas%22%2C%22lastName%22%3A%22Daniilidis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22Tracking%20any%20point%20%28TAP%29%20recently%20shifted%20the%20motion%20estimation%20paradigm%20from%20focusing%20on%20individual%20salient%20points%20with%20local%20templates%20to%20tracking%20arbitrary%20points%20with%20global%20image%20contexts.%20However%2C%20while%20research%20has%20mostly%20focused%20on%20driving%20the%20accuracy%20of%20models%20in%20nominal%20settings%2C%20addressing%20scenarios%20with%20difficult%20lighting%20conditions%20and%20high-speed%20motions%20remains%20out%20of%20reach%20due%20to%20the%20limitations%20of%20the%20sensor.%20This%20work%20addresses%20this%20challenge%20with%20the%20first%20event%20camera-based%20TAP%20method.%20It%20leverages%20the%20high%20temporal%20resolution%20and%20high%20dynamic%20range%20of%20event%20cameras%20for%20robust%20high-speed%20tracking%2C%20and%20the%20global%20contexts%20in%20TAP%20methods%20to%20handle%20asynchronous%20and%20sparse%20event%20measurements.%20We%20further%20extend%20the%20TAP%20framework%20to%20handle%20event%20feature%20variations%20induced%20by%20motion%20--%20thereby%20addressing%20an%20open%20challenge%20in%20purely%20event-based%20tracking%20--%20with%20a%20novel%20feature-alignment%20loss%20which%20ensures%20the%20learning%20of%20motion-robust%20features.%20Our%20method%20is%20trained%20with%20data%20from%20a%20new%20data%20generation%20pipeline%20and%20systematically%20ablated%20across%20all%20design%20decisions.%20Our%20method%20shows%20strong%20cross-dataset%20generalization%20and%20performs%20136%25%20better%20on%20the%20average%20Jaccard%20metric%20than%20the%20baselines.%20Moreover%2C%20on%20an%20established%20feature%20tracking%20benchmark%2C%20it%20achieves%20a%2020%25%20improvement%20over%20the%20previous%20best%20event-only%20method%20and%20even%20surpasses%20the%20previous%20best%20events-and-frames%20method%20by%204.1%25.%20Our%20code%20is%20available%20at%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Ftub-rip%5C%2FETAP%22%2C%22date%22%3A%222024%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FARXIV.2412.00133%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2412.00133%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-12T06%3A43%3A51Z%22%7D%7D%2C%7B%22key%22%3A%22YXZHPWX4%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20et%20al.%22%2C%22parsedDate%22%3A%222023%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20Ghosh%2C%20S.%2C%20Martinez%2C%20I.%20J.%2C%20Hart%2C%20T.%2C%20Kacelnik%2C%20A.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282023%29.%20%26lt%3Bi%26gt%3BLow-power%2C%20Continuous%20Remote%20Behavioral%20Localization%20with%20Event%20Cameras%26lt%3B%5C%2Fi%26gt%3B.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2312.03799%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2312.03799%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Low-power%2C%20Continuous%20Remote%20Behavioral%20Localization%20with%20Event%20Cameras%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Suman%22%2C%22lastName%22%3A%22Ghosh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ignacio%20Juarez%22%2C%22lastName%22%3A%22Martinez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tom%22%2C%22lastName%22%3A%22Hart%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alex%22%2C%22lastName%22%3A%22Kacelnik%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22Researchers%20in%20natural%20science%20need%20reliable%20methods%20for%20quantifying%20animal%20behavior.%20Recently%2C%20numerous%20computer%20vision%20methods%20emerged%20to%20automate%20the%20process.%20However%2C%20observing%20wild%20species%20at%20remote%20locations%20remains%20a%20challenging%20task%20due%20to%20difficult%20lighting%20conditions%20and%20constraints%20on%20power%20supply%20and%20data%20storage.%20Event%20cameras%20offer%20unique%20advantages%20for%20battery-dependent%20remote%20monitoring%20due%20to%20their%20low%20power%20consumption%20and%20high%20dynamic%20range%20capabilities.%20We%20use%20this%20novel%20sensor%20to%20quantify%20a%20behavior%20in%20Chinstrap%20penguins%20called%20ecstatic%20display.%20We%20formulate%20the%20problem%20as%20a%20temporal%20action%20detection%20task%2C%20determining%20the%20start%20and%20end%20times%20of%20the%20behavior.%20For%20this%20purpose%2C%20we%20recorded%20a%20colony%20of%20breeding%20penguins%20in%20Antarctica%20for%20several%20weeks%20and%20labeled%20event%20data%20on%2016%20nests.%20The%20developed%20method%20consists%20of%20a%20generator%20of%20candidate%20time%20intervals%20%28proposals%29%20and%20a%20classifier%20of%20the%20actions%20within%20them.%20The%20experiments%20show%20that%20the%20event%20cameras%26%23039%3B%20natural%20response%20to%20motion%20is%20effective%20for%20continuous%20behavior%20monitoring%20and%20detection%2C%20reaching%20a%20mean%20average%20precision%20%28mAP%29%20of%2058%25%20%28which%20increases%20to%2063%25%20in%20good%20weather%20conditions%29.%20The%20results%20also%20demonstrate%20the%20robustness%20against%20various%20lighting%20conditions%20contained%20in%20the%20challenging%20dataset.%20The%20low-power%20capabilities%20of%20the%20event%20camera%20allow%20it%20to%20record%20significantly%20longer%20than%20with%20a%20conventional%20camera.%20This%20work%20pioneers%20the%20use%20of%20event%20cameras%20for%20remote%20wildlife%20observation%2C%20opening%20new%20interdisciplinary%20opportunities.%20https%3A%5C%2F%5C%2Ftub-rip.github.io%5C%2Feventpenguins%5C%2F%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22%22%2C%22date%22%3A%222023%22%2C%22DOI%22%3A%2210.48550%5C%2FARXIV.2312.03799%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2312.03799%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-12T09%3A25%3A25Z%22%7D%7D%2C%7B%22key%22%3A%22S6I3BBS3%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Wang%20et%20al.%22%2C%22parsedDate%22%3A%222023%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BWang%2C%20Z.%2C%20Hamann%2C%20F.%2C%20Chaney%2C%20K.%2C%20Jiang%2C%20W.%2C%20Gallego%2C%20G.%2C%20%26amp%3B%20Daniilidis%2C%20K.%20%282023%29.%20%26lt%3Bi%26gt%3BEvent-based%20Continuous%20Color%20Video%20Decompression%20from%20Single%20Frames%26lt%3B%5C%2Fi%26gt%3B.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2312.00113%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2312.00113%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Event-based%20Continuous%20Color%20Video%20Decompression%20from%20Single%20Frames%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziyun%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kenneth%22%2C%22lastName%22%3A%22Chaney%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wen%22%2C%22lastName%22%3A%22Jiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kostas%22%2C%22lastName%22%3A%22Daniilidis%22%7D%5D%2C%22abstractNote%22%3A%22We%20present%20ContinuityCam%2C%20a%20novel%20approach%20to%20generate%20a%20continuous%20video%20from%20a%20single%20static%20RGB%20image%20and%20an%20event%20camera%20stream.%20Conventional%20cameras%20struggle%20with%20high-speed%20motion%20capture%20due%20to%20bandwidth%20and%20dynamic%20range%20limitations.%20Event%20cameras%20are%20ideal%20sensors%20to%20solve%20this%20problem%20because%20they%20encode%20compressed%20change%20information%20at%20high%20temporal%20resolution.%20In%20this%20work%2C%20we%20tackle%20the%20problem%20of%20event-based%20continuous%20color%20video%20decompression%2C%20pairing%20single%20static%20color%20frames%20and%20event%20data%20to%20reconstruct%20temporally%20continuous%20videos.%20Our%20approach%20combines%20continuous%20long-range%20motion%20modeling%20with%20a%20neural%20synthesis%20model%2C%20enabling%20frame%20prediction%20at%20arbitrary%20times%20within%20the%20events.%20Our%20method%20only%20requires%20an%20initial%20image%2C%20thus%20increasing%20the%20robustness%20to%20sudden%20motions%2C%20light%20changes%2C%20minimizing%20the%20prediction%20latency%2C%20and%20decreasing%20bandwidth%20usage.%20We%20also%20introduce%20a%20novel%20single-lens%20beamsplitter%20setup%20that%20acquires%20aligned%20images%20and%20events%2C%20and%20a%20novel%20and%20challenging%20Event%20Extreme%20Decompression%20Dataset%20%28E2D2%29%20that%20tests%20the%20method%20in%20various%20lighting%20and%20motion%20profiles.%20We%20thoroughly%20evaluate%20our%20method%20by%20benchmarking%20color%20frame%20reconstruction%2C%20outperforming%20the%20baseline%20methods%20by%203.61%20dB%20in%20PSNR%20and%20by%2033%25%20decrease%20in%20LPIPS%2C%20as%20well%20as%20showing%20superior%20results%20on%20two%20downstream%20tasks.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22%22%2C%22date%22%3A%222023%22%2C%22DOI%22%3A%2210.48550%5C%2FARXIV.2312.00113%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2312.00113%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-12T08%3A37%3A01Z%22%7D%7D%2C%7B%22key%22%3A%22ZQ2YPXZD%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hamann%20and%20Gallego%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHamann%2C%20F.%2C%20%26amp%3B%20Gallego%2C%20G.%20%282022%29.%20%26lt%3Bi%26gt%3BStereo%20Co-capture%20System%20for%20Recording%20and%20Tracking%20Fish%20with%20Frame-%20and%20Event%20Cameras%26lt%3B%5C%2Fi%26gt%3B%20%5BPoster%5D.%20Visual%20observation%20and%20analysis%20of%20Vertebrate%20And%20Insect%20Behavior%20%28VAIB%29%20Workshop%20at%20the%2026th%20International%20Conference%20on%20Pattern%20Recognition%20%28ICPR%29.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2207.07332%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2207.07332%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22presentation%22%2C%22title%22%3A%22Stereo%20Co-capture%20System%20for%20Recording%20and%20Tracking%20Fish%20with%20Frame-%20and%20Event%20Cameras%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22presenter%22%2C%22firstName%22%3A%22Friedhelm%22%2C%22lastName%22%3A%22Hamann%22%7D%2C%7B%22creatorType%22%3A%22presenter%22%2C%22firstName%22%3A%22Guillermo%22%2C%22lastName%22%3A%22Gallego%22%7D%5D%2C%22abstractNote%22%3A%22This%20work%20introduces%20a%20co-capture%20system%20for%20multi-animal%20visual%20data%20acquisition%20using%20conventional%20cameras%20and%20event%20cameras.%20Event%20cameras%20offer%20multiple%20advantages%20over%20frame-based%20cameras%2C%20such%20as%20a%20high%20temporal%20resolution%20and%20temporal%20redundancy%20suppression%2C%20which%20enable%20us%20to%20efficiently%20capture%20the%20fast%20and%20erratic%20movements%20of%20fish.%20We%20furthermore%20present%20an%20event-based%20multi-animal%20tracking%20algorithm%2C%20which%20proves%20the%20feasibility%20of%20the%20approach%20and%20sets%20the%20baseline%20for%20further%20exploration%20of%20combining%20the%20advantages%20of%20event%20cameras%20and%20conventional%20cameras%20for%20multi-animal%20tracking.%22%2C%22date%22%3A%222022%22%2C%22sessionTitle%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2207.07332%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-09-11T07%3A32%3A54Z%22%7D%7D%5D%7D
Hamann, F., Mededovic, E., Gülhan, F., Wu, Y., Stegmaier, J., He, J., Wang, Y., Zhang, K., Li, L., Jiao, L., Ma, M., Huang, H., Yan, Y., Ren, H., Lin, X., Huang, Y., Cheng, B., Lee, S. H., Ham, G. S., … Gallego, G. (2025). SIS-Challenge: Event-Based Spatio-Temporal Instance Segmentation Challenge at the CVPR 2025 Event-Based Vision Workshop. 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 4734–4742. https://doi.org/10.1109/ICCVW69036.2025.00491
Guo, S., Hamann, F., & Gallego, G. (2025). Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), 7980–7989. https://doi.org/10.1109/ICCV51701.2025.00748
Hamann, F., Gehrig, D., Febryanto, F., Daniilidis, K., & Gallego, G. (2025). ETAP: Event-based Tracking of Any Point. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 27186–27196. https://doi.org/10.1109/CVPR52734.2025.02532
Wang, Z., Hamann, F., Chaney, K., Jiang, W., Gallego, G., & Daniilidis, K. (2025). Event-Based Continuous Color Video Decompression from Single Frames. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 4968–4978. https://doi.org/10.1109/CVPRW67362.2025.00490
Hamann, F., Ghosh, S., Juárez Martínez, I., Hart, T., Kacelnik, A., & Gallego, G. (2025). Fourier‐Based Action Recognition for Wildlife Behavior Quantification with Event Cameras. Advanced Intelligent Systems, 7(2), 2400353. https://doi.org/10.1002/aisy.202400353
Guo, S., Hamann, F., & Gallego, G. (2025). Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras. arXiv. https://doi.org/10.48550/ARXIV.2503.17262
Hamann, F., Wang, Z., Asmanis, I., Chaney, K., Gallego, G., & Daniilidis, K. (2025). Motion-Prior Contrast Maximization for Dense Continuous-Time Motion Estimation. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision – ECCV 2024 (pp. 18–37). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-72646-0_2
Hamann, F., Li, H., Mieske, P., Lewejohann, L., & Gallego, G. (2025). MouseSIS: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice. In A. Del Bue, C. Canton, J. Pont-Tuset, & T. Tommasi (Eds.), Computer Vision – ECCV 2024 Workshops (pp. 156–173). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-92460-6_10
Hamann, F., Ghosh, S., Martínez, I. J., Hart, T., Kacelnik, A., & Gallego, G. (2024). Low-power, Continuous Remote Behavioral Localization with Event Cameras. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 18612–18621. https://doi.org/10.1109/CVPR52733.2024.01761
Shiba, S., Hamann, F., Aoki, Y., & Gallego, G. (2024). Event-Based Background-Oriented Schlieren. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4), 2011–2026. https://doi.org/10.1109/TPAMI.2023.3328188
Hamann, F., Gehrig, D., Febryanto, F., Daniilidis, K., & Gallego, G. (2024). ETAP: Event-based Tracking of Any Point. https://doi.org/10.48550/ARXIV.2412.00133
Hamann, F., Ghosh, S., Martinez, I. J., Hart, T., Kacelnik, A., & Gallego, G. (2023). Low-power, Continuous Remote Behavioral Localization with Event Cameras. arXiv. https://doi.org/10.48550/ARXIV.2312.03799
Wang, Z., Hamann, F., Chaney, K., Jiang, W., Gallego, G., & Daniilidis, K. (2023). Event-based Continuous Color Video Decompression from Single Frames. arXiv. https://doi.org/10.48550/ARXIV.2312.00113
Hamann, F., & Gallego, G. (2022). Stereo Co-capture System for Recording and Tracking Fish with Frame- and Event Cameras [Poster]. Visual observation and analysis of Vertebrate And Insect Behavior (VAIB) Workshop at the 26th International Conference on Pattern Recognition (ICPR). https://doi.org/10.48550/arXiv.2207.07332
NSF AccelNet NeuroPac Fellowship
For the Public -
Science of Intelligence Fair 2023
All electronics – June 2023 - Neue Art von Kamera bringt Robotern das menschliche Sehen bei
Tu Berlin – June 2023 - Eventkameras: eine völlig neue Art zu fotografieren
