Florian Blume
Photo: SCIoI
Florian Blume has obtained his Master’s degree in Computer Science from the Technische Universität Dresden in December 2019. His main focus was on machine learning and computer vision, working with deep neural networks for pose estimation and biological image denoising. At SCIoI, he was a PhD student carrying our research on knowledge-augmented face perception.
Final dissertation: “Knowledge-augmented and context-sensitive face perception”, 21/01/2025.
Projects
Florian Blume is member of:
Publications +
6984777
Blume
1
apa
50
date
desc
year
19809
https://www.scienceofintelligence.de/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22KXJWLR47%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier%20et%20al.%22%2C%22parsedDate%22%3A%222025-04-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier%2C%20M.%2C%20Leonhardt%2C%20A.%2C%20Blume%2C%20F.%2C%20Bideau%2C%20P.%2C%20Hellwich%2C%20O.%2C%20%26amp%3B%20Abdel%20Rahman%2C%20R.%20%282025%29.%20Neural%20dynamics%20of%20mental%20state%20attribution%20to%20social%20robot%20faces.%20%26lt%3Bi%26gt%3BSocial%20Cognitive%20and%20Affective%20Neuroscience%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B20%26lt%3B%5C%2Fi%26gt%3B%281%29%2C%20nsaf027.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1093%5C%2Fscan%5C%2Fnsaf027%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1093%5C%2Fscan%5C%2Fnsaf027%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Neural%20dynamics%20of%20mental%20state%20attribution%20to%20social%20robot%20faces%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Maier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexander%22%2C%22lastName%22%3A%22Leonhardt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Blume%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pia%22%2C%22lastName%22%3A%22Bideau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olaf%22%2C%22lastName%22%3A%22Hellwich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rasha%22%2C%22lastName%22%3A%22Abdel%20Rahman%22%7D%5D%2C%22abstractNote%22%3A%22Abstract%5Cn%20%20%20%20%20%20%20%20%20%20%20%20The%20interplay%20of%20mind%20attribution%20and%20emotional%20responses%20is%20considered%20crucial%20in%20shaping%20human%20trust%20and%20acceptance%20of%20social%20robots.%20Understanding%20this%20interplay%20can%20help%20us%20create%20the%20right%20conditions%20for%20successful%20human%5Cu2013robot%20social%20interaction%20in%20alignment%20with%20societal%20goals.%20Our%20study%20shows%20that%20affective%20information%20about%20robots%20describing%20positive%2C%20negative%2C%20or%20neutral%20behaviour%20leads%20participants%20%28N%5Cu2009%3D%5Cu200990%29%20to%20attribute%20mental%20states%20to%20robot%20faces%2C%20modulating%20impressions%20of%20trustworthiness%2C%20facial%20expression%2C%20and%20intentionality.%20Electroencephalography%20recordings%20from%2030%20participants%20revealed%20that%20affective%20information%20influenced%20specific%20processing%20stages%20in%20the%20brain%20associated%20with%20early%20face%20perception%20%28N170%20component%29%20and%20more%20elaborate%20stimulus%20evaluation%20%28late%20positive%20potential%29.%20However%2C%20a%20modulation%20of%20fast%20emotional%20brain%20responses%2C%20typically%20found%20for%20human%20faces%20%28early%20posterior%20negativity%29%2C%20was%20not%20observed.%20These%20findings%20suggest%20that%20neural%20processing%20of%20robot%20faces%20alternates%20between%20being%20perceived%20as%20mindless%20machines%20and%20intentional%20agents%3A%20people%20rapidly%20attribute%20mental%20states%20during%20perception%2C%20literally%20seeing%20good%20or%20bad%20intentions%20in%20robot%20faces%2C%20but%20are%20emotionally%20less%20affected%20than%20when%20facing%20humans.%20These%20nuanced%20insights%20into%20the%20fundamental%20psychological%20and%20neural%20processes%20underlying%20mind%20attribution%20can%20enhance%20our%20understanding%20of%20human%5Cu2013robot%20social%20interactions%20and%20inform%20policies%20surrounding%20the%20moral%20responsibility%20of%20artificial%20agents.%22%2C%22date%22%3A%222025-04-02%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1093%5C%2Fscan%5C%2Fnsaf027%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Facademic.oup.com%5C%2Fscan%5C%2Farticle%5C%2Fdoi%5C%2F10.1093%5C%2Fscan%5C%2Fnsaf027%5C%2F8069429%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%221749-5016%2C%201749-5024%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-20T06%3A59%3A45Z%22%7D%7D%2C%7B%22key%22%3A%228FP86LMX%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Blume%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBlume%2C%20F.%20%282025%29.%20%26lt%3Bi%26gt%3BKnowledge-augmented%20and%20context-sensitive%20face%20perception%26lt%3B%5C%2Fi%26gt%3B%20%5BTechnische%20Universit%26%23xE4%3Bt%20Berlin%5D.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.14279%5C%2FDEPOSITONCE-23757%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.14279%5C%2FDEPOSITONCE-23757%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22thesis%22%2C%22title%22%3A%22Knowledge-augmented%20and%20context-sensitive%20face%20perception%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Blume%22%7D%2C%7B%22creatorType%22%3A%22contributor%22%2C%22name%22%3A%22Technische%20Universit%5Cu00e4t%20Berlin%22%7D%2C%7B%22creatorType%22%3A%22contributor%22%2C%22firstName%22%3A%22Olaf%22%2C%22lastName%22%3A%22Hellwich%22%7D%5D%2C%22abstractNote%22%3A%22Gesichtsausdr%5Cu00fccke%20spielen%20in%20der%20menschlichen%20Kommunikation%20eine%20entscheidende%20Rolle%2C%20denn%20sie%20vermitteln%20essenzielle%20Informationen%20%5Cu00fcber%20den%20inneren%20emotionalen%20Zustand%20einer%20Person.%20In%20der%20Vergangenheit%20hat%20die%20psychologische%20Forschung%20Gesichtsausdr%5Cu00fccke%20prim%5Cu00e4r%20in%20Isolation%20betrachtet.%20In%20der%20neueren%20Forschung%20gab%20es%20jedoch%20einen%20Paradigmenwechsel%20hin%20zu%20der%20Erkenntnis%2C%20dass%20die%20menschliche%20Wahrnehmung%20und%20menschliche%20Denkprozesse%20stark%20von%20Kontext%20und%20%28Vor-%29Wissen%20beeinflusst%20sind.%20So%20%5Cu00e4ndert%20sich%20zum%20Beispiel%20mit%20biografischen%20Informationen%20%5Cu00fcber%20die%20Person%2C%20wie%20deren%20L%5Cu00e4cheln%20wahrgenommen%20wird.%20Daher%20ist%20es%20unerl%5Cu00e4sslich%2C%20Wissen%20und%20Kontextinformationen%2C%20dynamisch%20und%20inspiriert%20durch%20menschliche%20Wahrnehmung%2C%20in%20Gesichtsausdruckserkennung%20%28GAE%29%20einzuarbeiten%2C%20um%20eine%20Diskrepanz%20zwischen%20k%5Cu00fcnstlicher%20Verarbeitung%20und%20menschlicher%20Wahrnehmung%20zu%20verhindern.%20Eine%20enge%20Zusammenarbeit%20von%20Forschern%20in%20Psychologie%20und%20Informatik%20ist%20notwendig.%20In%20dieser%20Dissertation%20arbeiten%20wir%20daran%2C%20Gesichtsausdr%5Cu00fccke%20automatisch%20mit%20tiefen%20neuronalen%20Netzen%20zu%20erkennen%2C%20aus%20einer%20interdisziplin%5Cu00e4ren%20und%20auf%20den%20Menschen%20fokussierten%20Perspektive%20heraus%20mit%20dem%20Ziel%2C%20GAE%20an%20die%20Dynamiken%20menschlicher%20Gesichtsausdruckswahrnehmung%20zu%20adaptieren%20und%20somit%20die%20Kommunikation%20zwischen%20Mensch%20und%20Maschine%20zu%20erleichtern%20und%20zu%20erm%5Cu00f6glichen.%20Um%20die%20Kommunikation%20zwischen%20allen%20Feldern%20zu%20erm%5Cu00f6glichen%2C%20begutachten%20wir%20die%20existierende%20Literatur%20unter%20Bezugnahme%20auf%20ein%20von%20uns%20entwickeltes%20Framework%20und%20einer%20gemeinsamen%20Terminologie.%20In%20diesem%20Framework%20identifizieren%20wir%20Kennzeichen%20menschlicher%2C%20kontextualisierter%20und%20wissenserweiterter%20Wahrnehmung%20und%20deduzieren%20programmatische%20Konzepte%20f%5Cu00fcr%20synthetische%20GAE%20Systeme.%20Wir%20stellen%20zwei%20verschiedene%20Methoden%20vor%2C%20um%20Kontextsensitivit%5Cu00e4t%20in%20GAE%20zu%20realisieren%3A%20Ein%20selbst-%5Cu00fcberwachter%20Lernansatz%2C%20der%20den%20Repr%5Cu00e4sentationsraum%20unter%20Zuhilfenahme%20von%20Kontextinformation%2C%20in%20Form%20von%20zus%5Cu00e4tzlichen%20Audio-%20und%20Textmodalit%5Cu00e4ten%20strukturiert.%20Eine%20zweite%20Methode%20adaptiert%20dynamisch%20die%20Repr%5Cu00e4sentation%20des%20Gesichtsausdrucks%20anhand%20von%20Audiokontext%20w%5Cu00e4hrend%20der%20Inferenz.%20Wir%20evaluieren%20beide%20Methoden%20auf%20aktuellen%20GAE%20Datens%5Cu00e4tzen%20und%20demonstrieren%2C%20dass%20unsere%20Ans%5Cu00e4tze%20Konkurrenzmethoden%20%5Cu00fcbertreffen%2C%20wobei%20unsere%20entweder%20un%5Cu00fcberwacht%20ohne%20Klassen%20trainiert%20werden%2C%20oder%20aber%20mit%20zus%5Cu00e4tzlichen%20generativen%20F%5Cu00e4higkeiten%20ausgestattet%20sind.%20Letzteres%20erlaubt%20Ann%5Cu00e4herungen%20an%20mentale%20Abbildungen%20von%20Gesichtsausdr%5Cu00fccken%20zu%20generieren.%20Automatische%20Generierung%2C%20wie%20wir%20sie%20in%20der%20vorliegenden%20Arbeit%20erreichen%2C%20kann%20f%5Cu00fcr%20die%20Forschung%20mit%20Gesichtern%20n%5Cu00fctzlich%20sein.%20Gleichzeitig%20erhalten%20wir%20so%20ein%20Werkzeug%20f%5Cu00fcr%20Erkl%5Cu00e4rbarkeit%20f%5Cu00fcr%20k%5Cu00fcnstliche%20Intelligenz%20%28KI%29%20Systeme%20und%20k%5Cu00f6nnen%20die%20synthetischen%20Repr%5Cu00e4sentationen%20von%20Gesichtern%20mit%20menschlichen%20vergleichen.%22%2C%22thesisType%22%3A%22%22%2C%22university%22%3A%22Technische%20Universit%5Cu00e4t%20Berlin%22%2C%22date%22%3A%222025%22%2C%22DOI%22%3A%2210.14279%5C%2FDEPOSITONCE-23757%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdepositonce.tu-berlin.de%5C%2Fhandle%5C%2F11303%5C%2F24941%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-20T12%3A17%3A15Z%22%7D%7D%2C%7B%22key%22%3A%22I47S7QBP%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Blume%20et%20al.%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBlume%2C%20F.%2C%20Qu%2C%20R.%2C%20Bideau%2C%20P.%2C%20Maier%2C%20M.%2C%20Rahman%2C%20R.%20A.%2C%20%26amp%3B%20Hellwich%2C%20O.%20%282025%29.%20How%20Do%20You%20Perceive%20My%20Face%3F%20Recognizing%20Facial%20Expressions%20in%26%23xA0%3BMulti-modal%20Context%20by%26%23xA0%3BModeling%20Mental%20Representations.%20In%20D.%20Cremers%2C%20Z.%20L%26%23xE4%3Bhner%2C%20M.%20Moeller%2C%20M.%20Nie%26%23xDF%3Bner%2C%20B.%20Ommer%2C%20%26amp%3B%20R.%20Triebel%20%28Eds.%29%2C%20%26lt%3Bi%26gt%3BPattern%20Recognition%26lt%3B%5C%2Fi%26gt%3B%20%28pp.%2020%26%23x2013%3B36%29.%20Springer%20Nature%20Switzerland.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2F978-3-031-85187-2_2%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2F978-3-031-85187-2_2%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22How%20Do%20You%20Perceive%20My%20Face%3F%20Recognizing%20Facial%20Expressions%20in%5Cu00a0Multi-modal%20Context%20by%5Cu00a0Modeling%20Mental%20Representations%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Blume%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Runfeng%22%2C%22lastName%22%3A%22Qu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pia%22%2C%22lastName%22%3A%22Bideau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Maier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rasha%20Abdel%22%2C%22lastName%22%3A%22Rahman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olaf%22%2C%22lastName%22%3A%22Hellwich%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Cremers%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Zorah%22%2C%22lastName%22%3A%22L%5Cu00e4hner%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Moeller%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Nie%5Cu00dfner%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Bj%5Cu00f6rn%22%2C%22lastName%22%3A%22Ommer%22%7D%2C%7B%22creatorType%22%3A%22editor%22%2C%22firstName%22%3A%22Rudolph%22%2C%22lastName%22%3A%22Triebel%22%7D%5D%2C%22abstractNote%22%3A%22Facial%20expression%20perception%20in%20humans%20inherently%20relies%20on%20prior%20knowledge%20and%20contextual%20cues%2C%20contributing%20to%20efficient%20and%20flexible%20processing.%20For%20instance%2C%20multi-modal%20emotional%20context%20%28such%20as%20voice%20color%2C%20affective%20text%2C%20body%20pose%2C%20etc.%29%20can%20prompt%20people%20to%20perceive%20emotional%20expressions%20in%20objectively%20neutral%20faces.%20Drawing%20inspiration%20from%20this%2C%20we%20introduce%20a%20novel%20approach%20for%20facial%20expression%20classification%20that%20goes%20beyond%20simple%20classification%20tasks.%20Our%20model%20accurately%20classifies%20a%20perceived%20face%20and%20synthesizes%20the%20corresponding%20mental%20representation%20perceived%20by%20a%20human%20when%20observing%20a%20face%20in%20context.%20With%20this%2C%20our%20model%20offers%20visual%20insights%20into%20its%20internal%20decision-making%20process.%20We%20achieve%20this%20by%20learning%20two%20independent%20representations%20of%20content%20and%20context%20using%20a%20VAE-GAN%20architecture.%20Subsequently%2C%20we%20propose%20a%20novel%20attention%20mechanism%20for%20context-dependent%20feature%20adaptation.%20The%20adapted%20representation%20is%20used%20for%20classification%20and%20to%20generate%20a%20context-augmented%20expression.%20We%20evaluate%20synthesized%20expressions%20in%20a%20human%20study%2C%20showing%20that%20our%20model%20effectively%20produces%20approximations%20of%20human%20mental%20representations.%20We%20achieve%20State-of-the-Art%20classification%20accuracies%20of%2081.01%25%20on%20the%20RAVDESS%20dataset%20and%2079.34%25%20on%20the%20MEAD%20dataset.%20We%20make%20our%20code%20publicly%20available.%22%2C%22proceedingsTitle%22%3A%22Pattern%20Recognition%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222025%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1007%5C%2F978-3-031-85187-2_2%22%2C%22ISBN%22%3A%22978-3-031-85187-2%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A16Z%22%7D%7D%2C%7B%22key%22%3A%22QKAHMER8%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Halawa%20et%20al.%22%2C%22parsedDate%22%3A%222024-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHalawa%2C%20M.%2C%20Blume%2C%20F.%2C%20Bideau%2C%20P.%2C%20Maier%2C%20M.%2C%20Rahman%2C%20R.%20A.%2C%20%26amp%3B%20Hellwich%2C%20O.%20%282024%29.%20Multi-Task%20Multi-Modal%20Self-Supervised%20Learning%20for%20Facial%20Expression%20Recognition.%20%26lt%3Bi%26gt%3B2024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20Workshops%20%28CVPRW%29%26lt%3B%5C%2Fi%26gt%3B%2C%204604%26%23x2013%3B4614.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPRW63382.2024.00463%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPRW63382.2024.00463%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Multi-Task%20Multi-Modal%20Self-Supervised%20Learning%20for%20Facial%20Expression%20Recognition%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marah%22%2C%22lastName%22%3A%22Halawa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Blume%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pia%22%2C%22lastName%22%3A%22Bideau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Maier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rasha%20Abdel%22%2C%22lastName%22%3A%22Rahman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olaf%22%2C%22lastName%22%3A%22Hellwich%22%7D%5D%2C%22abstractNote%22%3A%22Human%20communication%20is%20multi-modal%3B%20e.g.%2C%20face-to-face%20interaction%20involves%20auditory%20signals%20%28speech%29%20and%20visual%20signals%20%28face%20movements%20and%20hand%20gestures%29.%20Hence%2C%20it%20is%20essential%20to%20exploit%20multiple%20modalities%20when%20designing%20machine%20learning-based%20facial%20expression%20recognition%20systems.%20In%20addition%2C%20given%20the%20ever-growing%20quantities%20of%20video%20data%20that%20capture%20human%20facial%20expressions%2C%20such%20systems%20should%20utilize%20raw%20unlabeled%20videos%20without%20requiring%20expensive%20annotations.%20Therefore%2C%20in%20this%20work%2C%20we%20employ%20a%20multitask%20multi-modal%20self-supervised%20learning%20method%20for%20facial%20expression%20recognition%20from%20in-the-wild%20video%20data.%20Our%20model%20combines%20three%20self-supervised%20objective%20functions%3A%20First%2C%20a%20multi-modal%20contrastive%20loss%2C%20that%20pulls%20diverse%20data%20modalities%20of%20the%20same%20video%20together%20in%20the%20representation%20space.%20Second%2C%20a%20multi-modal%20clustering%20loss%20that%20preserves%20the%20semantic%20structure%20of%20input%20data%20in%20the%20representation%20space.%20Finally%2C%20a%20multi-modal%20data%20reconstruction%20loss.%20We%20conduct%20a%20comprehensive%20study%20on%20this%20multimodal%20multi-task%20self-supervised%20learning%20method%20on%20three%20facial%20expression%20recognition%20benchmarks.%20To%20that%20end%2C%20we%20examine%20the%20performance%20of%20learning%20through%20different%20combinations%20of%20self-supervised%20tasks%20on%20the%20facial%20expression%20recognition%20downstream%20task.%20Our%20model%20ConCluGen%20outperforms%20several%20multi-modal%20self-supervised%20and%20fully%20supervised%20baselines%20on%20the%20CMU-MOSEI%20dataset.%20Our%20results%20generally%20show%20that%20multi-modal%20self-supervision%20tasks%20offer%20large%20performance%20gains%20for%20challenging%20tasks%20such%20as%20facial%20expression%20recognition%2C%20while%20also%20reducing%20the%20amount%20of%20manual%20annotations%20required.%20We%20release%20our%20pre-trained%20models%20as%20well%20as%20source%20code%20publicly1.%22%2C%22proceedingsTitle%22%3A%222024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20Workshops%20%28CVPRW%29%22%2C%22conferenceName%22%3A%222024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20Workshops%20%28CVPRW%29%22%2C%22date%22%3A%222024-06%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FCVPRW63382.2024.00463%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F10678641%22%2C%22ISSN%22%3A%222160-7516%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T12%3A01%3A10Z%22%7D%7D%2C%7B%22key%22%3A%22MQW2YAL9%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier%2C%20M.%2C%20Blume%2C%20F.%2C%20Bideau%2C%20P.%2C%20Hellwich%2C%20O.%2C%20%26amp%3B%20Abdel%20Rahman%2C%20R.%20%282022%29.%20Knowledge-augmented%20face%20perception%3A%20Prospects%20for%20the%20Bayesian%20brain-framework%20to%20align%20AI%20and%20human%20vision.%20%26lt%3Bi%26gt%3BConsciousness%20and%20Cognition%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B101%26lt%3B%5C%2Fi%26gt%3B%2C%20103301.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.concog.2022.103301%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.concog.2022.103301%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Knowledge-augmented%20face%20perception%3A%20Prospects%20for%20the%20Bayesian%20brain-framework%20to%20align%20AI%20and%20human%20vision%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Maier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Blume%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pia%22%2C%22lastName%22%3A%22Bideau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olaf%22%2C%22lastName%22%3A%22Hellwich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rasha%22%2C%22lastName%22%3A%22Abdel%20Rahman%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%2205%5C%2F2022%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.concog.2022.103301%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Flinkinghub.elsevier.com%5C%2Fretrieve%5C%2Fpii%5C%2FS1053810022000332%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%2210538100%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-08-20T05%3A55%3A11Z%22%7D%7D%5D%7D
Maier, M., Leonhardt, A., Blume, F., Bideau, P., Hellwich, O., & Abdel Rahman, R. (2025). Neural dynamics of mental state attribution to social robot faces. Social Cognitive and Affective Neuroscience, 20(1), nsaf027. https://doi.org/10.1093/scan/nsaf027
Blume, F. (2025). Knowledge-augmented and context-sensitive face perception [Technische Universität Berlin]. https://doi.org/10.14279/DEPOSITONCE-23757
Blume, F., Qu, R., Bideau, P., Maier, M., Rahman, R. A., & Hellwich, O. (2025). How Do You Perceive My Face? Recognizing Facial Expressions in Multi-modal Context by Modeling Mental Representations. In D. Cremers, Z. Lähner, M. Moeller, M. Nießner, B. Ommer, & R. Triebel (Eds.), Pattern Recognition (pp. 20–36). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-85187-2_2
Halawa, M., Blume, F., Bideau, P., Maier, M., Rahman, R. A., & Hellwich, O. (2024). Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 4604–4614. https://doi.org/10.1109/CVPRW63382.2024.00463
Maier, M., Blume, F., Bideau, P., Hellwich, O., & Abdel Rahman, R. (2022). Knowledge-augmented face perception: Prospects for the Bayesian brain-framework to align AI and human vision. Consciousness and Cognition, 101, 103301. https://doi.org/10.1016/j.concog.2022.103301
