Danny Driess
Photo: SCIoI
Danny Driess worked as a doctoral researcher in Marc Toussaint’s team. His PhD focused on learning in Task and Motion Planning with a special focus on connecting perception and planning for sequential manipulation tasks through machine learning. Within SCIoI, he investigated novel ways to couple sequential manipulation planning with reactive policies for execution. He received his Bachelor of Science (with distinction) in Simulation Technology in 2016 and his Master of Science (with distinction) in Simulation Technology in 2019, both from the University of Stuttgart.
Final dissertation: “Learning for sequential manipulation”, 21/05/2024.
Projects
Danny Driess is member of:
• Project 39: Physical manipulation planning with differentiable closed-loop manipulation primitives
Publications +
6984777
Driess
1
apa
50
date
desc
year
19841
https://www.scienceofintelligence.de/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22TRZCTPB5%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222023-07-03%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Xia%2C%20F.%2C%20Sajjadi%2C%20M.%20S.%20M.%2C%20Lynch%2C%20C.%2C%20Chowdhery%2C%20A.%2C%20Ichter%2C%20B.%2C%20Wahid%2C%20A.%2C%20Tompson%2C%20J.%2C%20Vuong%2C%20Q.%2C%20Yu%2C%20T.%2C%20Huang%2C%20W.%2C%20Chebotar%2C%20Y.%2C%20Sermanet%2C%20P.%2C%20Duckworth%2C%20D.%2C%20Levine%2C%20S.%2C%20Vanhoucke%2C%20V.%2C%20Hausman%2C%20K.%2C%20Toussaint%2C%20M.%2C%20Greff%2C%20K.%2C%20%26%23x2026%3B%20Florence%2C%20P.%20%282023%29.%20PaLM-E%3A%20An%20Embodied%20Multimodal%20Language%20Model.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%2040th%20International%20Conference%20on%20Machine%20Learning%20%28ICML%29%26lt%3B%5C%2Fi%26gt%3B%2C%208469%26%23x2013%3B8488.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv202%5C%2Fdriess23a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv202%5C%2Fdriess23a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22PaLM-E%3A%20An%20Embodied%20Multimodal%20Language%20Model%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fei%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mehdi%20S.%20M.%22%2C%22lastName%22%3A%22Sajjadi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Corey%22%2C%22lastName%22%3A%22Lynch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aakanksha%22%2C%22lastName%22%3A%22Chowdhery%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brian%22%2C%22lastName%22%3A%22Ichter%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ayzaan%22%2C%22lastName%22%3A%22Wahid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Tompson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Quan%22%2C%22lastName%22%3A%22Vuong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tianhe%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wenlong%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yevgen%22%2C%22lastName%22%3A%22Chebotar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Sermanet%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Duckworth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sergey%22%2C%22lastName%22%3A%22Levine%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vincent%22%2C%22lastName%22%3A%22Vanhoucke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karol%22%2C%22lastName%22%3A%22Hausman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Greff%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andy%22%2C%22lastName%22%3A%22Zeng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Igor%22%2C%22lastName%22%3A%22Mordatch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pete%22%2C%22lastName%22%3A%22Florence%22%7D%5D%2C%22abstractNote%22%3A%22Large%20language%20models%20excel%20at%20a%20wide%20range%20of%20complex%20tasks.%20However%2C%20enabling%20general%20inference%20in%20the%20real%20world%2C%20e.g.%20for%20robotics%20problems%2C%20raises%20the%20challenge%20of%20grounding.%20We%20propose%20embodied%20language%20models%20to%20directly%20incorporate%20real-world%20continuous%20sensor%20modalities%20into%20language%20models%20and%20thereby%20establish%20the%20link%20between%20words%20and%20percepts.%20Input%20to%20our%20embodied%20language%20model%20are%20multimodal%20sentences%20that%20interleave%20visual%2C%20continuous%20state%20estimation%2C%20and%20textual%20input%20encodings.%20We%20train%20these%20encodings%20end-to-end%2C%20in%20conjunction%20with%20a%20pre-trained%20large%20language%20model%2C%20for%20multiple%20embodied%20tasks%20including%20sequential%20robotic%20manipulation%20planning%2C%20visual%20question%20answering%2C%20and%20captioning.%20Our%20evaluations%20show%20that%20PaLM-E%2C%20a%20single%20large%20embodied%20multimodal%20model%2C%20can%20address%20a%20variety%20of%20embodied%20reasoning%20tasks%2C%20from%20a%20variety%20of%20observation%20modalities%2C%20on%20multiple%20embodiments%2C%20and%20further%2C%20exhibits%20positive%20transfer%3A%20the%20model%20benefits%20from%20diverse%20joint%20training%20across%20internet-scale%20language%2C%20vision%2C%20and%20visual-language%20domains.%20Our%20largest%20model%20with%20562B%20parameters%2C%20in%20addition%20to%20being%20trained%20on%20robotics%20tasks%2C%20is%20a%20visual-language%20generalist%20with%20state-of-the-art%20performance%20on%20OK-VQA%2C%20and%20retains%20generalist%20language%20capabilities%20with%20increasing%20scale.%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%2040th%20International%20Conference%20on%20Machine%20Learning%20%28ICML%29%22%2C%22conferenceName%22%3A%22International%20Conference%20on%20Machine%20Learning%22%2C%22date%22%3A%222023-07-03%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv202%5C%2Fdriess23a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T13%3A37%3A18Z%22%7D%7D%2C%7B%22key%22%3A%22Z7BVH4H4%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Huang%2C%20Z.%2C%20Li%2C%20Y.%2C%20Tedrake%2C%20R.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282023%29.%20Learning%20Multi-Object%20Dynamics%20with%20Compositional%20Neural%20Radiance%20Fields.%20%26lt%3Bi%26gt%3BProceedings%20of%20The%206th%20Conference%20on%20Robot%20Learning%20%28CoRL%202022%29%26lt%3B%5C%2Fi%26gt%3B%2C%201755%26%23x2013%3B1768.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv205%5C%2Fdriess23a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv205%5C%2Fdriess23a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Learning%20Multi-Object%20Dynamics%20with%20Compositional%20Neural%20Radiance%20Fields%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhiao%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yunzhu%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Russ%22%2C%22lastName%22%3A%22Tedrake%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22We%20present%20a%20method%20to%20learn%20compositional%20multi-object%20dynamics%20models%20from%20image%20observations%20based%20on%20implicit%20object%20encoders%2C%20Neural%20Radiance%20Fields%20%28NeRFs%29%2C%20and%20graph%20neural%20networks.%20NeRFs%20have%20become%20a%20popular%20choice%20for%20representing%20scenes%20due%20to%20their%20strong%203D%20prior.%20However%2C%20most%20NeRF%20approaches%20are%20trained%20on%20a%20single%20scene%2C%20representing%20the%20whole%20scene%20with%20a%20global%20model%2C%20making%20generalization%20to%20novel%20scenes%2C%20containing%20different%20numbers%20of%20objects%2C%20challenging.%20Instead%2C%20we%20present%20a%20compositional%2C%20object-centric%20auto-encoder%20framework%20that%20maps%20multiple%20views%20of%20the%20scene%20to%20a%20set%20of%20latent%20vectors%20representing%20each%20object%20separately.%20The%20latent%20vectors%20parameterize%20individual%20NeRFs%20from%20which%20the%20scene%20can%20be%20reconstructed.%20Based%20on%20those%20latent%20vectors%2C%20we%20train%20a%20graph%20neural%20network%20dynamics%20model%20in%20the%20latent%20space%20to%20achieve%20compositionality%20for%20dynamics%20prediction.%20A%20key%20feature%20of%20our%20approach%20is%20that%20the%20latent%20vectors%20are%20forced%20to%20encode%203D%20information%20through%20the%20NeRF%20decoder%2C%20which%20enables%20us%20to%20incorporate%20structural%20priors%20in%20learning%20the%20dynamics%20models%2C%20making%20long-term%20predictions%20more%20stable%20compared%20to%20several%20baselines.%20Simulated%20and%20real%20world%20experiments%20show%20that%20our%20method%20can%20model%20and%20learn%20the%20dynamics%20of%20compositional%20scenes%20including%20rigid%20and%20deformable%20objects.%20Video%3A%20https%3A%5C%2F%5C%2Fdannydriess.github.io%5C%2Fcompnerfdyn%5C%2F%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20The%206th%20Conference%20on%20Robot%20Learning%20%28CoRL%202022%29%22%2C%22conferenceName%22%3A%22Conference%20on%20Robot%20Learning%22%2C%22date%22%3A%222023-03-06%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv205%5C%2Fdriess23a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T13%3A39%3A15Z%22%7D%7D%2C%7B%22key%22%3A%22Q2Z3NLG2%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Harris%20et%20al.%22%2C%22parsedDate%22%3A%222022-10-23%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHarris%2C%20J.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20FC%26lt%3Bsup%26gt%3B3%26lt%3B%5C%2Fsup%26gt%3B%26%23x202F%3B%3A%20Feasibility-Based%20Control%20Chain%20Coordination.%20%26lt%3Bi%26gt%3B2022%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%20%28IROS%29%26lt%3B%5C%2Fi%26gt%3B%2C%2013769%26%23x2013%3B13776.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9981758%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9981758%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22FC%3Csup%3E3%3C%5C%2Fsup%3E%20%3A%20Feasibility-Based%20Control%20Chain%20Coordination%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jason%22%2C%22lastName%22%3A%22Harris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%222022%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%20%28IROS%29%22%2C%22conferenceName%22%3A%222022%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%20%28IROS%29%22%2C%22date%22%3A%222022-10-23%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FIROS47612.2022.9981758%22%2C%22ISBN%22%3A%22978-1-6654-7927-1%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9981758%5C%2F%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A28Z%22%7D%7D%2C%7B%22key%22%3A%227GMQQ8NR%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Toussaint%20et%20al.%22%2C%22parsedDate%22%3A%222022-10-23%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BToussaint%2C%20M.%2C%20Harris%2C%20J.%2C%20Ha%2C%20J.-S.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20H%26%23xF6%3Bnig%2C%20W.%20%282022%29.%20Sequence-of-Constraints%20MPC%3A%20Reactive%20Timing-Optimal%20Control%20of%20Sequential%20Manipulation.%20%26lt%3Bi%26gt%3B2022%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%20%28IROS%29%26lt%3B%5C%2Fi%26gt%3B%2C%2013753%26%23x2013%3B13760.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9982236%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9982236%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Sequence-of-Constraints%20MPC%3A%20Reactive%20Timing-Optimal%20Control%20of%20Sequential%20Manipulation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jason%22%2C%22lastName%22%3A%22Harris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wolfgang%22%2C%22lastName%22%3A%22H%5Cu00f6nig%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%222022%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%20%28IROS%29%22%2C%22conferenceName%22%3A%222022%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%20%28IROS%29%22%2C%22date%22%3A%222022-10-23%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FIROS47612.2022.9982236%22%2C%22ISBN%22%3A%22978-1-6654-7927-1%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9982236%5C%2F%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A28Z%22%7D%7D%2C%7B%22key%22%3A%22ZUQKM6LP%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222022-01-11%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Ha%2C%20J.-S.%2C%20Toussaint%2C%20M.%2C%20%26amp%3B%20Tedrake%2C%20R.%20%282022%29.%20Learning%20Models%20as%20Functionals%20of%20Signed-Distance%20Fields%20for%20Manipulation%20Planning.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%205th%20Conference%20on%20Robot%20Learning%20%28CoRL%202021%29%26lt%3B%5C%2Fi%26gt%3B%2C%20245%26%23x2013%3B255.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fdriess22a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fdriess22a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Learning%20Models%20as%20Functionals%20of%20Signed-Distance%20Fields%20for%20Manipulation%20Planning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Russ%22%2C%22lastName%22%3A%22Tedrake%22%7D%5D%2C%22abstractNote%22%3A%22This%20work%20proposes%20an%20optimization-based%20manipulation%20planning%20framework%20where%20the%20objectives%20are%20learned%20functionals%20of%20signed-distance%20fields%20that%20represent%20objects%20in%20the%20scene.%20Most%20manipulation%20planning%20approaches%20rely%20on%20analytical%20models%20and%20carefully%20chosen%20abstractions%5C%2Fstate-spaces%20to%20be%20effective.%20A%20central%20question%20is%20how%20models%20can%20be%20obtained%20from%20data%20that%20are%20not%20primarily%20accurate%20in%20their%20predictions%2C%20but%2C%20more%20importantly%2C%20enable%20efficient%20reasoning%20within%20a%20planning%20framework%2C%20while%20at%20the%20same%20time%20being%20closely%20coupled%20to%20perception%20spaces.%20We%20show%20that%20representing%20objects%20as%20signed-distance%20fields%20not%20only%20enables%20to%20learn%20and%20represent%20a%20variety%20of%20models%20with%20higher%20accuracy%20compared%20to%20point-cloud%20and%20occupancy%20measure%20representations%2C%20but%20also%20that%20SDF-based%20models%20are%20suitable%20for%20optimization-based%20planning.%20To%20demonstrate%20the%20versatility%20of%20our%20approach%2C%20we%20learn%20both%20kinematic%20and%20dynamic%20models%20to%20solve%20tasks%20that%20involve%20hanging%20mugs%20on%20hooks%20and%20pushing%20objects%20on%20a%20table.%20We%20can%20unify%20these%20quite%20different%20tasks%20within%20one%20framework%2C%20since%20SDFs%20are%20the%20common%20object%20representation.%20%20Video%3A%20https%3A%5C%2F%5C%2Fyoutu.be%5C%2Fga8Wlkss7co%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%205th%20Conference%20on%20Robot%20Learning%20%28CoRL%202021%29%22%2C%22conferenceName%22%3A%22Conference%20on%20Robot%20Learning%22%2C%22date%22%3A%222022-01-11%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fdriess22a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A09Z%22%7D%7D%2C%7B%22key%22%3A%226FMEQ52D%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ortiz-Haro%20et%20al.%22%2C%22parsedDate%22%3A%222022-01-11%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BOrtiz-Haro%2C%20J.%2C%20Ha%2C%20J.-S.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20Structured%20deep%20generative%20models%20for%20sampling%20on%20constraint%20manifolds%20in%20sequential%20manipulation.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%205th%20Conference%20on%20Robot%20Learning%20%28CoRL%202021%29%26lt%3B%5C%2Fi%26gt%3B%2C%20213%26%23x2013%3B223.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fortiz-haro22a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fortiz-haro22a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Structured%20deep%20generative%20models%20for%20sampling%20on%20constraint%20manifolds%20in%20sequential%20manipulation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Joaquim%22%2C%22lastName%22%3A%22Ortiz-Haro%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22Sampling%20efficiently%20on%20constraint%20manifolds%20is%20a%20core%20problem%20in%20robotics.%20We%20propose%20Deep%20Generative%20Constraint%20Sampling%20%28DGCS%29%2C%20which%20combines%20a%20deep%20generative%20model%20for%20sampling%20close%20to%20a%20constraint%20manifold%20with%20nonlinear%20constrained%20optimization%20to%20project%20to%20the%20constraint%20manifold.%20The%20generative%20model%20is%20conditioned%20on%20the%20problem%20instance%2C%20taking%20a%20scene%20image%20as%20input%2C%20and%20it%20is%20trained%20with%20a%20dataset%20of%20solutions%20and%20a%20novel%20analytic%20constraint%20term.%20To%20further%20improve%20the%20precision%20and%20diversity%20of%20samples%2C%20we%20extend%20the%20approach%20to%20exploit%20a%20factorization%20of%20the%20constrained%20problem.%20We%20evaluate%20our%20approach%20in%20two%20problems%20of%20robotic%20sequential%20manipulation%20in%20cluttered%20environments.%20Experimental%20results%20demonstrate%20that%20our%20deep%20generative%20model%20produces%20diverse%20and%20precise%20samples%20and%20outperforms%20heuristic%20warmstart%20initialization.%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%205th%20Conference%20on%20Robot%20Learning%20%28CoRL%202021%29%22%2C%22conferenceName%22%3A%22Conference%20on%20Robot%20Learning%22%2C%22date%22%3A%222022-01-11%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fortiz-haro22a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-04T13%3A54%3A36Z%22%7D%7D%2C%7B%22key%22%3A%22FSWYC9SD%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ha%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHa%2C%20J.-S.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20Deep%20Visual%20Constraints%3A%20Neural%20Implicit%20Models%20for%20Manipulation%20Planning%20From%20Visual%20Input.%20%26lt%3Bi%26gt%3BIEEE%20Robotics%20and%20Automation%20Letters%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B7%26lt%3B%5C%2Fi%26gt%3B%284%29%2C%2010857%26%23x2013%3B10864.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FLRA.2022.3194955%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FLRA.2022.3194955%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Deep%20Visual%20Constraints%3A%20Neural%20Implicit%20Models%20for%20Manipulation%20Planning%20From%20Visual%20Input%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%2210%5C%2F2022%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FLRA.2022.3194955%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9844753%5C%2F%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%222377-3766%2C%202377-3774%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A27Z%22%7D%7D%2C%7B%22key%22%3A%22H2UBVSA3%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Schubert%2C%20I.%2C%20Florence%2C%20P.%2C%20Li%2C%20Y.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20Reinforcement%20Learning%20with%20Neural%20Radiance%20Fields.%20%26lt%3Bi%26gt%3BAdvances%20in%20Neural%20Information%20Processing%20Systems%20%28NeurIPS%29%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B35%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.neurips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2022%5C%2Ffile%5C%2F6c294f059e3d77d58dbb8fe48f21fe00-Paper-Conference.pdf%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.neurips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2022%5C%2Ffile%5C%2F6c294f059e3d77d58dbb8fe48f21fe00-Paper-Conference.pdf%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Reinforcement%20Learning%20with%20Neural%20Radiance%20Fields%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ingmar%22%2C%22lastName%22%3A%22Schubert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pete%22%2C%22lastName%22%3A%22Florence%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yunzhu%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22Advances%20in%20Neural%20Information%20Processing%20Systems%20%28NeurIPS%29%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222022%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.neurips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2022%5C%2Ffile%5C%2F6c294f059e3d77d58dbb8fe48f21fe00-Paper-Conference.pdf%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-05-26T15%3A31%3A56Z%22%7D%7D%2C%7B%22key%22%3A%22CBM6GXGF%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222021%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Ha%2C%20J.-S.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282021%29.%20Learning%20to%20solve%20sequential%20physical%20reasoning%20problems%20from%20a%20scene%20image.%20%26lt%3Bi%26gt%3BThe%20International%20Journal%20of%20Robotics%20Research%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B40%26lt%3B%5C%2Fi%26gt%3B%2812%26%23x2013%3B14%29%2C%201435%26%23x2013%3B1466.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1177%5C%2F02783649211056967%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1177%5C%2F02783649211056967%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Learning%20to%20solve%20sequential%20physical%20reasoning%20problems%20from%20a%20scene%20image%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22In%20this%20article%2C%20we%20propose%20deep%20visual%20reasoning%2C%20which%20is%20a%20convolutional%20recurrent%20neural%20network%20that%20predicts%20discrete%20action%20sequences%20from%20an%20initial%20scene%20image%20for%20sequential%20manipulation%20problems%20that%20arise%2C%20for%20example%2C%20in%20task%20and%20motion%20planning%20%28TAMP%29.%20Typical%20TAMP%20problems%20are%20formalized%20by%20combining%20reasoning%20on%20a%20symbolic%2C%20discrete%20level%20%28e.g.%2C%20first-order%20logic%29%20with%20continuous%20motion%20planning%20such%20as%20nonlinear%20trajectory%20optimization.%20The%20action%20sequences%20represent%20the%20discrete%20decisions%20on%20a%20symbolic%20level%2C%20which%2C%20in%20turn%2C%20parameterize%20a%20nonlinear%20trajectory%20optimization%20problem.%20Owing%20to%20the%20great%20combinatorial%20complexity%20of%20possible%20discrete%20action%20sequences%2C%20a%20large%20number%20of%20optimization%5C%2Fmotion%20planning%20problems%20have%20to%20be%20solved%20to%20find%20a%20solution%2C%20which%20limits%20the%20scalability%20of%20these%20approaches.%20To%20circumvent%20this%20combinatorial%20complexity%2C%20we%20introduce%20deep%20visual%20reasoning%3A%20based%20on%20a%20segmented%20initial%20image%20of%20the%20scene%2C%20a%20neural%20network%20directly%20predicts%20promising%20discrete%20action%20sequences%20such%20that%20ideally%20only%20one%20motion%20planning%20problem%20has%20to%20be%20solved%20to%20find%20a%20solution%20to%20the%20overall%20TAMP%20problem.%20Our%20method%20generalizes%20to%20scenes%20with%20many%20and%20varying%20numbers%20of%20objects%2C%20although%20being%20trained%20on%20only%20two%20objects%20at%20a%20time.%20This%20is%20possible%20by%20encoding%20the%20objects%20of%20the%20scene%20and%20the%20goal%20in%20%28segmented%29%20images%20as%20input%20to%20the%20neural%20network%2C%20instead%20of%20a%20fixed%20feature%20vector.%20We%20show%20that%20the%20framework%20can%20not%20only%20handle%20kinematic%20problems%20such%20as%20pick-and-place%20%28as%20typical%20in%20TAMP%29%2C%20but%20also%20tool-use%20scenarios%20for%20planar%20pushing%20under%20quasi-static%20dynamic%20models.%20Here%2C%20the%20image-based%20representation%20enables%20generalization%20to%20other%20shapes%20than%20during%20training.%20Results%20show%20runtime%20improvements%20of%20several%20orders%20of%20magnitudes%20by%2C%20in%20many%20cases%2C%20removing%20the%20need%20to%20search%20over%20the%20discrete%20action%20sequences.%22%2C%22date%22%3A%2212%5C%2F2021%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1177%5C%2F02783649211056967%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fjournals.sagepub.com%5C%2Fdoi%5C%2F10.1177%5C%2F02783649211056967%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%220278-3649%2C%201741-3176%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-29T10%3A40%3A29Z%22%7D%7D%2C%7B%22key%22%3A%225IFJTUK6%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Schubert%20et%20al.%22%2C%22parsedDate%22%3A%222021%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BSchubert%2C%20I.%2C%20Driess%2C%20D.%2C%20Oguz%2C%20O.%20S.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282021%29.%20Learning%20to%20execute%3A%20efficiently%20learning%20universal%20plan-conditioned%20policies%20in%20robotics.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%2035th%20International%20Conference%20on%20Neural%20Information%20Processing%20Systems%2C%20NIPS%20%26%23x2019%3B21%26lt%3B%5C%2Fi%26gt%3B%2C%201912%26%23x2013%3B1924.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fargmin.lis.tu-berlin.de%5C%2Fpapers%5C%2F21-schubert-NeurIPS.pdf%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fargmin.lis.tu-berlin.de%5C%2Fpapers%5C%2F21-schubert-NeurIPS.pdf%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Learning%20to%20execute%3A%20efficiently%20learning%20universal%20plan-conditioned%20policies%20in%20robotics%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ingmar%22%2C%22lastName%22%3A%22Schubert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ozgur%20S.%22%2C%22lastName%22%3A%22Oguz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22Applications%20of%20Reinforcement%20Learning%20%28RL%29%20in%20robotics%20are%20often%20limited%20by%20high%20data%20demand.%20On%20the%20other%20hand%2C%20approximate%20models%20are%20readily%20available%20in%20many%20robotics%20scenarios%2C%20making%20model-based%20approaches%20like%20planning%20a%20data-efficient%20alternative.%20Still%2C%20the%20performance%20of%20these%20methods%20suffers%20if%20the%20model%20is%20imprecise%20or%20wrong.%20In%20this%20sense%2C%20the%20respective%20strengths%20and%20weaknesses%20of%20RL%20and%20model-based%20planners%20are%20complementary.%20In%20the%20present%20work%2C%20we%20investigate%20how%20both%20approaches%20can%20be%20integrated%20into%20one%20framework%20that%20combines%20their%20strengths.%20We%20introduce%20Learning%20to%20Execute%20%28L2E%29%2C%20which%20leverages%20information%20contained%20in%20approximate%20plans%20to%20learn%20universal%20policies%20that%20are%20conditioned%20on%20plans.%20In%20our%20robotic%20manipulation%20experiments%2C%20L2E%20exhibits%20increased%20performance%20when%20compared%20to%20pure%20RL%2C%20pure%20planning%2C%20or%20baseline%20methods%20combining%20learning%20and%20planning.%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%2035th%20International%20Conference%20on%20Neural%20Information%20Processing%20Systems%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%22Dezember%206%2C%202021%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22978-1-7138-4539-3%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fargmin.lis.tu-berlin.de%5C%2Fpapers%5C%2F21-schubert-NeurIPS.pdf%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-04T14%3A08%3A43Z%22%7D%7D%2C%7B%22key%22%3A%22QN3L779A%22%2C%22library%22%3A%7B%22id%22%3A6984777%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222020%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Ha%2C%20J.-S.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282020%29.%20Deep%20Visual%20Reasoning%3A%20Learning%20to%20Predict%20Action%20Sequences%20for%20Task%20and%20Motion%20Planning%20from%20an%20Initial%20Scene%20Image.%20%26lt%3Bi%26gt%3BCorvalis%2C%20Oregon%2C%20USA%26lt%3B%5C%2Fi%26gt%3B.%20Robotics%3A%20Science%20and%20Systems.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.15607%5C%2FRSS.2020.XVI.003%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.15607%5C%2FRSS.2020.XVI.003%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Deep%20Visual%20Reasoning%3A%20Learning%20to%20Predict%20Action%20Sequences%20for%20Task%20and%20Motion%20Planning%20from%20an%20Initial%20Scene%20Image%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22In%20this%20paper%2C%20we%20propose%20a%20deep%20convolutional%20recurrent%20neural%20network%20that%20predicts%20action%20sequences%20for%20task%20and%20motion%20planning%20%28TAMP%29%20from%20an%20initial%20scene%20image.%20Typical%20TAMP%20problems%20are%20formalized%20by%20combining%20reasoning%20on%20a%20symbolic%2C%20discrete%20level%20%28e.g.%20first-order%20logic%29%20with%20continuous%20motion%20planning%20such%20as%20nonlinear%20trajectory%20optimization.%20Due%20to%20the%20great%20combinatorial%20complexity%20of%20possible%20discrete%20action%20sequences%2C%20a%20large%20number%20of%20optimization%5C%2Fmotion%20planning%20problems%20have%20to%20be%20solved%20to%20find%20a%20solution%2C%20which%20limits%20the%20scalability%20of%20these%20approaches.%20To%20circumvent%20this%20combinatorial%20complexity%2C%20we%20develop%20a%20neural%20network%20which%2C%20based%20on%20an%20initial%20image%20of%20the%20scene%2C%20directly%20predicts%20promising%20discrete%20action%20sequences%20such%20that%20ideally%20only%20one%20motion%20planning%20problem%20has%20to%20be%20solved%20to%20find%20a%20solution%20to%20the%20overall%20TAMP%20problem.%20A%20key%20aspect%20is%20that%20our%20method%20generalizes%20to%20scenes%20with%20many%20and%20varying%20number%20of%20objects%2C%20although%20being%20trained%20on%20only%20two%20objects%20at%20a%20time.%20This%20is%20possible%20by%20encoding%20the%20objects%20of%20the%20scene%20in%20images%20as%20input%20to%20the%20neural%20network%2C%20instead%20of%20a%20fixed%20feature%20vector.%20Results%20show%20runtime%20improvements%20of%20several%20magnitudes.%20Video%3A%20https%3A%5C%2F%5C%2Fyoutu.be%5C%2Fi8yyEbbvoEk%22%2C%22proceedingsTitle%22%3A%22Corvalis%2C%20Oregon%2C%20USA%22%2C%22conferenceName%22%3A%22Robotics%3A%20Science%20and%20Systems%22%2C%22date%22%3A%222020%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%2210.15607%5C%2FRSS.2020.XVI.003%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222026-06-22T11%3A59%3A35Z%22%7D%7D%5D%7D
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., … Florence, P. (2023). PaLM-E: An Embodied Multimodal Language Model. Proceedings of the 40th International Conference on Machine Learning (ICML), 8469–8488. https://proceedings.mlr.press/v202/driess23a.html
Driess, D., Huang, Z., Li, Y., Tedrake, R., & Toussaint, M. (2023). Learning Multi-Object Dynamics with Compositional Neural Radiance Fields. Proceedings of The 6th Conference on Robot Learning (CoRL 2022), 1755–1768. https://proceedings.mlr.press/v205/driess23a.html
Harris, J., Driess, D., & Toussaint, M. (2022). FC3 : Feasibility-Based Control Chain Coordination. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 13769–13776. https://doi.org/10.1109/IROS47612.2022.9981758
Toussaint, M., Harris, J., Ha, J.-S., Driess, D., & Hönig, W. (2022). Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulation. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 13753–13760. https://doi.org/10.1109/IROS47612.2022.9982236
Driess, D., Ha, J.-S., Toussaint, M., & Tedrake, R. (2022). Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning. Proceedings of the 5th Conference on Robot Learning (CoRL 2021), 245–255. https://proceedings.mlr.press/v164/driess22a.html
Ortiz-Haro, J., Ha, J.-S., Driess, D., & Toussaint, M. (2022). Structured deep generative models for sampling on constraint manifolds in sequential manipulation. Proceedings of the 5th Conference on Robot Learning (CoRL 2021), 213–223. https://proceedings.mlr.press/v164/ortiz-haro22a.html
Ha, J.-S., Driess, D., & Toussaint, M. (2022). Deep Visual Constraints: Neural Implicit Models for Manipulation Planning From Visual Input. IEEE Robotics and Automation Letters, 7(4), 10857–10864. https://doi.org/10.1109/LRA.2022.3194955
Driess, D., Schubert, I., Florence, P., Li, Y., & Toussaint, M. (2022). Reinforcement Learning with Neural Radiance Fields. Advances in Neural Information Processing Systems (NeurIPS), 35. https://proceedings.neurips.cc/paper_files/paper/2022/file/6c294f059e3d77d58dbb8fe48f21fe00-Paper-Conference.pdf
Driess, D., Ha, J.-S., & Toussaint, M. (2021). Learning to solve sequential physical reasoning problems from a scene image. The International Journal of Robotics Research, 40(12–14), 1435–1466. https://doi.org/10.1177/02783649211056967
Schubert, I., Driess, D., Oguz, O. S., & Toussaint, M. (2021). Learning to execute: efficiently learning universal plan-conditioned policies in robotics. Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, 1912–1924. https://argmin.lis.tu-berlin.de/papers/21-schubert-NeurIPS.pdf
Driess, D., Ha, J.-S., & Toussaint, M. (2020). Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image. Corvalis, Oregon, USA. Robotics: Science and Systems. https://doi.org/10.15607/RSS.2020.XVI.003
