Physical manipulation planning with differentiable closed-loop manipulation primitives
Reuniting planning and reacting
Courtesy of Marc Toussaint
Optimization-based Task and Motion Planning (TAMP) approaches show remarkable capabilities in finding paths given a scene description and (intuitive) physics models. However, the result of a TAMP algorithm is usually an open-loop trajectory, which, when executed in the real world, is likely to fail under disturbances or other sources of
uncertainty. The objective of this project is to bring the generality and computational strength of our TAMP framework to real-world execution. Instead of trying to find controllers that execute a planning result, this project investigates a different, novel approach: Can the behavior of closed-loop (perception-based) control primitives be directly embedded into the planning framework itself such that the result is a sequence of reactive closed-loop control policies instead of open-loop paths? We aim to demonstrate the robustness of our control strategies to severe perturbations, e.g., human interventions, in real-world sequential manipulation tasks such as the escape room scenario.
2756394
proj039
1
apa
50
creator
desc
year
20212
https://www.scienceofintelligence.de/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22PLGW2GBV%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Zhou%20et%20al.%22%2C%22parsedDate%22%3A%222023%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BZhou%2C%20H.%2C%20Schubert%2C%20I.%2C%20Toussaint%2C%20M.%2C%20%26amp%3B%20Oguz%2C%20O.%20S.%20%282023%29.%20Spatial%20Reasoning%20via%20Deep%20Vision%20Models%20for%20Robotic%20Sequential%20Manipulation.%20%26lt%3Bi%26gt%3BIROS%202023%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2Firos55552.2023.10342010%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2Firos55552.2023.10342010%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Spatial%20Reasoning%20via%20Deep%20Vision%20Models%20for%20Robotic%20Sequential%20Manipulation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hongyou%22%2C%22lastName%22%3A%22Zhou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ingmar%22%2C%22lastName%22%3A%22Schubert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ozgur%20S.%22%2C%22lastName%22%3A%22Oguz%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222023%22%2C%22proceedingsTitle%22%3A%22IROS%202023%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2Firos55552.2023.10342010%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A18Z%22%7D%7D%2C%7B%22key%22%3A%2274WGLYYF%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Toussaint%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BToussaint%2C%20M.%2C%20Harris%2C%20J.%2C%20Ha%2C%20J.-S.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20H%26%23xF6%3Bnig%2C%20W.%20%282022%29.%20Sequence-of-Constraints%20MPC%3A%20Reactive%20Timing-Optimal%20Control%20of%20Sequential%20Manipulation.%20%26lt%3Bi%26gt%3BIROS%202022%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9982236%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9982236%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Sequence-of-Constraints%20MPC%3A%20Reactive%20Timing-Optimal%20Control%20of%20Sequential%20Manipulation.%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jason%22%2C%22lastName%22%3A%22Harris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wolfgang%22%2C%22lastName%22%3A%22H%5Cu00f6nig%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222022%22%2C%22proceedingsTitle%22%3A%22IROS%202022%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FIROS47612.2022.9982236%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9982236%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A13Z%22%7D%7D%2C%7B%22key%22%3A%228HI2W9KY%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Harris%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHarris%2C%20J.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20FC3%3A%20Feasibility-Based%20Control%20Chain%20Coordination.%20%26lt%3Bi%26gt%3BIROS%202022%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9981758%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS47612.2022.9981758%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22FC3%3A%20Feasibility-Based%20Control%20Chain%20Coordination%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jason%22%2C%22lastName%22%3A%22Harris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222022%22%2C%22proceedingsTitle%22%3A%22IROS%202022%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FIROS47612.2022.9981758%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9981758%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A16Z%22%7D%7D%2C%7B%22key%22%3A%22K3Y3WZPY%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ha%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHa%2C%20J.-S.%2C%20Driess%2C%20D.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20Deep%20Visual%20Constraints%3A%20Neural%20Implicit%20Models%20for%20Manipulation%20Planning%20from%20Visual%20Input.%20%26lt%3Bi%26gt%3BIEEE%20Robotics%20and%20Automation%20Letters%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FLRA.2022.3194955%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FLRA.2022.3194955%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Deep%20Visual%20Constraints%3A%20Neural%20Implicit%20Models%20for%20Manipulation%20Planning%20from%20Visual%20Input%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222022%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FLRA.2022.3194955%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9844753%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A16Z%22%7D%7D%2C%7B%22key%22%3A%22GZNH5FFX%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Grote%20et%20al.%22%2C%22parsedDate%22%3A%222023%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGrote%2C%20P.%2C%20Ortiz-Haro%2C%20J.%2C%20Toussaint%2C%20M.%2C%20%26amp%3B%20Oguz%2C%20O.%20S.%20%282023%29.%20%26lt%3Bi%26gt%3BNeural%20Field%20Representations%20of%20Articulated%20Objects%20for%20Robotic%20Manipulation%20Planning%26lt%3B%5C%2Fi%26gt%3B.%20arXiv%3A%202309.07620.%20https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2309.07620%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22document%22%2C%22title%22%3A%22Neural%20Field%20Representations%20of%20Articulated%20Objects%20for%20Robotic%20Manipulation%20Planning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Phillip%22%2C%22lastName%22%3A%22Grote%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Joaquim%22%2C%22lastName%22%3A%22Ortiz-Haro%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ozgur%20S.%22%2C%22lastName%22%3A%22Oguz%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222023%22%2C%22language%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A12Z%22%7D%7D%2C%7B%22key%22%3A%2296QH5J7K%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Schubert%2C%20I.%2C%20Florence%2C%20P.%2C%20Li%2C%20Y.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20Reinforcement%20Learning%20with%20Neural%20Radiance%20Fields.%20%26lt%3Bi%26gt%3BNeurIPS%202022%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2206.01634%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2206.01634%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Reinforcement%20Learning%20with%20Neural%20Radiance%20Fields%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ingmar%22%2C%22lastName%22%3A%22Schubert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pete%22%2C%22lastName%22%3A%22Florence%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yunzhu%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222022%22%2C%22proceedingsTitle%22%3A%22NeurIPS%202022%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2206.01634%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.neurips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2022%5C%2Fhash%5C%2F6c294f059e3d77d58dbb8fe48f21fe00-Abstract-Conference.html%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A18Z%22%7D%7D%2C%7B%22key%22%3A%22PRCWG3XK%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222023%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Xia%2C%20F.%2C%20Sajjadi%2C%20M.%20S.%20M.%2C%20Lynch%2C%20C.%2C%20Chowdhery%2C%20A.%2C%20Ichter%2C%20B.%2C%20Wahid%2C%20A.%2C%20Tompson%2C%20J.%2C%20Vuong%2C%20Q.%2C%20Yu%2C%20T.%2C%20Huang%2C%20W.%2C%20Chebotar%2C%20Y.%2C%20Sermanet%2C%20P.%2C%20Duckworth%2C%20D.%2C%20Levine%2C%20S.%2C%20Vanhoucke%2C%20V.%2C%20Hausman%2C%20K.%2C%20Toussaint%2C%20M.%2C%20Greff%2C%20K.%2C%20%26%23x2026%3B%20Florence%2C%20P.%20%282023%29.%20PaLM-E%3A%20An%20Embodied%20Multimodal%20Language%20Model.%20%26lt%3Bi%26gt%3BICML%202023%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2303.03378%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2303.03378%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22PaLM-E%3A%20An%20Embodied%20Multimodal%20Language%20Model%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fei%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mehdi%20S.%20M.%22%2C%22lastName%22%3A%22Sajjadi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Corey%22%2C%22lastName%22%3A%22Lynch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aakanksha%22%2C%22lastName%22%3A%22Chowdhery%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brian%22%2C%22lastName%22%3A%22Ichter%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ayzaan%22%2C%22lastName%22%3A%22Wahid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Tompson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Quan%22%2C%22lastName%22%3A%22Vuong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tianhe%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wenlong%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yevgen%22%2C%22lastName%22%3A%22Chebotar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Sermanet%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Duckworth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sergey%22%2C%22lastName%22%3A%22Levine%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vincent%22%2C%22lastName%22%3A%22Vanhoucke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karol%22%2C%22lastName%22%3A%22Hausman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Greff%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andy%22%2C%22lastName%22%3A%22Zeng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Igor%22%2C%22lastName%22%3A%22Mordatch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pete%22%2C%22lastName%22%3A%22Florence%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222023%22%2C%22proceedingsTitle%22%3A%22ICML%202023%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2303.03378%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A18Z%22%7D%7D%2C%7B%22key%22%3A%22FE5HXUBR%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Huang%2C%20Z.%2C%20Li%2C%20Y.%2C%20Tedrake%2C%20R.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282022%29.%20Learning%20Multi-Object%20Dynamics%20with%20Compositional%20Neural%20Radiance%20Fields.%20%26lt%3Bi%26gt%3BCoRL%202022%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2202.11855%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2202.11855%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Learning%20Multi-Object%20Dynamics%20with%20Compositional%20Neural%20Radiance%20Fields%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhiao%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yunzhu%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Russ%22%2C%22lastName%22%3A%22Tedrake%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222022%22%2C%22proceedingsTitle%22%3A%22CoRL%202022%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2202.11855%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv205%5C%2Fdriess23a.html%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A18Z%22%7D%7D%2C%7B%22key%22%3A%22NPUJQTWN%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222021%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Ha%2C%20J.-S.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282021%29.%20Learning%20to%20solve%20sequential%20physical%20reasoning%20problems%20from%20a%20scene%20image.%20%26lt%3Bi%26gt%3BThe%20International%20Journal%20of%20Robotics%20Research%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B40%26lt%3B%5C%2Fi%26gt%3B%2812%26%23x2013%3B14%29%2C%201435%26%23x2013%3B1466.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1177%5C%2F02783649211056967%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1177%5C%2F02783649211056967%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Learning%20to%20solve%20sequential%20physical%20reasoning%20problems%20from%20a%20scene%20image%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222021%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1177%5C%2F02783649211056967%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fjournals.sagepub.com%5C%2Fdoi%5C%2F10.1177%5C%2F02783649211056967%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A13Z%22%7D%7D%2C%7B%22key%22%3A%22RRIAEERL%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222020%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Ha%2C%20J.-S.%2C%20%26amp%3B%20Toussaint%2C%20M.%20%282020%29.%20%26lt%3Bi%26gt%3BDeep%20Visual%20Reasoning%3A%20Learning%20to%20Predict%20Action%20Sequences%20for%20Task%20and%20Motion%20Planning%20from%20an%20Initial%20Scene%20Image%26lt%3B%5C%2Fi%26gt%3B.%20arXiv.%20https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2006.05398%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22document%22%2C%22title%22%3A%22Deep%20Visual%20Reasoning%3A%20Learning%20to%20Predict%20Action%20Sequences%20for%20Task%20and%20Motion%20Planning%20from%20an%20Initial%20Scene%20Image%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222020%22%2C%22language%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A16%3A12Z%22%7D%7D%2C%7B%22key%22%3A%22B2AMAPGX%22%2C%22library%22%3A%7B%22id%22%3A2756394%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Driess%20et%20al.%22%2C%22parsedDate%22%3A%222021%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDriess%2C%20D.%2C%20Ha%2C%20J.-S.%2C%20Toussaint%2C%20M.%2C%20%26amp%3B%20Tedrake%2C%20R.%20%282021%29.%20Learning%20Models%20as%20Functionals%20of%20Signed-Distance%20Fields%20for%20Manipulation%20Planning.%20%26lt%3Bi%26gt%3BCoRL%202021%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2110.00792%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2110.00792%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Learning%20Models%20as%20Functionals%20of%20Signed-Distance%20Fields%20for%20Manipulation%20Planning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danny%22%2C%22lastName%22%3A%22Driess%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jung-Su%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Toussaint%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Russ%22%2C%22lastName%22%3A%22Tedrake%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222021%22%2C%22proceedingsTitle%22%3A%22CoRL%202021%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2110.00792%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv164%5C%2Fdriess22a.html%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-09-26T13%3A23%3A54Z%22%7D%7D%5D%7D
Zhou, H., Schubert, I., Toussaint, M., & Oguz, O. S. (2023). Spatial Reasoning via Deep Vision Models for Robotic Sequential Manipulation. IROS 2023. https://doi.org/10.1109/iros55552.2023.10342010
Toussaint, M., Harris, J., Ha, J.-S., Driess, D., & Hönig, W. (2022). Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulation. IROS 2022. https://doi.org/10.1109/IROS47612.2022.9982236
Harris, J., Driess, D., & Toussaint, M. (2022). FC3: Feasibility-Based Control Chain Coordination. IROS 2022. https://doi.org/10.1109/IROS47612.2022.9981758
Ha, J.-S., Driess, D., & Toussaint, M. (2022). Deep Visual Constraints: Neural Implicit Models for Manipulation Planning from Visual Input. IEEE Robotics and Automation Letters. https://doi.org/10.1109/LRA.2022.3194955
Grote, P., Ortiz-Haro, J., Toussaint, M., & Oguz, O. S. (2023). Neural Field Representations of Articulated Objects for Robotic Manipulation Planning. arXiv: 2309.07620. https://doi.org/10.48550/arXiv.2309.07620
Driess, D., Schubert, I., Florence, P., Li, Y., & Toussaint, M. (2022). Reinforcement Learning with Neural Radiance Fields. NeurIPS 2022. https://doi.org/10.48550/arXiv.2206.01634
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., … Florence, P. (2023). PaLM-E: An Embodied Multimodal Language Model. ICML 2023. https://doi.org/10.48550/arXiv.2303.03378
Driess, D., Huang, Z., Li, Y., Tedrake, R., & Toussaint, M. (2022). Learning Multi-Object Dynamics with Compositional Neural Radiance Fields. CoRL 2022. https://doi.org/10.48550/arXiv.2202.11855
Driess, D., Ha, J.-S., & Toussaint, M. (2021). Learning to solve sequential physical reasoning problems from a scene image. The International Journal of Robotics Research, 40(12–14), 1435–1466. https://doi.org/10.1177/02783649211056967
Driess, D., Ha, J.-S., & Toussaint, M. (2020). Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image. arXiv. https://doi.org/10.48550/arXiv.2006.05398
Driess, D., Ha, J.-S., Toussaint, M., & Tedrake, R. (2021). Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning. CoRL 2021. https://doi.org/10.48550/arXiv.2110.00792
