Workshop
6th Robot Learning Workshop: Pretraining, Fine-Tuning, and Generalization with Large Scale Models
Dhruv Shah · Paula Wulkop · Claas Voelcker · Georgia Chalvatzaki · Alex Bewley · Hamidreza Kasaei · Ransalu Senanayake · Julien PEREZ · Jonathan Tompson
Hall B2 (level 1)
Sat 16 Dec, 6:15 a.m. PST
The proposed workshop focuses on the intersection of machine learning (ML) and robotics, under this year’s focus topic: “Pretraining, Fine-Tuning, and Generalization with Large Scale Models.” Embodied AI and robotics pose unique challenges and opportunities for utilizing large pre-trained models. We seek to host a diverse set of views and approaches from across the robotics domain and dive deep into questions such as: What sources of data can be used for training large models in robotics? What role should pre-training play in robotics pipelines? How far can pre-trained models generalize when faced with novel tasks and environments? What is currently missing to the pre-training paradigm for embodied systems?
Schedule
Sat 6:15 a.m. - 6:20 a.m.
|
Opening Remarks
(
Presentation
)
>
SlidesLive Video |
🔗 |
Sat 6:20 a.m. - 6:45 a.m.
|
Keynote Masha Itkina
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 6:45 a.m. - 7:10 a.m.
|
Keynote Jesse Thomason
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 7:10 a.m. - 7:35 a.m.
|
Keynote Dhruv Batra and Arjun Majumdar
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 7:35 a.m. - 8:00 a.m.
|
Keynote Deepak Pathak
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 8:00 a.m. - 9:00 a.m.
|
Poster Session + Robot Demos I
(
Poster Session
)
>
|
🔗 |
Sat 9:00 a.m. - 9:40 a.m.
|
Oral Spotlights
(
Presentation
)
>
SlidesLive Video |
🔗 |
Sat 9:40 a.m. - 10:10 a.m.
|
Panel: How much are physical robots still needed in current robot learning research?
(
Panel
)
>
SlidesLive Video |
🔗 |
Sat 10:10 a.m. - 11:30 a.m.
|
Lunch Break
|
🔗 |
Sat 11:30 a.m. - 11:35 a.m.
|
Keynote Suraj Nair
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 11:55 a.m. - 12:20 p.m.
|
Keynote Matt Barnes
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 12:20 p.m. - 12:45 p.m.
|
Keynote Keerthana Gopalakrishnan and Montserrat Gonzalez Arenas
(
Talk
)
>
SlidesLive Video |
🔗 |
Sat 12:45 p.m. - 2:15 p.m.
|
Poster Session + Robot demos II
(
Poster Session
)
>
|
🔗 |
Sat 2:15 p.m. - 3:15 p.m.
|
Debate: Scaling models and data size is sufficient for deploying robots in the real world
(
Debate
)
>
SlidesLive Video |
🔗 |
Sat 3:15 p.m. - 3:30 p.m.
|
Closing remarks & Awards
(
Presentation
)
>
SlidesLive Video |
🔗 |
-
|
Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation ( Poster ) > link | Annie Xie · Lisa Lee · Ted Xiao · Chelsea Finn 🔗 |
-
|
Pre-Trained Binocular ViTs for Image-Goal Navigation ( Poster ) > link | Guillaume Bono · Leonid Antsfeld · Boris Chidlovskii · Philippe Weinzaepfel · Christian Wolf 🔗 |
-
|
Sample-Efficient Online Imitation Learning using Pretrained Behavioural Cloning Policies ( Poster ) > link | Joe Watson · Jan Peters 🔗 |
-
|
DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models ( Poster ) > link | Norman Di Palo · Edward Johns 🔗 |
-
|
MultiReAct: Multimodal Tools Augmented Reasoning-Acting Traces for Embodied Agent Planning ( Poster ) > link | Zhouliang Yu · Jie Fu · Yao Mu · Chenguang Wang · Lin Shao · Yaodong Yang 🔗 |
-
|
MultiReAct: Multimodal Tools Augmented Reasoning-Acting Traces for Embodied Agent Planning ( Spotlight ) > link | Zhouliang Yu · Jie Fu · Yao Mu · Chenguang Wang · Lin Shao · Yaodong Yang 🔗 |
-
|
Reinforcement-learning robotic sailboats: simulator and preliminary results ( Poster ) > link | Eduardo Vasconcellos · Ronald M. Sampaio · ANDRE PAULO ARAUJO · Esteban CLUA · philippe preux · Luiz Marcos Garcia Goncalves · Luis Martí 🔗 |
-
|
Learning to Act from Actionless Videos through Dense Correspondences ( Poster ) > link | Po-Chen Ko · Jiayuan Mao · Yilun Du · Shao-Hua Sun · Josh Tenenbaum 🔗 |
-
|
EvIL: Evolution Strategies for Generalisable Imitation Learning ( Poster ) > link | Silvia Sapora · Chris Lu · Gokul Swamy · Yee Whye Teh · Jakob Foerster 🔗 |
-
|
A Statistical Guarantee for Representation Transfer in Multitask Imitation Learning ( Poster ) > link | Bryan Chan · James Bergstra · Karime Pereida 🔗 |
-
|
CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller ( Poster ) > link | Yuxiang Yang · Guanya Shi · Xiangyun Meng · Wenhao Yu · Tingnan Zhang · Jie Tan · Byron Boots 🔗 |
-
|
Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks ( Poster ) > link | Murtaza Dalal · Tarun Chiruvolu · Devendra Singh Chaplot · Russ Salakhutdinov 🔗 |
-
|
TD-MPC2: Scalable, Robust World Models for Continuous Control ( Poster ) > link | Nicklas Hansen · Hao Su · Xiaolong Wang 🔗 |
-
|
Reasoning with Latent Diffusion in Offline Reinforcement Learning ( Poster ) > link | Siddarth Venkatraman · Shivesh Khaitan · Ravi Tej Akella · John Dolan · Jeff Schneider · Glen Berseth 🔗 |
-
|
Knolling bot 2.0: Enhancing Object Organization with Self-supervised Graspability Estimation ( Poster ) > link | Yuhang Hu · Zhizhuo Zhang · Hod Lipson 🔗 |
-
|
Open X-Embodiment: Robotic Learning Datasets and RT-X Models ( Poster ) > link |
174 presentersQuan Vuong · Ajinkya Jain · Alex Bewley · Alexander Irpan · Alexander Khazatsky · Anant Rai · Anikait Singh · Antonin Raffin · Ayzaan Wahid · Beomjoon Kim · Bernhard Schölkopf · brian ichter · Cewu Lu · Charles Xu · Chelsea Finn · Chenfeng Xu · Cheng Chi · Chenguang Huang · Chuer Pan · Chuyuan Fu · Coline Devin · Danny Driess · Deepak Pathak · Dhruv Shah · Dieter Büchler · Dmitry Kalashnikov · Dorsa Sadigh · Edward Johns · Federico Ceola · Fei Xia · Freek Stulp · Gaoyue Zhou · Gaurav Sukhatme · Gautam Salhotra · Ge Yan · Giulio Schiavi · Hao Su · Hao-Shu Fang · Haochen Shi · Heni Ben Amor · Henrik Christensen · Hiroki Furuta · Homer Walke · Hongjie Fang · Igor Mordatch · Ilija Radosavovic · Isabel Leal · Jacky Liang · Jaehyung Kim · Jan Schneider · Jasmine Hsu · Jeannette Bohg · Jiajun Wu · Jialin Wu · Jianlan Luo · Jiayuan Gu · Jie Tan · Jitendra Malik · Jonathan Tompson · Jonathan Yang · Joseph Lim · João Silvério · Junhyek Han · Kanishka Rao · Karl Pertsch · Karol Hausman · Keegan Go · Keerthana Gopalakrishnan · Ken Goldberg · Kevin Zhang · Keyvan Majd · Krishan Rana · Krishnan Srinivasan · Lawrence Yunliang Chen · Lerrel Pinto · Liam Tan · Lionel Ott · Lisa Lee · Masayoshi TOMIZUKA · Michael Ahn · Mingyu Ding · Mohan Kumar Srirama · Mohit Sharma · Moo J Kim · Nicklas Hansen · Nicolas Heess · Nikhil Joshi · Niko Suenderhauf · Norman Di Palo · Nur Muhammad Shafiullah · Oier Mees · Oliver Kroemer · Pannag Sanketi · Paul Wohlhart · Peng Xu · Pierre Sermanet · Priya Sundaresan · Rafael Rafailov · Ran Tian · Ria Doshi · Roberto Martín-Martín · Russell Mendonca · Rutav Shah · Ryan Hoque · Ryan Julian · Samuel Bustamante · Sean Kirmani · Sergey Levine · Sherry Q Moore · Shikhar Bahl · Shivin Dass · Shuran Song · Sichun Xu · Siddhant Haldar · Simeon Adebola · Simon Guist · Soroush Nasiriany · Stefan Schaal · Stefan Welker · Stephen Tian · Sudeep Dasari · Suneel Belkhale · Takayuki Osa · Tatsuya Harada · Tatsuya Matsushima · Ted Xiao · Tianhe Yu · Tianli Ding · Todor Davchev · Tony Zhao · Trevor Darrell · Vidhi Jain · Vincent Vanhoucke · Wei Zhan · Wenxuan Zhou · Wolfram Burgard · Xi Chen · Xiaolong Wang · Xinghao Zhu · Xuanlin Li · Yao Lu · Yevgen Chebotar · Yifan Zhou · Yifeng Zhu · Yonatan Bisk · Yoonyoung Cho · Youngwoon Lee · Yuchen Cui · Yueh-Hua Wu · Yujin Tang · Yuke Zhu · Yunzhu Li · Yusuke Iwasawa · Yutaka Matsuo · Zhuo Xu · Zichen Cui · Alexander Herzog · Abhishek Padalkar · Acorn Pooley · Anthony Brohan · Ben Burgess-Limerick · Christine Chan · Jeffrey Bingham · Jihoon Oh · Kendra Byrne · Kenneth Oslund · Kento Kawaharazuka · Maximilian Du · Mingtong Zhang · Naoaki Kanazawa · Travis Armstrong · Ying Xu · Yixuan Wang · Jan Peters |
-
|
T3GDT: Three-Tier Tokens to Guide Decision Transformer for Offline Meta Reinforcement Learning ( Poster ) > link | Zhe Wang · Haozhu Wang · Yanjun Qi 🔗 |
-
|
Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models ( Poster ) > link | Ivan Kapelyukh · Yifei Ren · Ignacio Alzugaray · Edward Johns 🔗 |
-
|
Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models ( Spotlight ) > link | Ivan Kapelyukh · Yifei Ren · Ignacio Alzugaray · Edward Johns 🔗 |
-
|
Policy-Guided Diffusion ( Poster ) > link | Matthew T Jackson · Michael Matthews · Cong Lu · Jakob Foerster · Shimon Whiteson 🔗 |
-
|
$\texttt{PREMIER-TACO}$ is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Loss ( Poster ) > link | Ruijie Zheng · Yongyuan Liang · Xiyao Wang · Shuang Ma · Hal Daumé III · Huazhe Xu · John Langford · Praveen Palanisamy · Kalyan Basu · Furong Huang 🔗 |
-
|
IG-Net: Image-Goal Network for Offline Visual Navigation on A Large-Scale Game Map ( Poster ) > link | Pushi Zhang · Baiting Zhu · Xin-Qiang Cai · Li Zhao · Masashi Sugiyama · Jiang Bian 🔗 |
-
|
Robotic Task Generalization via Hindsight Trajectory Sketches ( Poster ) > link |
17 presentersJiayuan Gu · Sean Kirmani · Paul Wohlhart · Yao Lu · Montserrat Gonzalez Arenas · Kanishka Rao · Wenhao Yu · Chuyuan Fu · Keerthana Gopalakrishnan · Zhuo Xu · Priya Sundaresan · Peng Xu · Hao Su · Karol Hausman · Chelsea Finn · Quan Vuong · Ted Xiao |
-
|
Hybrid Inverse Reinforcement Learning ( Poster ) > link | Juntao Ren · Gokul Swamy · Steven Wu · J. Bagnell · Sanjiban Choudhury 🔗 |
-
|
RoboAgent: Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking ( Poster ) > link | Homanga Bharadhwaj · Jay Vakil · Mohit Sharma · Abhinav Gupta · Shubham Tulsiani · Vikash Kumar 🔗 |
-
|
Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment ( Poster ) > link | Annie Chen · Govind Chada · Laura Smith · Archit Sharma · Zipeng Fu · Sergey Levine · Chelsea Finn 🔗 |
-
|
D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation ( Poster ) > link | Yixuan Wang · Zhuoran Li · Mingtong Zhang · Katherine Driggs-Campbell · Jiajun Wu · Fei-Fei Li · Yunzhu Li 🔗 |
-
|
How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies ( Poster ) > link |
15 presentersMontserrat Gonzalez Arenas · Ted Xiao · Sumeet Singh · Vidhi Jain · Allen Z. Ren · Quan Vuong · Jake Varley · Alexander Herzog · Isabel Leal · Sean Kirmani · Dorsa Sadigh · Vikas Sindhwani · Kanishka Rao · Jacky Liang · Andy Zeng |
-
|
How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies ( Spotlight ) > link |
15 presentersMontserrat Gonzalez Arenas · Ted Xiao · Sumeet Singh · Vidhi Jain · Allen Z. Ren · Quan Vuong · Jake Varley · Alexander Herzog · Isabel Leal · Sean Kirmani · Dorsa Sadigh · Vikas Sindhwani · Kanishka Rao · Jacky Liang · Andy Zeng |
-
|
World Model Based Sim2Real Transfer for Visual Navigation ( Poster ) > link | Kiran Lekkala · Chen Liu · Laurent Itti 🔗 |
-
|
Robotic Offline RL from Internet Videos via Value-Function Pre-Training ( Poster ) > link | Chethan Bhateja · Derek Guo · Dibya Ghosh · Anikait Singh · Manan Tomar · Quan Vuong · Yevgen Chebotar · Sergey Levine · Aviral Kumar 🔗 |
-
|
RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot ( Poster ) > link | Hao-Shu Fang · Hongjie Fang · Zhenyu Tang · Jirong Liu · Chenxi Wang · Junbo Wang · Haoyi Zhu · Cewu Lu 🔗 |
-
|
RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot ( Spotlight ) > link | Hao-Shu Fang · Hongjie Fang · Zhenyu Tang · Jirong Liu · Chenxi Wang · Junbo Wang · Haoyi Zhu · Cewu Lu 🔗 |
-
|
Exploitation-Guided Exploration for Semantic Embodied Navigation ( Poster ) > link | Justin Wasserman · Girish Chowdhary · Abhinav Gupta · Unnat Jain 🔗 |
-
|
Exploitation-Guided Exploration for Semantic Embodied Navigation ( Spotlight ) > link | Justin Wasserman · Girish Chowdhary · Abhinav Gupta · Unnat Jain 🔗 |
-
|
A$^2$Nav: Action-Aware Zero-Shot Robot Navigation Using Vision-Language Ability of Foundation Models ( Poster ) > link | Peihao Chen · Xinyu Sun · Hongyan Zhi · Runhao Zeng · Thomas Li · Mingkui Tan · Chuang Gan 🔗 |
-
|
LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion ( Poster ) > link | Firas Al-Hafez · Davide Tateo · Jan Peters 🔗 |
-
|
Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design ( Poster ) > link | Aoran Jiao · Tanmay Patel · Sanjmi Khurana · Anna-Mariya Korol · Lukas Brunke · Vivek Adajania · Utku Culha · SiQi Zhou · Angela Schoellig 🔗 |
-
|
Human Scene Transformer ( Poster ) > link | Tim Salzmann · Hao-Tien Lewis Chiang · Markus Ryll · Dorsa Sadigh · Carolina Parada · Alex Bewley 🔗 |
-
|
Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild ( Poster ) > link | Hongjie Fang · Hao-Shu Fang · Yiming Wang · Jieji Ren · Jingjing Chen · Ruo Zhang · Weiming Wang · Cewu Lu 🔗 |
-
|
Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models ( Poster ) > link | Tsun-Hsuan Johnson Wang · Alaa Maalouf · Wei Xiao · Yutong Ban · Alexander Amini · Guy Rosman · Sertac Karaman · Daniela Rus 🔗 |
-
|
LLM Augmented Hierarchical Agents ( Poster ) > link | Bharat Prakash · Tim Oates · Tinoosh Mohsenin 🔗 |
-
|
Formalizing Lines of Research on Generalization in Deep Reinforcement Learning ( Poster ) > link | Ezgi Korkmaz 🔗 |
-
|
Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning ( Poster ) > link | Jingyun Yang · Max Sobol Mark · Brandon Vu · Archit Sharma · Jeannette Bohg · Chelsea Finn 🔗 |
-
|
Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning ( Spotlight ) > link | Jingyun Yang · Max Sobol Mark · Brandon Vu · Archit Sharma · Jeannette Bohg · Chelsea Finn 🔗 |
-
|
Vision-Language Models Provide Promptable Representations for Reinforcement Learning ( Poster ) > link | William Chen · Oier Mees · Aviral Kumar · Sergey Levine 🔗 |
-
|
Trajeglish: Learning the Language of Driving Scenarios ( Poster ) > link | Jonah Philion · Xue Bin Peng · Sanja Fidler 🔗 |
-
|
Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models ( Poster ) > link | Kevin Black · Mitsuhiko Nakamoto · Pranav Atreya · Homer Walke · Chelsea Finn · Aviral Kumar · Sergey Levine 🔗 |
-
|
Causal Influence Aware Counterfactual Data Augmentation ( Poster ) > link | Núria Armengol Urpí · Georg Martius 🔗 |
-
|
LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation ( Poster ) > link | Shengqiang Zhang · Philipp Wicke · Lütfi Kerem Senel · Luis Figueredo · Abdeldjallil Naceri · Sami Haddadin · Barbara Plank · Hinrich Schuetze 🔗 |