Recent advances in AI and Machine Learning have made great strides in developing robust and adaptive agents in the real world. Nonetheless, unlike the recent remarkable multi-task consolidations in Natural Language Processing and Computer Vision, today’s Embodied AI research has mainly focused on building siloed systems for narrow tasks. We argue that the crux of building generalist agents is harnessing massive, diverse, and multimodal data altogether. This talk will examine various sources of training data available for training embodied agents, from Internet-scale corpora to task demonstrations. We will discuss the complementary values and limitations of these data in a pyramid structure and introduce our recent efforts in building generalist agents with this data pyramid.
Yuke Zhu is an Assistant Professor in the Computer Science department of UT-Austin, where he directs the Robot Perception and Learning Lab. He is also a core faculty at Texas Robotics and a senior research scientist at NVIDIA. His research lies at the intersection of robotics, machine learning, and computer vision. He received his Master's and Ph.D. degrees from Stanford University. His research works have won several awards and nominations, including the Best Conference Paper Award in ICRA 2019, Outstanding Learning Paper at ICRA 2022, Outstanding Paper at NeurIPS 2022, and Best Paper Finalists in IROS 2019, 2021. He is the recipient of the NSF CAREER Award and the Amazon Research Awards.