09.05.24: Real-to-Sim-to-Real: A Scalable Data Diet for Robot Learning
Speaker: Abhishek Gupta
Host: Sanjiban Choudhury
Abstract: Robotic automation, powered by machine learning driven methods, has the potential to build systems that change the future of work, daily life and society at large by acting intelligently in human centric environments. As with most modern machine learning methods, a key component in building such a robotic system the availability of data, abundant, diverse and high quality. In domains of nature language or computer vision, data of this form has scaled passively with internet scale, since people naturally interact through the medium of language or images. In contrast, robots are hardly deployed in human-centric settings and certainly are not collecting internet scale data passively. The key question I will ask is - how can we develop a data diet for robotic learning that scales passively? In particular, I will discuss how simulation, despite being fundamentally inaccurate, can provide a scalable source of data for robotic learning. We will discuss a class of real-to-sim-to-real methods that are able to construct simulation content on the fly from cheap real-world data, enabling scalable robust robot training. In doing so, I hope to shed some light on the unique challenge that data acquisition plays in robot learning and discuss how developing truly open-world robotic learning systems will necessitate a careful consideration of data quality and quantity.
09.12.24: Learning to build an actionable, composable, and controllable digital twin
Speaker: Wei-Chiu Ma
Host: Sanjiban Choudhury
Abstract: Simulation has been the driving force behind robot development. With recent advances in computer vision and graphics, simulating sensor observations has particularly drawn wide attention across the community, since it may enable end-to-end testing of full autonomy systems. Unfortunately, existing sensor simulators, while impressive, still suffer from realism and can neither effectively model the outcomes of actions nor hallucinate counterfactual scenarios. In this talk, I will summarize our recent efforts to enable this goal.
First, I will discuss how we develop a high-fidelity closed-loop sensor simulator for self-driving vehicles. Our key insight is to build a digital twin directly from real-world data and leverage the compositional structure of the world to decompose the scene into foreground actors and background. This not only allows us to synthesize extremely high-quality sensor observations to avoid domain gap, but also facilitates better modeling of the interactions among the actors and the scene. Next, I will discuss how we can further expand the simulator to generate physically plausible sensor observations under different lighting conditions and improve the robustness of autonomous systems. Finally, I will present our recent efforts on pushing the boundaries of digital twins with generative models. I will showcase how we distill knowledge from multimodal LLMs into existing 3D systems, making them interactable, actionable, and thus suitable for physical intelligence.
09.26.24: Do We Need Social Robots? Towards Impactful and Long-term Interaction in Social Robotics and Personalized Healthcare/Medical Systems
Speaker: Chung Hyuk Park
Host: Tapomayukh Bhattacharjee
Abstract: The exponential growth of research efforts in robotics and human-robot interaction (HRI) has unveiled an increasing number of collaborative fields. One of the highly impactful areas that has shown a steep increase of interests and collaboration is the healthcare and medical domain, which has welcomed the technological advancement in assistive robotics, artificial intelligence (AI) and machine learning (ML). Assistive robotics is an expanding field of research that holds many potentials for impacting human health and quality of life, along with the advancement of AI/ML. In this presentation, I will discuss my research activities focused on four main themes related to assistive robotics and AI/ML: (1) My vision for socially assistive robots and initial approaches of SARs as embodied agents with multi-modal perception and interactions will be shared; (2) We will then delve into the realm of robotic learning, specifically targeting interactive learning and socio-emotional interactions for autistic individua; (3) I will examine contextual and mutual learning for personalized interaction, with the ultimate goal of facilitating clinical outcomes and long-term human-robot interaction; and (4) Our latest findings and new approaches will be shared, with several directions addressing AI/ML aided healthcare and telemedicine applications. Along with my research endeavors, I will also impart the experiences and knowledge gained from cross-disciplinary studies and translational research, aimed at advancing assistive robotics for healthcare and personalized interventions.
10.03.24: The Impact of VLMs on Semantic Navigation: A Before and After View of Object Search
Speaker: Bernadette Bucher
Host: Kuan Fang
Abstract: Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. For example, when looking for a fork, a person would look near refrigerators and ovens, not beds and sofas. To perform similar reasoning, a robot needs to have and use priors about the expected semantic layout of the environment. In this talk, I will present two of my solutions to this object search problem which leverage semantic priors developed directly before (ICLR 2022) and directly after (ICRA 2024) the recent rapid improvements in LLMs and vision-language models (VLMs). I will discuss how these advances in natural language and computer vision changed our solution to this robotics problem, and I will also talk about the connection between these solutions to the object search problem and other unsolved semantic reasoning challenges in robotics.
10.10.24: Toward Flexible and Effective Human-Robot Teaming
Speaker: Mike Hagenow
Host: Tapomayukh Bhattacharjee
Abstract: Despite nearly seventy years of development, robots are not yet realizing their promise of handling the undesirable day-to-day tasks of skilled industrial workers. Recent studies indicate that today’s robots are still too inflexible and difficult to program, particularly for less structured and high-variability tasks. In this talk, I will present three recent approaches to human-robot teaming that aim to unlock new opportunities for robots. These approaches address key questions in human-robot teaming, such as how to optimize human input during teaming and how skilled workers can teach robots complex behaviors. I will conclude by discussing open problems in the area and outlining next steps toward more widespread human-robot teaming.
10.17.24: Attention and Inattention for Minimalist Robot Learners
Speaker: Dinesh Jayaraman
Host: Kuan Fang
Abstract: Industry is placing big bets on "brute forcing" robotic control through scaling data, compute, and models, but a key blind spot of such scaling methods is that they are profligate in their use of expensive resources: power, compute, time, data, etc. Towards developing more minimalist robotic control stacks, my research group studies how agents can select and attend to task-relevant information during sensing, representation, decision making, and policy learning. I will speak about my group's work on exploiting privileged sensors at training time, combining pre-training language and vision models to compute task-relevant representations, and task-relevant world model learning.
10.24.24: Scaling Robot Learning with Passively-Collected Human Data
Speaker: Danfei Xu
Host: Kuan Fang
Abstract: The foundation of modern AI is scalable knowledge transfer from humans to machines. While Computer Vision and NLP can glean such knowledge from exabytes of human-generated data on the Internet, Robot Learning still heavily relies on resource-intensive teleoperation for data collection. Can we capture real-world human interactions as effortlessly as the Internet captures the virtual world? We propose that passive human data collection is a crucial step towards this future. Just as the Internet evolved into an unintentional data repository for AI, an ideal data collection system should capture sensorimotor data from everyday human activities, without humans’ conscious participation.
10.31.24: What Foundation Models Can and Cannot Do for Bringing Helpful Robotic Assistants into Our Lives
Speaker: Roberto Martin-Martin
Host: Tapomayukh Bhattacharjee
Abstract: The past few years have seen remarkable advancements in AI. What began with the NLP revolution has sparked new ideas across many fields, including robotics, driving the search for a "RobotGPT." But is this all we need to finally have robots assist humans in everyday tasks? What challenges have been addressed, and which remain unsolved as we look ahead? In this talk, I will discuss recent ways we have integrated Foundation Models into robotic solutions, as well as the limitations of the current FM-based paradigm—particularly in learning from few human demonstrations and in seeking information for complex manipulation tasks.
11.14.24: A Vision-Language-Action Flow Model for General Robot Control
Speaker: Quan Vuong
Host: Kuan Fang
Abstract: Robot learning has the potential to unlock flexible, general, and dexterous systems while addressing key AI challenges. However, achieving the generality needed for real-world applications faces obstacles like data, generalization, and robustness. This talk will describe the journey in building our flagship model, Pi_0 [1]. We propose a novel flow-matching architecture built on a pre-trained vision-language model to leverage Internet-scale semantic knowledge. The model is trained on diverse datasets from various dexterous robots, including single-arm, dual-arm, and mobile manipulators. We evaluate its zero-shot performance, ability to follow language instructions, and capacity to learn new skills through fine-tuning across tasks like laundry folding, table cleaning, and box assembly.
11.21.24: Generative Simulation for Embodied AI in Urban (Micro)mobility
Speaker: Bolei Zhou
Host: Kuan Fang
Abstract: Public urban spaces like streetscapes and plazas accommodate human social life in all vibrant variations. Advances in micromobility make public urban spaces no longer exclusive to humans: Food delivery bots and electric wheelchairs are sharing sidewalks with pedestrians, while robot dogs and humanoids have recently emerged in the street. Embodied AI plays a transformative role in shaping the future of urban micromobility, by assisting human operator of these mobile machines to navigate through the unpredictable sidewalks safely. In this talk, I will introduce our effort of building MetaDriverse, a simulation platform that facilitates the computer vision and autonomy research for urban mobility and micromobility. It incorporates generative AI capabilities to simulates diverse urban environments, encompassing a wide range of visual appearances, behavioral dynamics, and terrain structures. It enables the scalable training of embodied AI agents and safety evaluation before real-world deployment. Relevant projects are available at https://metadriverse.github.io/.
12.05.24: Moving from Data Collection to Data Generation: Addressing the Need for Data in Robotics
Speaker: Ajay Mandlekar
Host: Kuan Fang
Abstract: Imitation learning from human demonstrations has emerged as a widely adopted paradigm for teaching robots manipulation skills. However, data collection for imitation learning is costly and resource-intensive, often spanning teams of human operators, fleets of robots, and months of persistent data collection effort. Instead, in this talk, I will advocate for the use of automated data generation methods and simulation platforms as a scalable alternative to fuel this need for data. I will introduce a suite of automated data generation tools that make use of robot planning methods and small sets of human demonstrations, to synthesize new demonstrations automatically. These tools are broadly applicable to a wide range of manipulation problems, including high-precision and long-horizon manipulation, and can be used to produce performant, often near-perfect agents for such tasks. The data generated in simulation can also be used to address real-world robotic manipulation, making synthetic data generation a compelling tool for imitation learning in robotics.
02.08.24: Expressive robot swarms and multi-human-swarm interaction for social applications
Speaker: Merihan Alhafnawi
Host: Tapomayukh Bhattacharjee
Abstract: Robot swarms can be used as a new and exciting medium through which people can express themselves to one another. The swarm can help facilitate social group tasks that people might face in their daily lives. A robot swarm could especially be useful due to its distributed nature that helps with scaling to many robots and many users. In her talk, Merihan will present her research that focuses on designing, building and applying expressive robot swarms that support simultaneous interactions with many people in social tasks. During her PhD, Merihan designed and built MOSAIX, a swarm of 100 robot Tiles, that was applied in group tasks such as brainstorming, decision-making and collective art. MOSAIX was used in many public events by more than 400 people. In her postdoctoral research, Merihan combines swarm robotics and architecture to create self-adaptive structures that respond to the environment and human interaction.
02.22.24: Efficient Reductions for Inverse Reinforcement Learning
Speaker: Gokul Swamy
Host: Sanjiban Choudhury
Abstract: Interactive approaches to imitation learning like inverse reinforcement learning (IRL) have become the preferred approach for problems that range from autonomous driving to mapping. Despite its impressive empirical performance, robustness to compounding errors + causal confounders, and sample efficiency, IRL comes with a strong computational burden: the requirement to repeatedly solve a reinforcement learning (RL) problem in the inner loop. If we pause and take a step back, this is rather odd: we’ve reduced the easier problem of imitation to the harder problem of RL. In this talk, we will discuss a new paradigm for IRL that leverages a more informed reduction to expert competitive RL (rather than to globally optimal RL), allowing us to provide strong guarantees at a lower computational cost. Specifically, we will present a trifecta of efficient algorithms for IRL that use information from the expert demonstrations during RL to curtail unnecessary exploration, allowing us to dramatically speed up the overall procedure, both in theory and practice.
02.29.24: Bad Robot, Good Robot - Rethinking the Agency of Our Artificial Teammates
Speaker: Reuth Mirsky
Host: Sanjiban Choudhury
Abstract: Should autonomous robots always obey instructions and comply with our expectations? Most existing research on collaborative robots and agents assumes that a “good” robot should abide by the instructions it is given and should act in a way that will not interfere with its surroundings: a fetching robot should do its work seamlessly, a navigating robot should always avoid collisions, and a guide robot should always do as it is instructed. In this talk, I will question this assumption by presenting the Guide Robot Grand Challenge and discussing the components required to design and build a service robot that can replace or surpass the functionalities of a guide dog. This challenge encompasses a variety of research problems, for each of which I will present a novel contribution: reasoning about other agents, initiating an interaction, teaching teammates, and more. Finally, I will discuss the many remaining challenges towards achieving a guide robot and how I plan to tackle these challenges.
03.07.24: On Building General-Purpose Home Robots
Speaker: Lerrel Pinto
Host: Sanjiban Choudhury
Abstract: The concept of a "generalist machine" in homes — a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective — has long been a goal in robotics that has been steadily pursued for decades. In this talk, I will present our recent efforts towards building such capable home robots. First, I will discuss how large, pretrained vision-language models can induce strong priors for mobile manipulation tasks like pick-and-drop. But pretrained models can only take us so far. To scale beyond basic picking, we will need systems and algorithms to rapidly learn new skills. This requires creating new tools to collect data, improving representations of the visual world, and enabling trial-and-error learning during deployment. While much of the work presented focuses on two-fingered hands, I will briefly introduce learning approaches for multi-fingered hands which support more dexterous behaviors and rich touch sensing combined with vision. Finally, I will outline unsolved problems that were not obvious initially, which, when solved, will bring us closer to general-purpose home robots.
03.14.24: Dexterous Multimodal Robotic Tool-use: From Compliant Tool Representations to High-Resolution Tactile Perception and Control
Speaker: Nima Fazeli
Host: Sanjiban Choudhury
Abstract: Dexterous tool manipulation is a dance between tool motion, deformation, and force transmission choreographed by the robot's end-effector. Take for example the use of a spatula. How should the robot reason jointly over the tool’s geometry and forces imparted to the environment through vision and touch? In this talk, I will present two new tools in our tool-box for dexterous tool manipulation: multimodal compliant tool representations via neural implicit representations and our recent progress on tactile control with high-resolution and highly deformable tactile sensors. Our methods seek to address two fundamental challenges in object manipulation. First, the frictional interactions between these objects and their environment is governed by complex non-linear mechanics, making it challenging to model and control their behavior. Second, perception of these objects is challenging due to both self-occlusions and occlusions that occur at the contact location (e.g., when wiping a table with a sponge, the contact is occluded). We will demonstrate how implicit functions can seamlessly integrate with robotic sensing modalities to produce high-quality tool deformation and contact patches and how high-resolution tactile controllers can enable robust tool-use behavior despite the complex dynamics induced by the sensor mechanical substrate. We’ll conclude the talk by discussing future directions for dexterous tool-use.
03.21.24: Robot Foundation Model via Simulation
Speaker: Pulkit Agrawal
Host: Sanjiban Choudhury
Abstract: Unlike natural language and image processing where internet data is easily available for training foundation models, data for robot learning is unavailable. I will discuss how simulators can be used to learn complex and generalizable sensorimotor skills in a manner that reduces human effort and is easily scaled to many tasks. I will elaborate using the following case studies:
(i) a dexterous manipulation system capable of re-orienting novel objects of complex shapes and peeling vegetables.
(ii) a quadruped robot capable of fast locomotion, manipulation, and whole-body control on diverse natural terrains.
(iii) a lifelong learning robotic agent that can request and learn new rigid-object manipulation skills in a few minutes.
Next, I will discuss some algorithmic ideas aimed at mitigating human effort in reward design, hyper-parameter tuning and enabling seamless combination of learning signals from demonstrations, rewards, and the agent's self-exploration. The resulting framework provides a way to collect high-quality data for multiple tasks that can be used to train a robotic foundation model.
03.28.24: Trust, Robots, and Trust Repair
Speaker: Connor Esterwood
Host: Tapomayukh Bhattacharjee
Abstract: Trust is a dynamic force, evolving over time. When a trustee excels, trust flourishes; conversely, a trustee's failures can erode trust. While managing trust increases is relatively straightforward, trust declines can have enduring consequences, souring collaborative endeavors. Though trust breakdowns are inevitable they can be mitigated through verbal repairs such as apologies, denials, explanations, and promises. However, it remains unclear whether these strategies effectively restore trust in robots. This gap in understanding hinders the development of resilient robots capable of gracefully recovering from inevitable failures, thus limiting the potential of human-robot collaboration. This presentation highlights and summarizes my recent work in trying to solve the problem of trust violations and determining trust repairs in the context of human--robot interaction.
04.11.24: Human-Centered AI for Accessible and Assistive Robotics: Towards a Disability-Centered HRI
Speaker: Elaine Short
Host: Tapomayukh Bhattacharjee
Abstract: Powered by advances in AI, especially machine learning, robots are becoming smarter and more widely used. Robots can provide critical assistance to people in a variety of contexts, from improving the efficiency of workers to helping people with disabilities in their day-to-day lives. However, inadequate attention to the needs of users in developing these intelligent robots results in systems that are both less effective at their core tasks and more likely to do unintended harm. The Assistive Agent Behavior and Learning (AABL) Lab at Tufts University seeks to apply human-centered design thinking, especially disability ethics, to the design of state-of-the-art robot learning algorithms and interaction frameworks. This talk will explore how disability-community-centered thinking can be used to inspire new directions for intelligent interactive robotics and review recent work from the AABL lab at the intersection of assistive robotics, robot learning, and human-robot interaction.
04.25.24: Robot-Assisted Feeding: Recent Advances and Future Directions in the Personal Robotics Lab
Speaker: Taylor Kessler-Faulkner
Host: Tapomayukh Bhattacharjee
Abstract: Eating is a personal and intricate task that we perform every day. However, approximately 1.8 million people in the US alone cannot eat without assistance. The ADA (Assistive Dextrous Arm) project in the Personal Robotics Lab (PRL) at UW seeks to address this through a robot-assisted feeding system. In this presentation, I will discuss PRL's recent advances in both the human-robot interaction and autonomous robotics components of assistive feeding, as well as promising future research avenues.
05.02.24: Design and Perception of Wearable Multi-Contact Haptic Devices for Social Communication
Speaker: Cara Nunez
Host: Tapomayukh Bhattacharjee
Abstract: During social interactions, people use auditory, visual, and haptic (touch) cues to convey their thoughts, emotions, and intentions. Current technology allows humans to convey high-quality visual and auditory information but has limited ability to convey haptic expressions remotely. However, as people interact more through digital means rather than in person, it becomes important to have a way to be able to effectively communicate emotions through digital means as well. As online communication becomes more prevalent, systems that convey haptic signals could allow for improved distant socializing and empathetic remote human-human interaction.
Due to hardware constraints and limitations in our knowledge regarding human haptic perception, it is difficult to create haptic devices that completely capture the complexity of human touch. Wearable haptic devices allow users to receive haptic feedback without being tethered to a set location and while performing other tasks, but have stricter hardware constraints regarding size, weight, comfort, and power consumption. In this talk, I will present how I address these challenges through a cyclic process of (1) developing novel designs, models, and control strategies for wearable haptic devices, (2) evaluating human haptic perception using these devices, and (3) using prior results and methods to further advance design methodologies and understanding of human haptic perception