Open-vocabulary mobile manipulation is the problem of picking up 𝑨𝑵𝒀 object, and placing it in 𝑨𝑵𝒀 location, in 𝑨𝑵𝒀 home, given just names. HomeRobot is a REAL and SIMULATED benchmark, with

and

: github.com/facebookresear…

In simulation, we implement different methods in complex, realistic, human-authored home environments, populated with thousands of different object instances.These environments can be used to train RL policies which transfer to the real world.

In the real world, we perform 20 predetermined experiments for each method, with a mixture of unseen objects, some from categories the robot had seen during training, some from categories it had never seen before. Final experiments are run in a single, controlled apartment.

HomeRobot’s combination of large, multi-room, human-authored environments, with a large diversity of objects and spaces, and both real-world and simulation support, makes it unique, and we hope it will help drive research forward.

In the real world, HomeRobot provides code for navigation, exploration, grasping, and placing, letting you run experiments with a low-cost mobile robot in any environment you want to. Find it here: github.com/facebookresear…

We’re aiming to run the first large-scale sim-to-real mobile manipulation challenge, and we want you to participate! Try out HomeRobot, download our simulation, and implement your best solution. Check out the competition website here: aihabitat.org/challenge/2023…