Why Robots Need 6D Vision to Survive the Real World

The conversation around Artificial Intelligence has shifted from the digital screens of LLMs to the physical world. In a recent episode of The Physical AI, host Tommaso Di Bartolo sat down with Benjamin (Ben) Busam, a professor at the Technical University of Munich (TUM) and a leading voice in computer vision, to discuss the next frontier: 6D Pose Estimation and the rise of Embodied AI.

While we experience the world in three dimensions, a robot navigating that world requires a more complex understanding. Busam explains that for any rigid object, there are six degrees of freedom (6DoF):

1. Translation: Its location in space (X, Y, and Z coordinates).

2. Rotation: Its orientation along those axes (roll, pitch, and yaw).

For an “embodiment”—the industry term for a physical AI agent like a robot—knowing where an object is isn’t enough. To grasp a tool or avoid an obstacle safely, the machine must understand exactly how that object is tilted or turned.

Traditionally, computer vision relied on “handcrafted” pipelines. Engineers would feed a robot a perfect 3D CAD model of a part, and the robot would look for that exact match on a conveyor belt.

However, Busam highlights a radical shift toward Implicit Models.

“The explicit forms of having replicas... become more and more implicit, such that their reasoning about what to do—grasping, manipulating, or sticking things together—is not something where the pipeline needs to be designed in modules. The models reason about this themselves by being trained.”

This transition is powered by Large World Models and technologies like:

• Neural Radiance Fields (NeRFs): Storing 3D scene information within the weights of a neural network rather than as a simple list of points.

• 3D Gaussian Splatting: A newer technique that allows for high-fidelity, photorealistic 3D reconstruction.

A recurring theme in the discussion was the tension between physical correctness and visual realism. Busam argues that the priority depends entirely on the application:

PriorityUse CaseWhy it MattersGeometryAutonomous Vehicles / Industrial RobotsA car doesn’t care if the obstacle is red or blue; it only cares that it must not “bump” into it.PhotorealismHuman-Robot Collaboration / Digital TwinsFor humans to trust and understand a robot’s intent, the simulation needs to look intuitive and realistic.

We are moving away from “closed-set” environments—like a laboratory with constant lighting—into the “open world” of beer gardens and city streets. Busam notes that while a robot server in a lab is easy to build today, the real world presents chaotic variables: changing light, weather, and unpredictable human behavior.

As machines become capable of solving problems end-to-end, the role of the human changes. Historically, tools like hammers or bicycles required human “energy and experience” to function. Now, we are entering a phase where the “cognitive muscle” of humanity might shift as we delegate entire tasks to embodied assistants.

Despite the breakthroughs in “in-silico” (simulated) AI, Busam remains grounded about the difficulties of the physical realm.

“Software is easy, hardware is hard. Moving hardware through an environment has many more interdependencies... we often can’t wait for a long time; we need to be very focused and have constrained resources.”

The future of Physical AI, according to Busam, lies in interdisciplinary collaboration. The next generation of experts must bridge the gap between computer vision, natural language processing, and mechanical robotics to move these systems out of the lab and into our daily lives.