#36 a practical first-timer's guide to finetuning a Physical AI model (π0.5)

After reading a bunch of articles about World Models, VLAs etc. I figured the best way to learn is by getting a robot and actually trying the tech.

This goal of this guide is to get things done with minimal hassle.

The cheapest you can get is the SO-101 ($300 here). It is a pair of robot arms: a leader arm which you control, and a follower arm which copies your movement.

Each arm is made of 3D-printed parts and 6 servo motors. It’s all open-source so you can 3D print and assemble it yourself, but paying $300 saves you time.

Each motor acts as a joint, so this is a 5 DOF arm (5 arm joints + 1 gripper ) which mimics a human arm. The joints are the shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll, and gripper.

First you need a setup: Two light sources, a webcam+tripod, a cube and a box (the task). Put it all on a table that’s at least 120cm x 60cm (the 140x60 Lagkapten from Ikea works well).

Create a HuggingFace account and get your user access token. Everything here onwards runs on the HuggingFace open-source robotics library LeRobot (@LeRobotHF).

Clone LeRobot repo and install via the command line interface (CLI). Everything that follows are just lerobot commands on CLI.

CLI may be intimidating for non-technical folks, but you can just “claude --dangerously-skip-permissions” and chat with Claude. It troubleshoots very well.

Connect both robot arms + wrist cam + scene webcam to your laptop (you will likely need a USB hub). Run “lerobot-calibrate” to calibrate, then hit the “lerobot-teleoperate” command.

Now, you can start recording teleoperation data for your task (”pick up the cube and place it in the box”). Complete 50 episodes, vary the position of the cube and box on the table for each episode, and grab the cube from slightly different angles each time so the model can learn.

If you make mistakes such as failing to grip the cube or moving the box with the gripper, re-record the episode. It should take 30-40mins in total.

Make sure to keep the lighting and camera angle consistent (do not bump the camera!).

After completing 50 episodes, LeRobot automatically uploads your dataset to HuggingFace.

^ bare minimum teleop setup in my bedroom (before i got my lighting set up)

Rent a GPU on Runpod.io to either train the ACT model (Action Chunking with Transformers, released 2023) from scratch, or finetune the frontier model π0.5 by @physical_int (model open-sourced Sep 2025, note the latest π0.6 and π0.7 are not open-source).

ACT is usually the first model practiced on by beginners. The model is small, trains on a single GPU, and has no pretrained robotics knowledge to lean on. It learns to imitate teleoperation demos through plain supervised regression.

It introduced two new ideas back then: 1) instead of predicting one action at a time, it predicts a chunk of the next ~50 to 100. 2) it encodes multimodal human demos as a “latent style variable” rather than averaging them into mush.

π0.5 is a vision-language-action (VLA) foundation model. It is pretrained on heterogeneous data, robot demonstrations across multiple embodiments, web image-text, object detections, and high-level semantic subtask labels, co-trained so the policy learns not just how to move but what scene it is looking at.

Architecturally it is a VLM backbone driving an action expert that emits continuous motor commands. What we are doing here is finetuning it on additional 50 demos, so the general manipulation prior gets specialised into a single SO-101 cube-in-box policy.

For ACT you will need an RTX4090 (~$0.70/hr) and training takes ~2hrs for ~15-20k steps, costing ~$1.40 in total.

For π0.5, you will need an A100 80GB (~$1.40/hr). They can take ~8hours to complete ~15-20k steps, so total cost is ~$10.

Once your training or finetuning is complete, your model will be uploaded to HuggingFace automatically.

Pro tip: Just run Claude to help you install all the dependencies on the Runpod instance.

This is the fun part. Time to see if your training / finetuning works!

For ACT, you can run inference on your laptop via CLI. If all goes well, success rate for in-distribution data (same object color, position, target, lighting as training data) should be 70-80%. Out-of-distribution data varies, but can be as low as 10-30%.

I was speedrunning this and used whatever I had at home -- a 2x2 rubik’s cube and an old reflective metallic box. The tests were fairly out-of-distribution given the colours/orientation of the cube, giving a success rate of ~20% (4 out of 20 tries).

This is somewhat expected for a small 50-episode dataset, and the multi-colored cube likely let ACT overfit to colour.

As for π0.5 you will need to rent an RTX 4090 on Runpod to run inference in the cloud (note how inference is far lighter than training, where we used an A100). The success rate is much higher, mine worked ~40% of the time (8 out of 20 tries).

^ fully autonomous robot using π0.5 finetuned on 50 teleoperated episodes

Overall I learned a lot about the practical difficulties and limitations of deploying robots in the real world: hardware limitations (max reach, slippery / transparency, contact-based tasks), data collection ops (manhour scales linearly, teleop vs egocentric vs UMI data), and software (long context window limitations, tradeoff between latency and edge compute).

Now that I have the basics, there are lots of things to experiment with:

• More complicated tasks: Conditional pick-and-place, stacking blocks, pouring beads from one cup to another, contact-rich tasks such as driving a screw, installing a drive belt, wiping a surface clean, peeling, plugging cables

• Try different robots (embodiments): Bi-manual, more DOF per arm (typically 7 DOF), longer reach, heavier payload. (Next cheapest one is YAM at >$3000)

• Try other models: VLA-JEPA apparently works well after

  finetuning with only ~13 episodes

All of this has also given me a profound appreciation of an infant’s ability to learn to manipulate objects through visual cues, proprioception, and force feedback. The human brain is truly amazing.