SO-101 — learning the shell game
A team-built robotic shell game: human demonstrations, imitation learning and evaluation on a physical SO-101 arm.
A small task with real-world complexity
The task is to find the ball hidden under one of three cups, then choose and lift the corresponding cup with a physical SO-101 arm. The team printed robot parts at LabElec, calibrated the motors and set up leader/follower teleoperation.
Learning from demonstrations
The team manually recorded hundreds of episodes with a two-camera setup, fine-tuned SmolVLA on its own scene and also tried ACT. After observing failures, the team adjusted and extended the demonstrations before training again.
An agent with a visual interface was added for interaction with the player. The exact connection between that agent and the movement policy is not documented here.
The process
01 / HUMAN DEMONSTRATIONS
Good learning starts
with good examples.
The team recorded hundreds of episodes using a leader/follower setup and two cameras. Demonstrations were collected in the project’s own physical scene.
02 / IMITATION LEARNING
Observe. Adapt.
Train again.
The team fine-tuned SmolVLA on its demonstrations and also tried ACT. Failures on the real robot informed the next data collection and training rounds.
03 / REAL-WORLD EVALUATION
Completion is a status.
Reliability is a measure.
Historical team evaluations reported full-task success improving from 20% to 50%. Trial counts and detailed conditions were not provided; later improvements have not been quantified.
- 01 Reach the right place
- 02 Grip the cup
- 03 Lift the cup
- 04 Complete the task
Evaluating what the robot actually does
Manual trials tracked four stages: reaching the right location, gripping, lifting and fully completing the task. Reported full-task success improved from 20% to 50% in historical evaluations.
Trial counts, precise evaluation dates and detailed conditions were not supplied. Later improvements were reported without a new measured percentage. A completed project does not imply 100% reliability.
The difficult part: generalisation
Camera configuration, overfitting, background and lighting sensitivity, and compatibility between macOS and Windows were recurring difficulties. The practical response was iterative: observe the failures, collect better examples and retrain.
A separate next idea: chess
A personal chess-playing robot is a future project, distinct from this completed team effort. Automatic simulation evaluation and more extensive RGB camera use are possibilities rather than completed capabilities.
Read about the future chess project →