ralph balazs
← Back to selected work
RoboticsImitation learningTeam project

SO-101 — learning the shell game

A team-built robotic shell game: human demonstrations, imitation learning and evaluation on a physical SO-101 arm.

Status: Completed, confirmed November 2026 · Context: Team project with LabElec support

Stack: Python · LeRobot · SmolVLA · ACT experiments · Two cameras

Work described here belongs to the team. Individual ownership of the components is not specified. Repository link not available.

A small task with real-world complexity

The task is to find the ball hidden under one of three cups, then choose and lift the corresponding cup with a physical SO-101 arm. The team printed robot parts at LabElec, calibrated the motors and set up leader/follower teleoperation.

SO-101 leader and follower arms on a workbench
The physical leader/follower setup, not a demonstration of autonomous completion.

Learning from demonstrations

The team manually recorded hundreds of episodes with a two-camera setup, fine-tuned SmolVLA on its own scene and also tried ACT. After observing failures, the team adjusted and extended the demonstrations before training again.

An agent with a visual interface was added for interaction with the player. The exact connection between that agent and the movement policy is not documented here.

The process

01 / HUMAN DEMONSTRATIONS

Good learning starts
with good examples.

The team recorded hundreds of episodes using a leader/follower setup and two cameras. Demonstrations were collected in the project’s own physical scene.

Hardware teleoperation footage — not autonomous shell-game execution.

02 / IMITATION LEARNING

Observe. Adapt.
Train again.

The team fine-tuned SmolVLA on its demonstrations and also tried ACT. Failures on the real robot informed the next data collection and training rounds.

RECORDED EPISODES↓SMOLVLA FINE-TUNING↓REAL ROBOT TRIALSObserve errors → improve demonstrations ↺

03 / REAL-WORLD EVALUATION

Completion is a status.
Reliability is a measure.

Historical team evaluations reported full-task success improving from 20% to 50%. Trial counts and detailed conditions were not provided; later improvements have not been quantified.

  1. 01 Reach the right place
  2. 02 Grip the cup
  3. 03 Lift the cup
  4. 04 Complete the task

Evaluating what the robot actually does

Manual trials tracked four stages: reaching the right location, gripping, lifting and fully completing the task. Reported full-task success improved from 20% to 50% in historical evaluations.

Trial counts, precise evaluation dates and detailed conditions were not supplied. Later improvements were reported without a new measured percentage. A completed project does not imply 100% reliability.

The difficult part: generalisation

Camera configuration, overfitting, background and lighting sensitivity, and compatibility between macOS and Windows were recurring difficulties. The practical response was iterative: observe the failures, collect better examples and retrain.

A separate next idea: chess

A personal chess-playing robot is a future project, distinct from this completed team effort. Automatic simulation evaluation and more extensive RGB camera use are possibilities rather than completed capabilities.

Read about the future chess project →