Can a robot's task graph be built from videos of people?

NEOPATH represents a task as a graph in which the nodes are objects, the edges are actions and the weights are learned costs. Every decision of the robot can be traced back to an edge and a number, and that number can be edited without retraining. The work asks whether the labels that train the graph can be extracted automatically from videos of people cooking. On people not seen during training, three different classifiers are between 48.7 and 65.0% accurate over eight action classes, and the graph reproduces the actual action in about half of the events. The planner works in simulation, not on a real robot.
Context
Vision-language-action (VLA) models take the image and the instruction directly to control, with no intermediate step that can be inspected. Symbolic planners are readable, but their costs are written by an engineer. NEOPATH seeks an explicit graph whose costs are learned from demonstrations, because in education and in collaboration with people it is necessary to explain why the robot chose an action.
The graph
A semantic graph joins concepts (cup, knife, the robot's hands) with the actions of a fixed vocabulary. A perceptual graph stores the detected objects, their 3D position and their spatial relations. A graph attention network multiplies the base cost of each action by a factor between 0.1 and 2.0. The network only rescales costs; the plan comes from a shortest-path search with Dijkstra's algorithm.
From video to labels
Each frame goes through an object segmenter, a hand tracker and a depth estimator. When the hand is less than 150 px from an object, four frames are taken, about 0.3 s. The 97 verbs of EPIC-KITCHENS-100 were grouped into eight actions the arm can execute, such as grasp, move, cut or pour.
Classifier architecture: CLIP over four frames, temporal pooling and hand-motion features
On top of CLIP, averaging four frames instead of one raised accuracy by 16.8 points, and adding hand aperture, which separates grasping from moving, brought it to 65.0% on one participant.
The data leak
Those figures were inflated. Each segment produced up to four nearly identical windows, and when splitting by window some ended up in training and others in testing. When grouping by participant, accuracy dropped:
| Approach | Split by window | Split by participant |
|---|---|---|
| Frozen embeddings + GBM | 56.8% | 48.7% |
| Fine-tuning of 8 layers | 70.7% | 49.6% |
The model with more capacity to memorize is the one that lost the most. The paper warns that the same error probably affects other systems that sample windows from video.
Results
With the corrected split, a LoRA adaptation of CLIP ViT-L/14 reached 65.0%, an optimistic figure because the checkpoint was chosen with the same test participants. Balanced accuracy is 31.8%. The confusion matrix in the header shows why: grasp has 0% recall, and cut, pour and displace are mostly confused with move.
On three videos from an unseen participant (342 events), the graph matched the actual distribution of actions in 48.5% of events. Predicting the costs takes 5.8 ms per event.
On a Jetcobot arm simulated in MuJoCo, 96 of 102 plans finished and 3 of 207 steps failed. After each failure the edge was penalized and the new plan was completed.
What is missing
The figures come from a single split with no validation group, so there are no confidence intervals. How the systematic errors of the classifier change the plans has not been measured. The graph network was trained with two scenarios from one video and has not been compared with fixed costs. The simulation uses cubes and scripted grasps, with no comparison with other planners and no real robot.
Since three architectures stayed in the same range, the paper concludes that the limit is the diversity of people: only 18 of the 37 participants in EPIC-KITCHENS-100 have public labels. It proposes adding Ego4D and evaluating the educational use with students: editing a cost, replanning and observing the change.
How it fits in Robiolab
NEOPATH uses the same structure of a semantic and a perceptual graph as the autonomous reasoning for a humanoid, where actions and their constraints were defined by hand; here the weight of each action is learned from video. With the humanoid that imitates the arms of a person it shares the idea of learning from the human body as seen by a camera, and its educational purpose brings it close to the acrylic robot arm. It leaves public code, a vocabulary of executable actions and a warning about how to evaluate video classifiers. The work was presented at SAMYRAD 2026, a conference on artificial intelligence and education.
