Video alone can't train a policy.
A manipulation policy has to output actions — end-effector motion and gripper commands — that a video never contains. So the question isn't 'is video enough?' It's 'which action and sensor signals does your embodiment actually need?'
Essential vs. nice-to-have, by robot type
Gyroscope, force/torque, joystick/manipulator traces, hand pose — each is essential for some embodiments and pure noise for others. Match the stack to the robot.
| Robot type / task | Essential modalities | Nice-to-have | Video-only role |
|---|---|---|---|
Manipulation (VLA, parallel gripper) | Multi-view RGB + proprioception + 7-DoF action + language + calibration | Depth, wrist cam | Pretraining vision backbones only |
Dexterous / 5-finger manipulation | + per-frame 3D hand / finger pose (the recovered action) | Tactile | — |
Contact-rich assembly | + 6-axis force / torque + tactile (shear, slip, compression) | — | — |
Legged locomotion | IMU + joint encoders (proprioception core) | Vision / LiDAR for terrain | Minimal |
Drone / UAV | IMU + depth / RGB + state estimation → control commands | LiDAR, GPS | Minimal |
AV / ADAS not our focus | LiDAR + camera + radar + GPS/IMU + CAN-bus, time-synced | — | — |
swipe to compare →
Every modality, and when it earns its place
Multi-view RGB(-D)
All manipulationTwo or more synchronized camera views, optionally with depth.
The observation stream. Depth helps but rarely replaces multi-view for occlusion.
Proprioception
Manipulation · LeggedJoint angles, end-effector pose, gripper width — the robot's sense of its own state.
Essential context for manipulation; the primary signal for blind legged locomotion.
7-DoF action label
All manipulation6-DoF Δpose + gripper, per timestep — what the operator commanded.
The value the policy is trained to output. Without it, you have video, not training data.
Per-frame 3D hand pose
Dexterous21–25 finger joints tracked in 3D each frame.
The recovered action for dexterous, multi-finger manipulation. Over-specified for 2-finger grippers.
Force / torque + tactile
Contact-rich6-axis contact forces and moments; tactile arrays.
Exposes shear, slip and compression vision can't see. Publicly scarce.
IMU + joint encoders
Legged · DroneGyroscope, accelerometer, high-rate joint state.
Proprioception core for locomotion and drones; nice-to-have context elsewhere.
Language & calibration
AllTask instructions; per-camera intrinsics/extrinsics.
Language conditions VLA policies; calibration makes multi-view geometrically usable.
Human demonstration vs. robot teleoperation
Raw human egocentric video is not robot training data on its own — it lacks target-robot action labels. It becomes trainable once an action (typically a recovered 3D hand/finger pose) is attached. That's the load-bearing step, and it's exactly what we capture — not just footage.
Where video-only still helps
Human video is a proven co-training source for pretraining vision backbones, learning affordances and world models (Apple EgoDex: 829h, 194 tasks, with paired 3D hand tracking). Use it for the base — buy action-paired capture for the policy.