ComparisonJul 13, 20265 min read
Synthetic vs. real-world data for robotics: what actually trains a policy
Simulation is cheap and infinite; real-world data is expensive and finite. Here is how the two actually compare for robotic manipulation, and why the strongest training sets use both.
Every robotics team eventually reaches the same fork in the road. You need far more training data than you have, and you have to decide where it comes from: generate it in simulation, or capture it in the real world.
The honest answer is that this is a false binary. The best programs use both. But the tradeoffs are real, and getting the mix wrong is one of the more expensive mistakes a robotics effort can make. This is a practical look at what each source is good for, where each one fails, and how to combine them.
The case for synthetic data
Simulation is seductive, and for good reasons:
- Scale is effectively free. Once a scene is built, you can generate millions of trajectories overnight across randomized objects, lighting, and layouts.
- Labels are perfect. Segmentation masks, depth, object poses, and contact points come out of the renderer exactly, with no annotation cost or human error.
- Rare events are controllable. You can deliberately sample the dangerous, the unusual, and the near-miss, cases that almost never show up in a real capture session.
- It is safe and repeatable. No hardware to break, no reset labor, and a bug is a re-run rather than a damaged gripper.
For perception, this works remarkably well. Object detectors and pose estimators trained largely in simulation now transfer to the real world with modest fine-tuning. The pixels are close enough.
Where simulation breaks: the sim-to-real gap
Manipulation is a different story, because manipulation is about contact, and contact is exactly what simulators model worst.
Friction, compliance, deformation, backlash, cable drag, the way a zip-lock seal resists and then suddenly gives: these are precisely the dynamics that get approximated away. A policy can master a task in simulation and then fail on the real robot because the real world pushed back in a way the simulator never did.
A policy trained only in simulation learns the physics of the simulator, not the physics of the world.
Three gaps show up again and again:
- Contact-rich dynamics. Insertion, folding, wiping, and assembly depend on force and slip that rigid-body engines render only approximately.
- Sensor realism. Real depth cameras produce noise, dropouts, and reflections that clean synthetic depth never reproduces, so policies over-trust their inputs.
- The long tail of the real distribution. Worn parts, unusual grasps, clutter, and human-in-the-loop interruptions are hard to imagine and easy to under-sample.
What real-world data brings
Real-world capture inverts every one of those weaknesses. The distribution is the true one by construction. Contact is real contact. Sensor streams carry the exact noise your deployed robot will see. And when the demonstrations come from skilled people, the data encodes judgment: how an expert recovers from a slip, re-grasps, or adjusts force mid-motion, the tacit skill that was never written down anywhere to be scraped.
The cost is equally real: capture rigs, operator time, resets, and quality control. Real-world data is finite and it is earned. That is the whole tension.
Side by side
| Dimension | Synthetic data | Real-world data |
|---|---|---|
| Marginal cost per sample | Very low | High |
| Achievable scale | Effectively unlimited | Bounded by capture capacity |
| Label quality | Exact, automatic | Needs annotation or instrumentation |
| Contact & physics fidelity | Approximate | Ground truth |
| Sensor realism | Idealized | Native to deployment |
| Rare-event coverage | Fully controllable | Opportunistic |
| Distribution match to reality | Requires domain transfer | Correct by construction |
| Expert judgment / dexterity | Absent | Captured directly |
Read down the two columns and the pattern is clear: the strengths are almost perfectly complementary. That is the argument for using both, not for picking a side.
The pragmatic stack: pretrain in sim, ground in reality
The approach that works in practice looks like a pyramid.
At the base, a large synthetic corpus teaches broad visual and geometric priors and lets you brute-force coverage of scenes and object variation. Techniques like domain randomization widen this base so the policy stops depending on any single rendering artifact.
At the top, a smaller, higher-value layer of real-world data grounds the policy in true dynamics. This is where you spend on quality: instrumented expert demonstrations, real sensor streams, and honest coverage of the long tail. Co-training on both, rather than a naive pretrain-then-finetune, consistently transfers better, because the real data corrects the simulator's systematic biases while the synthetic data supplies volume.
And critically: your evaluation set must be real. Simulation can flatter a policy endlessly. Only real-world holdout data tells you whether it will work on the robot.
What this means if you're building a manipulation policy
- Use simulation for what it is genuinely good at: perception pretraining, scene and object coverage, and rare-event curricula.
- Do not trust simulation for contact-rich skills or for final evaluation.
- Invest your real-world budget where it compounds most: high-quality expert demonstrations and a faithful real evaluation set.
- Measure the mix. Track how much each data source moves real-world success rate, and let that ratio, not intuition, set your spend.
Synthetic data will keep getting better, and the base of the pyramid will keep growing. But the top, the deliberate, instrumented, expert real-world data that grounds a policy in the physics of the actual world, does not come for free, and it is not going away. It has to be produced.
That is the work we do at Synthsis Labs: producing real-world data for physical AI, deliberately and at scale.