X Square Robot launches X2Real, a simulation test for robot AI

A robot arm has to be run, watched and scored one attempt at a time. Each new version of a robot AI model means another round of that work, and the time and cost add up quickly. X Square Robot, an embodied AI startup, has released X2Real, a simulation benchmark meant to measure how robot foundation models — large AI models trained to control robots — will perform in the physical world, and to keep expanding its test set as those models improve.

The starting point is a familiar trade-off. Real-robot evaluation is expensive and slow, while simulation is fast but often unrealistic, and models can learn to score well on it without getting better at the real thing. To make the simulation faithful, the team recorded movements from physical robot arms, replayed the same control commands in simulation, and tuned the joint responses until the two matched. The company says this yields millimeter-level agreement between the trajectories. Visual appearance and physical contact were calibrated as well. The benchmark supports the single-arm Franka, the dual-arm ArtiXon Arm-6A, the ARX R5 and the Quanta X1 mobile manipulator.

The headline figures need reading with care. On general manipulation tasks, X Square Robot reports a 0.96 correlation between simulated and real-world task success rates, and 0.99 for progress scores. A correlation close to 1 means that the simulated scores move with the real ones, so a model that scores higher in the simulator should also score higher on the robot. But these numbers come from the company's own testing on that task subset. Independent trials across other tasks have not been reported in the material available.

The benchmark itself is broad. It covers 44 tasks in 10 categories: four on reasoning, including generalization, visual understanding, language understanding and memory, and five on manipulation, including fine manipulation, bimanual coordination, tool use, dynamic objects and mobile manipulation. A separate challenge set targets precision, long-horizon memory and planning, with tasks such as stacking blocks, pressing buttons in a color sequence and solving the Tower of Hanoi. The company says most current models still struggle there. To avoid overfitting, appearance, task setup and robot type are randomized independently, and the training data, nearly 300 hours collected in a uniform green-screen studio, is kept separate from the evaluation scenes. The simulation runs on the Mana ecosystem, built on NVIDIA Isaac Lab-Arena, which packages scenes, robots and task steps as reusable modules.

What changes for teams building robot models is the speed of the feedback loop. Developers run inference on their own machines and connect to the platform, which handles scoring and trajectory replay online, so they do not need to set up a local simulation. Two AI agents keep the test set moving: one writes new tasks and scenes from a constrained menu of options, with automatic checks removing invalid setups, and the other breaks each task into subtasks, judges each step from object poses, contacts and grasp states, and generates targeted new tests wherever a model fails. For researchers, that means failures can surface before a physical trial is scheduled. The limits are equally clear: the correlation figures are the company's own, they cover one subset of tasks, and the hardest tests are the ones where most models still fail.