For decades, the great milestones of artificial intelligence have happened on screens. Deep Blue beat Kasparov at chess in 1997. AlphaGo took down Lee Sedol in 2016. AI agents have since dispatched the world's best at StarCraft, Dota and poker. All of it, however, has unfolded in the tidy, frictionless world of pixels and rules.

Now Sony AI says its robot, Ace, has done something harder. It has won at a real, physical sport, against real, elite human opponents, on a real table, with a real ball.

In research published on the cover of Nature on 23 April, Sony AI reports that Ace defeated three of five elite table tennis players in full matches played under International Table Tennis Federation rules. In follow-up matches in December and March, it beat professionals too. It is, Sony says, the first autonomous system ever to play a competitive physical sport at expert human level.

Why a robot at the table is so hard

Table tennis looks deceptively casual. In fact, it is one of the cruellest tests engineers can set a machine. A ball can travel at over 70mph, spin at hundreds of revolutions per second, and arrive with a trajectory bent by aerodynamic forces too subtle to model cleanly. A human player has perhaps a few hundred milliseconds to read it, plan a shot, and execute.

To meet that standard, Ace's designers had to crack three problems at once: see the ball, decide what to do about it, and move fast enough to actually do it.

The "seeing" comes from a bristling array of Sony Semiconductor Solutions hardware. Nine high-speed cameras, built around the IMX273 global shutter sensor, fix the ball's 3D position. Three separate "gaze control" rigs — event-based IMX636 vision sensors mounted on pan-tilt mirrors with tunable telephoto lenses — track angular velocity and spin in real time. Event sensors only register changes in the scene, which is how Ace keeps up with a ball that is, in conventional camera terms, a blur.

The "deciding" is done by reinforcement learning. Rather than programming Ace with a hand-tuned model of physics and tactics, the team let it learn by playing — refining its policy through experience until it could improvise against shots the engineers had never anticipated, including balls clipping the net.

The "doing" is high-speed robotic hardware capable of swinging a paddle with the precision of a competitive player and the timing of a metronome.

From the screen into the room

"This breakthrough is much bigger than table tennis," said Peter Stone, Sony AI's chief scientist, in the company's announcement. He called it "a landmark moment", arguing that an AI which can perceive, reason and act in fast, unpredictable real-world conditions opens "an entirely new class of real-world applications".

Peter Dürr, who led the Ace project from Sony AI's Zürich office, framed the challenge in plainer terms. Table tennis, he said, demands "split-second decisions as well as speed and power" — the same combination required of any robot expected to share space with humans at human speeds.

That is the part worth dwelling on. The technologies that let Ace return a 450 rad/s topspin loop are also the technologies that could let a warehouse robot dodge a falling box, a surgical assistant track a moving instrument, or an assistive device hand a cup to someone whose hand is shaking. Sony's own research lineage points the same way: Ace builds on Gran Turismo Sophy, the company's superhuman racing AI, which lived only inside a PlayStation. Ace does not.

It is also worth keeping perspective. Ace has not "solved" robotics any more than AlphaGo solved intelligence. It plays one game, on one table, with extraordinarily expensive sensors and a small army of researchers behind it. The elite players it faced still won two of five matches in the Nature trials.

But for the first time, the score is on the board. A machine has stepped off the screen, picked up a paddle, and won.