Outplaying elite table tennis players with an autonomous robot

Coordinate system

We use right-handed conventions with the origin of the coordinate system on the centre of the taking part in floor of the table, by which the x-axis factors in the direction of the human participant facet of the table and the z-axis factors upwards.

Perception

Ball triangulation

We use 9 cameras synchronized with the actuators of the robot with a 200 Hz set off sign to precisely find the ball within the quantity of the Olympic-sized courtroom. At every set off occasion, cameras seize 1,440 × 1,080 pixel Bayer8 color photographs. To cut back knowledge switch and enhance scalability (extra cameras can improve robustness and accuracy⁴⁴), every digicam is supplied with a hardware-accelerated discipline programmable gate array to facilitate two-dimensional (2D) ball detection. The discipline programmable gate arrays course of the photographs by a segmentation pipeline to supply a compressed 2D detection masks, which is streamed to a central server by an embedded CPU. The server verifies the form of the ball and triangulates its 3D place utilizing pre-calibrated digicam parameters. The whole course of is accomplished inside 10.2 ms.

Camera placement is optimized utilizing a customized covariance matrix adaptation evolution technique (CMA-ES) algorithm⁴⁵. The optimizer determines the lens choice, mounting peak and orientation for every digicam, topic to constraints such because the variety of towers, desired protection quantity and a minimal projected 2D ball radius (5 pixels).

Spin estimation

The angular velocity of the ball is estimated by observing the motion of the brand printed on the floor of the official ball. To precisely seize the high-speed transferring and rotating brand, we develop a mirror-based occasion imaginative and prescient monitoring system known as the gaze management system (GCS). The GCS includes three elements: (1) an occasion digicam⁴ for low-latency, low-motion-blur imaging; (2) a telephoto, electrically tunable lens to amplify the ball and preserve it in focus; and (3) a set of rotatable mirrors to trace the ball easily (Fig. 2d). Given the 3D triangulation outcomes, the mirrors and lens are managed to trace and deal with the ball with the system delay compensated by predicting the ball trajectory utilizing the ball aerodynamics. With the ball being tracked, its contour on the occasion digicam body is first detected by a CNN⁴⁶. Then the occasions on the ball are processed by two spin estimators, particularly, a low-latency estimator primarily based on one other CNN³³ and a high-accuracy however slower estimator primarily based on CMax³⁴. The CNN estimates the angular velocities with heteroscedastic uncertainties from collected occasions and is educated on pseudo-ground-truth knowledge obtained by CMax utilizing heteroscedastic regression⁴⁷.

Events are aggregated right into a polarity-separated floor of lively occasions⁴⁸ of 15 ms accumulation time window by which timestamps are minimal/most normalized to a variety between 0 and 1. We use a centred 320 × 320 pixel {hardware} crop of the unique 1,280 × 720 pixel.

The angular velocities estimated by the CNN are refined asynchronously by CMax. To obtain each low-latency and excessive accuracy, the robot agent Ace makes use of the angular velocities obtained by the CNN at first of the trajectory and switches to those obtained by CMax as quickly as they change into accessible with low uncertainty. Because the spin estimation uncertainty will increase when the brand is invisible, we place three GCSs to trace the ball from a number of views, as proven in Fig. 2a, and mix the multi-view measurements primarily based on the respective uncertainties.

Simulation

Ball aerodynamics

The aerodynamics of the ball in flight are ruled by the drag f_d, Magnus f_M and gravitational f_g forces. Given that the ball’s angular velocity ω is roughly fixed over brief flight intervals, the flight dynamics will be modelled as

the place v is the ball velocity, ρ_air = 1.204 kg m⁻³ (density of dry air at room temperature and commonplace stress), m = 2.7 × 10⁻³ kg (ball mass), r = 0.02 m (ball radius), c_d = 0.55 (drag coefficient), and g = [0, 0, −9.81]^T m s⁻² (gravitational acceleration). Unlike the bottom mannequin⁴⁹, which treats the Magnus coefficient c_M as fixed, we modelled it as ({c}_{{rm{M}}}=0.1frac{Vert {bf{v}}Vert }{rVert {boldsymbol{omega }}Vert }-0.001).

Ball–table contact mannequin

The table contact mannequin⁴⁹, which assumes instantaneous level contact, is enhanced to seize some results of floor contacts on the coefficient of restitution, ε^table, by modelling it as ε^table = 0.98 − 0.02v_z.

the place superscripts ‘−’ and ‘+’ are pre- and post-contact portions, respectively, and

the place the contact sort is decided as sliding if ν_s > 0 and rolling if ν_s ≤ 0, with

ε^table and μ are the coefficient of restitution and the dynamic coefficient of friction between ball and table, respectively, modelled as ({varepsilon }^{{rm{table}}}={varepsilon }^{{rm{table}}}({v}_{z}^{-})=0.98-0.02{v}_{z}^{-}) and μ = 0.25 from experimental knowledge.

Ball–racket contact mannequin

The linear mannequin proposed within the literature⁴⁹ is prolonged to deal with the broad ranges of linear and angular velocities encountered in professional-level table tennis by incorporating (1) a velocity-dependent coefficient of restitution and (2) a residual correction neural community to right mannequin errors. The base linear mannequin shares the identical construction as in equations (2) and (3) and is outlined as

with

the place R is the rotation matrix from the native body of the racket to the worldwide body of reference, v^racket is the racket linear velocity at impression and okay is a coefficient relating tangential portions. ({varepsilon }^{mathrm{racket}}={gamma }_{1}{{rm{e}}}^{{gamma }_{2}|{v}_{z}^{-{prime} }|}) is modelled as a perform of the traditional relative velocity ({v}_{z}^{-{prime} }) by becoming coefficients γ₁, γ₂ on knowledge collected from video games. The residual correction neural community is a small multilayer perceptron educated on sport knowledge and corrects each velocity and angular velocity error by 4% on common.

Sensor modelling

To mannequin the ball triangulation obtained from APS cameras, we pattern latency from a uniform distribution and noise from a zero-mean Gaussian distribution, and apply dropout of sensor measurements with a hard and fast chance. For the spin estimation, latency and dropout are modelled equally, with extra dropout utilized immediately after racket contact to replicate monitoring lack of GCS round these occasions. Both precision (sensor noise) and accuracy (sensor bias) of GCS are modelled utilizing separate zero-mean Gaussian distributions for spin magnitude and axis. However, accuracy is sampled as soon as per contact occasion to imitate the bias launched by GCS reinitialization at these occasions.

Physics perturbations

To enhance the simulation-to-reality switch, ball state perturbations are added after table contact. Each Cartesian element of ball linear and angular velocities is perturbed independently utilizing a zero-mean Gaussian distribution.

Robot dynamics

Robot joints are modelled as decoupled, delayed linear time-invariant methods, by which every joint i is described by

with ({{boldsymbol{zeta }}}_{i}

Episode definition

An episode begins when the ball is in free flight, transferring in the direction of the robot. An episode ends when the ball meets one in every of 4 circumstances: (1) the ball is out of play or not authorized; (2) the robot hits the ball; (3) the ball passes the racket of the robot; and (4) the joint trajectory produced by Ace would end in a collision with itself or the table.

Rewards

The reward perform used throughout coaching consists of a number of phrases, all of that are calculated after the episode has completed, that’s, as a perform of the terminal state. Although reward phrases differ throughout insurance policies to induce totally different expertise, they are often categorized by assigning particular rewards for (1) lacking the ball; (2) hitting the ball however failing to return it; or (3) efficiently returning the ball:

A subset of the insurance policies use a reward formulation for ({R}_{,{rm{hit}}}^{{rm{return}}}) that may be parameterized by a desired y-landing place (y_desired) and a set of reward weights (w_reward = [w_p, w_s], the place w_p ∈ [0, 1] and w_s ∈ [ − 1, 1]). y_desired is used to calculate a reward primarily based on the gap between y_desired and the achieved y-landing place, w_p is used to weight this distance reward and w_s is used to weight a time period proportional to the angular velocity within the y-axis of the ball body on touchdown. By sampling these conditioning variables, these insurance policies can exhibit a wide range of totally different behaviours reminiscent of aiming, topspin and backspin.

States

The state in our RL framework will be written as ({{bf{s}}}_{t}=[{{bf{s}}}_{t}^{mathrm{ball}},{{bf{s}}}_{t}^{mathrm{robot}},{{bf{s}}}_{t}^{mathrm{skill}}]). ({{bf{s}}}_{t}^{{rm{ball}}}) is the ball state consisting of ball place and spin histories of size N, alongside with their related time-stamps. ({{bf{s}}}_{t}^{{rm{robot}}}) is the robot state and consists of the joint states (place, velocity and acceleration) and finish effector state (pose and twist) related with the terminal state of ({Q}_{t-1}^{ast [1:T]}) (for additional particulars see Supplementary Information part 1.4.1). For insurance policies educated with parameterized reward features, the state is additional augmented with ({{bf{s}}}_{t}^{{rm{ability}}}), which is the fastened ability state composed of y_desired and w_reward. s_t is used to deduce actions a_t, that are subsequently mapped to joint trajectories and reset plans (see Supplementary Information sections 1.4.2 and 1.4.3). This course of requires a time price range of 5 ms and so s_t have to be constructed 5 ms earlier than the following set of instructions is shipped to the robot (Extended Data Fig. 1).

Actions

Actions, ({{bf{a}}}_{t}in {[-1,1]}^{2{N}_{q}}), are sampled from a tanh squashed multivariate Gaussian distribution. This varieties an summary area, by which for every joint there are two actions that outline a goal joint place and velocity 32 ms into the long run (see Supplementary Information part 1.4.2).

Transition chance perform

The transition chance perform is as follows:

the place f_ball is a stochastic perform that relies on sensors and physics modelling within the simulator, and f_robot is a deterministic perform relying on ({Q}_{t-1}^{* [1:T]}) and the robot dynamics.

Initial state coaching distribution

During coaching in simulation, an episode begins with an preliminary state that’s sampled from three impartial distributions:

1. 1.

   Initial ball state ({{bf{s}}}_{0}^{{rm{ball}}}): the preliminary state of the ball is sampled from a kernel density estimation (KDE) mannequin match both to artificial or human knowledge. For the artificial dataset, photographs are uniformly sampled from a variety of preliminary ball states and checked for validity. KDE fashions are generated for each returns (that’s, photographs carried out throughout a rally) and serves. During coaching, serves and returns are sampled at a ratio of three:7, and the preliminary state is sampled with a hard and fast chance from both the artificial or the human KDE fashions (Supplementary Information part 1.4.3).

2. 2.

   Initial robot state ({{bf{s}}}_{0}^{{rm{robot}}}): the preliminary robot state will be static or dynamic. Static states are sampled with the arm in a impartial configuration, and prismatic actuators are initialized uniformly inside their allowed vary, whereas dynamic states are sampled from reset plans saved throughout earlier coaching episodes.

3. 3.

   Initial ability state ({{bf{s}}}_{0}^{{rm{ability}}}): y_desired is sampled uniformly inside the bounds of the opponent’s facet of the table. w_reward is sampled in a approach that’s biased in the direction of sparse reward weight vectors and boundary values (Supplementary Information part 1.4.5).

Algorithm

To prepare the deep RL coverage, we use SAC³⁶ asynchronously with a number of knowledge assortment duties in parallel² (see Supplementary Table 8 for hyperparameters). We use uneven actor–critic^27,28,29, offering the ground-truth ball state from the simulator to the critic and sequences of sensor measurements to the actor. Apart from the usual coverage loss, an auxiliary loss is added to the coverage to reconstruct the bottom reality ball state from its ball state embedding. When amassing expertise, we apply three totally different types of knowledge augmentation as follows:

1. 1.

   Symmetric augmentation to reflect all states, actions and rewards with respect to the XZ aircraft (that’s, the aircraft containing the centre-line of the table and perpendicular to each the table and the online).

2. 2.

   Event tables⁵⁰ to retailer transitions resulting in predetermined occasions in separate replay buffers for stratified sampling of the mini-batch. The occasions utilized in our coaching pipeline are outlined primarily based on heuristics and embody the next occasions: close to miss, ball hit, ball returned, high-speed return, high-top-spin return, high-back-spin return (see Supplementary Information part 1.4.8).

3. 3.

   Hindsight expertise replay⁵¹ to reinforce RL transitions with an extra copy by which y_desired is the same as the achieved place, w_p is the same as 1, and the utmost position-based reward is given.

Feasible motion for optimum management

Mapping algorithm

The motion a_t sampled from the deep RL coverage is mapped from the summary set ({[-1,1]}^{2{N}_{q}}) to the possible set of joint place and velocity pairs 32 ms sooner or later, utilizing a mapping algorithm. The generic mapping algorithm will be said as follows: Let ({mathbb{X}}subset {{mathbb{R}}}^{n}) be the compact base set with centre (bar{{bf{x}}}) and ({mathbb{Y}}subset {{mathbb{R}}}^{n}) be the compact goal set with centre (bar{{bf{y}}}). For a given mapping (({{bf{x}}}_{i}in {mathbb{X}},{{bf{y}}}_{i}in {mathbb{Y}})), if ({{bf{y}}}_{i}=bar{{bf{y}}}) then ({{bf{x}}}_{i}=bar{{bf{x}}}), in any other case

the place (partial {mathbb{X}}) and (partial {mathbb{Y}}) are the boundaries of ({mathbb{X}}) and ({mathbb{Y}}), respectively, (bar{{bf{y}}}) the centre of ({mathbb{Y}}). The ratio (frac{{beta }_{i}}{{alpha }_{i}}) determines the situation of y_i between (bar{{bf{y}}}) and (partial {mathbb{Y}}), whereas the perform f(⋅) modifies δ_i to account for the form variations between ({mathbb{X}}) and ({mathbb{Y}}). The mapping is bijective, invertible and centre to centre and boundary to boundary of the map units.

Optimization downside formulation

We use the results of the mapping as a terminal place and velocity constraint for an optimization downside that computes reference trajectories for every robot joint as cubic splines that reduce jerk. By definition of the issue, the results of the mapping is at all times inside the utmost management invariant set, which is the biggest subset of the possible state area containing the preliminary states from which the related MPC downside is recursively possible⁵² (Supplementary Information part 1.5.1). The results of the mapping varieties the preliminary state for the following optimization downside. The optimization is solved utilizing DAQP⁵³ and sampled at 1 okayHz to generate ({Q}_{t}^{* [1:T]}).

Reset trajectories

For each ({Q}_{t}^{* [1:T]}) produced, a reset trajectory is required that strikes the robot from the terminal state of ({Q}_{t}^{* [1:T]}) to a goal stationary reset place. Ace makes use of a close to time-optimal variation of MPC (see Supplementary Information part 1.5.2) to generate these reset trajectories. They are executed as quickly as one of many termination standards for the RL episode is glad (Supplementary Information part 1.4.1). If the episode is terminated because of a predicted collision, then the reset trajectory from the earlier RL step is executed.

The goal reset place is chosen as both a continuing impartial configuration or a configuration computed by a put together coverage community. The put together coverage is educated utilizing a dataset constructed from elite-level rallies for high-dexterity shot execution. From every recorded rally, we extract (1) the ball state at the beginning of an episode, ({{bf{s}}}_{0}^{{rm{ball}}}); (2) y_desired; and (3) subsequent racket place ({{bf{x}}}_{{t}_{{rm{c}}}}^{{rm{r}}{rm{a}}{rm{c}}{rm{okay}}{rm{e}}{rm{t}}}) executed by robot at contact time t_c. For every ({{bf{x}}}_{{t}_{{rm{c}}}}^{{rm{r}}{rm{a}}{rm{c}}{rm{okay}}{rm{e}}{rm{t}}}), we compute offline the optimum reset configuration ({{bf{q}}}_{,{rm{r}}{rm{e}}{rm{s}}{rm{e}}{rm{t}}}^{ast }) that maximizes a dexterity goal (D({bf{q}},{{bf{x}}}_{{t}_{{rm{c}}}}^{{rm{r}}{rm{a}}{rm{c}}{rm{okay}}{rm{e}}{rm{t}}})), contemplating kinematic constraints. This course of yields a coaching dataset ({mathcal{D}}={{{({{bf{s}}}_{0}^{{rm{b}}{rm{a}}{rm{l}}{rm{l}}},{y}_{{rm{d}}{rm{e}}{rm{s}}{rm{i}}{rm{r}}{rm{e}}{rm{d}}},{{bf{q}}}_{{rm{r}}{rm{e}}{rm{s}}{rm{e}}{rm{t}}}^{ast })}^{i}}}_{i=1}^{M}) of M samples. During deployment for every shot, the agent receives (({{bf{s}}}_{0}^{{rm{ball}}},{y}_{{rm{desired}}})) as enter and predicts the optimum ({{bf{q}}}_{,{rm{r}}{rm{e}}{rm{s}}{rm{e}}{rm{t}}}^{{rm{d}}{rm{e}}{rm{s}}{rm{i}}{rm{r}}{rm{e}}{rm{d}}}) that helps dexterous execution of subsequent actions. ({{bf{q}}}_{,{rm{r}}{rm{e}}{rm{s}}{rm{e}}{rm{t}}}^{{rm{d}}{rm{e}}{rm{s}}{rm{i}}{rm{r}}{rm{e}}{rm{d}}}) is sampled from a Gaussian distribution estimated from the N nearest reset configurations ({{{{bf{q}}}_{,{rm{r}}{rm{e}}{rm{s}}{rm{e}}{rm{t}}}^{ast j}}}_{j=1}^{N}) to ({{bf{q}}}_{{rm{reset}}}^{* }) of (({{bf{s}}}_{0}^{{rm{ball}}},{y}_{{rm{desired}}})) discovered by KD-tree (okay-dimensional tree) search on the dataset.

Policy sampler

Ace makes use of a number of rally-specific insurance policies educated to optimize totally different aims and subsequently requires a sampling technique throughout matches. Ace makes use of 4 totally different methods for sampling the insurance policies (see Supplementary Information part 1.7.1 for particulars):

1. 1.

   Fixed: a single coverage is sampled with a hard and fast chance of 1.

2. 2.

   Random: a coverage is chosen at random on a shot-by-shot foundation from a subset of insurance policies.

3. 3.

   Heuristic: a set of heuristics dictates the coverage sampling on a shot-by-shot foundation. The heuristics map the traits of the incoming ball to probably the most acceptable coverage.

4. 4.

   Data-driven: a supervised studying mannequin is educated to categorise profitable and dropping photographs primarily based on knowledge from elite table tennis players aside from the seven players within the analysis. The mannequin is used to establish photographs with the very best predicted win charge, and the coverage most able to producing these photographs is sampled.

For insurance policies conditioned on y_desired and w_reward, Ace samples them from the identical fastened chance distribution used throughout coaching, that’s, uniform for y_desired and sparse however biased in the direction of boundary values for w_reward. As these insurance policies are conditioned on y_desired, in addition they afford using the put together coverage, which requires y_desired as enter.

Serve design

Ace achieves ITTF-compliant serves by executing a single-arm toss utilizing the ball cup mounted on its finish effector (Fig. 2c), adopted by placing the ball throughout its free fall. Although commonplace ITTF guidelines require a free-hand toss, one-handed serves are permitted when a participant has a bodily incapacity that impedes them from correctly tossing the ball with the free hand, offering a precedent for our implementation.

For the serve tossing, we acquire human serve demonstrations and re-target them to the kinematics of the robot utilizing an optimization process⁵⁴. The ensuing movement is a trajectory of joint instructions u_toss(t) that produces a legitimate ball toss when executed by the robot. We outline t_elevate because the time index of the tossing trajectory by which the acceleration of the ball approximates that of gravity, that means that the ball has been launched from the cup.

In simulation, the ball-striking movement u_strike(t) is obtained by connecting the present state of the robot at t = t_elevate to a racket state produced by a genetic algorithm (GA). This connection trajectory is generated by an MPC in racket area, for which the GA searches the optimum parameters ({{boldsymbol{xi }}}_{s}^{ast }) (Supplementary Information part 1.8.2) that maximize a health perform ({{mathcal{F}}}_{theta }(cdot )). This health perform is designed to seize serve metrics of curiosity (for instance, ball velocity, spin, touchdown place), which circumstances the kind of serves produced. The remaining serve trajectory is the concatenation of u_toss(t), t ∈ [0, t_lift] adopted by u_strike(t).

As u_strike(t) relies on simulated physics, we assess the effectiveness of every serve on the actual robot in devoted periods with a coach. Serves deemed sufficiently difficult for matches endure repeated open-loop execution (not less than 20 instances) to confirm their reliability. If the failure chance of a serve is lower than 5%, it’s added to the library for open-loop use throughout matches. If the chance exceeds 5%, we try closed-loop MPC execution, the place the parameters ({{boldsymbol{xi }}}_{s}^{ast }) are up to date on-line with precise hitting states output by a ball flight predictor. If this process efficiently decreases the failure charges to five% or much less, the serve is added to the library for closed-loop use. Details on how the serves have been chosen from the library will be present in Supplementary Information part 1.8.1.

Experimental protocol

Players heat up with one other participant for as much as 15 min earlier than the match. The participant practices with the robot instantly earlier than the beginning of the match for two min, as directed by the foundations of ITTF (https://www.ittf.com). During this observe interval, Ace makes use of a coverage that returns balls to a hard and fast place with average high spin, as is widespread observe. Players have been knowledgeable that in the event that they enter the courtroom facet of the robot, a security mild curtain triggers an emergency system cease. However, crossing the centre line is uncommon in high-level video games, and such a set off by no means occurred in our experiments. They have been instructed to put on goggles to guard their eyes. They select goggles from a wide range of sizes and colors to attenuate the impression on their efficiency. The participant decides whether or not to serve or obtain first. The participant is eligible to name a 1-min time-out in the course of the match. Elite D used this selection in the course of the third sport. Following the ITTF guidelines, the robot racket is proven to the participant and umpire earlier than the match. All tools, together with table (SAN-EI), internet (Butterfly), balls (Nittaku), racket (VICTAS and Butterfly) and ground mat, is accepted by the ITTF. We use subtle lights to make sure a uniform mild depth over the entire taking part in space with round 1,400 lux as directed within the guidelines of ITTF.