AMD Runs Zero-Shot Sim-to-Real Pick-and-Lift on Instinct GPUs

AMD published a blog on September 29, 2026 describing a sim-to-real reinforcement learning pipeline for a UFactory XArm-6 robotic arm. The pick-and-lift policy was trained with proximal policy optimization in MuJoCo Playground using the MuJoCo Warp backend, with simulation executed on Instinct GPUs via a ROCm implementation. Training ran on single-GPU systems: one MI300X and one MI355X. The trained policy transferred zero-shot to the real robot, meaning no fine-tuning on physical hardware was needed.

Simulation and Calibration

The simulation setup included several steps to close the gap between virtual and real. AMD measured the arm and gripper dynamics and fed those measurements into MuJoCo, a process they call real-to-sim calibration. They enabled gravity compensation in MuJoCo to match the XArm-6 firmware behavior, and fitted per-joint dynamics models by measuring the robot's response to commanded motions. Hand-built collision primitives, boxes, capsules, and cylinders, approximated the XArm-6 meshes.

Domain randomization sampled mass, poses, and actuation delay each episode. The randomization also varied block mass and scene layout using real-measured ranges. Actuation delay was modeled and randomized in simulation to mimic firmware latency. The reward function was adapted from MetaWorld and added an orientation-aware caging term. The observation space included proprioceptive state and a privileged cube pose in simulation, while on the real robot, perception estimates replaced that privileged pose. The action space consisted of six joint-angle deltas plus one gripper delta, clipped and scaled. The policy stacked short state history and included a smoothness reward for stable servo streaming.

Perception and Deployment

The real robot used two RealSense D435i cameras, one wrist-mounted and one overhead. The wrist camera was calibrated with an eye-in-hand method, and the scene camera was calibrated by least-squares with about 4 mm residual. The perception pipeline thresholded RGB in HSV to find a green cube and fused that with depth on an AMD GPU, using a ROCm librealsense pull request. Fusion combined camera estimates with confidence weighting and fell back to a single view if needed.

For deployment, the policy was exported as an ONNX model and queried at 40 Hz on an AMD Ryzen AI 300 Mini-PC. Commands were sent to the XArm-6 via ROS 2 and the xArm SDK. AMD also released a robotics.zip archive with patches, code, and a README for reproducing the experiment.

Future work includes embedding a vision encoder in the policy and randomizing rendered scenes using a GPU batch renderer. This work shows a full pipeline from GPU-accelerated simulation to edge inference, with calibration and randomization as the key enablers for zero-shot transfer.