BeyondMimic: The 1st Revolutionary Way Guided Diffusion Controls Humanoids

Why humanoids are so hard to make graceful

The pitch for humanoid robots is that their body looks like ours, so they should be able to do what we do, move through spaces built for people, use tools made for our hands, and pick up new skills by watching. The reality has been clumsier. A humanoid has dozens of joints and a floating base that has to balance while everything else moves, which makes coordinated, natural motion genuinely hard to produce. Most working controllers get there by narrowing the ambition, one carefully tuned controller for walking, another for stairs, each hand built with its own reward terms and its own quirks.

Two failure modes have dogged the field. The model based approach pairs a simplified physics model with a low level controller, which keeps the computation tractable but tends to produce stiff, unnatural motions and struggles with contact rich skills like rolling on the ground. The learning based approach, using reinforcement learning, can produce agile and natural motion, but the usual recipe bakes in handcrafted rewards tuned for one specific behavior. Want a new skill and you are back to designing rewards, and worse, the resulting policy cannot be reused or composed with others to solve a task it was not trained for.

That word, composed, is the crux. People do not just have skills. We chain them. We walk, then break into a run, then dodge around a person, then stop, all fluidly, treating our motions as building blocks for whatever the moment demands. A controller that can only replay one motion at a time, however agile, has missed the point. Qiayuan Liao and colleagues set out to build a humanoid controller that keeps the agility and the natural look of learning from humans while gaining the versatility to combine skills for tasks it never saw in training.

Stage one, learning to move like a person

BeyondMimic is a two stage framework, and the first stage is about acquiring skills faithfully. The authors take roughly two and a half hours of diverse human motion capture and train the robot to track it. The key claim here is scalability under one recipe. Rather than hand tune a reward for each motion, they use a single shared formulation, the same reward, the same observations, the same hyperparameters, and train a separate tracking policy for every motion with reinforcement learning. Any new motion goes through the same pipeline with no bespoke tuning.

The reward is deliberately plain. It has one task term that measures how well the robot’s body matches the reference motion, plus only three lightweight penalties that apply across all motions, one to keep joints within safe limits, one to discourage jittery actions, and one to penalize self collisions. This restraint matters. Prior work leaned on stacks of ad hoc shaping terms to force natural looking motion, and those terms are exactly what fails to generalize. By keeping the reward general, the same formulation trains a cartwheel and a slow walk without changes.

A few design choices carry the sim to real transfer, and the authors are clear that the integration is the contribution rather than any single trick. They use an accurate model of the actuators, minimal randomization of the simulation, a continuous orientation representation called Rot6D that transfers better than quaternions or axis angle, a short history of past observations, and a deployment stack tuned for low latency. An adaptive sampling scheme during training focuses effort on the hard parts of a motion, the moment of a cartwheel rather than the easy standing before it, which sharply speeds up learning on difficult clips.

The key element was not any single component but their careful integration and the per motion tuning that normally accompanies sim to real transfer, and allowed one formulation to scale to hundreds of motions. Liao and colleagues, on what actually made the tracking work

Human level agility, measured

The agility results are the eye catching part, and they hold up to measurement rather than just looking impressive on video. The robot performed aerial cartwheels, two consecutive 180 degree side spin kicks, and 360 degree flip kicks, outdoors on soft soil and uneven ground that was never part of training. During the airborne phase of a cartwheel it reached a peak acceleration of 31 meters per second squared and a pelvic angular velocity of 15.7 radians per second, with a mean around 7 radians per second. For comparison, the authors note that skilled human aerial motions average about 7.75 radians per second. The robot is spinning in the same league as a trained human acrobat, and its landings were clean enough to need minimal recovery.

It also handled contact rich moves that stiff controllers struggle with, two cartwheels in a row, crawling on the ground and getting back up, and it repeated Cristiano Ronaldo’s celebration jump turn five times in succession without losing stability, where a prior method managed a single execution. These are not carefully staged single takes. They are demonstrations of repeatable, robust dynamic control.

Stage two, one diffusion model to compose them all

Skills are only half the story. The second stage is what turns a library of motions into a versatile controller, and it is where the diffusion model comes in. The authors train a single latent diffusion model on the trajectories produced by all those tracking policies. A diffusion model learns to generate data by starting from noise and repeatedly denoising it, and here what it generates is short trajectories of states and actions, the raw material of humanlike motion. Crucially it is trained on unlabeled, task agnostic data. It learns the distribution of natural motion itself, not a mapping to any particular goal.

The reason to use a diffusion model rather than a simpler policy is a property called classifier guidance. Because a diffusion model implicitly learns the gradient of the data distribution, you can nudge its generation toward any objective you can write as a differentiable cost, at inference time, without retraining. The mechanism is a clean application of Bayes rule. The guided score is just the unguided score plus the gradient of the task cost.

The first term keeps the generated motion inside the distribution of natural human movement the model learned. The second term pulls it toward the task. Associating the task likelihood with a cost through \(p(\tau^{*}\mid\tau) \propto e^{-G(\tau)}\), the guidance reduces to subtracting the gradient of the cost.

That is the whole idea in one line. Write a cost \(G\) that scores how well a trajectory satisfies your task, take its gradient during denoising, and the model produces a motion that both looks human and does what you asked. Because the model already carries a rich prior over agile, feasible motion, the cost can be simple. No behavior shaping, no regularization stacks, just a local signal that says here is the goal.

The versatility this unlocks is the headline result. The same trained model, with different costs plugged in at run time, performed zero shot commanded walking to a velocity or a waypoint, joystick teleoperation with live recovery from kicks, obstacle avoidance using a signed distance field cost, and motion inpainting where a few future keyframes guide a smooth transition into a cartwheel and back to walking. None of these tasks were in the training data. They are costs, evaluated at inference, steering a fixed model.

Why the latent space matters

A detail that turns out to be load bearing is that the diffusion runs in a latent space, not directly on raw states and actions. The action space of a humanoid is jerky, full of sharp torque spikes, which violates the smoothness that diffusion models assume and makes direct learning unstable. So the authors first train a variational autoencoder to compress the motions into a smooth latent representation, then run the diffusion there, and decode back to precise actions with a lightweight decoder.

An ablation shows how much this matters. On the demanding cartwheel, a baseline without the latent encoding reached only a 23 percent success rate in simulation, while the latent diffusion model hit 96 percent, and that high performance carried over to the physical robot. Modeling smooth latents rather than raw twitchy actions is what makes the whole guided diffusion approach stable enough to trust on hardware.

On naturalness, the authors ran a forced choice study with 77 participants who watched paired clips of BeyondMimic and the robot’s vendor supplied native controller and picked which looked more human. Across 1539 choices, BeyondMimic was preferred overall about 70.8 percent of the time, with the gap widest on running at 84.7 percent. People do not just tolerate these motions. They read them as noticeably more human than the default.

Honest limitations

The authors are candid about the boundaries, and the limits are real. The diffusion model inherits the quality of the robot’s state estimation, so errors in sensing its own joints and pose propagate straight into the generated motion. Better sensor fusion or learned estimators would help, and the current latent approach only partly cushions noisy observations.

The planning horizon is short. The model predicts trajectories about 0.64 seconds ahead, which is fine for reactive control and dodging a nearby obstacle but not enough for long horizon tasks that need to reason about a distant goal or anticipate early. Extending the horizon, and including enough history to stabilize predictions without the model getting trapped in repetitive loops, is unfinished business. Under guidance the robot is stable once a gait is established but tends to stumble at the very start and end of motions.

Latency is unforgiving. The team found that even small delays in the deployment pipeline degrade tracking sharply, with a 5 millisecond delay causing one failure and a 10 millisecond delay causing failures in two of three trials. That places a heavy demand on the real time engineering around the model, and it is a reminder that a beautiful policy still lives or dies by the plumbing it runs on. Finally, the guidance works well for coarse grained objectives but still needs some manual tuning of guidance weights for finer grained control, so the zero tuning promise is not yet total.

Reproducing the guided diffusion idea

The engineering that puts this on a real humanoid is enormous, but the core idea, steer a diffusion model over trajectories toward a task cost at inference time, is compact and reproducible. The implementation below writes exactly that. It trains a small diffusion model on a prior of smooth, natural trajectories, then samples with and without classifier guidance toward a waypoint, and includes a composable obstacle cost like the one the paper uses. A runnable smoke test shows guidance pulling the endpoint to the goal while the motion stays smooth, the same mechanism that lets BeyondMimic solve tasks it never trained on.

# Classifier guided trajectory diffusion, the core idea behind
# BeyondMimic from Liao, Truong, et al., "BeyondMimic: From motion
# tracking to versatile humanoid control via guided diffusion"
# (Science Robotics 2026). A small diffusion model learns a prior over
# smooth trajectories, then a task cost steers sampling at inference,
# with no retraining. The real system does this in a learned latent
# space over humanoid state-action trajectories.

import torch
import torch.nn as nn
import torch.nn.functional as F

T = 24            # trajectory length
D = 2             # position dimension
K = 50            # diffusion steps
betas = torch.linspace(1e-4, 0.02, K)
alphas = 1 - betas
abar = torch.cumprod(alphas, 0)


def sample_natural(n):
    """A prior of smooth trajectories, the stand in for human motions."""
    v = torch.randn(n, T, D) * 0.15
    k = torch.tensor([0.25, 0.5, 0.25]).view(1, 1, 3)
    for d in range(D):                       # smooth the velocities
        v[:, :, d] = F.conv1d(v[:, :, d].unsqueeze(1), k, padding=1).squeeze(1)
    pos = torch.cumsum(v, dim=1)
    return (pos - pos[:, 0:1]).reshape(n, T * D)   # start at origin


class Denoiser(nn.Module):
    """Predicts the noise added to a trajectory at diffusion step t."""
    def __init__(self, dim=T * D, h=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim + 1, h), nn.SiLU(),
            nn.Linear(h, h), nn.SiLU(),
            nn.Linear(h, h), nn.SiLU(),
            nn.Linear(h, dim))

    def forward(self, x, t):
        te = (t.float() / K).view(-1, 1)
        return self.net(torch.cat([x, te], 1))


def train(model, iters=1500, lr=2e-3):
    """Standard denoising diffusion training on the natural prior."""
    opt = torch.optim.Adam(model.parameters(), lr)
    for _ in range(iters):
        x0 = sample_natural(256)
        t = torch.randint(0, K, (256,))
        noise = torch.randn_like(x0)
        ab = abar[t].view(-1, 1)
        xt = ab.sqrt() * x0 + (1 - ab).sqrt() * noise
        loss = ((model(xt, t) - noise) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()


def task_cost(x, goal, obstacle=None):
    """Differentiable task cost G. Reach a waypoint, optionally avoid a disk.

    The obstacle term is a signed distance style penalty, the same kind of
    composable cost the paper sums for scene aware navigation.
    """
    p = x.view(-1, T, D)
    cost = ((p[:, -1] - goal) ** 2).sum(1)          # reach the waypoint
    if obstacle is not None:
        center, radius = obstacle
        dist = ((p - center) ** 2).sum(-1).clamp(min=1e-6).sqrt()
        cost = cost + 3.0 * torch.clamp(radius - dist, min=0.0).pow(2).sum(1)
    return cost.sum()


def sample(model, n, goal=None, guide=0.0, obstacle=None):
    """Reverse diffusion with optional classifier guidance toward a cost."""
    x = torch.randn(n, T * D)
    for t in reversed(range(K)):
        b, a, ab = betas[t], alphas[t], abar[t]
        with torch.no_grad():
            eps = model(x, torch.full((n,), t))
            mean = (x - b / (1 - ab).sqrt() * eps) / a.sqrt()
        if goal is not None and guide > 0:
            xin = x.detach().requires_grad_(True)
            grad, = torch.autograd.grad(task_cost(xin, goal, obstacle), xin)
            mean = mean - guide * b * grad         # the guidance step
        x = mean + (b.sqrt() * torch.randn_like(x) if t > 0 else 0.0)
    return x.detach().view(n, T, D)


def smoothness(p):
    """Mean squared acceleration. Lower means smoother, more natural."""
    acc = p[:, 2:] - 2 * p[:, 1:-1] + p[:, :-2]
    return (acc ** 2).mean().item()


def smoke_test():
    """Guidance steers the natural prior to a waypoint, no retraining."""
    torch.manual_seed(0)
    model = Denoiser()
    train(model)

    goal = torch.tensor([[2.0, 1.0]])
    unguided = sample(model, 64)
    guided = sample(model, 64, goal=goal, guide=25.0)

    d_un = ((unguided[:, -1] - goal) ** 2).sum(1).sqrt().mean().item()
    d_gd = ((guided[:, -1] - goal) ** 2).sum(1).sqrt().mean().item()

    # Composable guidance, add an obstacle cost to the same model.
    obstacle = (torch.tensor([1.0, 0.5]), 0.4)
    composed = sample(model, 64, goal=goal, guide=25.0, obstacle=obstacle)
    d_co = ((composed[:, -1] - goal) ** 2).sum(1).sqrt().mean().item()

    assert d_gd < 0.6 * d_un, "guidance should pull the endpoint to the goal"
    print("endpoint distance to goal  unguided {:.2f}  guided {:.2f}".format(d_un, d_gd))
    print("smoothness  natural {:.4f}  guided {:.4f}".format(
        smoothness(sample_natural(64).view(-1, T, D)), smoothness(guided)))
    print("composed waypoint plus obstacle cost, goal distance {:.2f}".format(d_co))
    print("smoke test passed")


if __name__ == "__main__":
    smoke_test()

The honest note is scale. This runs a small diffusion model on two dimensional toy trajectories, where the real system runs a latent diffusion model over high dimensional humanoid state action trajectories decoded through a variational autoencoder. The mechanism is the same, a learned prior over natural motion plus a gradient from a task cost steering generation at inference, and it is exactly what the smoke test shows when the guided endpoints land near the goal while the paths stay smooth. Swap the toy prior for the latent motion model and the toy cost for a velocity or obstacle cost and you have the shape of BeyondMimic’s second stage.

What this changes for humanoid robotics

The practical shift is away from building one controller per behavior and toward building one model that holds many behaviors and gets steered. For anyone deploying a humanoid, that means a new task can often be a new cost function evaluated at run time rather than a new training run, which collapses the effort of adding capabilities. The open source release has already been picked up widely, reproduced across several different humanoid platforms, and a follow up scaled the same recipe to hundreds of hours of motion, evidence that the formulation travels rather than being tied to one robot.

The deeper idea, learn a rich prior over feasible behavior and then guide it toward goals, is one the field keeps rediscovering because it works. It shows up in how planners reason about motion and in how robots learn manipulation from demonstration, and BeyondMimic is a striking instance of it for whole body control. For readers tracking that thread across robotics, it sits alongside work on how robots learn manipulation skills in the DySL-VLA vision language action model, the contact physics of releasing an object mid throw, and the planning side of coordinating robots through tight spaces. You can browse the wider set through the robotics and autonomous systems pillar.

Conclusion

The core achievement of BeyondMimic is a humanoid controller that keeps the agility and natural look of learning from human motion while gaining the versatility to compose skills into tasks it never trained on. A two stage design does the work, a scalable reinforcement learning recipe that masters hundreds of motions under one shared formulation, and a guided diffusion model that steers those skills toward run time objectives without retraining or fine tuning.

The conceptual shift is treating control as guidance over a learned prior rather than a bespoke policy per task. Because the diffusion model already carries a rich distribution of agile, feasible motion, a new task becomes a simple differentiable cost added during sampling, which sidesteps the reward engineering and behavior shaping that made older approaches brittle. That a single model can walk to a waypoint, dodge an obstacle, take a joystick command, and recover from a shove, all from the same weights, is the payoff of that shift.

The results back it up where it counts. The robot spun through cartwheels at speeds above the average skilled human acrobat, landed clean on natural terrain it never trained on, and won a naturalness study against its own vendor controller by wide margins. The latent space ablation, 96 percent success against 23 for a raw baseline, shows the design choices are load bearing rather than incidental.

The limits are stated plainly. The model depends on good state estimation, its short prediction horizon rules out long range planning for now, it is sensitive to deployment latency, and fine grained tasks still need some guidance tuning. None of these undo the result, but they sketch the next problems, longer horizons, better estimation, and tighter real time engineering.

What lingers is how much the framing did. Faced with a robot that has a human shape but none of the human ease, the answer was not a cleverer controller for each trick but a model that learns how people move and then lets a task quietly pull those movements where they need to go. Guide the prior rather than program the behavior, and a machine built like us starts to move a little more like us too.

Frequently asked questions