DrawMotion Turns a Doodle Into 3D Character Motion

Why text alone keeps failing to describe the motion in your head

Text to motion systems have gotten genuinely good at turning a sentence like walk forward and turn right into a plausible looking animated character. The trouble shows up at the edges of what language is good for. A prompt like a high kick forward tells a diffusion model almost nothing about exactly how the arms should swing through the kick, whether the torso twists, how far the supporting leg bends. Prior work tried to close this gap by asking users to write more, appending extra clauses, describing individual body parts, even generating small chunks of code to specify joint rotations directly. Every one of those fixes asks the user to translate a mental image into more precise language, and precise language about body mechanics is exactly the kind of thing most people struggle to produce on demand.

DrawMotion’s starting bet is that this translation step is the actual bottleneck, not the underlying generative model. If someone can picture the motion clearly enough to critique a generated animation, they can usually sketch its rough shape with a pen, a path across the screen showing where the character travels, and a few stick figures showing what the body looks like at key moments. That sketch carries spatial information text was never well suited to convey, while the text prompt still carries whatever semantic meaning benefits from being said in words. The paper treats these as genuinely complementary channels rather than as one replacing the other.

Sketch as a second input, what a stickman actually encodes

A hand drawing condition in DrawMotion has two parts. A 2D trajectory, drawn as a single continuous path on screen, gets resampled to match the number of frames in the target motion sequence, either preserving the original drawing speed or normalizing it depending on the setting. Along that trajectory, the user can drop in stick figures at any point to pin down exactly what the body should look like at that moment, useful for a kick, a jump, or any pose too specific to describe in a sentence.

Turning a scribble into training data

Training a model to interpret hand drawn stick figures runs into an immediate problem, no dataset of real hand drawn stickmen paired with motion sequences exists. The authors solved this with a Stickman Generation Algorithm that converts the 3D joint coordinates already present in standard motion datasets into synthetic sketches designed to mimic how an actual person draws. That meant reasoning carefully about what makes a real sketch look like a sketch rather than a computer generated diagram, stroke smoothness that varies with device and individual style, misplacement where a hand slips slightly off target, and scaling errors where a person focuses on getting one limb right while losing track of the whole figure’s proportions. The algorithm also standardizes viewing angle, generating each stickman as if observed from directly in front of the body, since the same pose can look confusingly different from other angles.

Six strokes instead of two hundred points

The naive way to represent a stick figure, as a set of roughly 200 two dimensional coordinate points with connectivity information, would faithfully capture the drawing but at a computational cost the authors call prohibitive, since attention over that many points means tracking pairwise interactions across all of them. Their fix compresses each figure down to six single stroke lines, one for the head, one for the torso, and one for each limb, drawn in any order. Each of those six lines gets encoded separately and then aggregated by a small transformer encoder into a single stickman embedding. The paper reports this compact encoding both reduces computational cost and improves how accurately the model recognizes the intended pose, a rare case where the cheaper representation also performed better.

Fusing a sketch and a sentence without paying twice for attention

Once text and drawing conditions are both encoded, the diffusion model needs a way to combine them, and to handle every situation where a user supplies only one, both, or neither. Earlier systems handled this by running one shared self attention layer over all conditions at once and masking out the parts corresponding to whichever conditions were absent for a given sample. It works, but the authors point out it wastes computation on masked out tokens and constrains every condition to pass through the same uniform attention structure regardless of whether that structure actually suits the condition.

The Multi Condition Module, four combinations from two decoders

DrawMotion’s answer, the Multi Condition Module, splits incoming data along the batch dimension into four segments representing the four possible condition combinations, text plus drawing, text alone, drawing alone, and neither. Rather than running four separate decoders, a Condition Fusion step processes only the text input for the segments that need it and only the drawing input for the segments that need that, using just two decoders total, then sums the resulting offsets back onto the appropriate motion features along the batch dimension. A Latent Encoder afterward encodes the fused result again for further integration. The practical payoff is that all four combinations get produced from two decoders instead of four, with no wasted attention over irrelevant tokens, and the paper reports this reduces computational complexity while also improving performance compared to the traditional masked self attention baseline.

Picking the right attention for each condition

The stickman and trajectory conditions get standard dot product attention, since they encode frame by frame local pose and position information where the query needs to explicitly attend across all tokens to find the right temporal match. The text condition instead gets a more efficient linear complexity attention mechanism, justified by the fact that text carries fairly global semantic meaning rather than frame specific detail, so the model does not need the full pairwise attention pattern to make good use of it. An ablation comparing these choices against their reverse confirms the pairing, dot product attention suits the local, frame aligned drawing conditions, while efficient attention suits the global, holistic text condition.

Training free guidance, steering generation without retraining

Even with a well designed fusion module, competing conditions can pull the generation away from what the user actually drew. Text carries global semantic pressure, the trajectory carries global spatial pressure, and when the two disagree even slightly the generated path can drift from the sketch. DrawMotion’s fix for this borrows an idea from motion editing research but adapts it to avoid that field’s usual costs.

Why earlier motion editing tricks either break fidelity or cost too much

Two families of existing motion editing approaches handle this kind of spatial correction. One, exemplified by OmniControl, directly overwrites the predicted clean motion at each denoising step and then relies on an additional ControlNet style module to pull the result back toward a realistic distribution, at the cost of extra trained parameters. The other, exemplified by DNO, backpropagates the spatial loss all the way to the initial random noise the diffusion process starts from, which keeps the final result within a well behaved distribution but requires repeated gradient steps through the entire denoising chain, a genuinely expensive operation.

A continuous feature space you can safely push around

DrawMotion’s authors noticed something specific about the Multi Condition Module’s own intermediate features, the outputs of its Condition Fusion step form a continuous, densely packed region of feature space rather than the scattered, cluster like distribution typical of ordinary model layers. They demonstrate this experimentally by shuffling and interpolating between feature vectors within a batch and checking whether the resulting motion quality collapses, and it does not, unlike a comparison baseline built on ReMoDiffuse, whose equivalent features do degrade sharply under the same test. A feature space that tolerates this kind of perturbation is one where you can nudge features directly with a gradient from a spatial loss, without the ControlNet style correction module OmniControl needs and without the expensive backpropagation all the way to the initial noise that DNO relies on.

That is the Mahalanobis distance the method uses to keep its updates honest. During the guided denoising steps, the algorithm estimates the mean and covariance of the intermediate features it expects to see, then measures how far a proposed update has drifted from that expected distribution, accounting for correlations between feature dimensions rather than treating every dimension as equally important. Whenever an update would push the feature past a set boundary, the algorithm clips it, keeping only a small fraction, around 0.01 in the authors’ best configuration, of the excess gradient rather than discarding the update outright or letting it run unchecked. The result, which the authors call Intermediate Feature Guidance, gets the trajectory alignment benefits of the more expensive editing methods at a fraction of their computational cost, and the paper reports it also shortens overall inference time compared to those alternatives.

What the benchmarks and the user study actually show

DrawMotion is evaluated on the standard KIT-ML and HumanML3D text to motion benchmarks using the field’s usual metrics, alongside two measures specific to sketch conditioned generation, Stickman Similarity and 2D trajectory error.

The professional animator comparison is worth sitting with, since it is a rarer kind of evidence in a generative modeling paper. Five professional animators were asked to manually produce stick figure based 3D animations under the same trajectory constraints DrawMotion was given. They took roughly 3 hours per sample and scored a mean of 7.4 on the same rating scale, and while their handmade results tracked the target trajectory faithfully, the animators themselves reported that the AI generated motions looked richer and more natural than what pure manual keyframe editing produced, while DrawMotion’s own generation latency was, in their words, acceptable in practice.

What still goes wrong

The authors are direct about where DrawMotion’s added control becomes a liability rather than a feature. Because the system gives users the freedom to specify both a trajectory and character poses at arbitrary points, nothing stops a user from drawing a trajectory that contradicts their own text prompt, or sketching a stick figure pose that is not physically achievable by a human body. When that happens, the paper reports the generated motion tends to deviate from the input and lose fidelity, essentially the model has to resolve a genuine contradiction in what it was asked to produce, and it does not always resolve it gracefully.

The authors’ proposed mitigation is refreshingly practical rather than purely technical, since the final guidance loss computed during Intermediate Feature Guidance can itself be reported back to the user as a rough signal of how much conflict existed between their inputs, giving them something concrete to adjust rather than leaving them to guess why a result looks off. There is also a structural trade off worth naming directly, letting users place stick figures anywhere shifts real responsibility onto them for keeping the input roughly physically reasonable, whereas the more restrictive StickMotion, which only allowed 3 automatically placed stick figures, sacrificed flexibility for a design that was harder for users to break.

Why this idea travels beyond animation

The broader lesson sitting underneath DrawMotion’s specific design choices is about where to put the burden of precision in a generative system. Rather than asking users to get better at describing spatial detail in words, a request that keeps failing across the whole text to motion literature, the paper hands spatial precision to the modality that is actually good at it, a hand drawn mark on a screen, while leaving semantic nuance to the modality that is actually good at that, natural language. Neither channel is asked to do the other’s job.

That division of labor, and the specific mechanism that makes it efficient, batching condition combinations so that a handful of decoders can cover every combination rather than training one decoder per combination, seems like a genuinely reusable pattern anywhere a generative model needs to accept several partially redundant, partially complementary control signals. The Intermediate Feature Guidance idea is arguably even more portable. Any diffusion model whose intermediate features happen to form a continuous rather than clustered space could in principle borrow the same trick, gradient guidance from a spatial or structural loss applied directly to those features, with a Mahalanobis distance boundary keeping the updates honest, without needing a separate trained correction network or expensive backpropagation through the full denoising chain.

Complete PyTorch implementation

The following code is an original, simplified educational implementation inspired by the mechanisms described in the paper, the six stroke stickman encoder, the Multi Condition Module’s batch partitioning trick, and Intermediate Feature Guidance with Mahalanobis distance clipping. It is not the authors’ released code, and it runs on small dummy tensors as a sanity check of the logic. The real, full implementation from the authors is publicly available and linked at the end of this article.

# drawmotion_lite.py
# Educational reimplementation of the core DrawMotion mechanisms.
# Not the authors' official code, see the real repository linked in this article.

import torch
import torch.nn as nn
import torch.nn.functional as F


class StickmanEncoder(nn.Module):
    """Encodes six single stroke lines, head, torso, and four limbs, into
    one stickman embedding via a small transformer encoder, matching
    the paper's compact stroke based representation."""

    def __init__(self, points_per_stroke=8, dim=64, num_heads=4):
        super().__init__()
        self.point_proj = nn.Linear(2, dim)
        encoder_layer = nn.TransformerEncoderLayer(d_model=dim, nhead=num_heads, batch_first=True)
        self.stroke_encoder = nn.TransformerEncoder(encoder_layer, num_layers=1)
        self.stroke_pool = nn.Linear(dim, dim)
        self.aggregate = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=dim, nhead=num_heads, batch_first=True), num_layers=1
        )
        self.output_proj = nn.Linear(dim, dim)

    def forward(self, strokes):
        """strokes has shape (batch, 6, points_per_stroke, 2)."""
        b, num_strokes, num_points, _ = strokes.shape
        flat = strokes.view(b * num_strokes, num_points, 2)
        tokens = self.point_proj(flat)
        encoded = self.stroke_encoder(tokens)
        pooled = self.stroke_pool(encoded.mean(dim=1))
        pooled = pooled.view(b, num_strokes, -1)
        aggregated = self.aggregate(pooled)
        stickman_embedding = self.output_proj(aggregated.mean(dim=1))
        return stickman_embedding


class ConditionFusion(nn.Module):
    """A simplified Multi Condition Module. Splits a batch into four
    segments representing (text, draw), (text, none), (none, draw) and
    (none, none), then produces all four outputs using only a text
    decoder and a draw decoder, matching the paper's batch
    partitioning trick rather than running four separate decoders."""

    def __init__(self, dim=64):
        super().__init__()
        self.text_decoder = nn.Linear(dim, dim)
        self.draw_decoder = nn.Linear(dim, dim)
        self.latent_encoder = nn.Linear(dim, dim)

    def forward(self, motion_feature, text_embedding, draw_embedding):
        """Each input has shape (4, dim), one row per condition segment,
        in the order (text+draw), (text only), (draw only), (neither)."""
        text_offset = self.text_decoder(text_embedding)
        draw_offset = self.draw_decoder(draw_embedding)

        # segment 0 (text+draw) gets both offsets, segment 1 (text only) gets
        # only the text offset, segment 2 (draw only) gets only the draw
        # offset, segment 3 (neither) gets no offset at all
        offsets = torch.zeros_like(motion_feature)
        offsets[0] = text_offset[0] + draw_offset[0]
        offsets[1] = text_offset[1]
        offsets[2] = draw_offset[2]
        # offsets[3] stays zero, the unconditional case

        fused = motion_feature + offsets
        return self.latent_encoder(fused)


def mahalanobis_distance(feature, mean, cov_inv):
    """Measures how far a feature has drifted from an expected
    distribution, accounting for correlations between dimensions,
    matching Equation for M(F) used in Intermediate Feature Guidance."""
    diff = feature - mean
    dist_sq = torch.einsum("i,ij,j->", diff, cov_inv, diff)
    return torch.sqrt(torch.clamp(dist_sq, min=0.0))


def intermediate_feature_guidance(feature, target_trajectory, mean, cov_inv,
                                  md_boundary=1.0, clip_scale=0.01, repeat=5, lr=0.05):
    """A simplified stand in for Algorithm 1, Intermediate Feature
    Guidance. Uses SGD to move a feature toward better trajectory
    alignment, then applies Mahalanobis distance clipping so the
    update never pushes the feature far outside its expected
    distribution."""
    working_feature = feature.clone().detach().requires_grad_(True)

    for _ in range(repeat):
        predicted_position = working_feature[:2]
        loss = F.mse_loss(predicted_position, target_trajectory)
        grad = torch.autograd.grad(loss, working_feature)[0]

        with torch.no_grad():
            proposed = working_feature - lr * grad
            distance = mahalanobis_distance(proposed.detach(), mean, cov_inv)
            base_distance = mahalanobis_distance(feature, mean, cov_inv)

            if distance > base_distance + md_boundary:
                # retain only a small fraction of the update once the
                # Mahalanobis boundary is exceeded, per the paper's clip scale
                delta = proposed - working_feature
                proposed = working_feature + clip_scale * delta

            working_feature = proposed.clone().requires_grad_(True)

    return working_feature.detach()


def condition_mixture(eps_text_draw, eps_draw, eps_text, eps_none, w=1.6, w_hat=1.2):
    """Reimplements the paper's Equation 6, a weighted mixture of noise
    predictions from four condition combinations, adjustable toward
    the drawing condition through the probability controlled weight."""
    w1, w2, w3, w4 = w, w_hat, w_hat, 1 - 2 * w_hat
    mixed = w1 * eps_text_draw + w2 * eps_draw + w3 * eps_text + w4 * eps_none
    return mixed


def make_dummy_batch(dim=64):
    motion_feature = torch.randn(4, dim)
    text_embedding = torch.randn(4, dim)
    draw_embedding = torch.randn(4, dim)
    strokes = torch.randn(2, 6, 8, 2)
    return motion_feature, text_embedding, draw_embedding, strokes


if __name__ == "__main__":
    torch.manual_seed(0)

    # part one, stickman encoder smoke test
    stickman_encoder = StickmanEncoder(points_per_stroke=8, dim=64, num_heads=4)
    motion_feature, text_embedding, draw_embedding, strokes = make_dummy_batch()
    stickman_embedding = stickman_encoder(strokes)
    print("stickman embedding shape", stickman_embedding.shape)

    # part two, Multi Condition Module smoke test
    fusion = ConditionFusion(dim=64)
    fused_output = fusion(motion_feature, text_embedding, draw_embedding)
    print("fused motion feature shape", fused_output.shape)

    # part three, Intermediate Feature Guidance smoke test
    single_feature = fused_output[0]
    mean = torch.zeros(64)
    cov_inv = torch.eye(64)
    target_trajectory = torch.tensor([0.5, -0.2])
    guided_feature = intermediate_feature_guidance(
        single_feature, target_trajectory, mean, cov_inv, md_boundary=1.0, clip_scale=0.01, repeat=5, lr=0.05
    )
    print("guided feature, first two dims", guided_feature[:2])

    # part four, condition mixture smoke test
    eps_text_draw = torch.randn(64)
    eps_draw = torch.randn(64)
    eps_text = torch.randn(64)
    eps_none = torch.randn(64)
    mixed_eps = condition_mixture(eps_text_draw, eps_draw, eps_text, eps_none)
    print("mixed noise prediction, first five dims", mixed_eps[:5])

Running that file prints the shape of a stickman embedding built from six dummy stroke lines, confirms the Multi Condition Module correctly produces a fused feature for all four condition segments in one pass, shows the intermediate feature guidance step nudging a feature toward a target trajectory while respecting a Mahalanobis distance boundary, and finally checks that the condition mixture math combines four noise predictions into one weighted output.

Conclusion

The core achievement here is not a new diffusion architecture in the abstract sense, it is a specific, well argued claim about where the bottleneck in controllable motion generation actually sits. Text is genuinely bad at conveying spatial and postural detail, and rather than pushing users to write increasingly elaborate descriptions to compensate, DrawMotion hands that job to a hand drawn sketch, a modality humans are already fluent in for exactly this kind of spatial communication, while keeping text around for the semantic meaning it still does well.

The conceptual shift worth carrying forward is the batch partitioning trick inside the Multi Condition Module, producing every possible combination of two conditions from just two decoders rather than four, and the discovery that a model’s own intermediate feature space can be continuous enough to guide directly with gradients, no separate trained correction network required. Both ideas are narrower in scope than a totally new architecture, and that is exactly why they seem portable. Any system juggling multiple partially redundant control signals could reuse the batch partitioning idea, and any diffusion model with a sufficiently continuous intermediate feature space could reuse Intermediate Feature Guidance instead of reaching for a heavier editing method.

The honest limitation, that a sketch conflicting with its accompanying text or with basic human biomechanics will degrade the result, is not a flaw so much as an accurate description of what happens when you hand users more expressive freedom. StickMotion traded that freedom away for a safer, more constrained interface. DrawMotion bets that most users, most of the time, will sketch something reasonable, and backs that bet with a mechanism, the reported guidance loss, that at least gives users a signal when their inputs are working against each other.

What makes the user study numbers convincing is that they measure the thing that actually matters to a working animator, time spent getting from an idea to a usable result, rather than only an abstract quality score. Cutting that time by roughly 46.7 percent, and having professional animators independently judge the AI generated motion as richer than their own manual keyframe work, is a more concrete claim than most generative modeling papers manage to test at all.

For anyone building creative tools on top of motion generation, the practical takeaway is straightforward. A sketch based control channel is not a gimmick layered on top of text to motion generation, it is addressing a real and specific failure mode of text alone, and the underlying mechanisms, compact stroke encoding, efficient multi condition fusion, and gradient guidance through a continuous feature space, look reusable well beyond this one application.

Frequently asked questions

Related reading