Motion2VecSets: Teaching AI to Guess the Missing Motion in 3D Scans

Motion2VecSets · 4D Latent Set Diffusion

Why guessing a moving shape is harder than it sounds

Reconstructing a static object from a scan is already a well studied problem, going back to foundational techniques like Marching Cubes and Poisson Surface Reconstruction. Add motion to the picture and the difficulty compounds. Early systems such as DynamicFusion, VolumeDeform, and DoubleFusion tackled this by fusing depth camera frames together in real time while leaning on handcrafted deformation rules, most commonly As-Rigid-As-Possible regularization and Embedded Deformation. These rules assume nearby parts of a surface move together in roughly rigid chunks, an assumption that holds up fine for short, gentle motions and breaks down for the complex, highly non linear deformations that real bodies and real machines actually produce.

The deeper problem shows up whenever the input data itself is incomplete. A sparse point cloud, a low resolution voxel grid, or a single RGB camera angle simply does not contain enough information to pin down one correct answer. Many different complete shapes and motions are consistent with the same partial observation. Conventional feedforward neural networks, the kind that map an input directly to one output in a single pass, tend to respond to this ambiguity by averaging across the plausible answers, which produces a blurry, physically implausible compromise rather than a single crisp, correct looking result.

What came before, one code to represent an entire moving object

Parametric templates hit a topology wall

One family of solutions builds a dedicated parametric model for a specific object category. SMPL and STAR handle human bodies, FLAME handles faces, MANO handles hands, and SMAL handles four legged animals. These models are efficient and effective for tracking and animation within their own domain, but they all share a fixed mesh template, which makes them unable to represent an object whose topology changes or falls outside their narrow category definition.

Neural fields solved topology, then lost detail again

Neural implicit fields, representing a surface as a continuous function learned by a coordinate based multilayer perceptron rather than a fixed mesh, removed the topology restriction and could represent arbitrary shapes at high resolution. Extending this idea into 4D, methods like OFlow, NPMs, LPDC, and CaDeX represented an entire moving sequence using one single global latent vector, encoding the whole shape and its full motion trajectory into that one compressed code. The approach works reasonably well for a single, well behaved category like human bodies, but a single vector simply cannot hold onto fine local deformation detail, and the resulting models struggle badly when asked to generalize to an identity or a motion style they never saw during training.

The core idea, swap one global code for many local ones

Motion2VecSets starts from a simple observation, borrowed and extended from the earlier 3DShape2VecSet representation for static shapes. Objects with completely different overall topologies, a human torso and a dog’s flank, for instance, often exhibit similar local geometry and deformation patterns at a small enough scale. If a model can learn a vocabulary of these recurring local patterns rather than memorizing whole object categories, it should generalize far better to shapes and motions it has never encountered.

The team formalizes a moving mesh sequence using an occupancy field for the very first frame plus a chain of flow fields carrying that first frame forward through time.

Instead of compressing all of that into one vector, the model keeps a shape latent set for the first frame and a separate deformation latent set for every subsequent frame.

Both sets are built the same general way. The shape autoencoder farthest point samples a raw surface point cloud of 2048 points down to 512 representative points, embeds them, and compresses the result through a cross attention layer regularized with a KL divergence term, the same recipe used in standard latent diffusion pipelines to keep the latent space well behaved for later denoising. The deformation autoencoder does something slightly more involved, since it needs to encode a pairwise relationship rather than a single shape. It takes a source point cloud from the first frame and a target point cloud from a later frame, downsamples the source using farthest point sampling, then reuses those exact same sample indices on the target cloud. That reuse is what keeps every deformation latent tied to a consistent local surface region across the entire sequence, which is precisely what later lets the model stack deformation latents cleanly along the time axis and attend across frames.

Diffusing shape and motion together, not frame by frame

With a compact 4D representation in hand, training proceeds in two stages, first the autoencoders described above, then a pair of diffusion models that learn to generate plausible latents from noise, following the EDM noise scheduling approach from Karras and colleagues.

Shape diffusion reconstructs the reference frame

Synchronized deformation diffusion tracks the whole sequence at once

Rather than denoising each later frame in isolation, which risks jitter and drift from one frame to the next, or brute force attending across every frame and every latent simultaneously, which blows up computational cost to the square of the sequence length, Motion2VecSets treats the entire stack of deformation latents as one motion latent set and denoises all of them together in a single synchronized pass.

An attention block that alternates instead of brute forcing everything at once

The denoiser’s core building block, which the authors call an Interleaved Spatio-Temporal Attention block, runs three sublayers in sequence. A space self attention layer lets the 512 latents within a single frame talk to each other. A conditional cross attention layer injects the ambiguous input signal, whatever partial point cloud or image the model has to work with, into the denoising process. A time self attention layer then lets each of the 512 latents talk to its own counterpart across every other frame in the sequence, which is only possible because the shared farthest point sampling indices guarantee latent number 37, say, refers to the same patch of surface in frame one and frame sixteen alike. Alternating between spatial and temporal attention rather than running one giant combined attention operation drops the computational complexity from a cost that scales with the square of both the frame count and the latent count down to a cost that scales linearly with each dimension on its own, a meaningful saving once a sequence runs past a handful of frames.

What it was tested on, and how well it held up

The team evaluated on three public 4D datasets, Dynamic FAUST for human body motion, DeformingThings4D-Animals for a wide range of animal species, and Shape2Motion for articulated mechanical objects like laptops, staplers, and washing machines, comparing against three established baselines, OFlow, LPDC, and CaDeX, using Intersection over Union, Chamfer Distance, and point correspondence error as the evaluation metrics.

The qualitative pattern behind those numbers is consistent across every task. Prior global latent methods do reasonably well on the human dataset, where every training example shares roughly the same topology, but they visibly fall apart on animals and articulated machines, producing fragmented surfaces and losing track of fast moving parts such as a hinge or a swinging limb. Motion2VecSets keeps both geometry and motion coherent across all three categories, and on the voxel super resolution task it goes a step further, hallucinating plausible fine details, continuous antlers, smooth limb connections, that were never actually present in the coarse input grid at all, a direct consequence of having learned a genuine distribution over plausible shapes rather than a single deterministic mapping.

Taking the model apart, what each piece actually earns

A set of ablation experiments on the Dynamic FAUST dataset isolates exactly which design choices are doing the work. Removing the diffusion process entirely and replacing it with a deterministic feedforward network, using the identical autoencoder backbone, produces noticeably worse reconstructions, confirming that the probabilistic sampling step itself, not just the underlying representation, is responsible for handling ambiguous input gracefully. Shrinking the latent set down to a single global vector, effectively recreating the older approach, causes a clear drop in accuracy that grows even larger specifically on unseen identities, which is the cleanest evidence in the paper that local latents are what drive the generalization advantage rather than merely adding model capacity. Removing the time self attention layer degrades every metric the authors track, confirming its role in keeping motion smooth across frames rather than jittering from one denoising pass to the next.

A separate ablation on the animal dataset compares three configurations directly, a fully global shape and deformation representation, a hybrid with local shape latents but a global deformation code, and the full local plus local design. The fully global variant cannot even reconstruct a plausible first frame shape for an unfamiliar animal. The hybrid recovers reasonable static geometry but still loses track of fast, non linear motion such as a hinge swinging open. Only the complete local representation handles both halves of the problem at once. A cross dataset test pushes this further still, training only on animal motions and testing directly on human bodies with no fine tuning at all. Motion2VecSets stays stable in that transfer, while every baseline method’s accuracy collapses, and the gap between them actually widens compared with the in domain animal results, reinforcing that the local latent design is what generalizes rather than something specific to any one dataset’s quirks.

Past the benchmark, editing shapes, long sequences, and real footage

Because the shape and motion diffusion stages are cleanly separated, the same trained model supports a few capabilities the benchmark numbers alone do not capture. Given a static mesh and a handful of sparse control points dragged to new target positions, the motion diffusion stage alone can propagate those edits into a full, globally coherent deforming sequence, effectively turning the model into an interactive shape animation tool, and it does this even for shapes it never saw during training. In a deliberately extreme test, the team fed the model only upper body point clouds from the Dynamic FAUST dataset and asked it to reconstruct full body motion, a setup where a single visible upper body motion is genuinely consistent with many different plausible leg movements. Baseline methods produced distorted output, broken feet among the more visible failures, while Motion2VecSets generated complete, temporally coherent bodies and offered multiple distinct plausible completions for the same ambiguous input, a direct demonstration of the diffusion model’s ability to sample rather than average across an underdetermined problem.

Although training sequences were fixed at seventeen frames, the authors show the model extends to sequences of a hundred frames or more by splitting a long recording into overlapping seventeen frame windows and running the diffusion process autoregressively across them, preserving both geometric detail and temporal smoothness over the extended range. Real world validation came from two sources, the BEHAVE dataset, which captures real depth camera footage complete with occlusions and sensor noise, and ordinary monocular RGB video processed first through a separate 3D reconstruction tool called TripoSG to obtain per frame point clouds. The model handled both without any additional fine tuning, successfully inferring complete surfaces even where limbs were physically blocked from the camera’s view.

Where the method still comes up short

The authors document their own failure cases plainly rather than glossing over them. Reconstructing an identity that sits far outside the training distribution, their examples include a butterfly and a piece of draping cloth, produces noticeably worse results, which is expected for any learned prior but worth stating directly. Physical realism is a separate and more fundamental gap. Cloth, hair, and fluid motion follow physical simulation rules that this model has no access to, since its deformation priors come entirely from observed data rather than any physics based constraint, so its reconstructed cloth motion can look plausible in a generic sense without actually obeying how real fabric behaves.

The authors list four limitations directly in their conclusion. Training requires real 4D mesh sequences with dense temporal correspondence, which are expensive and difficult to collect at scale, meaning more diverse and larger datasets, or some hybrid supervision mixing mesh data with ordinary video, will likely be necessary to push the priors further. The current framework reconstructs geometry only and has no mechanism for texture or color. It handles one object at a time and does not yet extend to full dynamic scenes with multiple interacting objects. And its deformation priors are learned purely from data with no physical constraints built in, which the authors flag as the most promising direction for follow up work, particularly for materials like hair, cloth, and fluids where physics based priors could meaningfully improve realism. On the practical side, the multiple denoising steps required during inference, eighteen in the paper’s configuration, make the method slower at test time than a single pass feedforward network, though the authors note that distillation techniques built for image diffusion models could likely be adapted to shrink that gap.

Building a simplified Motion2VecSets pipeline in PyTorch

The implementation below captures the shape of the paper’s approach, a shape autoencoder built around farthest point sampling and cross attention, a deformation autoencoder that reuses sampling indices to preserve correspondence, a compact interleaved space and time attention denoiser, EDM style diffusion losses for both shape and motion, and a full crafting and evaluation loop. A smoke test at the bottom runs everything on randomly generated dummy point clouds so the code can be verified without downloading any dataset.

import torch
import torch.nn as nn
import torch.nn.functional as F

# ----------------------------------------------------------------------
# Farthest point sampling, simplified iterative implementation
# ----------------------------------------------------------------------
def farthest_point_sample(points, num_samples):
    # points: (B, N, 3)
    B, N, _ = points.shape
    device = points.device
    indices = torch.zeros(B, num_samples, dtype=torch.long, device=device)
    distance = torch.full((B, N), 1e10, device=device)
    farthest = torch.randint(0, N, (B,), device=device)
    batch_idx = torch.arange(B, device=device)
    for i in range(num_samples):
        indices[:, i] = farthest
        centroid = points[batch_idx, farthest, :].unsqueeze(1)
        dist = torch.sum((points - centroid) ** 2, dim=-1)
        distance = torch.minimum(distance, dist)
        farthest = torch.argmax(distance, dim=-1)
    return indices


def index_select_points(points, indices):
    B = points.shape[0]
    return torch.stack([points[b, indices[b]] for b in range(B)], dim=0)


# ----------------------------------------------------------------------
# Point embedding, a small MLP standing in for a positional encoder
# ----------------------------------------------------------------------
class PointEmbed(nn.Module):
    def __init__(self, in_dim=3, out_dim=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(in_dim, out_dim), nn.GELU(), nn.Linear(out_dim, out_dim))

    def forward(self, x):
        return self.net(x)


# ----------------------------------------------------------------------
# Cross attention compression layer used by both autoencoders
# ----------------------------------------------------------------------
class CrossAttention(nn.Module):
    def __init__(self, dim, heads=4):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm_q = nn.LayerNorm(dim)
        self.norm_kv = nn.LayerNorm(dim)

    def forward(self, query, key_value):
        q = self.norm_q(query)
        kv = self.norm_kv(key_value)
        out, _ = self.attn(q, kv, kv)
        return out


# ----------------------------------------------------------------------
# Shape autoencoder, compresses a point cloud into a shape latent set
# ----------------------------------------------------------------------
class ShapeAutoencoder(nn.Module):
    def __init__(self, latent_dim=32, num_latents=64, embed_dim=128):
        super().__init__()
        self.num_latents = num_latents
        self.embed = PointEmbed(out_dim=embed_dim)
        self.cross_attn = CrossAttention(embed_dim)
        self.to_latent = nn.Linear(embed_dim, latent_dim)
        self.from_latent = nn.Linear(latent_dim, embed_dim)
        self.query_embed = PointEmbed(out_dim=embed_dim)
        self.occ_head = nn.Sequential(nn.Linear(embed_dim, embed_dim), nn.GELU(), nn.Linear(embed_dim, 1))

    def encode(self, points):
        idx = farthest_point_sample(points, self.num_latents)
        sampled = index_select_points(points, idx)
        full_embed = self.embed(points)
        query_embed = self.embed(sampled)
        fused = self.cross_attn(query_embed, full_embed)
        latent = self.to_latent(fused)
        return latent

    def decode(self, latent, query_points):
        latent_embed = self.from_latent(latent)
        query_embed = self.query_embed(query_points)
        fused = self.cross_attn(query_embed, latent_embed)
        return self.occ_head(fused).squeeze(-1)

    def forward(self, points, query_points):
        latent = self.encode(points)
        occ = self.decode(latent, query_points)
        return latent, occ


# ----------------------------------------------------------------------
# Deformation autoencoder, encodes a source and target point cloud pair
# ----------------------------------------------------------------------
class DeformationAutoencoder(nn.Module):
    def __init__(self, latent_dim=32, num_latents=64, embed_dim=128):
        super().__init__()
        self.num_latents = num_latents
        self.embed = PointEmbed(out_dim=embed_dim)
        self.merge = nn.Linear(embed_dim * 2, embed_dim)
        self.cross_attn = CrossAttention(embed_dim)
        self.to_latent = nn.Linear(embed_dim, latent_dim)
        self.from_latent = nn.Linear(latent_dim, embed_dim)
        self.flow_head = nn.Sequential(nn.Linear(embed_dim, embed_dim), nn.GELU(), nn.Linear(embed_dim, 3))

    def encode(self, src, tgt):
        idx = farthest_point_sample(src, self.num_latents)
        src_sampled = index_select_points(src, idx)
        tgt_sampled = index_select_points(tgt, idx)

        src_full = self.embed(src)
        tgt_full = self.embed(tgt)
        full_fused = self.merge(torch.cat([src_full, tgt_full], dim=-1))

        src_q = self.embed(src_sampled)
        tgt_q = self.embed(tgt_sampled)
        query_fused = self.merge(torch.cat([src_q, tgt_q], dim=-1))

        fused = self.cross_attn(query_fused, full_fused)
        latent = self.to_latent(fused)
        return latent, idx

    def decode(self, latent, query_points):
        latent_embed = self.from_latent(latent)
        query_embed = self.embed(query_points)
        fused = self.cross_attn(query_embed, latent_embed)
        offset = self.flow_head(fused)
        return query_points + offset


# ----------------------------------------------------------------------
# Interleaved Spatio-Temporal Attention block, the denoiser's core unit
# ----------------------------------------------------------------------
class ISTABlock(nn.Module):
    def __init__(self, dim, heads=4):
        super().__init__()
        self.space_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.cond_attn = CrossAttention(dim, heads)
        self.time_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm1 = nn.LayerNorm(dim)
        self.norm2 = nn.LayerNorm(dim)

    def forward(self, motion_latents, cond_latents):
        # motion_latents: (B, T, M, C), cond_latents: (B, T, L, C)
        B, T, M, C = motion_latents.shape

        # space self attention, treat each frame independently
        x = motion_latents.reshape(B * T, M, C)
        x_norm = self.norm1(x)
        space_out, _ = self.space_attn(x_norm, x_norm, x_norm)
        x = x + space_out

        # conditional cross attention, inject the ambiguous input signal
        cond = cond_latents.reshape(B * T, -1, C)
        x = x + self.cond_attn(x, cond)

        # time self attention, let each latent talk to its own history
        x = x.reshape(B, T, M, C).permute(0, 2, 1, 3).reshape(B * M, T, C)
        x_norm = self.norm2(x)
        time_out, _ = self.time_attn(x_norm, x_norm, x_norm)
        x = x + time_out

        x = x.reshape(B, M, T, C).permute(0, 2, 1, 3)
        return x


# ----------------------------------------------------------------------
# Motion denoiser, stacks several ISTA blocks and conditions on shape
# ----------------------------------------------------------------------
class MotionDenoiser(nn.Module):
    def __init__(self, dim=32, depth=3):
        super().__init__()
        self.blocks = nn.ModuleList([ISTABlock(dim) for _ in range(depth)])
        self.sigma_embed = nn.Sequential(nn.Linear(1, dim), nn.GELU(), nn.Linear(dim, dim))
        self.shape_proj = nn.Linear(dim, dim)

    def forward(self, noisy_motion, sigma, shape_latents, cond_latents):
        B, T, M, C = noisy_motion.shape
        sigma_feat = self.sigma_embed(sigma.view(B, 1, 1, 1).expand(B, T, M, 1))
        x = noisy_motion + sigma_feat + self.shape_proj(shape_latents).unsqueeze(1)
        for block in self.blocks:
            x = block(x, cond_latents)
        return x


# ----------------------------------------------------------------------
# EDM style diffusion losses for the shape and motion denoisers
# ----------------------------------------------------------------------
def sample_noise_levels(batch_size, device, mu=-1.2, sigma=1.2):
    log_sigma = torch.randn(batch_size, device=device) * sigma + mu
    return log_sigma.exp()


def motion_diffusion_loss(denoiser, clean_motion, shape_latents, cond_latents):
    B = clean_motion.shape[0]
    sigmas = sample_noise_levels(B, clean_motion.device)
    noise = torch.randn_like(clean_motion) * sigmas.view(B, 1, 1, 1)
    noisy_motion = clean_motion + noise
    pred = denoiser(noisy_motion, sigmas, shape_latents, cond_latents)
    return F.mse_loss(pred, clean_motion)


# ----------------------------------------------------------------------
# Evaluation, a simple Intersection over Union on occupancy predictions
# ----------------------------------------------------------------------
def evaluate_iou(pred_occ_logits, gt_occ, threshold=0.0):
    pred_occ = (pred_occ_logits > threshold).float()
    intersection = (pred_occ * gt_occ).sum()
    union = ((pred_occ + gt_occ) > 0).float().sum()
    return (intersection / union.clamp(min=1.0)).item()


# ----------------------------------------------------------------------
# Smoke test on dummy data, no real dataset required
# ----------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(0)
    B, N, T, M, Cd = 2, 256, 5, 64, 32

    shape_ae = ShapeAutoencoder(latent_dim=Cd, num_latents=M)
    deform_ae = DeformationAutoencoder(latent_dim=Cd, num_latents=M)
    denoiser = MotionDenoiser(dim=Cd, depth=2)

    frame_one = torch.rand(B, N, 3)
    query_points = torch.rand(B, 128, 3)
    gt_occ = (torch.rand(B, 128) > 0.5).float()

    shape_latent, occ_pred = shape_ae(frame_one, query_points)
    occ_loss = F.binary_cross_entropy_with_logits(occ_pred, gt_occ)

    deform_latents = []
    for t in range(1, T):
        target_frame = torch.rand(B, N, 3)
        d_latent, _ = deform_ae.encode(frame_one, target_frame)
        deform_latents.append(d_latent)
    motion_latents = torch.stack(deform_latents, dim=1)  # (B, T-1, M, Cd)

    cond_latents = torch.rand(B, T - 1, 32, Cd)
    diffusion_loss = motion_diffusion_loss(denoiser, motion_latents, shape_latent, cond_latents)

    iou = evaluate_iou(occ_pred, gt_occ)

    print("Smoke test complete")
    print("Shape occupancy loss", occ_loss.item())
    print("Motion diffusion loss", diffusion_loss.item())
    print("Dummy occupancy IoU", iou)

Running this on random tensors confirms every stage connects correctly, farthest point sampling, correspondence preserving deformation encoding, the interleaved attention denoiser, and both loss functions. As with any smoke test on synthetic data, the numbers it prints are not meaningful in themselves, real training needs the actual mesh sequences and many more epochs than shown here, but the structure mirrors the paper’s actual pipeline closely enough to serve as a starting point.

Why this approach matters beyond one benchmark

The pattern underneath Motion2VecSets is broader than 4D mesh reconstruction specifically. Whenever a generative model needs to represent something with rich internal structure, a moving body, a large scene, a long document, there is a recurring choice between compressing everything into one global vector, which is simple but loses local detail and generalizes poorly, or keeping a set of many smaller vectors tied to consistent local regions, which costs more to design but tends to generalize far better once that correspondence is set up correctly. This paper is a clean, carefully ablated demonstration of that second choice paying off, with the correspondence preserving trick, reusing farthest point sampling indices across frames, doing much of the quiet structural work that makes the rest of the system function.

The practical implications for anyone working in 3D content creation, virtual reality, or motion capture are fairly direct. Sparse, low cost capture hardware, a single depth camera, a handful of RGB frames, a coarse voxel scan, becomes considerably more useful if a downstream model can fill in the missing detail plausibly rather than requiring dense, expensive multi camera rigs. The interactive shape editing capability, dragging a few control handles and getting back a full, coherent deformation, also points toward diffusion models increasingly functioning as authoring tools for artists and designers, not just reconstruction tools for existing footage.

The honest limitations matter just as much for anyone deciding whether to build on this work today. Data hunger is the most immediate constraint, dense 4D mesh supervision remains expensive to collect, which caps how far the current priors can generalize until larger and more varied datasets exist or someone works out a reliable way to supervise from ordinary video instead. The absence of texture, physics, and multi object scene support means this is a geometry and motion engine specifically, not yet a complete content generation pipeline, and integrating physics based priors for materials like cloth and hair remains explicitly unsolved future work rather than something this paper claims to have addressed.

Read as a single result, Motion2VecSets is a strong new benchmark number across three datasets. Read as a design lesson, it is a demonstration that the representation choice underneath a diffusion model, not just the diffusion process itself, is often what determines whether a generative system merely memorizes its training categories or genuinely learns something that transfers to shapes it has never encountered.

Frequently asked questions

Related reading