UniPart Finds 3D Object Parts From Plain Language

Why whole object understanding is not enough

General purpose robots are supposed to handle objects they have never encountered, guided by instructions a person gives in ordinary language. That goal runs into a wall at the level of parts. A robot must find the semantically relevant part of a previously unseen object, conditioned on what a human asked. Perception that stops at the object boundary cannot do this.

Existing 3D foundation models fall into two camps, and both leave a gap. One camp is promptable but coarse. Models that extend the Segment Anything style of prompting into 3D are good at separating whole object instances, but they do not resolve the internal, semantically meaningful parts inside a single object. The other camp is part aware but closed set. Models trained on datasets such as ShapeNetPart deliver fine part granularity, but only within a fixed taxonomy, so they cannot generalize to a part word they were never trained on. You either get open world coverage without parts, or parts without open world coverage.

UniPart, from a large team spanning the Institute of Automation at the Chinese Academy of Sciences, Peking University, Tsinghua University, Galbot, Shanghai Jiao Tong University, and the Beijing Academy of Artificial Intelligence, frames the missing capability cleanly. Treat fine grained 3D geometry as something you can query with language. Instead of predicting from a fixed label set, the robot takes a natural instruction and returns the corresponding 3D part mask. The paper calls this language grounded zero shot 3D part segmentation, and it attacks the problem from two sides at once, the data and the model.

The real bottleneck is data

Here is where it gets interesting. The hard part of open vocabulary part segmentation is not really the network. It is that nobody has a large, diverse, language labeled part dataset, because hand labeling parts on hundreds of thousands of 3D models is impractical, and fully automatic labeling introduces noise. UniPart’s answer is a dataset called LangPart-1M, built in two complementary tiers, and it is arguably the paper’s biggest contribution.

The automatic tier starts from roughly 800,000 Objaverse models, re rendered cleanly in Blender to produce over 8 million consistent images. A filtering pass, run by a large vision language model acting as an image judge against six fixed criteria, throws out low quality, abstract, scene level, and test leaking assets. What survives feeds a four stage generation pipeline that is worth understanding, because its cleverness is in keeping labels consistent across viewpoints.

First, a language model is asked to name each object and list its meaningful functional parts. Second, twelve random single views of each object are segmented with the Segment Anything model, and each candidate region gets a visual tag through Set-of-Mark prompting. Third, and this is the key step, a language model looks at the tagged views and merges the region tags that belong to the same named part, which enforces cross view coherence, fuses over segmented pieces, and drops outliers. Fourth, the two dimensional masks are back projected using camera geometry and depth into a single point cloud with a part label on every point.

That automatic tier is broad but slightly noisy, so a second tier adds fidelity. The team manually selected 4,000 objects with clear functional parts and had ten annotators verify and correct the auto generated masks, producing LangPart-4K. That set splits into a training portion and a held out benchmark called LangPart-1K, which deliberately includes both seen and unseen categories so open vocabulary generalization can actually be measured. All told, LangPart-1M spans more than 160,000 objects, over 250 categories, more than 1,000 part types, and about 8 million text to part pairs.

A deliberately simple model

The model, UniPart, is a feed forward cross modal 3D Transformer, and its restraint is the point. It takes a point cloud and a language phrase and predicts the matching part mask in a single forward pass, with no per part querying at inference and no slow multi view rendering.

On the geometry side, seed points are chosen by Farthest Point Sampling and grouped into local neighborhoods with nearest neighbors, and each neighborhood is encoded by a lightweight PointNet style extractor into a 3D token. On the language side, a frozen OpenAI CLIP text encoder turns the phrase into a single global embedding. The two streams meet inside a Transformer, and a decoder head upsamples back to a per point prediction, producing a binary mask over the points that match the phrase.

The interesting design decision is how language meets geometry. In principle you could fuse them with cross attention, adapters, or concatenation. UniPart instead uses the simplest possible move. It adds the text embedding to every point token at every layer.

Why does adding a vector beat fancier fusion? The authors reason that the language input is short, often just a phrase, so its meaning is already packed into one embedding, and adding that embedding to every token at every layer keeps the conditioning present throughout the network rather than letting it fade. The ablation backs this up. Swapping the addition for cross attention, multiplication, or concatenation all scored lower. Sometimes the boring option is the right one.

How it is trained

Training runs as a three stage curriculum. The first stage is a pretraining step that aligns point features with CLIP image patch features, so the 3D backbone inherits semantic structure before it ever sees a part label. That alignment uses a cosine similarity objective between each normalized point feature and its corresponding image patch feature.

The second and third stages fine tune for segmentation by minimizing a binary cross entropy loss between the predicted mask and the ground truth part label, first on the large automatic LangPart-1M and then on the smaller manually verified set to sharpen accuracy and undo the noise.

The staging matters. Alignment first grounds the geometry in language, the large noisy set teaches breadth, and the small clean set corrects the details. Each stage does a job the others cannot.

What the numbers say

Performance is measured with mean intersection over union, computed per object across its annotated parts and then averaged. The headline comparison is on Objaverse-General against open world baselines, and the gap is not subtle.

The number to sit with is the unseen column. UniPart scores 45.23 on categories it never trained on, against 27.41 for the previous best. That is the whole promise of zero shot part understanding, and the margin holds up rather than collapsing on novelty. The pattern repeats across classic transfer benchmarks. On ShapeNetPart-V2 UniPart reached 55.63 against FIND3D’s 42.15, and on PartNet-E it roughly matched or beat the field in both canonical and rotated settings, which matters because a robot never sees an object in a tidy canonical pose.

Speed is the quiet advantage. Because UniPart is feed forward, it segments an object in about 0.4 seconds, against 0.9 seconds for the nearest competitor and, tellingly, 174 and 296 seconds for the two methods that lean on repeated foundation model queries at inference. For a robot that has to act, the difference between half a second and five minutes is the difference between usable and not.

Untangling the dataset from the model

Whenever a paper ships both a big new dataset and a new architecture, the honest question is which one did the work. UniPart’s authors ask it directly with a controlled study, and this is one of the most useful tables in the paper.

Read it two ways and the answer is both. Hold the data fixed at the older FIND3D set and swap only the model, and UniPart improves the score by about 15 percent relative, so the additive injection architecture pulls real weight. Hold the model fixed at FIND3D and swap only the data to LangPart-1M, and the score jumps about 25 percent relative, so the dataset pulls even more. Put the good model on the good data and the two compound to the top result. It is refreshing to see a team quantify its own dataset’s contribution rather than let the architecture take all the credit.

Two smaller ablations round it out. Pretraining on single view partial point clouds helped fine tuning, with larger gains on partial inputs but consistent benefit on complete geometry, which is the right result since a real sensor only ever sees a partial cloud. And the fusion ablation above confirmed that plain addition beat every fancier alternative.

Does it work on a real robot?

Benchmarks are one thing, a gripper closing on the wrong end of a knife is another. The team tested UniPart on real hardware, a Franka Panda with a parallel gripper and a UR arm fitted with a Shadow Hand, using a single RealSense camera rather than an elaborate multi camera rig. UniPart predicts the target part from a language prompt, and off the shelf grasp generators plan the actual grasp on those points. This mirrors the broader move toward language conditioned robot manipulation, with part level perception as the missing front end.

Across 20 real objects named by language, UniPart reached 90.5 percent part segmentation accuracy and the full pipeline achieved 85.0 percent grasp success. The model delineated handles and grips under sensor noise and occlusion, which is exactly the messy condition where a benchmark trained model often falls apart. A single camera setup that still clears 85 percent grasp success is a practical result, not just a demo, because it lowers the cost and calibration burden of deploying the system.

Where it still falls short

The honest caveats start with the nature of the evidence. This is a preprint with a public project page but not yet peer review, so the results should be read as strong early signal rather than settled fact. The 20 object grasping evaluation is a real world test, but it is small, and the objects were chosen for clear functional parts, which is a friendlier setting than a cluttered drawer of ambiguous shapes.

There is also a dependency worth naming. The whole LangPart-1M pipeline leans on large vision language models, GPT-5.2-Pro specifically, to name parts, judge image quality, and merge regions across views. That is what made the scale possible, but it also means the dataset inherits whatever biases and blind spots those models carry, and the range of parts the pipeline can label is bounded by what the language model thinks a given object has. A part that the model does not think to name will not appear in the labels.

And the task itself is deliberately scoped. UniPart segments parts of isolated objects. It does not reason about a cluttered scene, about which object to act on among many, or about the sequence of actions a task requires. The authors are explicit that extending part understanding beyond isolated objects and closing the loop with robotic control is future work. UniPart is a strong perception primitive, not a full manipulation stack.

Why the approach travels

Strip away the robotics framing and UniPart carries two portable lessons. The first is that for open vocabulary tasks, the dataset is often the real model. A tractable network trained on 8 million well aligned examples beat clever architectures starved of data, and the controlled study proves the point rather than asserting it. The second is that conditioning a network on language can be as simple as adding an embedding everywhere, provided the language signal is compact and you keep it present at every layer.

The broader current is toward grounding language in 3D geometry for embodied agents, a theme that shows up across recent work on turning 3D scenes into something a language model can point at and on zero shot recognition that generalizes past a fixed label set. UniPart’s specific contribution is to push that grounding down to the part level, where manipulation actually happens, and to supply the dataset that makes the part level learnable at all.

Reference implementation in PyTorch

The code below is a runnable reconstruction of the UniPart idea based on the paper’s description, a point cloud tokenizer with additive text injection and a per point mask head. A stand in text encoder replaces the frozen CLIP backbone so the file runs without downloads. It includes Farthest Point Sampling, a lightweight neighborhood encoder, a Transformer with layer wise additive text conditioning, a per point decoder producing a binary mask, the binary cross entropy segmentation loss and the alignment objective, a training step, and a smoke test on dummy tensors. Swap in real CLIP and a dataset for actual experiments.

# unipart_reference.py
# Runnable reconstruction of the UniPart language grounded part segmenter.
# Replace TextEncoderStub with a frozen CLIP text encoder for real runs.

import torch
import torch.nn as nn
import torch.nn.functional as F


def farthest_point_sample(xyz, n_seed):
    """Pick n_seed points spread across the cloud. xyz is (B, N, 3)."""
    B, N, _ = xyz.shape
    idx = torch.zeros(B, n_seed, dtype=torch.long, device=xyz.device)
    dist = torch.full((B, N), 1e10, device=xyz.device)
    far = torch.zeros(B, dtype=torch.long, device=xyz.device)
    for i in range(n_seed):
        idx[:, i] = far
        centroid = xyz[torch.arange(B), far].unsqueeze(1)
        d = ((xyz - centroid) ** 2).sum(-1)
        dist = torch.minimum(dist, d)
        far = dist.argmax(-1)
    return idx


def knn_group(xyz, seed_idx, k):
    """Gather k nearest neighbors around each seed point."""
    B, N, _ = xyz.shape
    seeds = xyz[torch.arange(B).unsqueeze(1), seed_idx]      # (B, S, 3)
    d = torch.cdist(seeds, xyz)                                # (B, S, N)
    nn_idx = d.topk(k, largest=False).indices             # (B, S, k)
    grouped = xyz[torch.arange(B).view(B, 1, 1), nn_idx]     # (B, S, k, 3)
    return grouped - seeds.unsqueeze(2), seed_idx          # local coordinates


class NeighborhoodEncoder(nn.Module):
    """Mini PointNet style encoder for each local neighborhood."""
    def __init__(self, dim=256):
        super().__init__()
        self.mlp = nn.Sequential(nn.Linear(3, 64), nn.ReLU(inplace=True),
                                 nn.Linear(64, dim))

    def forward(self, grouped):                              # (B, S, k, 3)
        f = self.mlp(grouped)
        return f.max(dim=2).values                          # (B, S, dim) max pool


class TextEncoderStub(nn.Module):
    """Stand in for a frozen CLIP text encoder. One global vector."""
    def __init__(self, vocab=512, dim=256):
        super().__init__()
        self.embed = nn.Embedding(vocab, dim)
        for p in self.parameters():
            p.requires_grad = False                          # frozen

    def forward(self, token_ids):                           # (B, L)
        return self.embed(token_ids).mean(1)               # (B, dim) global embedding


class AdditiveBlock(nn.Module):
    """Transformer block that adds the text vector to every token."""
    def __init__(self, dim=256, heads=4):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm1 = nn.LayerNorm(dim)
        self.norm2 = nn.LayerNorm(dim)
        self.ff = nn.Sequential(nn.Linear(dim, 4 * dim), nn.GELU(),
                                nn.Linear(4 * dim, dim))

    def forward(self, x, t):
        x = x + t.unsqueeze(1)                              # additive text injection
        a, _ = self.attn(self.norm1(x), self.norm1(x), self.norm1(x))
        x = x + a
        return x + self.ff(self.norm2(x))


class UniPart(nn.Module):
    def __init__(self, dim=256, layers=6, n_seed=256, k=16):
        super().__init__()
        self.n_seed, self.k = n_seed, k
        self.enc = NeighborhoodEncoder(dim)
        self.text = TextEncoderStub(dim=dim)
        self.blocks = nn.ModuleList([AdditiveBlock(dim) for _ in range(layers)])
        self.point_head = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(inplace=True),
                                        nn.Linear(dim, 1))

    def forward(self, xyz, token_ids):
        seed_idx = farthest_point_sample(xyz, self.n_seed)
        grouped, seed_idx = knn_group(xyz, seed_idx, self.k)
        tok = self.enc(grouped)                               # (B, S, dim)
        t = self.text(token_ids)                              # (B, dim)
        for blk in self.blocks:
            tok = blk(tok, t)
        seed_logit = self.point_head(tok).squeeze(-1)        # (B, S)
        # propagate seed logits back to every input point by nearest seed
        seeds = xyz[torch.arange(xyz.shape[0]).unsqueeze(1), seed_idx]
        d = torch.cdist(xyz, seeds)                           # (B, N, S)
        nearest = d.argmin(-1)                              # (B, N)
        logits = torch.gather(seed_logit, 1, nearest)
        return torch.sigmoid(logits)                          # per point mask


def seg_loss(pred, gt):
    return F.binary_cross_entropy(pred, gt)


def align_loss(z, c):
    """Point to image alignment, cosine similarity hinge."""
    z = F.normalize(z, dim=-1)
    c = F.normalize(c, dim=-1)
    return torch.clamp(1 - (z * c).sum(-1).mean(), min=0.0)


if __name__ == "__main__":
    model = UniPart()
    trainable = [p for p in model.parameters() if p.requires_grad]
    opt = torch.optim.AdamW(trainable, lr=1e-4)
    B, N = 2, 2048
    xyz = torch.rand(B, N, 3)
    tokens = torch.randint(0, 512, (B, 8))              # "handle of a drill"
    gt = (torch.rand(B, N) > 0.7).float()             # target part points
    for step in range(3):
        pred = model(xyz, tokens)
        loss = seg_loss(pred, gt)
        opt.zero_grad(); loss.backward(); opt.step()
        print("step", step, "loss", round(loss.item(), 4))
    print("mask mean", round(pred.mean().item(), 4))

Conclusion

The core achievement of UniPart is to make a 3D part something you can select by name, on an object the model has never seen, fast enough for a robot to act on. On Objaverse-General it reached 45.23 mean intersection over union on unseen categories against a prior best of 27.41, and on a real robot it converted that perception into 85 percent grasp success from a single camera. Those two results, one on a benchmark and one on hardware, are the case that language grounded part segmentation has crossed from idea into working tool.

The conceptual shift worth remembering is that the dataset was the harder and larger half of the contribution, and the team proved it rather than assuming it. For years the reflex in this space was to design a cleverer network. UniPart’s controlled study shows a plain feed forward Transformer on 8 million well aligned text to part pairs beating more intricate designs, and it shows the same network gaining more from better data than from a better architecture. That is a useful corrective for a field that tends to credit the model.

The design also travels. The idea of conditioning a network on language by simply adding a compact embedding at every layer is not specific to point clouds, and the recipe of pretraining for alignment, then training broad on noisy scale, then sharpening on a small clean set, is a general curriculum for any task where clean labels are scarce and messy labels are cheap. The clean transfer across ShapeNetPart, PartNet-E, and rotated poses suggests the approach learned parts rather than benchmark quirks.

The honest limitations keep it grounded. This is a preprint, the real robot evaluation is small and used objects with clear parts, the dataset inherits the biases of the large language models that built it, and the task stops at isolated objects rather than cluttered scenes or multi step tasks. None of that undoes the result. It marks the road from a perception primitive toward a full manipulation system, which the authors name as the work ahead.

For anyone building embodied systems, the practical message is compact. Give your robot part level perception, not just object level, because that is where manipulation actually happens. Invest in the dataset at least as much as the model, because on open vocabulary tasks the data is often the model. And keep the conditioning simple, since a compact language signal added everywhere can outperform a heavier fusion scheme. UniPart is public on its project page, and the reference above is a place to start testing the idea on parts of your own.

Frequently asked questions

What is language grounded 3D part segmentation?

It is the task of taking a 3D point cloud and a free form language phrase, then returning a mask over the points that match the phrase. Instead of predicting from a fixed list of part labels, the model responds to whatever a person names, such as the handle of a drill, which makes it useful for robots acting on natural instructions.

What makes UniPart zero shot?

UniPart can segment parts of object categories and part types it never saw during training. It achieves this by conditioning on a frozen CLIP text embedding rather than a closed vocabulary, so a new part word maps into the same language space the model already understands. On the benchmark it scored 45.23 mean intersection over union on unseen categories.

What is LangPart-1M?

LangPart-1M is the dataset the team built to train UniPart. It contains more than 160,000 objects across over 250 categories, more than 1,000 part types, and about 8 million text to part pairs. It was generated by a multi view pipeline that segments rendered views and merges part labels consistently across viewpoints, with a manually verified subset for high fidelity supervision.

How does UniPart fuse language and geometry?

It uses simple additive injection. The frozen CLIP text embedding is added to every point token at every layer of the Transformer. In an ablation this plain addition beat cross attention, multiplication, and concatenation, which the authors attribute to the language input being short enough that one embedding carries its meaning and repeated addition keeps the conditioning strong throughout the network.

Does UniPart work on a real robot?

Yes. On 20 real objects named by language, using a single RealSense camera and off the shelf grasp generators, UniPart reached 90.5 percent part segmentation accuracy and the full pipeline achieved 85.0 percent grasp success. It ran in about 0.4 seconds per object, far faster than methods that query foundation models repeatedly at inference.

What are the main limitations?

The work is an unreviewed preprint, the real robot test is small and used objects with clear functional parts, and the dataset pipeline depends on large vision language models that impose their own biases on which parts get labeled. The task is also scoped to isolated objects rather than cluttered scenes or multi step manipulation, which the authors name as future work.