How Chat-Scene++ Turns 3D Scenes Into Object Sequences

Why 3D assistants keep tripping over words

Large language models learned to describe photographs fairly well, but a photograph is flat and a room is not. Moving from 2D captions to real 3D understanding means dealing with occlusion, depth, and the way objects relate to one another in space. A model has to know not just that there is a chair, but which chair, where it sits, and how it stands relative to the table beside it. Most earlier systems handled this by building narrow experts. One network for grounding, meaning localizing an object a user described. Another for captioning. Another for question answering. Each carried its own task specific head and none of them talked to a user in plain language.

The newer wave of 3D multimodal language models tried to unify all of that under one conversational interface. The trouble is that language is a clumsy way to point. When the answer to a grounding request is a bounding box, the model has to emit a set of coordinates, and predicting precise numbers as text is something language models are famously bad at. When the reference is a phrase like the chair at the southwest corner of the rightmost table, the description is long, subjective, and easy to misread. Earlier 3D language models such as 3D-LLM leaned on location tokens to name positions, and they still trailed the specialized expert models on grounding accuracy.

Chat-Scene++, from Haifeng Huang and Yilun Chen with colleagues at Zhejiang University and Shanghai AI Laboratory, comes at the problem from a different direction. Instead of teaching the model to speak in coordinates, it changes what the model is looking at.

A room as a numbered list

The core move is to stop treating a scene as an undifferentiated point cloud and start treating it as a sequence of objects. A 3D encoder first proposes the objects in the room. Each proposal gets a unique identifier drawn from a small set of learnable tokens, written in the paper as things like OBJ013 or OBJ032. From then on the model refers to that chair as object 13, full stop. There is no rightmost, no southwest corner, no room for the viewpoint to scramble the meaning.

This does more than tidy up the language. It gives the model a stable handle it can use in both directions. A user can name an object by its ID, and the assistant can ground its own answers by emitting the same ID. A counting question that used to end in a bare answer of four can now say four and list exactly which four objects it counted. The reference and the reasoning share one vocabulary.

There is a nice detail in how the identifiers are trained. The numbering itself carries no meaning, and the authors deliberately shuffle the object order during training so the model cannot lean on the sequence. At inference the objects can be numbered in any order with negligible effect. The ID is a name, not a rank.

What goes inside each object token

An identifier tells the model which object you mean. It says nothing about what the object looks like. So each object in the sequence carries a rich feature vector, and this is where Chat-Scene++ departs most sharply from its predecessor Chat-Scene. The earlier system ran an independent detector and then a separate per object encoder, which meant every object was described in isolation, blind to its neighbors, with a lot of redundant computation. Chat-Scene++ instead pulls context rich features from two large pre trained models and fuses them.

The 3D half uses a scene level transformer encoder built on Mask3D and pre trained with a contrastive language and scene objective called CLASP. Because the object queries attend across the whole scene, the resulting 3D feature already knows about the objects around it. That captures spatial relationships, geometry, and shape. In an honest ablation the authors show this scene aware feature beats the object level Uni3D encoder used in the original Chat-Scene, and that both beat using raw Mask3D detection features, which mostly encode category rather than rich semantics.

The 2D half adds what 3D point clouds struggle to convey, namely texture, color, and fine appearance. Chat-Scene++ projects each object’s points onto multi view images and reads local patch features from the pre trained DINOv2 encoder. Depth annotations decide visibility, so points hidden in a given view are simply left out of that view’s features. For a single image the object feature is the average of its masked patch features.

Features from every view are then combined, weighted by how much of the object each view actually saw, so a clear close view counts for more than a distant sliver.

The paper is careful about why DINOv2 rather than CLIP fills the 2D slot. CLIP was trained to match whole images to captions, which pushes it toward high level semantics and away from pixel detail. DINOv2 was trained with self supervision at both the image and patch level, so it keeps the local shape and texture cues that make one chair distinguishable from another. In the ablation DINOv2 wins.

Two small projectors map the 3D feature and the 2D feature into the language model’s embedding space, one for each modality.

The scene the language model finally reads is built object by object. For each one it sees the identifier embedding followed by the fused 3D and 2D object features, in a strict one to one order, and that ordered stream stands in for the entire room.

The point of an object identifier is that a language model can finally say exactly which thing it means, and prove it, rather than describe a thing and hope you agree. A plain reading of the Chat-Scene++ design

The token budget nobody talks about

Here is where the design earns its keep in practice. A real indoor scan can hold a hundred objects or more, and every token you spend describing one object multiplies across the room. The naive way to write an ID as plain text, say Obj001, splits into four separate tokens. Add the two feature tokens for 3D and 2D and a single object costs six tokens. In a scene with a hundred objects that is six hundred tokens before the user has even asked anything, and it grows the compute and latency accordingly.

By adding each identifier as one new learnable token in the vocabulary instead of spelling it out, Chat-Scene++ drops the per object cost from six tokens to three. The paper reports this as lowering the budget from 6N to 3N for N objects, and it also finds that these learnable tokens perform better than fixed random embeddings, not just cheaper. Halving the sequence length is the difference between a scene with hundreds of objects being practical or not. The scaling study backs this up. As the number of object proposals climbs from 150 toward 1000, inference time rises roughly linearly with the fixed three token budget, while peak memory stays nearly flat.

Reasoning that points at things

Once a scene is a list of nameable objects, a more interesting door opens. The model can reason in steps and tie each step to an object. The authors call this grounded chain of thought, or G-CoT. Ordinary chain of thought prompting has a model narrate its reasoning in text before answering. Grounded chain of thought does the same but anchors the intermediate steps to specific object IDs in the scene.

The reason this matters shows up in the training data. Benchmarks like ScanQA and ScanRefer usually supply only a terse answer, a single word or a bounding box, with none of the middle reasoning. Ask how many chairs are in a room and the label is four, which teaches the model nothing about how to find them. G-CoT fills that gap by generating structured multi step rationales from the existing annotations. For question answering the model first predicts the objects relevant to the question, then infers the answer from them. For grounding it can reason at the category level, first naming the target’s category, then listing all objects of that category, then locating the target.

The ablation offers a useful nuance. For grounding, category level reasoning beats space level reasoning, which defines the target’s neighbors as the five closest objects. Spatial proximity is not always the right context, because a description like the farthest lamp is precisely about something not nearby. Worth stressing, these templates exist only to build the training data. At inference the model is given a single generic prompt asking it to think step by step, not a menu of hand crafted templates, which keeps the method honest and general.

How the whole thing is trained

Every task, whether grounding, captioning, or question answering, is folded into one instruction following format of single turn user and assistant exchanges. Because of that unification there is exactly one training loss, the standard cross entropy of the language model over the target response.

The trainable parameters cover the two projectors, the new identifier token embeddings, and the language model itself, here a Vicuna-7B backbone. The team also uses a single stage of joint training rather than the common two stage recipe that first aligns a projector and then fine tunes. They report that skipping the separate alignment phase not only simplifies the pipeline but performs at least as well, which removes one of the more finicky choices in building these systems. Training draws on roughly 329 thousand samples assembled from the five benchmarks plus extra scene and object caption data.

What the numbers show

The headline is that a single unified model, carrying no task specific heads and receiving no per task fine tuning, tops five different 3D scene language benchmarks at once. That breadth is the point. A trick that only lifted one benchmark would be suspect, but improvements across grounding, captioning, and question answering suggest the object sequence representation is doing something real.

The grounding gains are the most telling, since grounding is exactly where earlier 3D language models fell short of expert systems. A 7.1 point jump in F1 on multi object grounding and a 3.2 point rise in strict accuracy on ScanRefer say the identifier scheme addresses the referring problem head on. The captioning and question answering gains show the same representation carries over to tasks that do not even require grounding, which is what you want from a general interface rather than a point solution.

The generalization tests are worth a look too. Trained only on the indoor ScanNet data, the model was evaluated zero shot on the 3RScan dataset, where it beat the prior LEO approach across all three of its tasks without any training in that domain. Pushed further out of distribution to the outdoor nuScenes-QA driving benchmark, it reached 39.8 percent overall accuracy zero shot. The authors are candid that methods actually fine tuned on driving data score higher, and that the indoor to outdoor gap is large, but a model built on indoor rooms transferring at all to street scenes is a reasonable signal that the representation is not memorizing one dataset.

Running without a 3D scan at all

Reconstructing a 3D point cloud from RGB-D images is slow and not always available. A phone captures video far more easily than it captures clean depth. So the team tested whether Chat-Scene++ can work from 2D video alone. The pipeline swaps the 3D detector for a video object tracker called DEVA, which finds and follows objects across frames, then extracts features for each tracked object. Grounding is scored against masks projected onto the video frames using a spatial and temporal overlap measure they call ST-IoU.

The results are a study in honest limits. On question answering the 2D only setting comes close to the full 3D inputs, because answering a question tolerates a roughly located object as long as the right one is identified. On precise localization it lags, because the video tracker sometimes loses an object that leaves the frame, causing identity switches that hurt the boundary accuracy. The authors say plainly that better video tracking would translate directly into better 2D grounding without any change to the reasoning framework. On the VSI-Bench spatial benchmark the model does best on object counting and absolute distance, the numerical and object centric questions its structured representation is built for, while larger 2D models pre trained on far more image and video data lead on broader scores. Given that Chat-Scene++ saw only about 300 thousand 3D samples, holding its own is the interesting part.

Where it still falls short

None of this is free of caveats, and the paper names most of them. The whole system inherits the quality of its object proposals. If the upstream detector or video tracker misses an object or splits one in two, the language model has no way to reason about a thing it never received. The 2D only localization gap is a direct symptom of that dependence.

There is also a scope limit worth stating plainly. Every benchmark here lives indoors on ScanNet derived data, rooms full of furniture. The driving and 3RScan transfer results are encouraging but they are probes, not a claim that the model is ready for autonomous navigation. Grounded chain of thought, for all its appeal, depends on annotations of related objects to build its training rationales, which is why the team could only run it zero shot on SQA3D rather than train it there. And while a Vicuna-7B backbone is a sensible common choice, it also means the language reasoning is bounded by a model that is modest by current standards.

Why the object sequence idea travels

Step back and the appeal is conceptual economy. Chat-Scene++ does not invent a new architecture so much as choose a better interface between the world and the language model. Turn the scene into a list of named, richly described objects and a general purpose language model can suddenly ground, caption, count, and reason over it with one shared vocabulary and one loss. The specialized heads fall away.

That framing has legs beyond indoor rooms. Any domain where you can detect discrete entities and want a language model to reason about them precisely, from a warehouse of parts to a medical scan of distinct structures to a road full of vehicles, could borrow the same recipe of stable identifiers plus fused features plus grounded reasoning. The token budget lesson travels too. If you are feeding many entities to a language model, spending one learnable token per entity instead of spelling out a name is the kind of unglamorous decision that decides whether a system is usable at scale.

Conclusion

Chat-Scene++ is a good reminder that the hardest part of connecting language models to the physical world is often not the perception and not the reasoning, but the handshake between them. Earlier 3D assistants could see rooms and could talk, yet they stumbled on the simple act of pointing, because natural language references are ambiguous and coordinate outputs are unreliable. Giving every object a stable identifier fixes the handshake, and almost everything else follows from that.

The engineering around the idea is disciplined. Fusing a scene aware 3D feature with an appearance rich 2D feature gives each object token real substance. Adding identifiers as single learnable tokens keeps the sequence short enough to handle crowded rooms. Unifying every task into one instruction format with one loss removes the tangle of task specific heads. And grounded chain of thought turns the identifier from a convenience into a reasoning tool, letting the model show which objects its answer rests on.

The evidence is broad rather than narrow. Five benchmarks improve at once, the same model transfers zero shot to new indoor scenes and even to outdoor driving data, and it degrades gracefully to a 2D only mode that needs no 3D scan. That combination is what separates a general interface from a benchmark specialist.

The honest limits keep it grounded. Proposal quality caps performance, the evaluations live mostly indoors, and the grounded reasoning still leans on annotations that are not always available. Those are the seams a follow up should work on, and the authors say as much. For now Chat-Scene++ makes a clean case that the right way to hand a room to a language model is not as a cloud of points but as a conversation about numbered things.

As embodied agents and 3D assistants keep maturing, the interface question will only grow more central. Chat-Scene++ offers one durable answer. Name the objects, describe them well, and let the language do the rest.

The listing below reconstructs the core mechanics of Chat-Scene++, namely the multi view 2D feature aggregation, the two feature projectors, the object sequence assembly with learnable identifier tokens, and a builder for the grounded chain of thought prompt. It uses small dummy tensors so it runs anywhere, and it ends with a smoke test. Swap the stand in encoders for real Mask3D, Uni3D, and DINOv2 features to reproduce the full pipeline.

# chat_scene_pp.py
# Minimal, runnable reconstruction of the Chat-Scene++ object sequence
# builder. Encoders are stand ins so the file runs without 3D data.

import torch
import torch.nn as nn


def aggregate_2d_feature(patch_feats, masks, mask_sizes, eps=1e-6):
    """Multi view 2D feature for one object.

    patch_feats : list over V views, each [P, d] DINOv2 patch features
    masks       : list over V views, each [P] with 1 for visible patches
    mask_sizes  : [V] how much of the object each view saw
    Per image feature is the mean of masked patches. Views are then
    combined weighted by mask size, as in the paper.
    """
    per_view = []
    for f, m in zip(patch_feats, masks):
        n = m.sum().clamp(min=1.0)
        per_view.append((f * m.unsqueeze(-1)).sum(0) / n)   # [d]
    per_view = torch.stack(per_view, 0)                     # [V, d]
    w = mask_sizes.clamp(min=0).unsqueeze(-1)               # [V, 1]
    return (per_view * w).sum(0) / (w.sum() + eps)          # [d]


class Projector(nn.Module):
    """Maps an object feature into the LLM embedding space."""

    def __init__(self, in_dim, llm_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim),
        )

    def forward(self, x):
        return self.net(x)


class ChatScenePP(nn.Module):
    """Builds scene embeddings as a sequence of object tokens.

    Each object contributes three tokens, one learnable identifier
    embedding plus a 3D feature token and a 2D feature token. This is
    the 3N token budget the paper adopts, down from 6N for text IDs.
    """

    def __init__(self, dim_3d, dim_2d, llm_dim=512, max_objs=200):
        super().__init__()
        self.proj_3d = Projector(dim_3d, llm_dim)
        self.proj_2d = Projector(dim_2d, llm_dim)
        # one learnable identifier token per possible object slot
        self.obj_ids = nn.Embedding(max_objs, llm_dim)

    def forward(self, z3d, z2d, order=None):
        """z3d [n, dim_3d], z2d [n, dim_2d]. Returns [3n, llm_dim]."""
        n = z3d.shape[0]
        if order is None:
            order = torch.randperm(n)          # shuffle IDs in training
        f_p = self.proj_3d(z3d)                 # [n, llm_dim]
        f_v = self.proj_2d(z2d)                 # [n, llm_dim]
        ids = self.obj_ids(order)               # [n, llm_dim]
        # interleave: id, 3D token, 2D token, per object, in order
        seq = torch.stack([ids, f_p, f_v], dim=1)   # [n, 3, llm_dim]
        return seq.reshape(n * 3, -1), order


def build_gcot_prompt(question, target_category):
    """Category level grounded chain of thought template.

    Used only to construct supervised training rationales. At inference
    a single generic 'think step by step' prompt is used instead.
    """
    return (
        f"Question: {question}\n"
        f"Step 1, the target category is {target_category}.\n"
        f"Step 2, list all objects of that category by ID.\n"
        f"Step 3, locate the target object and return its ID."
    )


def smoke_test():
    """Runs on dummy data, no 3D scan needed."""
    torch.manual_seed(0)
    n, dim_3d, dim_2d, llm_dim = 8, 128, 96, 64

    # stand in per object 3D and 2D features
    z3d = torch.randn(n, dim_3d)
    z2d = torch.randn(n, dim_2d)

    model = ChatScenePP(dim_3d, dim_2d, llm_dim, max_objs=64)
    seq, order = model(z3d, z2d)

    assert seq.shape == (n * 3, llm_dim)   # 3 tokens per object
    assert torch.isfinite(seq).all()

    # test the 2D aggregation on two dummy views
    patch_feats = [torch.randn(16, dim_2d), torch.randn(16, dim_2d)]
    masks = [torch.randint(0, 2, (16,)).float() for _ in range(2)]
    sizes = torch.tensor([float(m.sum()) for m in masks])
    obj2d = aggregate_2d_feature(patch_feats, masks, sizes)

    print("scene tokens:", seq.shape[0], "for", n, "objects (3N budget)")
    print("2D object feature dim:", obj2d.shape[0])
    print("sample G-CoT prompt:\n",
          build_gcot_prompt("where is the lamp?", "lamp"))
    print("smoke test passed")


if __name__ == "__main__":
    smoke_test()

Frequently asked questions

You can also open the source article through its IEEE TPAMI listing, which links the official version and the supplementary material.