Text4Seg++ Turns Image Segmentation Into Text Generation

Why teaching a language model to draw a mask is harder than it sounds

Multimodal language models have gotten remarkably good at looking at a picture and talking about it, answering questions, following instructions, even reasoning about implied relationships between objects. Segmentation, the task of marking exactly which pixels belong to which object, has resisted that same generative fluency. A caption is forgiving, close enough usually reads as correct. A segmentation mask is not forgiving in the same way. A boundary that drifts by a few pixels is a visibly wrong answer, and language models were never built with a native vocabulary for expressing pixel precise geometry.

The dominant workaround in recent years has been to bolt a specialist decoder onto the language model. The language model reads the prompt and the image, decides which object is being referred to, and emits a special token that acts as a cue. That cue gets handed off to a separate mask decoder, often built around Meta’s Segment Anything model, which does the actual pixel level work. The Text4Seg++ authors call this the embeddings as mask paradigm, and it is genuinely effective. It is also, in their telling, an admission of defeat for the language model itself. The heavy lifting still happens in a bolted on module, the training pipeline gets more complicated because now there are two loss functions to balance, and scaling the system up means carefully rebuilding the connection between two different architectures every time.

A second line of work tried to keep everything inside the language model by describing a mask as a sequence of polygon coordinates, essentially asking the model to write down a list of numbers tracing an outline. This stays purely generative, which is conceptually appealing, but the paper points out the practical problem. Language models are trained to predict discrete tokens from a vocabulary, not to regress continuous geometric shapes, and in practice they struggle to keep numeric coordinate sequences aligned with the actual boundary of an object. The result tends to be coarse or unstable masks, prompting some systems in this family to quietly reintroduce dedicated segmentation modules anyway, which defeats the original purpose.

From patches to sentences, how image wise semantic descriptors work

The starting idea borrows directly from how Vision Transformers already look at images. A Vision Transformer splits a photo into a grid of fixed size patches before processing it, and Text4Seg++ reuses that exact grid. Instead of stopping at a numeric label per patch the way a conventional segmentation map would, the authors assign each patch a piece of text. That text can be as short as a single word like sky or sand, a short phrase like brown dog, or, for genuinely complex scenes, a longer description like a brown dog on the left side. Flatten that grid into a single sequence and you get what the paper calls an image wise semantic descriptor, a purely textual stand in for a segmentation mask that slots directly into the same auto regressive training objective a language model already uses for ordinary text.

What this buys, and what it costs

The appeal is fairly direct. Because the representation is just text, no architectural surgery is required, the model can be fine tuned with the same infrastructure used for any other instruction following task, and a large vocabulary of semantic words comes for free instead of being restricted to a fixed label set. The cost shows up in sequence length. A 16 by 16 grid of patches already produces 256 tokens worth of labels, and once descriptions get longer than a single word, the sequence balloons further. The paper reports that on the RefCOCO dataset, the average token length of this full descriptor format reaches 583 tokens, which took roughly 19 seconds per query on an NVIDIA V100 GPU, mostly because generating text token by token is inherently sequential. A second, related problem is that in a typical photo the background occupies far more area than the object someone actually asked about, so a large share of that expensive sequence is spent repeating a background label like others over and over.

Row wise compression that respects the grid

The obvious fix for a long sequence of repeated tokens is compression, and the authors first tried the textbook approach, run length encoding applied across the entire flattened sequence. It backfired. Collapsing repeats globally ignores the fact that the descriptor grid is really a two dimensional layout, and the paper reports this naive approach cost nearly 4 points of cIoU on the RefCOCO validation split, evidence that the model was relying on the row by row spatial pattern of the uncompressed sequence more than expected.

Their working fix, row wise run length encoding, only compresses repeated labels within a single row of the grid, with each row kept separate by a newline character. That single constraint, respecting row boundaries, turned out to matter enormously. Token length on RefCOCO dropped from 583 to 154, a 74 percent reduction, and inference sped up by an average of 3 times, while the paper reports essentially no drop in segmentation accuracy compared to the uncompressed version. It is a small architectural decision with an outsized payoff, and one of the more convincing pieces of evidence in the paper that respecting an image’s actual 2D structure, rather than treating it as one long string, is what makes text based masks viable at all.

Box wise semantic descriptors and the next brick prediction idea

Image wise descriptors work well for dense, everyday semantic segmentation, but the authors identify three specific limits. Long text descriptions inflate sequence length and cap how finely a mask can be resolved. Background tokens dominate the sequence whenever the object of interest is small relative to the scene. And the whole approach assumes every patch has an explicit, predefined label available, which breaks down for reasoning driven requests where the model has to infer which object is meant without being told its category outright.

Assign and reference, naming objects the model was never told about

The refined version, Text4Seg++, restructures the task as two steps performed inside one generated response, first find the region, then describe its mask. Each detected instance gets wrapped in a compact structured format built from three parts. A reference tag names the object in natural language, either an explicit label like black dog or, for reasoning tasks where identity is implied rather than stated, an abstract placeholder like roi0 that the model assigns to an object earlier in its own response and then reuses as a tag later. A bounding box, quantized into a coarse 64 level grid, narrows attention to the relevant region of the image before any mask detail gets generated at all. And a semantic descriptor, now scoped only to that bounding box rather than the whole image, fills in the actual mask.

The practical benefit of narrowing the descriptor to a bounding box first is that background token dominance mostly disappears, since the descriptor no longer has to describe the entire scene, only the region the box already isolated. Quantitatively the paper shows this single change, moving from image wise to box wise descriptors at matched 32 by 32 resolution, adds 3.8 points of cIoU on RefCOCO, which the authors attribute directly to how much cleaner a signal the model gets once background noise is excluded from the sequence it has to generate.

Semantic bricks, a vocabulary built for one job

Compression still had more room to give. The authors noticed that a compressed descriptor string like others times 16 actually gets split by a standard tokenizer into four separate pieces, the word others, an asterisk, a 1 and a 6, which is wasteful given how repetitive these strings are. Their answer is to add 126 new tokens directly to the model’s vocabulary, 63 representing foreground segments of different lengths and 63 representing background segments, which the paper nicknames semantic bricks. A mask becomes a sequence of these bricks predicted one at a time, arranged left to right and top to bottom the way you would read a page, a process the authors call next brick prediction. The bricks make almost no difference to raw accuracy on their own, but they compress the sequence enough that scaling the descriptor resolution up to a much finer 64 by 64 grid becomes computationally practical, and that resolution increase is where the real accuracy gains show up, adding another 1.0 to 1.4 points of cIoU on top of the box wise gains already described.

What the benchmarks actually show

The paper’s experimental section is unusually broad, covering referring expression segmentation, generalized referring segmentation with multiple or absent targets, reasoning segmentation, open vocabulary segmentation, remote sensing segmentation, and plain object localization, all with the same trained model and no task specific architecture changes.

Two efficiency comparisons stand out beyond raw accuracy. Against a controlled polygon coordinate baseline trained on the exact same Qwen2-VL backbone for the same number of steps, the text as mask approach wins by 4.5 points of cIoU on RefCOCO while using roughly 150 tokens per mask against roughly 330 for polygons, which the authors credit to the descriptor format turning segmentation into something closer to a classification task, a better match for how a language model already predicts its next token. Against the earlier discriminative approaches that rely on a bolted on mask decoder, the strongest of those, GLaMM, reaches 75.6 average cIoU on the RefCOCO family, still short of Text4Seg++’s 79.3, despite carrying the extra architectural complexity of a separate decoder and loss function.

Where the two versions of the model actually differ

An easy assumption would be that the newer Text4Seg++ simply replaces the original Text4Seg everywhere, but the paper’s own numbers complicate that story in an interesting way. On the harder grefCOCO benchmark, once Text4Seg is equipped with a mask refiner and fine tuned specifically on that one benchmark, it edges out Text4Seg++, reaching 71.1 average score at the 7B scale and 71.5 at the 13B scale, both ahead of Text4Seg++’s 69.8 and 69.5 respectively. Text4Seg++ is built as a single generalist model trained once across many tasks and domains without per benchmark fine tuning, while these particular Text4Seg numbers reflect a model narrowly optimized for one benchmark. That is a reasonable trade to make depending on what a practitioner actually needs, a generalist model that handles referring expressions, reasoning tasks and remote sensing imagery with one set of weights, or a specialist model squeezed for maximum score on a single leaderboard.

What still doesn’t quite work

The paper is candid about at least two open gaps. On ReasonSeg, Text4Seg++ slightly trails a competing method called Seg-Zero, which the authors attribute to Seg-Zero’s dedicated reasoning chain mechanism built specifically for that style of task, a reminder that a general purpose text based approach will not automatically beat a narrowly specialized architecture on every single benchmark. On the RRSIS-D remote sensing benchmark, Text4Seg++ also trails the specialized SegEarth-R1 model, which the authors attribute to SegEarth-R1’s domain specific modules and its practice of training separately on that one dataset, versus Text4Seg++’s choice to train one unified model across both natural and aerial imagery at once.

There is also a documented trade off around general conversational ability. Specializing a multimodal model for segmentation through fine tuning introduces a distribution shift that can hurt performance on general visual question answering benchmarks like MMBench, though the authors report that mixing segmentation data with general instruction data during training substantially mitigates this, and even boosts performance on spatial reasoning benchmarks like GQA beyond what the base Qwen2-VL model achieves on its own.

Why the underlying idea travels beyond segmentation

Strip away the segmentation specific details and the paper is making a broader argument about how to hand a genuinely new modality to a language model without touching its architecture. Rather than inventing a new decoder, a new loss function, or a new attention mechanism to handle geometric output, the authors ask what happens if you simply find the right textual encoding for the thing you want the model to produce, one that respects the actual structure of the data, patches arranged in rows and columns, rather than flattening it naively. The row wise compression insight in particular, that global compression breaks spatial structure while row scoped compression preserves it, is a specific, transferable lesson about representing structured, grid shaped data as plain text without losing the properties that make it structured in the first place.

That lesson plausibly extends past image segmentation to other dense, grid shaped prediction problems that people have historically reached for specialized architectures to solve, depth estimation, optical flow, or any task where the natural output is a 2D field of values rather than a caption. If a carefully designed textual encoding can get a language model to next token predict its way to a competitive segmentation mask, the same recipe, patch aligned tokens, a vocabulary tuned for the specific output type, and compression that respects the grid rather than the raw sequence, seems worth testing on those neighboring problems too.

Complete PyTorch implementation

The following code is an original, simplified educational implementation inspired by the mechanisms described in the paper, the semantic descriptor grid, row wise run length encoding, and a toy next brick style autoregressive decoder. It is not the authors’ released code, since no public repository was cited in the version of the paper reviewed here, and it runs on small dummy tensors as a sanity check of the logic.

# text4seg_lite.py
# Educational reimplementation of the core Text4Seg++ mechanisms.
# Not the authors' official code. Runs a smoke test on random dummy data.

import torch
import torch.nn as nn
import torch.nn.functional as F


def mask_to_descriptor_grid(binary_mask, grid_size=16):
    """Downsamples a binary mask to a grid_size by grid_size label grid,
    the discrete analogue of an image-wise semantic descriptor before
    it gets turned into text tokens."""
    mask = binary_mask.unsqueeze(0).unsqueeze(0).float()
    pooled = F.adaptive_avg_pool2d(mask, (grid_size, grid_size))
    labels = (pooled.squeeze() > 0.5).long()
    return labels


def row_wise_rle_encode(label_row):
    """Row wise Run Length Encoding, R-RLE. Compresses one row of a label
    grid into (label, count) pairs, matching the paper's choice to
    compress within a row rather than across the whole flattened grid."""
    encoded = []
    prev = None
    count = 0
    for value in label_row.tolist():
        if value == prev:
            count += 1
        else:
            if prev is not None:
                encoded.append((prev, count))
            prev = value
            count = 1
    encoded.append((prev, count))
    return encoded


def row_wise_rle_decode(encoded, row_length):
    """Inverse of row_wise_rle_encode, expands (label, count) pairs back
    into a full row of per patch labels."""
    row = []
    for label, count in encoded:
        row.extend([label] * count)
    assert len(row) == row_length, "decoded row length mismatch"
    return torch.tensor(row)


def grid_to_rrle_sequence(label_grid):
    """Applies row wise RLE to every row of a grid, joining rows the way
    the paper separates them with a newline marker, here just a
    sentinel value of -1 between rows."""
    sequence = []
    for row_idx in range(label_grid.shape[0]):
        encoded = row_wise_rle_encode(label_grid[row_idx])
        for label, count in encoded:
            sequence.append(label)
            sequence.append(count)
        sequence.append(-1)
    return sequence


class NextBrickDecoder(nn.Module):
    """A drastically simplified stand in for a Qwen2-VL style backbone
    fine tuned with LoRA, just enough structure to demonstrate the
    next brick prediction training objective end to end. Predicts the
    next token in a semantic brick sequence given image patch features
    and the tokens generated so far."""

    def __init__(self, vocab_size=130, dim=128, num_heads=4, num_layers=2):
        super().__init__()
        self.token_embed = nn.Embedding(vocab_size, dim)
        self.patch_proj = nn.Linear(3, dim)
        decoder_layer = nn.TransformerDecoderLayer(
            d_model=dim, nhead=num_heads, batch_first=True
        )
        self.decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
        self.head = nn.Linear(dim, vocab_size)

    def forward(self, patch_features, token_ids):
        memory = self.patch_proj(patch_features)
        tgt = self.token_embed(token_ids)
        seq_len = tgt.shape[1]
        causal_mask = torch.triu(torch.ones(seq_len, seq_len) * float("-inf"), diagonal=1)
        decoded = self.decoder(tgt, memory, tgt_mask=causal_mask)
        logits = self.head(decoded)
        return logits


def next_brick_loss(model, patch_features, token_ids):
    """Standard next token cross entropy, the training objective behind
    next brick prediction. Predicts token[t+1] from tokens[0..t] and
    the image patch features."""
    logits = model(patch_features, token_ids[:, :-1])
    targets = token_ids[:, 1:]
    loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]), targets.reshape(-1))
    return loss


def train_step(model, optimizer, patch_features, token_ids):
    optimizer.zero_grad()
    loss = next_brick_loss(model, patch_features, token_ids)
    loss.backward()
    optimizer.step()
    return loss.item()


def mask_iou(pred_mask, true_mask):
    """Simple IoU between two binary masks, used as the smoke test
    evaluation metric standing in for cIoU in the paper."""
    intersection = (pred_mask & true_mask).sum().float()
    union = (pred_mask | true_mask).sum().float()
    return (intersection / union.clamp(min=1.0)).item()


def make_dummy_batch(batch_size=2, num_patches=16, seq_len=20, vocab_size=130):
    patch_features = torch.randn(batch_size, num_patches, 3)
    token_ids = torch.randint(0, vocab_size, (batch_size, seq_len))
    return patch_features, token_ids


if __name__ == "__main__":
    torch.manual_seed(0)

    # part one, row wise run length encoding sanity check
    dummy_mask = torch.zeros(32, 32)
    dummy_mask[10:20, 10:20] = 1.0
    grid = mask_to_descriptor_grid(dummy_mask, grid_size=8)
    print("label grid")
    print(grid)

    rrle_sequence = grid_to_rrle_sequence(grid)
    print("R-RLE sequence length", len(rrle_sequence))
    print("raw flattened length would have been", grid.numel())

    # round trip check on a single row
    row = grid[3]
    encoded_row = row_wise_rle_encode(row)
    decoded_row = row_wise_rle_decode(encoded_row, row_length=len(row))
    print("round trip matches original row", torch.equal(decoded_row, row))

    # part two, next brick prediction training smoke test
    model = NextBrickDecoder(vocab_size=130, dim=64, num_heads=4, num_layers=2)
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

    losses = []
    for step in range(5):
        patch_features, token_ids = make_dummy_batch()
        loss_value = train_step(model, optimizer, patch_features, token_ids)
        losses.append(loss_value)
    print("next brick prediction losses over 5 steps", losses)

    # part three, IoU smoke test between two random dummy masks
    pred = torch.randint(0, 2, (16, 16)).bool()
    truth = torch.randint(0, 2, (16, 16)).bool()
    print("dummy mask IoU", mask_iou(pred, truth))

Running that file prints the downsampled label grid, confirms the row wise encoding both shrinks the sequence and round trips back to the exact original row, prints five decreasing loss values from the toy next brick decoder, and finally reports an IoU score on two random masks, confirming the shapes and gradients flow correctly end to end.

Conclusion

The core achievement here is showing that a language model does not need a bolted on decoder or a shaky coordinate regression trick to produce a usable segmentation mask, it can simply write one out as words, provided the textual encoding respects the actual two dimensional structure of the thing being described. Text4Seg established that a patch aligned grid of semantic labels works well enough for dense segmentation, and Text4Seg++ shows that adding a bounding box and a compact brick vocabulary sharpens that same idea into something that also handles reasoning driven and multi object cases without losing the original approach’s simplicity.

The conceptual shift worth carrying forward is narrower than it first appears, and that is exactly what makes it durable. This is not a claim that language models are secretly good at geometry. It is a claim that geometry can often be re expressed as a carefully structured string, one that a language model’s existing next token prediction machinery can handle without modification, as long as the encoding respects the data’s real shape rather than flattening it carelessly. The row wise compression result, where naive global compression actively hurt performance while row scoped compression did not, is the clearest piece of evidence for that principle in the entire paper.

Whether this generalizes to other dense prediction problems, depth maps, flow fields, or other grid shaped outputs that currently require specialized architectures, remains an open question the authors gesture toward rather than answer directly. It is a reasonable bet given how cleanly the same recipe, patch alignment, a purpose built vocabulary, and grid respecting compression, transferred from a 16 by 16 image wise format to a much finer 64 by 64 box wise format within this one paper alone.

The honest limitations matter just as much as the headline numbers. A generalist Text4Seg++ model trained once across many tasks does not automatically beat every specialized, single benchmark competitor, and the paper is transparent about the specific cases, ReasonSeg against Seg-Zero, RRSIS-D against SegEarth-R1, where a narrower architecture still wins by a small margin. That is a healthy reminder that generality and peak single benchmark performance are still, at least for now, two different goals that occasionally pull in different directions.

For anyone building on this work, the practical takeaway is encouraging. Because Text4Seg++ requires no architectural surgery to the underlying multimodal model, it is a genuinely low friction way to add segmentation capability to an existing instruction tuned vision language model, and the same trained weights already generalize across referring expressions, reasoning queries, and even a completely different imaging domain like satellite photography. That combination of simplicity and reach is a rarer thing in this field than another few points on a benchmark table.

Frequently asked questions

Related reading