Diffuse2Seg: How Diffusion Models Segment Images Without Labels

The labelling problem behind segmentation

Segmenting an image means partitioning it into all of its coherent pieces, the countable objects like people and cars and the amorphous regions like sky and road, and doing it without being told in advance which classes to look for. This open ended version of the task is what lets a vision system work in the messy real world, where new kinds of objects show up all the time. Segment Anything, usually shortened to SAM, brought this idea to life and set the standard, and it did so on the back of an enormous, expensive annotation effort.

That expense is the bottleneck. Hand drawing masks for millions of images to train the next generation of these models is slow, costly, and does not scale, which is why so much recent research chases the same goal, high quality segmentation with little or no manual labelling. Our review of SAM2 and the segment anything family traces how far the supervised approach has come, and it is precisely that reliance on labels this work tries to escape.

The obvious escape route is to lean on models that have already learned about images for free. Self supervised vision models, trained without labels, are one option that prior work has mined. The team behind Diffuse2Seg bet on a different and, at the time, unexplored source, the generative diffusion models that power modern image synthesis. Their reasoning is elegant. A model that can draw a convincing photo of a street has, somewhere inside it, an understanding of where the car ends and the road begins, because it could not paint the scene otherwise. That understanding should be extractable, and Diffuse2Seg is the first method to pull it out for full open world segmentation without retraining the model at all.

Reading structure out of a diffusion model

To see how, it helps to recall what a diffusion model does. It learns to turn random noise into a realistic image, one denoising step at a time, and along the way its internal layers build a rich sense of the image’s layout. Buried in those layers is the self attention, the mechanism that lets every patch of the image decide which other patches it relates to. Patches belonging to the same object attend strongly to each other, which is exactly the grouping a segmenter needs.

Diffuse2Seg extracts this in a single pass. It encodes an image into the diffusion model’s latent space, runs just one denoising step rather than the hundreds used to generate a picture, and reads off the self attention from the highest resolution layers, blending them into a single affinity matrix that records how strongly every pair of image patches belongs together. No generation, no retraining, just a quick look under the hood.

An affinity matrix alone is not a set of masks, though. The raw attention is sparse and noisy, and simply thresholding it leaves ragged, broken regions. The clever second step treats that affinity as a graph and grows objects across it. The team scatter a regular grid of point prompts over the image, one seed per small region, and let each seed spread outward along the affinity, flowing freely across patches that belong together and stopping at the boundaries where objects meet.

The exponent in that formula is the heart of the trick. Set it to make the smoothing linear and prompts bleed across object edges into a blurry mess. Set it lower, as the team do, and the smoothing becomes edge preserving, spreading eagerly within an object but throttling sharply at its boundary, so each prompt grows into a clean, coherent soft mask. Because they run one prompt per region across the whole grid, they get a soft object map for every part of the image at once.

From soft maps to clean instances

The grid of soft maps is redundant, since neighbouring seeds often grow into the same object, and it captures objects at different scales, a whole car in one map and a wheel in another. Turning this into a tidy set of masks takes two more steps. First the team merge maps that describe the same thing, measuring how similar two maps are with a symmetric version of a standard statistical distance and clustering the similar ones together.

By cutting the clustering at several different thresholds, they keep objects at multiple granularities on purpose, coarse whole objects and their finer parts, which is what open world segmentation demands. A final cleanup removes duplicate masks and sharpens their boundaries. The output is a rich set of instance masks for a single image, produced entirely without labels, which the authors call pseudo labels because they will serve as training targets.

That last point is the second half of the method. Running the whole diffusion extraction for every image at inference time is slow, so the team use their generated masks to train a small, fast segmentation model, in two rounds. The first round learns from the raw pseudo labels, and the second round is a self improvement pass, where the model refines its own predictions on image crops to recover fine details the diffusion latent was too coarse to capture. The result is a lightweight model that segments a fresh image in one quick pass, having distilled the diffusion model’s knowledge into itself.

How well it works

The results come in three flavours, and each answers a different question. The first is whether the raw generated masks are any good.

On mask quality, the diffusion based labels beat every prior approach, including the previous best unsupervised method, which relies on self supervised vision features rather than a generative model. The margin held up across five very different datasets, from the SAM benchmark to scene parsing to densely annotated video frames to part segmentation, which is strong evidence that the diffusion features generalize rather than fitting one domain. The authors note their approach is especially good at the amorphous stuff regions like sky and road that many object focused methods neglect, and even outperforms a method trained on over a million labelled images despite using none.

The second question is whether the small model trained on those labels is any good on its own, and here the gains are larger still. Against the strongest label free baseline, the trained Diffuse2Seg model improved segmentation of objects, of objects and stuff together, and of parts by roughly seven to ten points, while using half the training images and a smaller backbone network. It even lands within a few points of the fully supervised Segment Anything on the general stuff and things task, a striking result for a model that saw no human labels.

A model trained on Diffuse2Seg labels provides a strong initialization for semi supervised learning, outperforming its fully supervised counterpart with already five thousand labelled images. Hummer and colleagues, arXiv preprint, 2026

The third question is the most practical. If you do have a modest labelling budget, does starting from Diffuse2Seg help? The team fine tuned their unsupervised model on small numbers of hand labelled images and found it remarkably data efficient. With just a hundred labelled images it recovered most of the accuracy of a model trained on a hundred thousand, and with a few thousand it began to beat fully supervised training outright. Learning the shape of objects without labels first, then adding a small dose of supervision, turns out to be a far cheaper path to a strong segmenter. This mix of a generative backbone with light fine tuning echoes the efficiency themes in our look at latent diffusion for 3D shape reconstruction.

Where the method falls short

The authors are honest about the limits, and a fair reading has to sit with them.

The clearest weakness is small objects. The diffusion model works in a compressed latent space, and fine details below a certain size simply are not represented there, so tiny objects tend to be missed or merged into their surroundings. The self improvement round and boundary sharpening help, but the coarse latent resolution is a real ceiling, and the authors flag it as the main open problem. Methods built on self supervised features, which keep a higher resolution, still have an edge on the smallest instances.

There is also a subtler issue with what the attention actually encodes. A diffusion model’s internal features capture not just semantic meaning but also texture, which sometimes leads the method to oversegment a large, textured region like the sky into several pieces that a human would call one. The authors point to borrowing a coarse to fine ranking idea from the self supervised literature as a possible fix, but it is a direction rather than a delivered solution.

Two practical caveats round it out. Running the diffusion extraction is computationally heavier than a single forward pass through a conventional network, which is exactly why the team distil it into a lightweight model rather than deploying the extraction directly. And this work is a preprint, not yet through peer review, so its numbers and claims should be read as a promising early report rather than a settled result. The code and the peer reviewed version, if and when they arrive, will be the real test of reproducibility.

Why it matters

Step back and the contribution is a new and free source of supervision. The dominant way to build a segmentation model is to label a mountain of images, and that mountain is the reason such models are expensive to make and slow to adapt to new domains. Diffuse2Seg shows that a generative diffusion model, trained for an entirely different purpose, already contains the structural knowledge a segmenter needs, and that this knowledge can be harvested without a single label. That reframes the labelling bottleneck as a modelling opportunity.

The broader significance is what it says about generative models in general. A model trained only to make pictures turns out to have learned, as a side effect, a usable map of what objects are and where they sit, which suggests these models understand far more about scene structure than their output alone reveals. As researchers keep finding perception hidden inside generation, from segmentation to depth to correspondence, the line between generative and analytical vision keeps blurring. Our coverage of that current, from reframing segmentation as text generation to distilling diffusion into a single step, keeps landing on the same lesson. The most capable vision systems of the next few years may not be trained to see at all, but to imagine, and made to see as a bonus. Diffuse2Seg is an early, concrete case of exactly that, and a reminder that the knowledge locked inside our biggest generative models is only beginning to be mined.

A reference implementation

Because the distinctive part of this work is the training free label generation rather than the standard segmentation model it later trains, the code below implements that core. It takes a self attention affinity, scatters a grid of point prompts, and grows each one into a soft object mask using the edge preserving propagation, then thresholds and merges the results, with a smoke test on a synthetic affinity. A full pipeline extracts the affinity from a real diffusion model, which this sketch stands in for. The authors work is linked under the block.

# Diffuse2Seg style training free mask generation, compact reference
# Grow a grid of point prompts across a diffusion self attention affinity
# using edge preserving propagation, then threshold and merge into masks.

import numpy as np


def propagate(affinity, seed, p=1.6, lam=1e-3, iters=200):
    # edge preserving smoothing of a seed over the affinity graph
    # p below 2 spreads within an object but stops at high contrast edges
    f = seed.copy()
    deg = affinity.sum(axis=1) + 1e-8
    for _ in range(iters):
        # per node edge energy raised to (p - 2) reweights the smoothing
        diff = f[None, :] - f[:, None]
        local = (affinity * diff ** 2).sum(axis=1) + 1e-8
        w = local ** (p / 2 - 1)                       # edge stopping weight
        num = (affinity * w[:, None] * f[None, :]).sum(axis=1) + lam * seed
        den = (affinity * w[:, None]).sum(axis=1) + lam
        f = num / den                                  # Gauss Jacobi update
    return f


def iou(a, b):
    inter = (a & b).sum()
    union = (a | b).sum() + 1e-8
    return inter / union


def generate_masks(affinity, n_nodes, n_prompts=12, thresh=0.5, dedup=0.9):
    # scatter equidistant seeds, propagate each, threshold to binary masks
    seeds = np.linspace(0, n_nodes - 1, n_prompts).astype(int)
    masks = []
    for s in seeds:
        seed = np.zeros(n_nodes)
        seed[s] = 1.0
        soft = propagate(affinity, seed)
        soft = soft / (soft.max() + 1e-8)
        m = soft > thresh
        if m.sum() > 1:
            masks.append(m)
    # remove duplicate masks by area sorted non maximum suppression
    masks.sort(key=lambda m: -m.sum())
    kept = []
    for m in masks:
        if all(iou(m, k) < dedup for k in kept):
            kept.append(m)
    return kept


if __name__ == "__main__":
    rng = np.random.default_rng(0)
    # synthetic affinity, three blocks stand in for three objects
    n = 60
    blocks = [range(0, 20), range(20, 40), range(40, 60)]
    A = np.full((n, n), 0.02)
    for b in blocks:
        idx = np.array(b)
        A[np.ix_(idx, idx)] = 0.9                       # strong within object affinity
    A = (A + A.T) / 2
    np.fill_diagonal(A, 0.0)

    masks = generate_masks(A, n)
    print(f"generated {len(masks)} masks from {n} nodes")
    for i, m in enumerate(masks):
        print(f"  mask {i} covers {int(m.sum())} nodes")

Conclusion

The core achievement here is turning a picture maker into a picture parser for free. By reading a pretrained diffusion model’s self attention as a map of what belongs together, and growing a grid of prompts across it with an edge preserving flow, Diffuse2Seg produces high quality segmentation masks with no labels and no retraining, then distils them into a fast, standalone model. Those masks beat the previous best unsupervised methods across five datasets, and the trained model generalizes strongly to images it never saw.

The conceptual shift worth remembering is that generation and perception are two sides of one coin. A model built only to synthesize images had, as an unavoidable byproduct, learned where objects are, and this work shows that knowledge can be harvested directly. That reframes the expensive labelling that segmentation has always demanded as something a generative model can partly supply for free, which is a genuinely different way to think about where supervision comes from.

The approach travels because it is not tied to one dataset or one kind of scene. The same extraction worked across objects, amorphous regions, and parts, and the resulting labels made a small model data efficient enough to beat fully supervised training with a few thousand human labels. The propagation framework is also flexible, and the authors point to extending it to video and to class conditioned prompting as natural next steps.

The honest limits keep it grounded. The method struggles with small objects because the diffusion latent is coarse, its texture sensitivity can oversegment smooth regions, the extraction is computationally heavy, and, importantly, this is a preprint whose results await peer review and public code. It is a promising early report, not a settled result.

Where this goes next is clear enough. Lift the small object ceiling with higher resolution features, tame the texture oversegmentation, and push the idea into video and other modalities where the same trick of reading structure out of a generative model should apply. If those steps hold up, the enormous cost of hand labelling images for segmentation could shrink dramatically, and a great deal of the visual knowledge locked inside our largest generative models could finally be put to work seeing rather than only imagining.

Frequently asked questions