SAUF-Net Separates Lesion Shape From Its Appearance

Why scarce labels lead a model astray

Deep learning has become very good at outlining anatomical structures and lesions in medical images, but the best methods still lean on large sets of pixel by pixel annotations. Producing those requires expert knowledge and careful manual tracing, which is expensive and time consuming, and that annotation burden is the main thing holding automated segmentation back in practice, especially where labeled data is scarce.

Semi supervised learning is the standard answer. Train on a small labeled set together with a large unlabeled set, and let the model extract signal from the unlabeled images through consistency and pseudo labels. Most methods do this at the level of predictions, enforcing that the output stays stable under perturbations or selecting pseudo labels by how confident the model is. That helps, but it regularizes the output while leaving the reliability of the model’s internal features unchecked, and that gap matters more in medicine than almost anywhere else. This lineage of low label medical segmentation runs through work like barely supervised 3D medical segmentation and one shot medical segmentation from a single labeled image.

Here is the specific trap the authors, from Nanchang Hangkong University, set out to close. In a medical image the cues that actually define the target, the lesion’s shape, its location, the continuity of its boundary, are easily entangled with unstable appearance cues, the background texture, color variation, imaging artifacts, and noise. When labeled data is scarce the model can produce a confident pseudo label that is really keyed on appearance rather than on the segmentation relevant structure. Once that biased prediction is used to train on unlabeled data, the mistake gets reinforced, and the errors accumulate progressively. A confident wrong answer, repeated across thousands of unlabeled images, is worse than no answer at all.

Splitting shape from appearance

SAUF-Net, which stands for Structure Appearance representation learning with Uncertainty Feedback, attacks the problem by refusing to let structure and appearance stay entangled. It is built on a SegFormer-B4 backbone, and its first move is to take the deepest encoder feature and decompose it with a Structure Appearance Decomposition Module.

That module is two parallel projection branches with independent parameters. One produces a structural representation designed to capture the target related cues, the shape, the spatial location, the region continuity, and the boundary, which are the things directly relevant to segmentation. The other produces an appearance representation that models the complementary information, the background texture, the intensity and color distribution, and the imaging noise. By pulling these apart explicitly, the network is encouraged to lean on the stable structural cues and to stop being dominated by unstable texture or intensity changes. The same instinct, separating what is essential to the target from what merely varies, drives related medical work such as separating shared disease knowledge from modality specific detail.

Once the two representations exist, a Disentangled Guidance Module injects them back into the decoder, and the order it does this in is deliberate. For each decoding stage the structural representation first generates an offset and a modulation mask.

The decoding feature is sampled by a deformable convolution using that offset and mask, which lets the decoder adaptively focus on structure related regions. Only then is the structurally guided feature refined by the appearance representation, through a second deformable step that produces the final guided feature for that stage.

The sequence is the point. Structure leads, appearance follows and refines, so the decoder is anchored on shape before it ever considers texture. An auxiliary decoder runs alongside, producing separate predictions from the structural and appearance features plus a fused one, and those separate predictions turn out to be the raw material for judging reliability later.

Proving the shape is stable

Splitting structure from appearance is only half the job. Nothing yet guarantees the structural representation is truly free of appearance information, so the authors add a test that forces the issue, called Appearance Swapped Consistency. For two images in a mini batch, it keeps the first image’s structural representation but replaces its appearance representation with the shuffled appearance of the second image, then asks the fused prediction to come out the same.

If the structural representation really carried the shape, then bolting on a stranger’s appearance should not change the outline the model draws. By demanding that the fused prediction preserve the structure from the first image despite the swapped appearance, this step actively pushes the structural features to become appearance invariant. It is a clean way to enforce the disentanglement rather than just hoping the two branches learned to separate.

A map of what to trust

The last piece addresses the pseudo label problem directly. Pseudo labels are unreliable in ambiguous or low contrast regions, exactly the places a lesion boundary tends to sit, so SAUF-Net builds a reliability map from the model’s own predictions and uses it to weight self training.

The two factors each carry a distinct signal. The first term measures agreement between the main prediction and the structural prediction, so it is high where the structure branch backs up the main output. The second term measures the response contrast between the structural and appearance branches, so it is high where the two branches disagree in a way that indicates the structure is carrying real, distinctive information. Multiply them and you get a per pixel score that is large where the prediction is genuinely well supported by structure. That score then scales how much each unlabeled pixel contributes to the consistency loss.

A confident pixel with a strong reliability score gets full weight, while an ambiguous or unreliable one is down weighted, and crucially the pseudo label and the weight carry no gradient, so the model cannot game its own reliability estimate. On top of this, a dual head discriminator adds feature level feedback, with one head judging whether a feature looks valid and another predicting where the model should be uncertain, guided by the reliability map. The whole apparatus, disentanglement, appearance swapping, reliability weighting, and the discriminator, is trained together but stripped away at inference, so the deployed model is just the segmentation network.

What the numbers say

SAUF-Net was evaluated on two public datasets, ISIC-2016, a set of dermoscopy images for skin cancer with 900 training and 379 test images, and Kvasir-SEG, a set of 1,000 polyp images split 80 to 20. Performance is reported as the Dice similarity coefficient, intersection over union, and pixel accuracy, at labeled ratios of 5, 10, and 20 percent. The headline comparison is against the previous best method, AdaptFRCNet.

The gains are consistent and larger on the harder polyp dataset. At 10 percent labels SAUF-Net improved Dice over the previous best by 0.51 on skin lesions and 1.18 on polyps, and the improvements in per image Dice were statistically significant across both datasets and every labeling ratio. The more striking result is at the bottom of the label budget.

With one image in twenty labeled, SAUF-Net reached a Dice of 90.91 on skin lesions and 89.71 on polyps, which the authors note is competitive with or better than several methods that used twice as many labels. Sitting only about two Dice points below the model trained on every label, while using a twentieth of them, is the whole promise of semi supervised learning made concrete. The qualitative comparisons show the same thing, with SAUF-Net preserving small lesions, blurry boundaries, and complex background textures where other methods break down.

Which piece did the work

The ablation is unusually clean about where the accuracy comes from, and the answer is that the disentanglement carries most of it.

The single biggest jump comes from the decomposition itself, lifting Dice from 82.16 to 89.91, which says plainly that separating structural cues from appearance is the load bearing idea. Each later addition stacks a smaller, real gain on top, the reliability map to 91.16, appearance swapped consistency to 91.38, and the uncertainty feedback to the full 91.98. The authors also note a nice interaction, that the uncertainty feedback branch on its own boosts accuracy but not Dice, and only becomes fully effective once appearance swapped consistency has regularized the structure, which is a sensible reason to keep both rather than pick one.

The clinical translation gap

Strong Dice scores on two benchmarks are not the same as a tool a clinic can use, and this work sits a clear distance from that, which the authors themselves flag. The most important gap is scope. The evaluation is limited to 2D binary lesion segmentation, one target against background, on two curated public datasets. Real clinical imaging is often 3D, frequently involves multiple structures at once, and varies enormously with scanner, staining, lighting, and patient population. A method that separates structure from appearance so cleanly on curated dermoscopy and colonoscopy images has not been shown to hold that separation on the messier distribution a hospital actually sees.

There is also a gap between the metric and the decision. A Dice of 91 on a research split is encouraging, but a segmentation that is usually excellent and occasionally wrong at a boundary, with no built in signal to the clinician about which case is which at inference, is a support tool rather than an autonomous one. The reliability map is used during training to weight learning, not surfaced at test time as a per case confidence a reader could act on. For skin cancer and polyp assessment, where a missed or mis outlined lesion has real consequences, that distinction matters, and the authors are appropriately modest about what the method is for.

Honest limitations

The headline limitation is the one the paper names first, that the current evaluation is limited to 2D binary lesion segmentation and does not assess cross dataset generalization. Both datasets are curated public benchmarks, which is the right place to start but not evidence that the method transfers across the staining, device, and population shifts that break medical models in the field. Until it is validated on independent data from other sources, the gains should be read as promising rather than deployable.

Dataset and design assumptions run through the method too. The whole approach presumes that structure and appearance can be cleanly separated in the bottleneck feature, which holds well on these two tasks but may not on modalities where the pathology itself is defined by texture or intensity rather than shape, where the appearance branch would be carrying the very signal you want. The reliability map is derived from the model’s own branch predictions, so if the model is confidently wrong across all its branches, the reliability estimate inherits that blind spot rather than catching it. And the reported numbers are single configuration results on fixed splits, so the usual caveats about run to run variance and a separate held out tuning split apply.

Generalization concerns are concrete rather than abstract. The authors explicitly leave 3D and multi class segmentation to future work, along with robustness across datasets and imaging domains. Those are not minor extensions, they are the difference between a benchmark result and a clinical tool, and the same appetite for that harder validation shows up across the field’s polyp work, from modeling the background as carefully as the foreground to handling blurry polyp boundaries. Above all, this is an unreviewed preprint, and its clinical claims should be read as provisional until peer review and external validation catch up.

Why the approach travels

Set the medical specifics aside and SAUF-Net carries two ideas that generalize well beyond lesions. The first is that in semi supervised learning, the reliability of internal features matters as much as the consistency of outputs, and you can act on it. Rather than only asking whether two predictions agree, the reliability map asks whether the structural evidence supports a prediction before letting it teach the model, which is a more honest gate on self training. The second is that disentangling a stable, task relevant factor from an unstable, incidental one, and then proving the separation with a swap test, is a general recipe for any setting where a model is tempted to take an appearance shortcut.

Both ideas would transfer to natural image segmentation under domain shift, to any task where labels are scarce and a spurious cue is available, and to modalities beyond skin and colon. SAUF-Net’s specific contribution is to wire structure appearance disentanglement together with a feature level reliability signal, and to show, through a clean ablation, that the disentanglement is doing most of the work and the reliability weighting adds a real increment on top.

Reference implementation in PyTorch

The code below is a runnable reconstruction of SAUF-Net’s core ideas, the structure appearance decomposition, the appearance swapped consistency, and the reliability weighted self training, based on the paper’s equations. A compact encoder and decoder stub stands in for the SegFormer-B4 backbone so the file runs without downloads. It includes the decomposition module producing structural and appearance features, an auxiliary decoder giving the branch predictions, the reliability map, the appearance swap, the supervised Dice plus cross entropy loss, and the reliability weighted consistency loss. A smoke test runs it on dummy tensors. Swap in the real backbone and medical datasets for actual experiments.

# sauf_net_reference.py
# Structure appearance decomposition, reliability weighting, appearance swap.
# Replace EncoderStub / DecoderStub with SegFormer-B4 for real runs.

import torch
import torch.nn as nn
import torch.nn.functional as F


class EncoderStub(nn.Module):
    """Stand in for the SegFormer encoder. Returns a bottleneck feature."""
    def __init__(self, cin=3, d=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(cin, d, 3, 2, 1), nn.BatchNorm2d(d), nn.ReLU(inplace=True),
            nn.Conv2d(d, d, 3, 2, 1), nn.BatchNorm2d(d), nn.ReLU(inplace=True))

    def forward(self, x):
        return self.net(x)


class Branch(nn.Module):
    """One SADM projection branch, two 3x3 blocks then a 1x1 block."""
    def __init__(self, d=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(d, d, 3, 1, 1), nn.BatchNorm2d(d), nn.ReLU(inplace=True),
            nn.Conv2d(d, d, 3, 1, 1), nn.BatchNorm2d(d), nn.ReLU(inplace=True),
            nn.Conv2d(d, d, 1), nn.BatchNorm2d(d), nn.ReLU(inplace=True))

    def forward(self, x):
        return self.net(x)


class DecoderStub(nn.Module):
    """Upsamples a feature to a single channel mask logit."""
    def __init__(self, cin, d=32):
        super().__init__()
        self.head = nn.Sequential(nn.Conv2d(cin, d, 3, 1, 1),
                                  nn.ReLU(inplace=True), nn.Conv2d(d, 1, 1))

    def forward(self, x, size):
        return F.interpolate(self.head(x), size=size, mode="bilinear", align_corners=False)


class SAUFNet(nn.Module):
    def __init__(self, d=64):
        super().__init__()
        self.enc = EncoderStub(3, d)
        self.struct = Branch(d)          # F_s, target shape and boundary
        self.appear = Branch(d)          # F_a, texture, color, noise
        self.dec_main = DecoderStub(d)
        self.dec_s = DecoderStub(d)
        self.dec_a = DecoderStub(d)
        self.dec_fuse = DecoderStub(2 * d)

    def forward(self, x, swap=False):
        size = x.shape[-2:]
        f = self.enc(x)
        fs, fa = self.struct(f), self.appear(f)
        if swap:                          # appearance swapped consistency, Eq 4
            fa = fa[torch.randperm(fa.shape[0])]
        p = torch.sigmoid(self.dec_main((fs + fa), size))
        p_s = torch.sigmoid(self.dec_s(fs, size))
        p_a = torch.sigmoid(self.dec_a(fa, size))
        p_aux = torch.sigmoid(self.dec_fuse(torch.cat([fs, fa], 1), size))
        return {"p": p, "p_s": p_s, "p_a": p_a, "p_aux": p_aux}


def reliability_map(p, p_s, p_a):
    """M_rel = (1 - |P - P_s|) * |P_s - P_a|, Eq 5."""
    return (1 - (p - p_s).abs()) * (p_s - p_a).abs()


def dice_bce(pred, target, eps=1.0):
    bce = F.binary_cross_entropy(pred, target)
    inter = (pred * target).sum((2, 3))
    dice = 1 - ((2 * inter + eps) / (pred.sum((2, 3)) + target.sum((2, 3)) + eps)).mean()
    return bce + dice


def supervised_loss(out, y, lam_aux=0.4):
    """Appearance branch is supervised by the complementary mask, Eq 6."""
    main = dice_bce(out["p"], y)
    aux = (dice_bce(out["p_s"], y) + dice_bce(out["p_a"], 1 - y)
           + dice_bce(out["p_aux"], y))
    return main + lam_aux * aux


def consistency_loss(out_aug, out_clean, tau=0.95, eps=1e-6):
    """Reliability weighted self training on unlabeled data, Eq 7."""
    p = out_clean["p"].detach()
    pseudo = (p > 0.5).float()
    conf = (torch.maximum(p, 1 - p) > tau).float()
    m_rel = reliability_map(p, out_clean["p_s"].detach(), out_clean["p_a"].detach())
    w = conf * m_rel                                  # stop gradient on both
    bce = F.binary_cross_entropy(out_aug["p"], pseudo, reduction="none")
    return (w * bce).sum() / (w.sum() + eps)


if __name__ == "__main__":
    model = SAUFNet()
    opt = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=1e-4)
    x_l = torch.rand(2, 3, 128, 128)
    y_l = (torch.rand(2, 1, 128, 128) > 0.6).float()
    x_u = torch.rand(2, 3, 128, 128)
    for step in range(3):
        sup = supervised_loss(model(x_l), y_l)
        con = consistency_loss(model(x_u + 0.1 * torch.randn_like(x_u)), model(x_u))
        loss = sup + 0.1 * con
        opt.zero_grad(); loss.backward(); opt.step()
        print("step", step, "sup", round(sup.item(), 4), "con", round(con.item(), 4))

Conclusion

The core achievement of SAUF-Net is to make semi supervised medical segmentation more honest about what it learns from unlabeled data. By splitting each feature into a structural part that captures a lesion’s shape and an appearance part that captures its texture and color, and then trusting the structure when it teaches itself, it stops the appearance shortcut that quietly corrupts pseudo labels when labels are scarce. On two datasets it beat the previous best at every labeling ratio, reaching a Dice of 91.88 on skin lesions and 90.53 on polyps with only a tenth of the images labeled, and staying strong down to a twentieth.

The conceptual contribution worth keeping is that reliability lives in the features, not just the outputs. Most semi supervised methods regularize predictions and hope the representations behind them are sound. SAUF-Net instead builds a reliability signal from the agreement and contrast among its own branches, and uses it to decide which unlabeled pixels have earned the right to teach the model. Pairing that with a disentanglement it actively verifies through an appearance swap is a cleaner discipline than confidence thresholding alone.

The design also travels. Separating a stable, task relevant factor from an unstable, incidental one and proving the separation with a swap test is a general defense against appearance shortcuts, and gating self training on feature level reliability rather than raw confidence would help any low label task tempted by a spurious cue. The clean ablation, which shows the decomposition doing most of the work and each later piece adding a real increment, makes the recipe easy to reason about and adopt.

The honest limitations keep it in proportion. This is a preprint on two 2D binary segmentation benchmarks, it does not yet handle 3D or multiple classes, it has not been tested for cross dataset generalization, and its reliability map inherits any blind spot the model’s own branches share. The authors name all of this, which is exactly the right posture for a result that is a strong step toward label efficient segmentation rather than a finished clinical tool.

For anyone building segmentation under scarce labels, the practical message is compact. Do not let your model learn a lesion by its color. Force it to separate shape from appearance, verify that separation, and let feature level reliability, not raw confidence, decide which unlabeled pixels get to shape the model. SAUF-Net shows the payoff is real at very low label budgets, and the reference above is a place to start testing the idea on a segmentation task of your own.

Frequently asked questions

What is semi supervised medical image segmentation?

It is training a segmentation model on a small set of labeled images together with a large set of unlabeled ones, so you do not have to pay experts to trace every image by hand. The model learns from the unlabeled data through techniques like consistency training and pseudo labels, which is valuable in medicine where pixel by pixel annotation is expensive and time consuming.

What does SAUF-Net do differently?

SAUF-Net separates each internal feature into a structural part that captures a lesion’s shape, location, and boundary and an appearance part that captures color, texture, and noise. It then trusts the structural part when learning from unlabeled data, verifies the separation by swapping appearances between images, and weights self training by a reliability map built from its own predictions rather than by raw confidence.

Why does separating structure from appearance help?

When labeled data is scarce a model can produce a confident pseudo label that is really keyed on unstable appearance cues like background texture or color rather than on the lesion’s actual shape. Training on those biased predictions reinforces the mistake across the unlabeled set. Forcing structure and appearance apart, and anchoring the decoder on structure first, keeps the model focused on the segmentation relevant shape.

How well did SAUF-Net perform?

With 10 percent of images labeled it reached a Dice of 91.88 on the ISIC-2016 skin lesion dataset and 90.53 on the Kvasir-SEG polyp dataset, beating the previous best method with statistically significant per image gains. With only 5 percent labeled it reached 90.91 and 89.71 Dice, competitive with or better than several methods that used twice as many labels.

Is SAUF-Net ready for clinical use?

No. It is an unreviewed preprint evaluated only on two curated 2D binary segmentation datasets, and it does not yet handle 3D or multi class tasks or demonstrate cross dataset generalization. Its reliability map is used during training rather than surfaced as a per case confidence at inference. The authors describe it as a step toward label efficient segmentation, and external validation would be needed before any clinical use.

What are the main limitations?

The evaluation is limited to 2D binary lesion segmentation on two public datasets and does not assess robustness across scanners, staining, or populations. The method assumes structure and appearance can be cleanly separated, which may fail where pathology is defined by texture rather than shape, and the reliability map can inherit a blind spot if all of the model’s branches are confidently wrong together. The results are also single configuration runs on fixed splits.