Long Tailed Weights Beat Thresholds For Pseudo Labels

The pseudo labels threshold problem nobody has fully solved

Most modern semi supervised learning systems, FixMatch and FreeMatch among the best known, work the same basic way. Take an unlabeled image, run it through the model with a light augmentation, and if the model’s confidence in its own prediction clears some threshold, treat that prediction as a pseudo label and train on it as if it were ground truth. It is a simple idea, and it works well enough to have become the default approach across the field.

The paper’s authors, Jiaqi Wu, Junbiao Pang, and Qingming Huang, lay out three specific ways this breaks down. Setting the threshold itself is genuinely hard. Too low, and noisy or wrong predictions leak into training as if they were correct. Too high, and the model only trains on the samples it was already confident about, which the authors note can worsen the Matthew effect, where classes the model already handles well keep improving while classes it struggles with keep falling further behind. Second, a hard cutoff throws away information. Even an incorrect pseudo label can carry a useful signal about where class boundaries sit, something prior entropy based work has also observed. Third, and probably the sharpest point, deep networks trained on scarce labeled data are frequently overconfident, so a high softmax score does not reliably mean a correct prediction. That gap between confidence and accuracy is especially visible in pose estimation, where a heatmap can look sharp and certain while still pointing at the wrong pixel entirely.

Two kinds of uncertainty, not one

UES builds on a prior method from the same research group called Channel based Ensemble, or CBE, which generates several low correlation prediction heads from a single backbone without meaningfully increasing parameter count or training time. UES adds a Mean based Uncertainty Estimator on top of that ensemble, and the estimator produces two distinct uncertainty signals rather than one.

Sample uncertainty

For a batch of unlabeled samples, UES first averages the predictions from all M prediction heads to form a reference distribution. It then measures how far each individual head’s prediction strays from that average. A sample where all the heads roughly agree gets a low uncertainty score. A sample where the heads scatter in different directions gets a high one. In the regression setting used for pose estimation, that disagreement is measured with mean squared error between each head’s prediction and the reference distribution.

Prediction head uncertainty

Separately, UES asks a different question about each prediction head as a whole, not sample by sample but averaged across an entire batch. How far does this particular head, on average, tend to drift from the ensemble consensus. Heads that drift a lot get downweighted when the ensemble prediction is computed, which the authors argue prevents the whole ensemble prediction from being dragged around by one erratic head.

That head level uncertainty is then converted into weights with a softmax over the negated uncertainty values, so heads that disagree with the consensus less often end up carrying more influence over the final ensemble prediction used as the pseudo label.

The long tailed weight, and why it never hits zero

Here is the part of the design that gives the method its name and its core philosophical bet. Once sample uncertainty is computed, it gets converted into a training weight using a specific formula.

Because the denominator is always at least one, no matter how large a sample’s uncertainty gets relative to the rest of the batch, the resulting weight can never actually reach zero. Even the single most uncertain sample in a batch, the one a threshold based method would discard outright, still contributes at roughly eighty percent the weight of the most confident sample, based on the curve the paper reports. That is a deliberate design choice, not an accident of the math. The authors argue that in the early stages of training, pulling in more samples, including ones the model is currently getting wrong, helps it learn general data features faster than being cautious would. As training progresses and the model’s predictions genuinely improve, the argument goes, the noise these low confidence samples introduce naturally shrinks relative to the signal.

How this plugs into an actual training loop

The supervised loss on labeled data stays completely conventional, an averaged cross entropy across all prediction heads.

For unlabeled data, the ensemble prediction that becomes the pseudo label is a weighted average across heads, gated by a confidence threshold that still exists in the background but plays a much smaller role than in a conventional pipeline.

The unsupervised loss then applies the sample level long tailed weight on top of that pseudo label, so a sample the model is uncertain about still trains the network, just with proportionally less influence than a sample it is confident about.

The authors are upfront that integrating UES into an existing SSL pipeline like FixMatch takes only two changes, swapping in the uncertainty weighted ensemble prediction and applying the sample weight inside the loss computation, which is a genuinely low integration cost for a method claiming to fix a structural problem with the entire field’s dominant approach.

Results on pose estimation, and where they get complicated

The paper validates UES on three keypoint datasets combined with a prior SSL pose method called DualPose, Sniffing, a small dataset of animal behavior recordings, FLIC, a human pose dataset built from film stills, and LSP, a sports pose dataset. The comparison baseline throughout is DualPose plus the earlier CBE ensemble without the new uncertainty weighting.

PCK figures use the paper’s own primary threshold for each split, PCK at 0.2 for Sniffing and PCK at 0.5 for FLIC and LSP. Full results, including additional label splits and PCK thresholds, appear in the paper’s Table I and Table II.

Look closely at the Sniffing dataset’s second split, sixty labeled out of two hundred total. UES improves the error metric, MSE drops from 4.15 to 3.99, but its keypoint accuracy metric, PCK at 0.2, actually falls from 70.34 to 68.75, a genuine regression against the paper’s own prior method. That result sits quietly in Table I with no separate comment or explanation anywhere in the text, even though the abstract’s headline number for the Sniffing dataset comes from the other split, the one where UES wins. A method that shows up as an improvement in the abstract and a mixed result inside its own results table is not unusual in machine learning research, but it is exactly the kind of detail that gets lost if you only read the summary. MSE and PCK are not always going to move together, since PCK counts a prediction correct only if it falls inside a normalized distance threshold, so a model can reduce its average error while shifting more of its predictions just outside that threshold, and this appears to be exactly what happened on this particular split.

Results on classification, where the gains are consistent but modest

On CIFAR-10 and CIFAR-100, the authors integrate UES with FixMatch rather than DualPose and test across a range of labeled data amounts, from extremely scarce to comparatively generous.

Every classification split moves in UES’s favor, which is a cleaner story than the pose estimation results, though the margins are genuinely small, error rate improvements between a tenth and a third of a percentage point. The authors frame this consistency as evidence that the long tailed weighting mechanism generalizes across task types, and the direction of the result does support that. It is worth being honest that gains this size sit close to the run to run variance the paper itself reports, with standard deviations in the same rough range as the improvements, so the classification results read more as a mild, consistent nudge than a decisive win.

What the ablation actually isolates

The ablation studies split UES into its two components, sample weights and prediction head weights, and add them one at a time on top of the CBE baseline. On the FLIC pose dataset, sample weights alone improve PCK by 2.39 percentage points over CBE, and adding prediction head weights on top pushes that improvement further, an additional 4.9 percentage points beyond the sample weights alone result. On LSP, the pattern repeats at smaller scale, 1.51 points from sample weights alone, another 2.4 points from adding head weights. On CIFAR-10 with 40 labels, the same ordering holds, sample weights cut error by 0.06 points and adding head weights cuts a further 0.14 points.

The consistent finding across every ablation is that prediction head weighting contributes more than sample weighting by itself, sometimes by a wide margin. That is a useful signal for anyone trying to replicate a slimmed down version of this method. If you only have budget to implement one half of UES, the paper’s own numbers suggest the head level reweighting of the ensemble prediction is doing more of the real work than the per sample long tailed weight that gives the method its name.

The caveat the authors state plainly and then move past

Section six of the paper includes a genuinely candid admission. The authors track the correlation between their uncertainty estimates and actual prediction accuracy over the course of training, and they report that early on, that correlation is weak. In other words, during the first stretch of training, a sample UES marks as low uncertainty is not reliably more accurate than one it marks as high uncertainty. The correlation strengthens as training progresses, which the authors attribute to the model’s overall predictions on unlabeled data becoming more stable once it has seen enough examples.

This matters because the entire mechanism rests on the premise that ensemble disagreement is a meaningful proxy for correctness. If that proxy is weak exactly when training first begins, then the long tailed weights during that early window are, by the authors’ own account, not doing what they are designed to do, at least not reliably. The paper does not propose a fix for this within its main method, and closes by naming it directly as a limitation, noting that the long tailed weight formula is static across training and does not adapt to the fact that early and late training have different reliability profiles. Dynamic, stage aware weighting is left as explicit future work rather than something the current version of UES addresses.

Honest limitations

Beyond the uncertainty correlation issue the authors raise themselves, a few other things are worth keeping in mind. The pose estimation gains are reported on small datasets, a maximum of three hundred total samples for LSP and two hundred for Sniffing, which makes the reported standard deviations proportionally large relative to the improvements being claimed, and makes it harder to be confident these gains hold at larger scale without further testing. The comparison baseline throughout is the authors’ own earlier CBE method rather than a broader field of competing uncertainty based SSL techniques, so the paper demonstrates that UES improves on its own predecessor more thoroughly than it demonstrates a clear win against the wider landscape of alternative approaches. And as the Sniffing 60 out of 200 result shows directly, the method’s benefit is not uniform across every metric and every label split, which is a reasonable thing to expect from any weighting scheme but is worth stating plainly rather than only citing the splits where the numbers look best.

A reference implementation of the core mechanism

The block below is a simplified, runnable PyTorch implementation of the Mean based Uncertainty Estimator, the long tailed sample weighting, the prediction head weighting, and the combined ensemble loss described above. It is built for learning and experimentation rather than as a reproduction of the authors’ exact codebase, and includes a smoke test against random dummy tensors.

import torch
import torch.nn as nn
import torch.nn.functional as F


# ---- Mean based uncertainty estimator ----

def sample_uncertainty_regression(head_preds: torch.Tensor) -> torch.Tensor:
    # head_preds shape [M, N, D], M prediction heads, N samples, D output dims
    reference = head_preds.mean(dim=0)  # [N, D]
    mse_per_head = ((head_preds - reference.unsqueeze(0)) ** 2).mean(dim=-1)  # [M, N]
    return mse_per_head.mean(dim=0)  # [N], sample uncertainty u_i^S


def head_uncertainty_regression(head_preds: torch.Tensor) -> torch.Tensor:
    reference = head_preds.mean(dim=0)
    mse_per_head = ((head_preds - reference.unsqueeze(0)) ** 2).mean(dim=-1)  # [M, N]
    return mse_per_head.mean(dim=1)  # [M], prediction head uncertainty u_m^H


def long_tailed_sample_weight(sample_uncertainty: torch.Tensor) -> torch.Tensor:
    max_u = sample_uncertainty.max().clamp(min=1e-8)
    return 1.0 / (sample_uncertainty / max_u + 1.0)


def head_weights(head_uncertainty: torch.Tensor) -> torch.Tensor:
    return F.softmax(-head_uncertainty, dim=0)  # [M]


def weighted_ensemble_prediction(head_preds: torch.Tensor, w_head: torch.Tensor, tau: float = 0.0) -> torch.Tensor:
    # head_preds [M, N, D], w_head [M]
    weighted = head_preds * w_head.view(-1, 1, 1)
    ensemble = weighted.sum(dim=0)  # [N, D]
    if tau > 0:
        confident = (ensemble.abs().max(dim=-1).values > tau).float().unsqueeze(-1)
        ensemble = ensemble * confident
    return ensemble


# ---- Ensemble supervised loss with long tailed weighting ----

def ensemble_supervised_loss(head_preds: torch.Tensor, pseudo_labels: torch.Tensor, sample_weight: torch.Tensor) -> torch.Tensor:
    # head_preds [M, N, D], pseudo_labels [N, D], sample_weight [N]
    m = head_preds.shape[0]
    per_head_mse = ((head_preds - pseudo_labels.unsqueeze(0)) ** 2).mean(dim=-1)  # [M, N]
    per_sample_loss = per_head_mse.mean(dim=0)  # [N]
    weighted_loss = sample_weight * per_sample_loss
    return weighted_loss.mean()


# ---- A minimal channel style multi head wrapper ----

class ChannelEnsembleHead(nn.Module):
    def __init__(self, in_dim: int, out_dim: int, num_heads: int = 5):
        super().__init__()
        self.num_heads = num_heads
        self.heads = nn.ModuleList([nn.Linear(in_dim, out_dim) for _ in range(num_heads)])

    def forward(self, feat: torch.Tensor) -> torch.Tensor:
        # feat [N, in_dim], returns [M, N, out_dim]
        outputs = [head(feat) for head in self.heads]
        return torch.stack(outputs, dim=0)


# ---- Smoke test ----

def smoke_test():
    torch.manual_seed(0)
    n_samples, in_dim, out_dim, m_heads = 8, 16, 4, 5

    model = ChannelEnsembleHead(in_dim, out_dim, num_heads=m_heads)
    feat = torch.randn(n_samples, in_dim)
    head_preds = model(feat)  # [M, N, D]

    u_sample = sample_uncertainty_regression(head_preds)
    u_head = head_uncertainty_regression(head_preds)
    w_sample = long_tailed_sample_weight(u_sample)
    w_head = head_weights(u_head)
    pseudo = weighted_ensemble_prediction(head_preds, w_head)
    loss = ensemble_supervised_loss(head_preds, pseudo, w_sample)

    assert u_sample.shape == (n_samples,)
    assert u_head.shape == (m_heads,)
    assert torch.all(w_sample > 0), "long tailed weights must never reach zero"
    assert torch.all(w_sample <= 1.0)
    assert abs(w_head.sum().item() - 1.0) < 1e-5, "head weights must sum to one"
    assert pseudo.shape == (n_samples, out_dim)
    assert torch.isfinite(loss)

    print("Smoke test passed.")
    print("Sample weight range", w_sample.min().item(), "to", w_sample.max().item())
    print("Head weights", w_head.tolist())
    print("Loss value", loss.item())

if __name__ == "__main__":
    smoke_test()

Conclusion

UES makes a specific, well scoped bet, that ensemble disagreement is a more honest signal of pseudo label trustworthiness than a single model’s softmax confidence, and that converting that signal into a weight which never fully discards a sample beats the conventional keep or reject threshold. The classification results back that bet cleanly, if modestly, across every label count the authors tested on both CIFAR datasets. The ablation studies make a genuinely useful secondary point, that weighting the prediction heads themselves contributes more to the improvement than weighting individual samples, which is a detail worth remembering for anyone tempted to implement a lighter version of this idea.

The pose estimation results are the more interesting half of the paper precisely because they are not uniformly positive. A method’s own results table quietly containing a split where it loses to its predecessor on the primary metric is not a reason to dismiss the method, methods rarely win everywhere, but it is a reason to read result tables in full rather than trusting a percentage figure lifted into an abstract. The authors deserve credit for reporting that split at all rather than omitting it, even if they do not call attention to it directly in the text.

The most transferable idea here, beyond the specific formulas, is treating uncertainty and reliability as continuous quantities to be weighted rather than binary properties to be thresholded. That framing shows up across a lot of adjacent problems, active learning sample selection, data cleaning for weakly labeled datasets, and out of distribution detection all wrestle with a similar tension between being too permissive and too conservative about which signal to trust.

The honest caveat that should travel with this idea is the one the authors state themselves. Ensemble disagreement is only a useful proxy for correctness once the ensemble has had enough training to actually diverge from random noise, and early in training that proxy is weak by the paper’s own measurement. A static, always on long tailed weight cannot fully account for that shifting reliability across the training timeline, which the authors acknowledge directly and leave as their proposed next step, dynamic weighting mechanisms that adapt to what stage of training the model is actually in.

Taken together, UES is a reasonable, low overhead addition to an existing semi supervised pipeline rather than a wholesale replacement for how the field thinks about pseudo labels. Its clearest contribution may end up being the demonstration that prediction head level reweighting, the less flashy half of the method, is doing more work than the long tailed sample weighting that gives the paper its title.

Frequently asked questions

Related reading