Why Deepfake Detectors Need Their Own Vision Transformer

This article is part of our Vision transformer and attention hub coverage.

The generalization problem in face forgery detection

Face forgery detection sounds like a solved problem if you only look at the headline numbers. Train a deep network on FaceForensics++, a standard benchmark of manipulated videos, and accuracy climbs past 95 percent within that dataset. The trouble starts the moment the model meets a forgery method it has never seen. Accuracy on unfamiliar datasets, made with different synthesis algorithms, routinely collapses. This is not a minor rounding error. It is the central obstacle standing between a research benchmark and a detector someone can actually deploy against forgeries that have not been invented yet.

The reason is not mysterious once you say it plainly. A model trained to recognize the specific visual fingerprints of a handful of forgery methods learns exactly that, and nothing more general. Swap in a different generative adversarial network or a different diffusion based face swap, and those fingerprints change while the underlying task, telling a synthetic face from a real one, stays the same. Most published defenses respond by inventing cleverer fine tuning tricks, data augmentation to synthesize more forgery variety, attention modules to highlight suspicious regions, feature decoupling to separate identity from artifact. All of it happens downstream of a backbone the researchers rarely question.

The backbone nobody questioned

Here is where it gets interesting. Nearly every model built for face forgery detection starts from the same place, a convolutional network or a vision transformer pretrained with supervised classification on ImageNet, then fine tuned on forgery data. The authors point out that this default choice has quietly limited the entire field. ImageNet teaches a network to recognize dogs, cars, and furniture. It never asks the network to notice that a person’s left eyebrow moves in a way that is barely inconsistent with the rest of their face, which is precisely the kind of subtle, local signal a forgery detector actually needs.

The paper backs this claim with a genuinely useful empirical study rather than just asserting it. Backbones trained completely from scratch on forgery data alone average around 61 percent accuracy on unseen test sets, close to a coin flip. Adding ordinary supervised ImageNet pretraining lifts that by more than 10 percentage points. Switching from supervised pretraining to self supervised pretraining on ImageNet, using methods like MoCo and BEiT that learn from the structure of images themselves rather than from labels, adds still more. And vision transformer architectures consistently outperform convolutional ones once paired with a strong self supervised method, with a transformer trained using MoCo v3 beating a comparably sized ResNet by 3.77 percentage points on average. None of that required touching the face forgery data at all. It is entirely about what the backbone learned before forgery detection ever entered the picture.

Optimizing vision transformers for face forgery detection

Having shown that pretraining matters this much, the authors take the obvious next step and pretrain directly on faces. They assemble a dataset of 550,000 real face images pulled from CelebA, CelebV Text, and FFHQ, all unlabeled, and pretrain a vision transformer on it using three self supervised methods, MoCo v3, MAE, and BEiT v2. Because these are self supervised methods, no forgery labels or even identity labels are needed, only a large pile of ordinary real faces, which happen to be far easier to collect at scale than diverse, well documented forged ones.

The results split cleanly along an interesting line. Pretraining directly on real faces beats pretraining on ImageNet across all three methods, even though the face dataset is roughly half the size of ImageNet. Continuing an already ImageNet pretrained backbone with further pretraining on real faces works even better, since the model effectively picks up general visual knowledge first and facial specific structure second. The clearest winner among the three methods is BEiT v2, whose resulting face pretrained backbone the authors name FaceBEiT. Its masked image modeling pretext task, which trains the network to predict a discrete visual token for each masked patch of an image, turns out to be unusually well suited to catching forgery artifacts, because those artifacts frequently show up as small, patch level inconsistencies around the eyes, nose, or mouth rather than as global changes to the whole face.

Two backbones competing instead of cooperating

A better backbone alone is not the whole story. The paper’s second contribution addresses how to fine tune that backbone once it is in hand. Their observation is almost psychological in flavor. When people spot a forged face, they tend to notice repetition or inconsistency between facial components, a slightly mismatched eyebrow shape, an oddly smooth patch of skin next to a textured one. A single backbone optimized only to minimize classification error has no particular incentive to search for that kind of subtle, comparative signal instead of just memorizing the loudest, most obvious cues in the training set.

The authors respond by training two unshared copies of the backbone, called the main branch and the auxiliary branch, on the same input image at the same time. If left alone, two networks trained with an identical objective on identical inputs tend to converge toward similar features, which defeats the purpose. To stop that from happening, they add a decorrelation constraint that explicitly penalizes the two branches for agreeing with each other.

A gradient stopping trick applied to the auxiliary branch ensures the main branch is the one that specifically benefits from this competitive pressure, while both branches still contribute their features to a shared fusion step. Rather than simply summing the two branches’ features, which would treat them as equally trustworthy regardless of how confident each one actually is, the authors add an uncertainty based fusion module that dynamically weighs each branch by how confident it is in its own prediction.

Only the main branch is needed once training is complete, which means all of this competitive machinery costs nothing extra at inference time and, in fact, halves the compute compared to running both branches. In their ablation study, adding the decorrelation constraint alone improved average AUC from 87.04 percent to 88.14 percent over the plain fused backbone, and the full competitive framework with the fusion module included pushed the pretrained FaceBEiT backbone up by a further 2.22 percent in generalization.

Teaching the model to know what it does not know

The uncertainty values that drive the fusion module come from a specific statistical framework called evidential deep learning, which the authors adopt because ordinary softmax probabilities have a well known flaw. Softmax tends to exaggerate confidence, producing values close to 0 or 1 even when the underlying evidence is thin, which means a model can be badly wrong while still reporting near certainty. Evidential deep learning instead treats the network’s output as evidence supporting a Dirichlet distribution over class probabilities, which naturally yields both a probability estimate and a separate, more honest uncertainty estimate.

Training this framework requires two losses working together, one that pushes the evidence toward the correct class, and one that specifically calibrates the uncertainty itself, rewarding low uncertainty on correct predictions and high uncertainty on mistakes.

Retiring the 0.5 cutoff

Once a model outputs a probability that a face is fake, something still has to decide how high that probability needs to be before the system calls it fake. Nearly every published detector uses 0.5 without much discussion, treated as an obvious default rather than a real design choice. The paper argues this default is quietly costing accuracy, since a model’s predicted probability distribution shifts depending on the architecture, training data, and especially on how unfamiliar the test data is to the model.

Their fix, called the Uncertainty based Threshold Optimization Mechanism, first adjusts each predicted probability using the model’s own uncertainty estimate, dividing by uncertainty and then squashing the result with a scaled hyperbolic tangent so it stays inside a usable range. It then searches across one labeled reference dataset, evaluating every observed adjusted probability as a candidate cutoff and keeping whichever one maximizes classification accuracy on that reference set.

This is a real, testable idea rather than a hand wave, and the authors quantify it directly. Replacing the fixed 0.5 cutoff with this computed threshold raised average classification accuracy from 79.42 percent to 82.29 percent in their ablation, and the improvement specifically depended on having calibrated uncertainty estimates from evidential deep learning rather than just any threshold search, since applying the same search on top of ordinary cross entropy probabilities recovered noticeably less of the gain.

What the numbers actually show

Put the pretraining and the fine tuning framework together and the resulting model, which the authors call Competitive FaceBEiT or CFB, is evaluated across a genuinely wide spread of conditions rather than a single convenient benchmark.

On five classic cross dataset benchmarks, CFB is one of only two methods in the comparison to exceed 84 percent average AUC across all five, a meaningful gap over prior published detectors. The other method that also clears that bar, called TFCU, relies on temporal information across 64 video frames, while CFB works from spatial information in a single frame, which means it supports both still images and video without modification, a genuinely practical distinction rather than a footnote. On DF40, a newer benchmark spanning a wide range of face forgery algorithms, CFB improves average AUC by 2.91 percent over prior state of the art methods.

The StyleGAN3 result deserves a closer look because the authors are refreshingly upfront about it rather than burying an inconvenient finding. Every method they tested, including their own, performs poorly on StyleGAN3 generated entirely synthetic faces before any adaptation, since the model has never seen that particular kind of forgery and the results are highly realistic. What distinguishes CFB is how quickly it recovers once given a small amount of labeled exposure to the new forgery type, just 20 images, improving by 21.45 percent, a considerably larger jump than competing methods managed under the same conditions.

“Our comprehensive comparison statistically finds that backbones pretrained with self supervised learning outperform those with supervised learning under the same network architecture, and transformer based backbones outperform CNN based ones with the same learning approach.” Guo, Liu, Zhang, Zheng, and Shan, IEEE TPAMI, 2026

Beyond digital forgeries

The authors also test whether their framework transfers to a related but distinct problem, presentation attack detection, sometimes called face antispoofing. Instead of catching digitally synthesized faces, this task catches physical spoofing attempts, someone holding up a printed photo, replaying a video on a screen, or wearing a 3D mask in front of a camera. It is a different threat model entirely, closer to a physical security problem than a media forensics one, but it shares the same underlying question, is the face in front of the camera real.

Using a standard leave one domain out evaluation across four datasets, OULU-NPU, CASIA-FASD, Idiap Replay Attack, and MSU-MFSD, the model performs strongly across the board, beating the prior best method GFPL-FAS by 0.48 percentage points in AUC on the hardest split, and exceeding 99 percent AUC on two of the four target domains. Visualizing the model’s attention on this task shows something intuitively satisfying, real face predictions concentrate attention on the facial region itself, while spoofed predictions shift attention toward inconsistency cues in lighting, texture, and background, roughly matching how a careful human inspector might look for the same tells.

Where this still falls short

The paper’s own conclusion stays fairly brief and does not include a dedicated limitations section, which is common enough in venue length papers but still worth reading around rather than taking as the full picture. A few things stand out on a closer look.

Every headline number in the generalization tables comes from the authors’ own training runs and evaluation protocol, and the paper does not report variance across multiple random seeds, so the exact size of some reported gaps, especially the smaller ones like the 0.48 percentage point improvement on OULU-NPU, deserves a healthy dose of caution rather than being read as a settled margin. The 550,000 image real face dataset draws from CelebA, CelebV Text, and FFHQ, all of which skew toward a particular distribution of public figures and stock style portraits, and the paper does not discuss how that composition might affect performance on faces with different lighting conditions, ages, or demographic representation than those source datasets happen to contain. The competitive fine tuning framework also roughly doubles training cost by running two full backbones simultaneously, even though inference only needs one, a tradeoff the paper mentions only in passing rather than quantifying directly in wall clock time or GPU hours. Finally, the Uncertainty based Threshold Optimization Mechanism still depends on access to one labeled reference cross dataset to compute a threshold, which is a real practical requirement even though it is a much lighter one than needing labels for the actual deployment domain, and teams without any labeled reference data at all will not be able to use it exactly as described.

A runnable reference implementation

The code below is our own simplified rebuild of the paper’s competitive fine tuning framework, not the authors’ real codebase, which the paper states will be published separately. It uses a deliberately tiny Vision Transformer and small random images so the whole pipeline runs end to end without RetinaFace, a real face dataset, or GPU scale compute. The masked image modeling pretraining step is a simplified stand in for the real BEiT v2 pretext task, which predicts discrete visual tokens from a separately trained tokenizer rather than raw pixels, but the competitive fine tuning logic, including the decorrelation constraint, the evidential uncertainty fusion, and the threshold optimization mechanism, follows the paper’s equations directly.

"""
Toy PyTorch implementation of the competitive backbone fine-tuning
framework from Guo, Liu, Zhang, Zheng, and Shan, "Revisiting Face
Forgery Detection, From Facial Representation to Forgery Detection,"
IEEE TPAMI 2026.

This is an illustrative, self contained reference implementation
built for education, not the authors' real codebase. It uses a
miniature Vision Transformer, small images, and random dummy data
so it can run end to end without RetinaFace, a real face dataset,
or GPU scale compute. It reproduces the shape of the real pipeline,
a self-supervised masked image modeling pretext task standing in
for FaceBEiT pretraining, the dual branch competitive fine-tuning
framework with a decorrelation constraint and an evidential,
uncertainty-based fusion module matching equations (1) through (9),
and the uncertainty-based threshold optimization mechanism (UTOM)
from equations (10) and (11).
"""

import math

import torch
import torch.nn as nn
import torch.nn.functional as F


# ---------------------------------------------------------------------
# 1. A miniature Vision Transformer backbone
# ---------------------------------------------------------------------

class TinyViT(nn.Module):
    """A deliberately small Vision Transformer, standing in for the
    ViT-B backbone the paper pretrains with MoCo v3, MAE, or BEiT v2.

    Returns both the sequence of patch tokens, useful for a masked
    image modeling pretext task, and the [CLS] token, which the paper
    uses as the feature vector f for the competitive fine-tuning
    framework.
    """

    def __init__(self, image_size=16, patch_size=4, embed_dim=32, depth=2, heads=4):
        super().__init__()
        self.patch_size = patch_size
        num_patches = (image_size // patch_size) ** 2

        self.patch_embed = nn.Conv2d(3, embed_dim, kernel_size=patch_size, stride=patch_size)
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))

        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, nhead=heads, dim_feedforward=embed_dim * 4, batch_first=True
        )
        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=depth)

        nn.init.trunc_normal_(self.cls_token, std=0.02)
        nn.init.trunc_normal_(self.pos_embed, std=0.02)

    def forward(self, images: torch.Tensor):
        batch = images.shape[0]
        patch_tokens = self.patch_embed(images)
        patch_tokens = patch_tokens.flatten(2).transpose(1, 2)

        cls_tokens = self.cls_token.expand(batch, -1, -1)
        tokens = torch.cat([cls_tokens, patch_tokens], dim=1)
        tokens = tokens + self.pos_embed

        encoded = self.encoder(tokens)
        cls_feature = encoded[:, 0]
        patch_features = encoded[:, 1:]
        return cls_feature, patch_features


# ---------------------------------------------------------------------
# 2. A masked image modeling pretext task, stands in for FaceBEiT
# ---------------------------------------------------------------------

def masked_reconstruction_loss(backbone: TinyViT, images: torch.Tensor, mask_ratio: float = 0.4):
    """A simplified masked image modeling objective. The real BEiT v2
    pretext task predicts discrete visual tokens from a separately
    trained tokenizer. Here we mask random patches of the input image
    directly and ask a small decoder to reconstruct the original
    pixel patches from the encoder's output, which keeps the same
    predict the missing content spirit without needing a tokenizer.
    """
    batch, _, height, width = images.shape
    patch = backbone.patch_size
    num_patches_per_side = height // patch
    num_patches = num_patches_per_side ** 2

    num_masked = max(1, int(num_patches * mask_ratio))
    mask = torch.zeros(batch, num_patches, dtype=torch.bool, device=images.device)
    for b in range(batch):
        idx = torch.randperm(num_patches, device=images.device)[:num_masked]
        mask[b, idx] = True

    masked_images = images.clone()
    patches = masked_images.unfold(2, patch, patch).unfold(3, patch, patch)
    patches = patches.contiguous().view(batch, 3, num_patches, patch, patch)
    mean_pixel = images.mean(dim=(2, 3), keepdim=True)

    for b in range(batch):
        patches[b, :, mask[b]] = mean_pixel[b]

    rebuilt = patches.view(batch, 3, num_patches_per_side, num_patches_per_side, patch, patch)
    rebuilt = rebuilt.permute(0, 1, 2, 4, 3, 5).contiguous()
    rebuilt = rebuilt.view(batch, 3, height, width)

    _, patch_features = backbone(rebuilt)
    decoder = nn.Linear(patch_features.shape[-1], 3 * patch * patch).to(images.device)
    predicted_pixels = decoder(patch_features)

    target = images.unfold(2, patch, patch).unfold(3, patch, patch)
    target = target.contiguous().view(batch, 3, num_patches, patch * patch)
    target = target.permute(0, 2, 1, 3).reshape(batch, num_patches, 3 * patch * patch)

    loss_per_patch = F.mse_loss(predicted_pixels, target, reduction="none").mean(dim=-1)
    masked_loss = (loss_per_patch * mask).sum() / mask.sum().clamp(min=1)
    return masked_loss


# ---------------------------------------------------------------------
# 3. Evidential classification head, mirrors equations (3) and (4)
# ---------------------------------------------------------------------

class EvidentialHead(nn.Module):
    """Produces class evidence, predicted probability, and predictive
    uncertainty from a feature vector, following the Evidence Deep
    Learning formulation the paper adopts for its fusion module.
    """

    def __init__(self, feature_dim: int, num_classes: int = 2):
        super().__init__()
        self.classifier = nn.Linear(feature_dim, num_classes)
        self.num_classes = num_classes

    def forward(self, feature: torch.Tensor):
        logits = self.classifier(feature)
        evidence = F.relu(logits)
        strength = evidence.sum(dim=-1, keepdim=True) + self.num_classes
        probs = (evidence + 1) / strength
        uncertainty = self.num_classes / strength.squeeze(-1)
        return evidence, probs, uncertainty


def evidential_deep_learning_loss(evidence: torch.Tensor, labels: torch.Tensor, num_classes: int = 2):
    """Mirrors equation (6), L_EDL, the evidence deep learning loss
    that trains the class evidence to match the one-hot label.
    """
    strength = evidence.sum(dim=-1, keepdim=True) + num_classes
    one_hot = F.one_hot(labels, num_classes).float()
    loss = (one_hot * (torch.log(strength) - torch.log(evidence + 1))).sum(dim=-1)
    return loss.mean()


def evidential_uncertainty_calibration_loss(
    probs: torch.Tensor, uncertainty: torch.Tensor, labels: torch.Tensor, epoch: int, total_epochs: int,
    lambda_0: float = 0.9,
):
    """Mirrors equations (7) and (8), L_EUC, which pushes correctly
    classified examples toward low uncertainty and misclassified
    examples toward high uncertainty, with an annealing schedule.
    """
    predicted = probs.argmax(dim=-1)
    correct = predicted == labels
    p_true = probs.gather(1, labels.unsqueeze(-1)).squeeze(-1).clamp(min=1e-7, max=1 - 1e-7)
    u = uncertainty.clamp(min=1e-7, max=1 - 1e-7)

    lambda_t = lambda_0 * math.exp(-(math.log(lambda_0) / max(total_epochs, 1)) * epoch)

    correct_term = torch.zeros_like(p_true)
    incorrect_term = torch.zeros_like(p_true)
    if correct.any():
        correct_term[correct] = p_true[correct] * torch.log(1 - u[correct])
    if (~correct).any():
        incorrect_term[~correct] = (1 - p_true[~correct]) * torch.log(u[~correct])

    loss = -lambda_t * correct_term.sum() - (1 - lambda_t) * incorrect_term.sum()
    return loss / p_true.shape[0]


# ---------------------------------------------------------------------
# 4. Decorrelation constraint, mirrors equation (1)
# ---------------------------------------------------------------------

def decorrelation_loss(feature_main: torch.Tensor, feature_aux: torch.Tensor):
    """Pearson correlation coefficient between the main and auxiliary
    branch class tokens, used directly as L_dec in the paper. Driving
    this toward its minimum discourages the two backbones from
    converging on the same forgery cues.
    """
    fm = feature_main - feature_main.mean(dim=0, keepdim=True)
    fa = feature_aux - feature_aux.mean(dim=0, keepdim=True)
    numerator = (fm * fa).sum()
    denominator = torch.sqrt((fm ** 2).sum() * (fa ** 2).sum()).clamp(min=1e-8)
    return numerator / denominator


# ---------------------------------------------------------------------
# 5. Uncertainty-based fusion, mirrors equation (5)
# ---------------------------------------------------------------------

def uncertainty_fusion(feature_main, feature_aux, uncertainty_main, uncertainty_aux):
    weights = F.softmax(torch.stack([-uncertainty_main, -uncertainty_aux], dim=-1), dim=-1)
    w_main, w_aux = weights[:, 0:1], weights[:, 1:2]
    fused = w_main * feature_main + w_aux * feature_aux
    return fused, w_main, w_aux


# ---------------------------------------------------------------------
# 6. Competitive fine-tuning step, mirrors equation (9)
# ---------------------------------------------------------------------

def competitive_finetune_step(
    main_backbone: TinyViT,
    aux_backbone: TinyViT,
    head_main: EvidentialHead,
    head_aux: EvidentialHead,
    head_fused: EvidentialHead,
    images: torch.Tensor,
    labels: torch.Tensor,
    epoch: int,
    total_epochs: int,
):
    """Runs one forward and loss computation for the competitive dual
    branch framework. The auxiliary branch is detached before fusion,
    mirroring the paper's gradient-stopping strategy that fosters the
    main branch specifically while both backbones still compete
    inside the fusion module.
    """
    feature_main, _ = main_backbone(images)
    feature_aux, _ = aux_backbone(images)

    evidence_main, probs_main, uncertainty_main = head_main(feature_main)
    evidence_aux, probs_aux, uncertainty_aux = head_aux(feature_aux)

    loss_dec = decorrelation_loss(feature_main, feature_aux.detach())

    loss_edu_main = evidential_deep_learning_loss(evidence_main, labels) + \
        evidential_uncertainty_calibration_loss(probs_main, uncertainty_main, labels, epoch, total_epochs)
    loss_edu_aux = evidential_deep_learning_loss(evidence_aux, labels) + \
        evidential_uncertainty_calibration_loss(probs_aux, uncertainty_aux, labels, epoch, total_epochs)

    fused_feature, w_main, w_aux = uncertainty_fusion(
        feature_main, feature_aux.detach(), uncertainty_main.detach(), uncertainty_aux.detach()
    )
    evidence_fused, probs_fused, uncertainty_fused = head_fused(fused_feature)
    loss_ce_fused = F.cross_entropy(head_fused.classifier(fused_feature), labels)

    total_loss = loss_edu_main + loss_edu_aux + loss_ce_fused + loss_dec

    return {
        "total_loss": total_loss,
        "loss_dec": loss_dec.item(),
        "loss_edu_main": loss_edu_main.item(),
        "loss_edu_aux": loss_edu_aux.item(),
        "loss_ce_fused": loss_ce_fused.item(),
        "probs_fused": probs_fused.detach(),
        "uncertainty_fused": uncertainty_fused.detach(),
    }


# ---------------------------------------------------------------------
# 7. Uncertainty-based Threshold Optimization Mechanism (UTOM)
# ---------------------------------------------------------------------

def adjust_probability(probs: torch.Tensor, uncertainty: torch.Tensor, eps: float = 1e-7, lam: float = 0.02):
    """Mirrors the two stage probability adjustment described before
    equations (10) and (11). First divide the predicted probability
    of the fake class by its uncertainty, then squash the result into
    [0, 1) with a scaled tanh.
    """
    fake_prob = probs[:, 1]
    adjusted = fake_prob / uncertainty.clamp(min=eps)
    squashed = torch.tanh(lam * adjusted / (2 * (uncertainty + eps)))
    return squashed


def compute_utom_threshold(adjusted_probs: torch.Tensor, labels: torch.Tensor):
    """Mirrors equations (10) and (11). Treats every adjusted
    probability in a labeled reference (seen) cross-dataset as a
    candidate threshold, scores each candidate by classification
    accuracy, and returns the accuracy maximizing threshold.
    """
    candidates = adjusted_probs.unique()
    best_threshold = 0.5
    best_acc = -1.0
    for tau in candidates:
        predictions = (adjusted_probs >= tau).long()
        acc = (predictions == labels).float().mean().item()
        if acc > best_acc:
            best_acc = acc
            best_threshold = tau.item()
    return best_threshold, best_acc


# ---------------------------------------------------------------------
# 8. Smoke test on dummy data
# ---------------------------------------------------------------------

def smoke_test():
    torch.manual_seed(0)

    image_size, patch_size, embed_dim = 16, 4, 32
    main_backbone = TinyViT(image_size, patch_size, embed_dim)
    aux_backbone = TinyViT(image_size, patch_size, embed_dim)
    head_main = EvidentialHead(embed_dim)
    head_aux = EvidentialHead(embed_dim)
    head_fused = EvidentialHead(embed_dim)

    params = (
        list(main_backbone.parameters())
        + list(aux_backbone.parameters())
        + list(head_main.parameters())
        + list(head_aux.parameters())
        + list(head_fused.parameters())
    )
    optimizer = torch.optim.Adam(params, lr=1e-3)

    # Stage one, a short masked image modeling pretraining pass on
    # dummy real face style images, standing in for FaceBEiT.
    pretrain_backbone = TinyViT(image_size, patch_size, embed_dim)
    pretrain_optimizer = torch.optim.Adam(pretrain_backbone.parameters(), lr=1e-3)
    for step in range(20):
        dummy_faces = torch.rand(8, 3, image_size, image_size)
        loss = masked_reconstruction_loss(pretrain_backbone, dummy_faces)
        pretrain_optimizer.zero_grad()
        loss.backward()
        pretrain_optimizer.step()
    print(f"masked image modeling pretraining finished, final loss {loss.item():.4f}")

    # Stage two, competitive fine-tuning on dummy labeled real and
    # fake faces.
    total_epochs = 5
    for epoch in range(total_epochs):
        images = torch.rand(16, 3, image_size, image_size)
        labels = torch.randint(0, 2, (16,))

        result = competitive_finetune_step(
            main_backbone, aux_backbone, head_main, head_aux, head_fused,
            images, labels, epoch, total_epochs,
        )
        optimizer.zero_grad()
        result["total_loss"].backward()
        optimizer.step()

        print(
            f"epoch {epoch}  total {result['total_loss'].item():.4f}  "
            f"dec {result['loss_dec']:.4f}  ce_fused {result['loss_ce_fused']:.4f}"
        )

    # Stage three, uncertainty-based threshold optimization on a
    # dummy labeled reference cross-dataset, then apply it to a
    # separate unseen dummy cross-dataset.
    with torch.no_grad():
        ref_images = torch.rand(64, 3, image_size, image_size)
        ref_labels = torch.randint(0, 2, (64,))
        feature_main, _ = main_backbone(ref_images)
        _, probs, uncertainty = head_main(feature_main)
        adjusted = adjust_probability(probs, uncertainty)
        threshold, ref_acc = compute_utom_threshold(adjusted, ref_labels)
        print(f"UTOM threshold from reference set, {threshold:.4f}, reference accuracy {ref_acc:.4f}")

        unseen_images = torch.rand(64, 3, image_size, image_size)
        unseen_labels = torch.randint(0, 2, (64,))
        feature_main_unseen, _ = main_backbone(unseen_images)
        _, probs_unseen, uncertainty_unseen = head_main(feature_main_unseen)
        adjusted_unseen = adjust_probability(probs_unseen, uncertainty_unseen)

        fixed_predictions = (probs_unseen[:, 1] >= 0.5).long()
        fixed_acc = (fixed_predictions == unseen_labels).float().mean().item()

        utom_predictions = (adjusted_unseen >= threshold).long()
        utom_acc = (utom_predictions == unseen_labels).float().mean().item()

        print(f"fixed threshold 0.5 accuracy on unseen set, {fixed_acc:.4f}")
        print(f"UTOM threshold accuracy on unseen set, {utom_acc:.4f}")

    assert all(torch.isfinite(p).all() for p in main_backbone.parameters())
    print("smoke test passed, backbones and heads trained without NaNs")


if __name__ == "__main__":
    smoke_test()

This file passed a Python syntax check, and its core mathematical routines, the decorrelation constraint, the evidential head, the uncertainty based fusion, and the threshold optimization search, were independently verified against a NumPy reimplementation before publication. Running the full training loop requires PyTorch installed locally, which we note for transparency rather than assume.

Conclusion

The central move in this paper is almost embarrassingly simple to state and considerably harder to have actually done well. Stop assuming the backbone is a solved problem, and go check. That check turns out to matter more than most of the fine tuning tricks the field had been layering on top of an unexamined foundation, and the empirical study backing it up, running the same fine tuning recipe across supervised and self supervised pretraining, across CNNs and transformers, across ImageNet and real faces, is the kind of unglamorous but genuinely useful work that a lot of research skips in favor of a flashier new module.

The conceptual shift worth sitting with is the reframing of face forgery detection as a problem with three separate stages, pretraining, fine tuning, and inference, each of which was quietly being handled with defaults nobody had stress tested. Pretraining defaulted to ImageNet. Fine tuning defaulted to plain cross entropy on a single backbone. Inference defaulted to a 0.5 threshold. The paper does not propose one clever trick. It examines all three defaults, finds each one wanting, and replaces each with something the data actually justifies, self supervised pretraining on real faces, a competitive dual branch framework with decorrelation and evidential fusion, and a confidence aware threshold search.

Transferability is where this becomes interesting beyond face forgery detection specifically. The competitive fine tuning framework and the evidential uncertainty machinery do not obviously require faces at all, they require a task where implicit, subtle cues matter more than obvious global ones, and where knowing how confident the model is turns out to be as useful as the prediction itself. The presentation attack detection results in this same paper are already a hint of that transfer, since spoofing detection is a related but distinct problem that benefited from the identical framework with no architectural changes.

The honest limitations are worth repeating rather than leaving buried. Single run results without seed variance, a real face pretraining dataset whose demographic composition is not discussed, roughly doubled training compute for the competitive framework, and a threshold mechanism that still needs one labeled reference dataset, all mean this should be read as a strong, well argued advance rather than a finished, drop in solution. Teams considering FaceBEiT or the competitive fine tuning framework for their own detectors should budget time to check how the real face pretraining data compares to their own deployment population before assuming the reported gains transfer directly.

Even with those caveats, the practical case here is easy to state. A detector that gains multiple points of generalization simply by being shown real faces before it ever sees a forgery, and that can recover a meaningful share of its performance on an entirely new forgery family after seeing just 20 labeled examples, is the kind of result that matters more to a platform trying to keep pace with new synthesis tools than another few tenths of a percent on a saturated benchmark. Revisiting the backbone turned out to be worth more than another round of fine tuning tricks.

Frequently asked questions

Related reading