S4ST: The Simple Scaling Trick That Fools AI Vision Models

S4ST · Scaling Based Adversarial Transfer

The problem with fooling a model you cannot see

Adversarial examples have been an open problem in deep learning since Szegedy and colleagues first described them over a decade ago. Add a small, carefully chosen pattern of noise to an image, invisible to a human eye, and a neural network that classified it correctly a moment ago suddenly gets it wrong. What makes this dangerous in practice is transferability. An attacker rarely gets to see the exact model running behind a face recognition gate, a content filter, or an autonomous driving system. Instead they train an adversarial example against a stand in model they do control, called a surrogate, and hope the same manipulated image also fools the real target.

Untargeted attacks, where any wrong answer counts as a win, are comparatively well understood at this point. Targeted attacks are a different animal. Forcing a model to output one specific wrong label, say convincing it that a vase is a church window rather than just anything other than a vase, requires steering the image into a narrow region of an unknown model’s decision space. That is a much harder needle to thread, and it explains why targeted transferable attacks have received far less research attention than their untargeted cousins, despite arguably posing the more serious real world threat.

Two flawed traditions, and the access they quietly assume

Data reliant methods

One family of solutions throws data and compute at the problem. Universal perturbation methods optimize a single manipulation across enormous datasets. Generative approaches, such as the M3D method the authors cite, train a dedicated image generator for every target label, a process that can eat ten hours of GPU time and fifty thousand training images per label. Other approaches fine tune the surrogate model itself to better match the statistics of the intended victim. All three paths share the same weakness. They typically rely on data drawn from the same distribution as the victim’s own training set, which means the attacker is quietly leaning on information a strict black box scenario is not supposed to provide.

Transformation based methods that still peek

A second, lighter weight family skips the training and instead applies input transformations, cropping, rotating, adjusting color, while optimizing the perturbation through gradient descent on the surrogate. These methods are branded data free, and in one sense they are, no victim training data required. But the authors point out an uncomfortable detail. Choosing which transformations to use, and how strongly to apply them, has historically depended on empirical trial and error against the very black box models the attack is supposed to be blind to. That tuning process is itself a form of information leakage, and it compromises any fair comparison between methods.

This is the gap the NUDT team set out to close. Could an attacker design and tune an entirely new transformation using only signals available from the surrogate model, with no queries to the target and no external data at all?

Two blind measurements that replace guesswork

The paper’s methodological contribution is a pair of diagnostic scores, both computable using nothing but the surrogate model the attacker already controls.

Self alignment, or does the transform undo overfitting

During a standard iterative attack, the surrogate’s confidence in the wrong target label rockets toward one very quickly. That rapid saturation is actually a symptom of overfitting to the surrogate, and it is exactly what destroys transferability, because the perturbation becomes finely tuned to quirks of one specific model rather than to features a different architecture would also respond to. The authors show that applying an image transformation during the attack pulls the surrogate’s internal features back toward how it represents the clean, unmodified image. They call this restoring effect self alignment, and it can be measured entirely inside the surrogate, using a nearest neighbor overlap between feature representations, without ever touching the target model.

When the team ran this measurement across twelve basic transformations, scaling stood apart. It produced the strongest alignment score and it also kept that strong score across a much wider range of scaling intensities than rotation, perspective warps, or color adjustments managed at their own effective settings.

Self transferability, or which transforms are actually redundant

Knowing that a transformation helps individually is not the same as knowing how to combine several of them without wasting effort on redundant choices. For that, the team introduces self transferability, measured by generating adversarial examples with one set of transformations and testing how well they hold up against the surrogate defended by a different set, treating the second set as a stand in defensive filter.

The correlation results carried a genuinely counterintuitive lesson. Geometric transformations, rotation, shear, translation, correlate strongly with each other even when they look unrelated on paper, meaning stacking several of them buys little. Color transformations behave the same way among themselves. But geometric and color families barely correlate with one another, which is precisely the gap S4ST was built to exploit, pairing a dominant scaling operation with a small set of color adjustments chosen because they add the least redundancy.

Inside S4ST, one core idea and two supporting acts

S4ST Base, bidirectional scaling

Rather than only shrinking or only enlarging an image the way earlier scaling based methods (DI and RDI in the paper’s terminology) did, S4ST Base moves in both directions. Given an intensity parameter r greater than one, the method samples a scale factor from a uniform range. When the sampled factor pulls the image smaller, it downsizes and pads the remainder with zeros. When it pulls larger, it crops a smaller patch from the image first and then enlarges that patch back to the original size, rather than naively upscaling the whole picture, which keeps the added computation small while forcing the attack to exploit local detail more aggressively. The authors trace this dual direction design to a concrete visual effect, shrinking an image tends to make the model latch onto dominant, whole scene semantics, while enlarging it exposes competition between multiple plausible local objects.

S4ST Aug, five carefully chosen companions

On top of the base scaling operation, S4ST occasionally layers one of five color and orientation transformations, flipping, brightness, contrast, saturation, or hue adjustment. None of these were chosen by intuition. Each candidate was scored using the self transferability measure described above, and the five with the lowest correlation to scaling were kept. Translation and cropping were deliberately left out, since both overlap too heavily with what scaling already does.

S4ST Block, dividing the canvas

The final component splits the image into a grid, six blocks arranged two by three in the tuned configuration, and applies the base scaling operation independently within each block. This adds diversity to the attack while keeping the overall image coherent, and the authors find its benefit grows with grid size up to a point before too many blocks start interfering with the optimization itself.

Every one of these choices, including the final numeric settings, probability 0.9 for applying the base scale, intensity 1.7, full application of the color pool, and a six block grid, came out of a black box Bayesian search that optimized average self transferability across the twelve basic transformations rather than any real target model. The full search took a little over two hours on a single RTX 4090, and the authors show the resulting proxy tracks true black box success closely enough to trust it as a stand in during design.

How well it actually holds up

The headline benchmark uses the ImageNet-Compatible dataset, a thousand images with target labels released for the original NIPS 2017 adversarial competition, attacking outward from a ResNet-50 surrogate toward fourteen unrelated victim models spanning both convolutional networks and vision transformers.

The team also tested S4ST against defenses that specifically try to blunt transfer attacks, JPEG compression, augmented training regimes, and adversarially trained models. S4ST kept a high success rate even as JPEG compression grew aggressive enough to noticeably hurt the victim model’s accuracy on clean images, a result the authors describe as complicating the usual generalization versus robustness bargain defenders rely on. Against ensembles of adversarially trained and stylized trained models, self ensemble variants of S4ST, which average gradients across several transformed copies per step rather than several separate surrogate models, came close to matching the performance of a true multi model ensemble, a meaningful practical advantage since gathering several genuinely diverse surrogate models is often the hardest part of mounting a real attack.

Why scaling wins, and the training habit that explains it

Neither CNNs nor ViTs possess inherent scale invariance. Any tolerance either architecture develops toward resized inputs has to be learned, and the authors point to one training ingredient shared almost universally across modern vision pipelines as the likely source, RandomResizedCrop. This augmentation randomly crops and resizes training images before they ever reach the network, and it appears not just in ordinary supervised classifier training but across self supervised approaches like MAE and DINO and vision language systems like CLIP. Because the augmentation is so widely adopted, it becomes a shared inductive bias baked into nearly every publicly available vision model, regardless of architecture family or training objective.

The team ran a direct controlled test to confirm the causal link, training three variants of ResNet-50 with standard RandomResizedCrop, half strength cropping, and no cropping at all. Accuracy dropped noticeably without the augmentation, as expected. But the model trained without it also became measurably harder to fool through scaling based attacks when used as a victim, while still remaining an effective attacker when used as a surrogate. That asymmetry points to something structural rather than accidental. Scale diversity baked in by training is doing real work for generalization, and the same diversity is exactly what a scaling centered attack learns to exploit.

Clinical translation gap

Among the paper’s boundary case experiments, one attacks a chest X-ray classifier trained on the NIH CXR14 dataset, specifically a cardiomegaly detection task, using a DenseNet-121 surrogate trained on ordinary photographs and transferring toward four victim models trained with masked image modeling on X-ray data. The evaluation covered five hundred test images, thirty three positive and four hundred sixty seven negative, a small, single label, single dataset slice used purely to probe whether the attack’s underlying logic survives a dramatic shift away from natural photographs. It succeeded well enough to matter as a research finding, but the setup is nowhere near the scale, diversity, or clinical validation rigor that would justify any claim about real diagnostic systems in use today. The gap between a laboratory transfer test on a public benchmark and the layered safeguards around an actual hospital imaging pipeline remains wide, and the authors do not claim otherwise.

What this means if you build or defend vision systems

A few implications follow directly from the results rather than from speculation. First, the absence of API access or query logs from an attacker does not mean a system is safe from targeted manipulation, since S4ST was designed and tuned without any of that access. Second, common lightweight defenses like JPEG recompression should not be treated as reliable filters against a well designed scaling centered attack. Third, the paper’s own framing of a generalization versus robustness bargain suggests that hardening a production model against this class of attack could mean giving up some of the accuracy gained from aggressive training time augmentation, a tradeoff worth budgeting for rather than discovering after deployment. Finally, because the underlying weakness is tied to a training practice shared across architecture families, swapping a convolutional backbone for a vision transformer is not, by itself, a meaningful defense.

Where the method runs into real limits

The authors are candid about where S4ST’s assumptions break down. When the target domain departs sharply from natural photographs, as with face verification, directly reusing the natural image tuned parameters performs poorly, the correlation between the self transferability proxy and true black box performance drops to a weak 0.37. Recovering usable guidance required simplifying the framework, dropping the block and augmentation components to isolate the base scaling operation, which raised that correlation to 0.78, and finally substituting a narrower proxy built only from scaling’s own self transferability, which reached 0.97. In other words, the method’s black box design philosophy holds up under domain shift, but the specific parameters tuned for natural images do not transfer automatically, and a fresh round of the same analysis is required for each new domain.

On vision language models specifically, S4ST trails the strongest data reliant rival by 3.2 percentage points, the one evaluation setting in the paper where it does not lead outright. The chest X-ray and face verification tests, as noted above, rely on modest sample sizes and single surrogate architectures, appropriate for a boundary case demonstration but not for drawing conclusions about deployed systems in either domain. The attack also assumes a fairly generous perturbation budget by common convention, and the authors themselves note a visible tension between how transferable an adversarial image becomes and how imperceptible its underlying changes remain, an open question they leave for dedicated defense research rather than resolving here.

Building S4ST from scratch in PyTorch

The block below implements the three components of S4ST, the margin style loss the paper pairs with a translation invariant momentum attack, a crafting loop that mirrors the paper’s baseline optimizer, and a small evaluation routine. A synthetic smoke test at the bottom runs the entire pipeline against randomly initialized dummy classifiers so the code can be verified without downloading any real dataset or pretrained weights.

import torch
import torch.nn as nn
import torch.nn.functional as F
import random

# ----------------------------------------------------------------------
# S4ST-Base: bidirectional scaling with crop-and-upscale for enlargement
# ----------------------------------------------------------------------
class S4STBase(nn.Module):
    def __init__(self, prob=0.9, r=1.7):
        super().__init__()
        self.prob = prob
        self.r = r

    def forward(self, x):
        # x: (B, C, H, W) float tensor in [0, 1]
        if random.random() > self.prob:
            return x
        B, C, H, W = x.shape
        scale = random.uniform(1.0 / self.r, self.r)
        if scale < 1.0:
            new_h, new_w = int(H * scale), int(W * scale)
            resized = F.interpolate(x, size=(new_h, new_w), mode="bilinear", align_corners=False)
            pad_h, pad_w = H - new_h, W - new_w
            top = random.randint(0, pad_h)
            left = random.randint(0, pad_w)
            out = F.pad(resized, (left, pad_w - left, top, pad_h - top))
        else:
            crop_h, crop_w = int(H / scale), int(W / scale)
            top = random.randint(0, H - crop_h)
            left = random.randint(0, W - crop_w)
            patch = x[:, :, top:top + crop_h, left:left + crop_w]
            out = F.interpolate(patch, size=(H, W), mode="bilinear", align_corners=False)
        return out.clamp(0.0, 1.0)


# ----------------------------------------------------------------------
# S4ST-Aug: five low-redundancy companion transformations
# ----------------------------------------------------------------------
class S4STAug(nn.Module):
    def __init__(self, prob=1.0):
        super().__init__()
        self.prob = prob
        self.ops = [self._flip, self._brightness, self._contrast, self._saturation, self._hue]

    def _flip(self, x):
        dim = random.choice([2, 3])
        return torch.flip(x, dims=[dim])

    def _brightness(self, x):
        factor = random.uniform(0.0, 2.0)
        return (x * factor).clamp(0.0, 1.0)

    def _contrast(self, x):
        factor = random.uniform(0.0, 2.0)
        mean = x.mean(dim=[2, 3], keepdim=True)
        return ((x - mean) * factor + mean).clamp(0.0, 1.0)

    def _saturation(self, x):
        factor = random.uniform(0.0, 2.0)
        gray = x.mean(dim=1, keepdim=True).expand_as(x)
        return (gray + (x - gray) * factor).clamp(0.0, 1.0)

    def _hue(self, x):
        shift = random.uniform(-0.5, 0.5)
        return torch.roll(x, shifts=int(shift * x.shape[1]), dims=1)

    def forward(self, x):
        if random.random() > self.prob:
            return x
        op = random.choice(self.ops)
        return op(x)


# ----------------------------------------------------------------------
# S4ST-Block: apply S4ST-Base independently within a grid of blocks
# ----------------------------------------------------------------------
class S4STBlock(nn.Module):
    def __init__(self, base_op, grid=(2, 3)):
        super().__init__()
        self.base_op = base_op
        self.gh, self.gw = grid

    def forward(self, x):
        B, C, H, W = x.shape
        bh, bw = H // self.gh, W // self.gw
        out = x.clone()
        for i in range(self.gh):
            for j in range(self.gw):
                block = x[:, :, i * bh:(i + 1) * bh, j * bw:(j + 1) * bw]
                out[:, :, i * bh:(i + 1) * bh, j * bw:(j + 1) * bw] = self.base_op(block)
        return out


# ----------------------------------------------------------------------
# Full S4ST transform, chaining Base, Aug, and Block
# ----------------------------------------------------------------------
class S4ST(nn.Module):
    def __init__(self, pr=0.9, r=1.7, p_aug=1.0, grid=(2, 3)):
        super().__init__()
        self.base = S4STBase(prob=pr, r=r)
        self.aug = S4STAug(prob=p_aug)
        self.block = S4STBlock(S4STBase(prob=1.0, r=r), grid=grid)

    def forward(self, x):
        x = self.base(x)
        x = self.aug(x)
        x = self.block(x)
        return x


# ----------------------------------------------------------------------
# Margin-style targeted loss, softened cross entropy variant from the paper
# ----------------------------------------------------------------------
def margin_targeted_loss(logits, target, margin=10.0):
    target_logit = logits.gather(1, target.unsqueeze(1)).squeeze(1)
    others = logits.clone()
    others.scatter_(1, target.unsqueeze(1), float("-inf"))
    runner_up = others.max(dim=1).values
    loss = F.relu(runner_up - target_logit + margin)
    return loss.mean()


# ----------------------------------------------------------------------
# Translation invariant momentum crafting loop (T-MI baseline from the paper)
# ----------------------------------------------------------------------
def craft_targeted_adversarial(surrogate, x, target_label, transform,
                                 eps=16/255, alpha=2/255, steps=300, mu=1.0):
    surrogate.eval()
    x_orig = x.clone().detach()
    x_adv = x.clone().detach()
    grad_momentum = torch.zeros_like(x)
    ti_kernel = _gaussian_kernel(size=5).to(x.device)

    for step in range(steps):
        x_adv.requires_grad_(True)
        transformed = transform(x_adv)
        logits = surrogate(transformed)
        loss = margin_targeted_loss(logits, target_label)
        grad = torch.autograd.grad(loss, x_adv)[0]
        grad = F.conv2d(grad, ti_kernel.expand(x.shape[1], 1, -1, -1),
                         padding=ti_kernel.shape[-1] // 2, groups=x.shape[1])
        grad = grad / (grad.abs().mean(dim=[1, 2, 3], keepdim=True) + 1e-12)
        grad_momentum = mu * grad_momentum + grad
        x_adv = x_adv.detach() - alpha * grad_momentum.sign()
        x_adv = torch.clamp(x_adv, x_orig - eps, x_orig + eps)
        x_adv = torch.clamp(x_adv, 0.0, 1.0).detach()

    return x_adv


def _gaussian_kernel(size=5, sigma=1.0):
    coords = torch.arange(size).float() - size // 2
    g = torch.exp(-(coords ** 2) / (2 * sigma ** 2))
    kernel_1d = g / g.sum()
    kernel_2d = kernel_1d[:, None] @ kernel_1d[None, :]
    return kernel_2d.unsqueeze(0).unsqueeze(0)


# ----------------------------------------------------------------------
# Evaluation: targeted success rate against one or more victim models
# ----------------------------------------------------------------------
def evaluate_targeted_success(x_adv, target_label, victim_models):
    results = {}
    with torch.no_grad():
        for name, model in victim_models.items():
            model.eval()
            preds = model(x_adv).argmax(dim=1)
            success_rate = (preds == target_label).float().mean().item()
            results[name] = success_rate
    return results


# ----------------------------------------------------------------------
# Smoke test on dummy data, no real dataset or pretrained weights needed
# ----------------------------------------------------------------------
class DummyClassifier(nn.Module):
    def __init__(self, num_classes=10, seed=0):
        super().__init__()
        torch.manual_seed(seed)
        self.net = nn.Sequential(
            nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1), nn.Flatten(),
            nn.Linear(16, num_classes)
        )

    def forward(self, x):
        return self.net(x)


if __name__ == "__main__":
    torch.manual_seed(42)
    batch = torch.rand(4, 3, 64, 64)
    target = torch.tensor([3, 3, 3, 3])

    surrogate = DummyClassifier(seed=1)
    victims = {
        "victim_a": DummyClassifier(seed=2),
        "victim_b": DummyClassifier(seed=3),
    }

    s4st = S4ST(pr=0.9, r=1.7, p_aug=1.0, grid=(2, 3))
    adv = craft_targeted_adversarial(surrogate, batch, target, s4st, steps=20)
    scores = evaluate_targeted_success(adv, target, victims)

    print("Smoke test complete, targeted success rate per victim")
    print(scores)

Running the smoke test produces a dictionary of success rates for each dummy victim, confirming the transformation stack, the loss, the crafting loop, and the evaluation function all fit together correctly. On randomly initialized dummy models the success numbers are not meaningful in themselves, since a real trial needs a properly trained surrogate and genuine victim architectures, but the plumbing mirrors the paper’s actual pipeline closely enough to serve as a starting point for a faithful reimplementation.

The bigger picture this paper is pointing at

Strip away the specific numbers and S4ST leaves behind a fairly uncomfortable structural insight. Two decades of vision research converged on RandomResizedCrop and its relatives as a near universal recipe for training models that generalize well, and that convergence created a shared blind spot across architecture families rather than a flaw isolated to any one model. The paper’s two blind measurement tools, self alignment and self transferability, matter beyond the immediate scaling result, since they hand attackers, and just as usefully defenders, a way to evaluate any new transformation idea using only a surrogate model, no queries, no external data, no assumptions about the target’s internals required.

That symmetry cuts both ways. The same measurements that let S4ST’s designers skip empirical guesswork could just as easily let a defense team audit which of their own training choices are quietly creating attack surface, before an outside party finds it first. The paper frames its contribution around attack effectiveness, but the diagnostic framework underneath is arguably the more durable idea, a repeatable way to ask whether a given transformation choice helps or hurts transferability, answerable without ever exposing the target system to a single probing query.

Where this leaves practical model builders is not comfortable, but it is at least concrete. Scale augmentation during training is not optional in the current playbook, dropping it costs meaningful accuracy across nearly every task the field cares about. Yet this paper demonstrates, with a controlled ablation rather than a hand wave, that the exact same augmentation opens a measurable transfer attack channel. Reconciling those two facts, rather than picking one and ignoring the other, is the open engineering problem the paper leaves on the table.

The remaining open threads are the ones the authors flag honestly rather than paper over. Extending the black box design philosophy to domains further from natural photographs, medical imaging and biometrics among them, still requires a fresh round of self alignment and self transferability analysis rather than a drop in reuse of the natural image parameters. Closing the small gap against the strongest data reliant rival on vision language models remains unresolved. And the deeper tension the authors name outright, between how transferable a perturbation becomes and how visible its underlying changes remain, is left as a question for future defense minded research rather than something this paper resolves.

Read narrowly, S4ST is a faster, more effective transfer attack. Read at the level the authors clearly intend, it is a demonstration that a shared, decades old training habit created a shared vulnerability across the entire field of modern vision models, one that a sufficiently careful black box analysis can locate and exploit without ever touching the systems it eventually breaks.

Frequently asked questions

Related reading