BlackVIP Adapts Foundation Models Without Their Weights

The two assumptions almost every efficient tuning method quietly makes

Parameter efficient transfer learning promised something appealing, adapt a huge pretrained model to a new task by training a tiny number of extra parameters instead of the whole network. Visual prompt tuning, prefix tuning, adapters, and text side prompt learning for vision language models all deliver on that promise in different ways. What the paper points out, plainly, is that essentially all of them still assume two things that are often false in practice. First, that you can see and modify the model’s actual parameters or at least its internal architecture. Second, that you have enough memory to cache every intermediate activation needed to compute a gradient through the network during backpropagation.

Neither assumption survives contact with how most people actually access a frontier vision model today. Commercial vision APIs, proprietary internal tools, and models a company will not open source for competitive reasons all hand you a prediction and nothing else. Even when parameters are technically available, a large modern model can demand more memory for training than a typical user has, tiny learnable parameter count or not, because the memory cost of backpropagation comes from the activations you must store, not from the parameters you update. BlackVIP’s premise is that a genuinely useful adaptation method has to work under both constraints simultaneously, no parameter access and no assumption of ample memory.

A prompt generator instead of a prompt, the Coordinator

Rather than directly learning a static set of pixel values, BlackVIP learns a small neural network, called the Coordinator, that looks at each incoming image and generates a prompt tailored to it. The generated prompt is added to the image, the combined result is clipped back into a valid pixel range, and that modified image is what actually gets sent to the black box model.

Two ways to build the Coordinator

The original BlackVIP version pairs a frozen, publicly available self supervised vision encoder, specifically an ImageNet pretrained masked autoencoder, with a small trainable decoder built from convolution layers. The frozen encoder compresses an input image into a compact feature vector, that feature vector gets combined with a separate learnable task specific vector, and the decoder turns that combination into a full size visual prompt shaped exactly like the input image. Using a frozen, publicly available encoder rather than training one from scratch keeps the number of trainable parameters small, which matters directly for how efficiently the black box optimizer described below can estimate a gradient.

The newer variant, BlackVIP-SE, for statistical feature guided efficient prompt, swaps that neural encoder out entirely. Instead of running images through a separate pretrained network, it fits principal component analysis directly on the small set of few shot training images available for a task, producing a simple fixed projection matrix. That projection plays the same role the neural encoder played before, feeding compact per image statistics into the same convolutional decoder, but at a fraction of the computational cost since there is no second neural network forward pass required at all. The paper reports this swap reduces runtime meaningfully during both training and inference while matching BlackVIP’s classification accuracy closely across most benchmarks.

Why replace a rich, semantically trained encoder with plain statistics and not lose much accuracy? The authors’ explanation is that transfer learning success does not come only from reusing high level semantic features, some of it comes from lower level statistical regularities in the data, and for many benchmarks those regularities are enough to steer the prompt generator effectively, particularly on datasets whose images occupy a comparatively low dimensional manifold of visual variation, such as SVHN’s fairly uniform digit photographs.

Estimating a gradient you are never allowed to compute directly

Generating a good prompt is only half the problem. Training the Coordinator’s parameters requires knowing which direction reduces the loss, and ordinary backpropagation is exactly what a black box setting rules out, since it requires access to the model’s internal computation graph.

SPSA, two queries and a gradient estimate

The paper builds on Simultaneous Perturbation Stochastic Approximation, SPSA, a decades old technique from the optimization literature that estimates a gradient in a high dimensional parameter space using just two evaluations of the loss function, regardless of how many parameters there are. The trick is to perturb every parameter simultaneously in a random direction, evaluate the loss at that perturbed point and at the point perturbed in the opposite direction, and use the difference between the two loss values, scaled appropriately, as an estimate of the gradient along that random direction.

With only two forward calls to the model API per optimization step, SPSA turns the difference between those two outputs into a usable gradient estimate for updating the Coordinator’s parameters. This is what makes the whole approach genuinely black box, the target model is only ever queried for its output given an input, never inspected internally.

Gradient correction, borrowing a trick from accelerated gradient descent

Plain SPSA works reasonably well in many applications, but the authors observed it converges slowly in practice, and that problem gets worse in the high dimensional parameter spaces typical of neural network optimization, since each individual gradient estimate is noisy, built from a single random perturbation direction rather than the true gradient. Their fix, SPSA-GC, borrows an idea from Nesterov’s accelerated gradient method, computing the update direction from a point slightly ahead of the current parameters rather than from the current parameters directly, then blending that estimate into a running momentum term.

The practical effect, demonstrated on the classic Rosenbrock optimization benchmark, is that SPSA-GC converges faster and more stably than a comparable one sided gradient estimator used in prior black box tuning work, and holds up well even when Gaussian noise is deliberately injected into the loss to simulate the kind of noisy, minibatch style observations a real training run would encounter. Under that noise, the paper reports SPSA-GC remains close to the performance of an idealized optimizer given the true gradient, while a plain zeroth order baseline degrades noticeably faster as the noise grows.

What actually happens across 14 benchmarks and two synthetic stress tests

The headline evaluation runs across 14 transfer learning benchmarks covering natural images, specialized domains, structured recognition, and fine grained categories, using CLIP as the frozen target model in the main experiments.

A frequency domain analysis of the learned prompts adds a genuinely useful piece of interpretability the paper did not have to include. Visualizing the average frequency spectrum of prompts learned by BAR versus BlackVIP shows BAR’s fixed, frame shaped prompts concentrate almost entirely in the low frequency range, while BlackVIP’s input dependent, full image prompts spread into the middle and high frequency range as well. That difference lines up with where each method wins, BlackVIP tends to edge out competitors on fine grained recognition tasks that hinge on subtle detail, while it can slightly trail BAR on coarse, scene level understanding tasks where a low frequency, broadly applied nudge is apparently enough.

Why an input dependent prompt is also a robustness mechanism

The paper’s most conceptually interesting contribution sits in its theory section, and it is worth taking seriously even if you never touch the optimizer code. The authors show, under a set of fairly standard assumptions about how stochastic gradient descent behaves near a loss minimum, that the parameters of a learned visual prompt converge toward samples from a Gaussian distribution centered at the optimum. That mathematical fact turns out to connect directly to randomized smoothing, a well established technique for proving formal robustness guarantees, where a classifier’s prediction is made more stable by averaging its output over many copies of an input corrupted with Gaussian noise.

The argument, in plain terms, is that a prompted classifier behaves like an adaptive version of a randomized smoothing classifier, one where the center of that Gaussian noise distribution has been shifted in exactly the direction that reduces the downstream classification loss. Because BlackVIP’s prompt depends on the specific input rather than being one fixed pattern for an entire dataset, its effective robustness radius, the size of perturbation a prediction can tolerate before flipping, can adapt per sample rather than being fixed once for the whole dataset the way a universal prompt like VP’s would be. The paper backs this up empirically on a 16 shot EuroSAT classification task, showing BlackVIP holds up better than plain VP under added L2 noise, consistent with the theory’s prediction that a more flexible, input aware prompt should translate into a more flexible robustness guarantee.

What still costs more, and where to be careful

The paper is direct about BlackVIP’s real limitations. Query efficiency is the clearest one, since estimating a gradient from only two black box queries per step is fundamentally less information dense than computing an exact gradient, BlackVIP needs roughly 50 times more optimization iterations than a method with true gradient access to reach comparable final accuracy. For anyone paying per API call, that is a real cost to budget for, even though the paper’s own cost estimate, about 12 US dollars for 10,000 calls to roughly double or triple zero shot accuracy on a task, suggests the absolute dollar cost still tends to stay modest for typical few shot adaptation budgets.

BlackVIP-SE’s efficiency gain also comes with a specific caveat the authors flag directly rather than glossing over. On synthetic datasets built to contain explicit spurious correlations, such as Biased MNIST, BlackVIP-SE noticeably underperforms the original BlackVIP, even though the two are closely matched on ordinary real world benchmarks. The likely explanation is that simple statistical features from PCA are a weaker foothold for recognizing and correcting a spurious correlation than the richer, semantically trained representation the original encoder provides, so a practitioner who suspects their deployment data has that kind of bias should lean toward the heavier BlackVIP rather than the faster BlackVIP-SE.

Why the underlying idea travels beyond vision APIs

Strip away the vision specific details and BlackVIP is making a broader argument about adaptation under real world constraints rather than idealized ones. Instead of assuming a user can see and modify a model’s internals, it treats the model purely as a function that maps inputs to outputs, and asks what the most information efficient way to steer that function is using only calls to it. That framing applies to any black box service, not just vision APIs, anywhere a useful model exists behind an interface the end user cannot open.

The Coordinator’s core idea, generating a per input adaptation signal from a lightweight network rather than learning one static signal for an entire dataset, is a pattern that seems worth testing wherever a fixed, one size fits all intervention is currently the default. And the theoretical bridge the authors draw between this style of prompting and randomized smoothing suggests a more general principle, that adaptation mechanisms which respond to the specific input in front of them may carry robustness properties that fixed, input agnostic mechanisms simply cannot match, a claim worth testing well outside of image classification.

Complete PyTorch implementation

The following code is an original, simplified educational implementation inspired by the mechanisms described in the paper, the Coordinator’s prompt generation, the SPSA gradient estimator, and the SPSA-GC momentum correction. It is not the authors’ released code, since no public repository URL was found in the version of the paper reviewed here, and it runs on small dummy tensors as a sanity check of the logic, treating a simple toy function as the black box model being queried.

# blackvip_lite.py
# Educational reimplementation of the core BlackVIP mechanisms.
# Not the authors' official code. Runs a smoke test on random dummy data.

import torch
import torch.nn as nn
import torch.nn.functional as F


class Coordinator(nn.Module):
    """A simplified stand in for the Coordinator. A frozen linear encoder
    compresses an image into a compact feature vector, which is
    combined with a learnable task vector and decoded into a
    per pixel visual prompt of the same shape as the input."""

    def __init__(self, image_dim=32, feature_dim=16, task_dim=8):
        super().__init__()
        self.encoder = nn.Linear(image_dim, feature_dim)
        for p in self.encoder.parameters():
            p.requires_grad_(False)
        self.task_vector = nn.Parameter(torch.randn(task_dim) * 0.01)
        self.decoder = nn.Sequential(
            nn.Linear(feature_dim + task_dim, 32),
            nn.ReLU(),
            nn.Linear(32, image_dim),
        )

    def forward(self, image):
        z_x = self.encoder(image)
        task = self.task_vector.unsqueeze(0).expand(image.shape[0], -1)
        combined = torch.cat([z_x, task], dim=-1)
        prompt = self.decoder(combined)
        return prompt

    def get_flat_params(self):
        return torch.cat([self.task_vector.data] + [p.data.view(-1) for p in self.decoder.parameters()])

    def set_flat_params(self, flat):
        idx = 0
        n = self.task_vector.numel()
        self.task_vector.data.copy_(flat[idx:idx + n].view_as(self.task_vector))
        idx += n
        for p in self.decoder.parameters():
            n = p.numel()
            p.data.copy_(flat[idx:idx + n].view_as(p))
            idx += n


def black_box_model(prompted_image, target):
    """A toy stand in for the black box target model. Only ever returns
    a scalar loss given a prompted image, mimicking an API that hands
    back a prediction and nothing else about its internals."""
    logits = prompted_image @ target
    loss = F.mse_loss(logits, torch.zeros_like(logits))
    return loss.item()


def query_loss(coordinator, flat_params, image, target, epsilon=0.1):
    """Applies a candidate parameter vector to the Coordinator, generates
    a prompt, clips the prompted image, and queries the black box
    model, exactly two of these calls are needed per SPSA-GC step."""
    coordinator.set_flat_params(flat_params)
    with torch.no_grad():
        prompt = coordinator(image)
        prompted_image = torch.clamp(image + epsilon * prompt, -1.0, 1.0)
    return black_box_model(prompted_image, target)


def spsa_gc_step(coordinator, image, target, momentum, a_i=0.05, c_i=0.05, beta=0.9):
    """One step of SPSA with Gradient Correction. Estimates the gradient
    from two black box queries at a Nesterov style lookahead point,
    then updates parameters with a momentum term, matching Equations
    4 and 5 in the paper."""
    flat_params = coordinator.get_flat_params()
    lookahead = flat_params + beta * momentum

    delta = torch.randint(0, 2, flat_params.shape).float() * 2 - 1  # Rademacher +/-1

    loss_plus = query_loss(coordinator, lookahead + c_i * delta, image, target)
    loss_minus = query_loss(coordinator, lookahead - c_i * delta, image, target)

    grad_estimate = (loss_plus - loss_minus) / (2 * c_i) * (1.0 / delta)

    new_momentum = beta * momentum - a_i * grad_estimate
    new_params = flat_params + new_momentum

    coordinator.set_flat_params(new_params)
    return new_momentum, 0.5 * (loss_plus + loss_minus)


def train_coordinator(steps=10, image_dim=32):
    torch.manual_seed(0)
    coordinator = Coordinator(image_dim=image_dim, feature_dim=16, task_dim=8)
    image = torch.randn(1, image_dim)
    target = torch.randn(image_dim, image_dim)

    momentum = torch.zeros_like(coordinator.get_flat_params())
    loss_history = []
    for step in range(steps):
        momentum, loss_value = spsa_gc_step(coordinator, image, target, momentum)
        loss_history.append(loss_value)
    return loss_history


if __name__ == "__main__":
    history = train_coordinator(steps=10)
    print("black box query loss over 10 SPSA-GC steps")
    for step, loss_value in enumerate(history):
        print("step", step, "loss", round(loss_value, 6))

Running that file prints the black box query loss at each of 10 training steps, and it should trend downward over the run, confirming that the Coordinator’s parameters are being nudged in a useful direction using nothing more than paired loss evaluations from a function treated as a sealed black box, the same constraint the real BlackVIP method is built to respect.

Conclusion

The core achievement in this paper is refusing to accept the two assumptions that quietly underlie most parameter efficient tuning research, full parameter access and abundant memory, and building a working method for the much more restrictive, and much more common, situation where neither holds. BlackVIP treats a foundation model exactly as most people actually encounter it, as an API that accepts an image and returns a prediction, and still manages to adapt that model meaningfully using only two queries per optimization step.

The conceptual shift worth remembering is the move from a single, fixed intervention to an input dependent one. Earlier visual prompting learned one pattern and applied it everywhere. BlackVIP’s Coordinator learns a function that decides, image by image, what intervention makes sense, and the paper’s theoretical section shows this is not just a modeling convenience, it is the reason BlackVIP’s robustness guarantee can flex per sample rather than being fixed for an entire dataset the way a universal prompt’s guarantee would be.

That theoretical connection to randomized smoothing is probably the most transferable idea in the paper. Any adaptation method whose learned parameters can be shown to approximate a Gaussian distribution near a loss minimum could, in principle, be analyzed through the same lens, and any method that conditions its intervention on the specific input rather than applying one fixed pattern everywhere is a candidate for the same kind of flexible, per sample robustness argument. That is a genuinely different kind of claim than most efficient tuning papers make, since it connects a practical engineering constraint to a formal robustness property rather than treating them as unrelated topics.

The honest limitations are worth taking at face value rather than glossing over. Query efficiency lags meaningfully behind true gradient access, by the paper’s own account roughly 50 times more iterations are needed, and the faster BlackVIP-SE variant should be used cautiously on data suspected to carry strong spurious correlations, where the original BlackVIP’s richer feature extractor still has a clear edge. Neither limitation undoes the core contribution, they simply mark out where the next round of improvement needs to focus.

For anyone building on top of a commercial or otherwise closed vision model, the practical takeaway is direct. You do not need the weights to meaningfully adapt a foundation model to your task, you need a principled way to query it, and a prompt generator that responds to what it actually sees rather than applying the same fixed nudge to every image. That combination, this paper shows, gets you real accuracy gains, real efficiency, and a measurable robustness benefit that nobody explicitly asked for but that falls out of designing the adaptation this way.

Frequently asked questions

Related reading