How ActiveKD Uses VLM Bias To Guide Active Learning

ActiveKD, one biased teacher, a coreset built in probability space

Why knowledge distillation and active learning rarely meet

Active learning exists to solve one specific pain point. Getting labels is expensive, so instead of labeling data at random, a model picks the unlabeled examples it expects to learn the most from, an expert labels just those, and the cycle repeats. Knowledge distillation solves a different pain point. A smaller model learns faster and often better when a larger, already capable teacher hands it soft probability targets instead of forcing it to learn from raw labels alone.

On paper these two ideas seem like they should combine easily. In practice they almost never get used together, and the paper’s authors point out exactly why. Active learning is built around the assumption that labels are scarce and must be earned through selective annotation. Knowledge distillation is built around the assumption that a strong, task specific teacher already exists, trained on enough labeled data to be worth learning from. Those two assumptions directly contradict each other in the setting active learning is meant for. If you already had a strong task specific teacher, you probably would not need active learning to build one.

Vision language models change that calculation. A model like CLIP was never trained on your specific eleven category flower dataset or your specific aircraft dataset, but its pretraining on enormous amounts of paired images and text gives it usable zero shot judgment about almost any visual category you can describe in a sentence. That makes it exactly the kind of teacher active learning was missing, one that requires no task specific labeled data at all to start being useful.

The ActiveKD training loop

The framework itself is close to what you would guess once the assumption gap is solved. In each active learning round, the student model trains on a combination of two losses, ordinary cross entropy on the labeled data collected so far, and a distillation loss that measures the divergence between the student’s predictions and the vision language teacher’s predictions on both the labeled and unlabeled pool.

The final loss is a weighted mix of the cross entropy term and this distillation term. The teacher’s probability vector itself comes from a simple, elegant construction, cosine similarity between an image embedding and a set of text embeddings built from class name prompts, passed through a softmax with a temperature term. No fine tuning of the teacher is strictly required, since CLIP’s zero shot capability is already enough to bootstrap useful soft labels on data it has never seen labels for.

What makes ActiveKD more than just a swap of teacher source is that the student model itself becomes the selection engine for choosing what gets labeled next. After each round of training, the framework runs a selection algorithm over the unlabeled pool using the now distilled student model, queries an oracle for labels on the chosen samples, and folds those into the labeled set for the next round. The teacher never disappears from the loop the way it would in a typical distill then discard pipeline. It keeps contributing soft supervision on unlabeled data every single round.

The bias nobody wanted to look at directly

Here is where the paper does something most teams building on top of a foundation model teacher would rather not do. Instead of treating CLIP as a clean oracle, the authors go looking for its flaws, and they find a specific, measurable one. Plot a vision language model’s predictions across a large unlabeled pool in probability space, and they do not spread out evenly. They cluster into a limited number of tight regions, shaped by whatever the pretraining data and the specific text prompts happened to emphasize.

The paper formalizes this as structured prediction bias, defined precisely as every teacher prediction falling inside a finite union of small balls in the probability simplex rather than being free to land anywhere in that space.

In plain terms, the teacher’s confidence pattern has a small number of favorite shapes it keeps repeating no matter which image comes in, centered at points mu with radius r, and K is typically much smaller than you would expect from a model with real expressive freedom over C classes.

The obvious next question is whether this bias is quarantined inside the teacher or leaks into the student through distillation. The paper proves, and then empirically confirms, that it leaks. If the student is trained with bounded approximation error to the optimal blend of ground truth label and teacher prediction, its own outputs are provably confined to a related, shifted set of clusters, at most C times K of them, still a small structured set rather than the full space of possible probability vectors.

PCoreSet, turning the bias into a targeting system

Most teams would treat this finding as a problem to patch around, maybe by trying to debias the teacher’s outputs before distilling them. The paper takes the opposite stance. If the student reliably inherits the teacher’s clustered structure, then samples that fall outside that structure are precisely the ones carrying information the student would not otherwise get. Rather than discarding the bias, the paper turns it into a targeting signal for which unlabeled samples are worth annotating.

The resulting method, Probabilistic Coreset, borrows its skeleton from a well known feature space coreset selection idea, greedily picking the unlabeled point that maximizes the minimum distance to every already labeled point, then repeating. The single change that matters is where that distance gets measured. Ordinary coreset selection measures distance in the model’s internal feature space. PCoreSet measures it in the probability simplex instead, the same space where the teacher’s bias lives.

By operating in probability space rather than feature space, PCoreSet greedily fills in exactly the regions of the simplex the teacher’s bias leaves underrepresented, which by the earlier proof are the regions the student would otherwise struggle to learn from labeled examples alone. There is also a quieter practical benefit buried in this design choice. Feature space coreset selection scales with the model’s hidden dimension H, which for a modern backbone can run into the hundreds or thousands. PCoreSet scales instead with the number of classes C, which for most real world classification tasks is far smaller, making it computationally cheaper in the common case where C is much less than H.

What the experiments actually show

The evaluation spans 11 datasets covering generic object recognition, fine grained categories like aircraft and car models, textures, satellite imagery, and action recognition, tested across 5 student architectures and 3 vision language teacher variants. The headline comparison is simple, standard active learning with no distillation at all against ActiveKD using either a fixed zero shot teacher or a teacher that itself gets fine tuned with a few shot method as labeled data accumulates.

Two things stand out immediately. First, ActiveKD helps every single selection strategy it is paired with, not just PCoreSet, which tells you the value of a vision language teacher is largely independent of how you choose samples. Second, PCoreSet consistently sits at or near the top of its column, and the paper reports it winning outright in 64 of 73 tested configurations across different student and teacher architecture pairings at 8 or 16 rounds, missing the top spot only in the first two rounds of active learning when almost no strategy has enough labeled data yet to differentiate itself.

The few shot teacher comparison surfaces an interesting wrinkle. Letting the teacher itself improve alongside the student, using the same newly labeled samples, helps on 10 of the 11 datasets but barely moves the needle on ImageNet specifically. The paper attributes this to ImageNet’s much larger class count, a thousand categories against roughly a hundred for the other datasets, which means each active learning round adds proportionally fewer labeled examples per class, diluting how much a slightly sharper teacher can contribute in that setting.

Perhaps the most surprising result comes from a virtuous cycle effect the paper documents almost as an aside. Because the few shot teacher gets updated using the same samples the selection strategy chose for the student, the authors could check whether PCoreSet’s choices also happen to help the teacher improve faster. They do. PCoreSet consistently outperforms the other selection strategies as a source of few shot training examples for the teacher too, meaning the same samples that patch the student’s blind spots also happen to be useful for sharpening the teacher, each side of the distillation relationship improving the other.

What this means beyond image classification

The transferable insight here is bigger than active learning specifically. Any time a foundation model gets used as a teacher for a smaller task specific model, that teacher brings its pretraining biases along for the ride whether anyone accounts for them or not. Most distillation pipelines never check for this, and the ones that do usually treat it purely as a source of error to be minimized. This paper’s contribution is showing that the bias itself is diagnosable, provably transferable, and usable as a targeting signal rather than something you can only try to average away.

For teams building any kind of human in the loop annotation pipeline around a foundation model teacher, whether that is CLIP for images or a large language model for text classification, the practical lesson is to measure where the teacher’s confidence clusters before designing a sample selection strategy, rather than assuming a general purpose uncertainty or diversity heuristic will automatically find the teacher’s blind spots. Probability space, not feature space, is where that blind spot actually lives, and PCoreSet’s core trick, just change which space you measure distance in, is cheap enough to try in almost any existing coreset based pipeline.

The efficiency argument is worth taking seriously on its own terms too. PCoreSet’s complexity scales with the number of classes rather than the feature dimension, which for most practical classification problems with dozens or hundreds of categories running on backbones with feature dimensions in the thousands, makes it a genuinely cheaper drop in replacement for feature space coreset selection, independent of whether the accuracy gains alone would have justified switching.

Honest limitations

Every experiment in this paper is a visual recognition classification task. The authors are explicit that this was a deliberate scope choice, since vision language models have their strongest zero shot and few shot track record specifically in image classification, but it leaves open how well structured prediction bias, and PCoreSet’s exploitation of it, would hold up for object detection, segmentation, or any task where the output is not a simple categorical probability vector living in a clean simplex.

The theoretical guarantee also leans on an approximation assumption, that the student converges to within a bounded error epsilon of the theoretically optimal blend of ground truth and teacher prediction. That is a reasonable assumption for a well trained model but not a certainty, and the paper’s own appendix shows the specific dual head training method it relies on, called DHO, requires its own separate justification for why single head and dual head training converge to the same target. If a different distillation setup were used, the bias propagation guarantee would need to be reestablished for that setup specifically, it does not follow automatically from the general framework.

There is also a boundary condition worth noting from the paper’s own ablation. When the researchers ran the same selection strategies in a traditional active learning setup with no distillation at all, PCoreSet lost its clear advantage, performing roughly on par with other diversity based baselines rather than winning outright. The paper interprets this as confirmation that probability space diversity is specifically valuable because of its relationship to teacher bias propagation, not as a generically superior active learning heuristic on its own. That is a more honest and more useful finding than claiming PCoreSet just works better everywhere, but it does mean the method’s value is tied tightly to the distillation setting it was designed for.

A minimal PyTorch implementation of PCoreSet

The code below implements a simplified but functionally faithful version of ActiveKD and PCoreSet. It defines a small CLIP style zero shot teacher using cosine similarity between image and text embeddings, a student classifier trained with the combined cross entropy and KL divergence loss from Equation 3 in the paper, the PCoreSet greedy selection algorithm from Algorithm 2, and a smoke test that runs one full active learning round on randomly generated dummy data.

import torch
import torch.nn as nn
import torch.nn.functional as F

# ---------------------------------------------------------------
# A minimal zero-shot style teacher, standing in for a real CLIP
# model. Produces a probability vector over C classes via cosine
# similarity between an image embedding and fixed text embeddings,
# matching Eq. 1 in the paper.
# ---------------------------------------------------------------
class ZeroShotTeacher(nn.Module):
    def __init__(self, embed_dim=32, num_classes=10, temperature=0.1):
        super().__init__()
        self.text_embeds = nn.Parameter(torch.randn(num_classes, embed_dim), requires_grad=False)
        self.temperature = temperature

    def forward(self, image_embeds):
        # image_embeds, shape (batch, embed_dim), already L2 normalized upstream
        img_norm = F.normalize(image_embeds, dim=-1)
        txt_norm = F.normalize(self.text_embeds, dim=-1)
        cos_sim = img_norm @ txt_norm.t()
        return F.softmax(cos_sim / self.temperature, dim=-1)


# ---------------------------------------------------------------
# A small student classifier. In the real paper this would be a
# ResNet or ViT backbone, here a compact MLP stands in.
# ---------------------------------------------------------------
class StudentModel(nn.Module):
    def __init__(self, input_dim=32, hidden_dim=64, num_classes=10):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, num_classes)
        )

    def forward(self, x):
        return F.softmax(self.net(x), dim=-1)


# ---------------------------------------------------------------
# The ActiveKD combined loss, following Eq. 2 and Eq. 3 in the
# paper. Cross entropy on labeled data plus KL divergence against
# the teacher on both labeled and unlabeled data.
# ---------------------------------------------------------------
def activekd_loss(student, teacher, x_labeled, y_labeled, x_unlabeled, lam=0.5):
    student_probs_l = student(x_labeled)
    ce_loss = F.nll_loss(torch.log(student_probs_l + 1e-8), y_labeled)

    with torch.no_grad():
        teacher_probs_l = teacher(x_labeled)
        teacher_probs_u = teacher(x_unlabeled)
    student_probs_u = student(x_unlabeled)

    kd_loss_l = F.kl_div(torch.log(student_probs_l + 1e-8), teacher_probs_l, reduction='batchmean')
    kd_loss_u = F.kl_div(torch.log(student_probs_u + 1e-8), teacher_probs_u, reduction='batchmean')
    kd_loss = kd_loss_l + kd_loss_u

    return lam * ce_loss + (1 - lam) * kd_loss


# ---------------------------------------------------------------
# PCoreSet, Algorithm 2 in the paper. Greedily selects unlabeled
# points that maximize the minimum probability-space distance to
# every already labeled point.
# ---------------------------------------------------------------
def pcoreset_select(student, x_labeled, x_unlabeled, query_size):
    with torch.no_grad():
        labeled_probs = student(x_labeled)
        unlabeled_probs = student(x_unlabeled)

    # D[i], the current minimum distance from unlabeled point i to the labeled set
    dists = torch.cdist(unlabeled_probs, labeled_probs, p=2)
    min_dists, _ = dists.min(dim=1)

    selected_indices = []
    remaining = set(range(unlabeled_probs.size(0)))

    for _ in range(query_size):
        remaining_list = list(remaining)
        local_dists = min_dists[remaining_list]
        best_local_idx = torch.argmax(local_dists).item()
        best_idx = remaining_list[best_local_idx]
        selected_indices.append(best_idx)
        remaining.remove(best_idx)

        # update min distances given the newly added point
        new_point = unlabeled_probs[best_idx].unsqueeze(0)
        new_dists = torch.cdist(unlabeled_probs, new_point, p=2).squeeze(-1)
        min_dists = torch.minimum(min_dists, new_dists)

    return selected_indices


# ---------------------------------------------------------------
# Smoke test, runs one ActiveKD training step followed by one
# PCoreSet selection round on dummy data end to end.
# ---------------------------------------------------------------
if __name__ == '__main__':
    torch.manual_seed(0)
    embed_dim, num_classes = 32, 10
    n_labeled, n_unlabeled, query_size = 20, 200, 10

    teacher = ZeroShotTeacher(embed_dim=embed_dim, num_classes=num_classes)
    student = StudentModel(input_dim=embed_dim, num_classes=num_classes)
    optimizer = torch.optim.Adam(student.parameters(), lr=1e-3)

    x_labeled = torch.randn(n_labeled, embed_dim)
    y_labeled = torch.randint(0, num_classes, (n_labeled,))
    x_unlabeled = torch.randn(n_unlabeled, embed_dim)

    for step in range(5):
        loss = activekd_loss(student, teacher, x_labeled, y_labeled, x_unlabeled, lam=0.5)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print('step', step, 'activekd loss', loss.item())

    queried = pcoreset_select(student, x_labeled, x_unlabeled, query_size=query_size)
    print('PCoreSet selected', len(queried), 'unlabeled indices for annotation')

    assert torch.isfinite(loss), 'loss should be a finite number'
    assert len(queried) == query_size, 'PCoreSet should return exactly query_size indices'
    assert len(set(queried)) == query_size, 'PCoreSet should not select duplicate indices'
    print('smoke test passed')

Conclusion

The real contribution of this paper is not that vision language models make good active learning teachers, that part is a relatively small step once you notice the assumption mismatch blocking the combination. The interesting move is refusing to treat the teacher’s imperfection as noise to be washed out. Structured prediction bias is defined precisely enough to prove, proved to propagate through distillation with a specific mathematical bound, and then turned directly into a selection criterion that gets better as training progresses rather than worse.

The conceptual shift worth carrying into other work is that a foundation model teacher’s failure mode is not necessarily random noise around the truth, it can be structured, measurable, and specific to that particular model’s training history and prompt design. Once you can characterize a bias precisely enough to write down its shape mathematically, you often gain more by exploiting that structure directly than by trying to average it away with more data or a bigger model.

Where this needs the most additional evidence is exactly where the authors say it does, tasks beyond simple classification, and distillation setups other than the specific dual head method the theoretical guarantee was built around. The virtuous cycle finding, where PCoreSet’s chosen samples improve the few shot teacher as much as the student, is a strong hint that the underlying mechanism is genuinely general rather than a quirk of one particular experimental setup, but it was only demonstrated inside this one visual classification testbed.

The honest limitations do not undercut what was shown, but they do draw a clear boundary around it. PCoreSet is not a universally better active learning heuristic, its own ablation shows that plainly when distillation is removed from the picture. It is a method that works because it targets a specific, provable weakness that only exists when a biased teacher is in the loop, and that specificity is exactly what makes the result credible rather than overclaimed.

Frequently asked questions

Related reading