Federated Learning Lets Surveillance Cameras Learn Without Sharing

The problem with training on everyone’s cameras at once

Weakly supervised video anomaly detection has become the dominant approach to flagging unusual events in surveillance footage, precisely because it sidesteps the need for frame by frame labels. A human only has to tag whether a whole video contains an anomaly somewhere, not exactly which frames, and the model learns to localize the anomaly on its own using a multiple instance learning setup. That is a real practical advantage, but it still assumes the model gets to see all the training videos in one place.

Real surveillance networks do not work that way. A school district’s cameras, a hospital’s cameras, and a retail chain’s cameras are typically siloed by design, and for good reason, surveillance footage often captures identifiable people going about private business. Even setting privacy law aside, most institutions simply are not willing to hand a vendor or a research partner a copy of their raw video. The paper’s authors point out the obvious downstream cost, models trained only on one institution’s narrow slice of scenes, camera angles, and anomaly types generalize poorly to anything outside that slice.

Federated learning is the natural fix for exactly this shape of problem. Instead of centralizing data, each institution trains a model locally and only ever sends the model’s learned weights to a coordinating server, which combines them into a shared global model and sends it back out. The raw footage never has to leave the building it was recorded in.

Building a client model light enough to actually federate

A federated system lives or dies on how much has to be communicated every round, so the authors deliberately kept the trainable part of their client network small. The backbone is CLIP, the widely used vision language model that aligns images and text in a shared embedding space, and CLIP’s own image and text encoders stay completely frozen throughout training. Freezing the encoders does two things at once, it preserves the broad visual and semantic knowledge CLIP already learned from massive pretraining, and it means those large encoder weights never need to be uploaded or downloaded during federated rounds at all.

What actually gets trained, and what actually gets communicated, are four lightweight modules layered on top of the frozen backbone.

An image prompt to close the domain gap

CLIP was pretrained mostly on clean, well composed natural images, the kind found in datasets like ImageNet. Surveillance footage looks nothing like that, it tends to be low resolution, poorly lit, and shot from an awkward downward angle. Rather than fine tuning CLIP’s own image encoder to close that gap, which would be expensive to train and expensive to communicate, the framework adds a small learnable image prompt that gets added directly to the frozen encoder’s output features, nudging them toward the surveillance domain without touching the encoder itself.

A temporal encoder to notice patterns across frames

A single frame’s features cannot capture events that unfold over time, a person loitering, a fight escalating, a crowd suddenly scattering. The client network adds a small temporal module, a shallow transformer followed by a two layer multilayer perceptron, that processes the sequence of frame level features together and lets the model pick up on exactly these fine grained temporal patterns.

A text prompt and an adapter, tuned specifically for security scenes

On the language side, CLIP’s frozen text encoder processes two prompts, one built around the phrase describing an abnormal scene and one describing a normal scene, each with its own learnable suffix that the training process tunes to fit the surveillance domain. A small adapter module then refines the resulting text features further, so the final text representation aligns as closely as possible with what the visual branch is producing.

Putting the two branches together, the model scores every frame by taking the dot product between its video feature and the text features for the abnormal and normal prompts. Training uses a hinge based multiple instance ranking loss, essentially teaching the model that the most anomalous looking frame in a video labeled abnormal should score meaningfully higher than the most anomalous looking frame in a video labeled normal, combined with a smaller regularization term that discourages the model from flagging too many frames at once or jumping around erratically between neighboring frames.

Three ways to combine everyone’s updates, and why one is not enough

The standard way to merge federated updates, FedAvg, simply averages every client’s weights together, weighted by how much data each client has. That works fine when every client’s data looks roughly similar. The moment institutions differ meaningfully in how good their local model turns out, in what kind of scenes they cover, or in how many of their videos actually contain anomalies, plain averaging starts pulling the shared model in inconsistent directions, a problem the federated learning literature calls client drift. Surveillance data is a near perfect storm for this, different institutions genuinely do cover different scenes, different camera setups, and wildly different ratios of normal to abnormal footage. So the authors design three alternative aggregation strategies, each aimed at a specific way client data can diverge.

Score based weighted averaging, for clients with uneven skill

The most direct fix is to trust better performing clients more. Score based weighted averaging measures each client’s performance on a shared test set after local training, converts those scores into weights using a softmax function, and averages the uploaded weights accordingly. A client whose local model happens to be more accurate gets proportionally more influence over the next global model.

Cluster based partial averaging, for clients whose data looks fundamentally different

Sometimes the issue is not skill but genuine distributional mismatch, one institution’s cameras cover a completely different mix of scenes than another’s. Cluster based partial averaging groups clients whose uploaded parameters look similar to each other, on the reasoning that similar parameters usually reflect similar underlying data, then only averages within the largest such cluster rather than across every client indiscriminately. This keeps a badly mismatched client from dragging the global model away from where most clients actually sit.

Anomaly ratio weighted averaging, for clients with lopsided anomaly counts

The last strategy addresses a more specific imbalance, the proportion of videos that actually contain an anomaly varies a lot from institution to institution. A client whose footage happens to include a richer share of genuine anomalies has more useful signal to contribute toward learning what anomalies look like, so anomaly ratio weighted averaging weights each client’s contribution by its own anomaly ratio, giving anomaly rich clients more say in shaping the global model’s understanding of abnormal behavior.

How well does the federated version actually perform

The authors test PPVAD on three well established benchmarks, UCF-Crime, a large collection of real world street and indoor surveillance footage covering thirteen categories of crime, XD-Violence, an even larger and more varied collection pulled from movies, sports broadcasts, and CCTV footage, and ShanghaiTech, a smaller campus surveillance dataset. Every result is reported twice, once under a centralized setting where all the data is pooled together as an upper bound, and once under the federated setting that actually respects data privacy.

The federated numbers are the ones that matter most here, since that is the setting the paper is actually built for. On UCF-Crime, PPVAD reaches an area under the curve score of 87.25 percent under federated training, ahead of the next best federated method, FedVAD, by a little over two percentage points, and ahead of CLAP, an earlier privacy focused baseline, by roughly four points. On XD-Violence, PPVAD’s federated area under the curve reaches 91.85 percent, more than six points ahead of CLAP and about three points ahead of FedVAD. On ShanghaiTech, PPVAD posts a federated area under the curve of 98.24 percent, ahead of the strongest prior federated approaches tested.

Just as telling is how close the federated numbers land to the centralized ones. On XD-Violence in particular, PPVAD’s federated result comes within about half a point of VadCLIP’s centralized result, a method that gets to see every institution’s data pooled together with no privacy constraint at all. That gap, or rather the near absence of one, is the paper’s central practical claim, that federated training does not have to mean settling for meaningfully worse detection just to protect privacy.

Does the custom aggregation actually beat off the shelf federated optimizers

It would be easy for a paper to claim its own aggregation strategies are better without checking them against the broader federated learning toolbox, so it is worth noting the authors do run that comparison directly. Against nine established aggregation methods, including FedAdam, FedProx, FedAvgM, and several Byzantine robust variants, the paper’s own anomaly ratio weighted averaging strategy comes out slightly ahead across random, category based, and scene based data splits on UCF-Crime. The margins are not enormous, typically well under a single percentage point over the next best generic method, but the consistency across three quite different ways of splitting the data suggests the improvement is not a fluke tied to one particular split.

The data splitting experiment itself is worth understanding, because it is a more realistic stress test than a single benchmark number implies. Random splitting distributes videos evenly and represents the easiest, most independent and identically distributed case. Category based splitting hands each simulated institution a different mix of anomaly types, mimicking a scenario where one hospital mostly sees theft while another mostly sees violence. Scene based splitting hands each institution a different physical environment entirely, mimicking schools, hotels, and streets each running their own cameras. Performance degrades somewhat as you move from random splitting toward these more realistic, more skewed splits, which the authors are candid about, this is exactly where non independent and identically distributed federated learning remains a genuinely open problem rather than a solved one.

What this actually costs to run

A privacy preserving method that is too expensive to deploy is not much use to anyone, so the paper’s complexity analysis deserves attention. On UCF-Crime, each client uploads roughly ten million parameters per round, which works out to about forty megabytes of communication per client per round. Local training per client takes about twenty two seconds, and a full communication round, including the server side aggregation step, takes roughly thirty two seconds. With five clients participating, that puts a full round at well under a minute of wall clock time, which is a genuinely practical cost profile for a system institutions might run on a recurring schedule rather than as a one off experiment.

The authors also stress test the system under conditions closer to a real deployment. When client hardware is deliberately made uneven, mixing older and newer GPUs across clients, synchronization latency creeps up slightly but stays in a similar range, around eleven to twelve seconds per round. When clients are allowed to randomly drop out of a training round entirely, simulating network instability, performance degrades gradually and gracefully rather than collapsing, PPVAD still reaches a respectable area under the curve even when only two of five clients participate in a given round.

Honest limitations worth keeping in mind

The gap between federated and centralized performance, while consistently small in this paper, is not zero, and on UCF-Crime specifically centralized methods still hold a real edge over every federated method tested, PPVAD included. The paper’s own data splitting experiments confirm that performance degrades as client data distributions become more skewed, and the authors are upfront that non independent and identically distributed federated learning remains an open research area rather than something this paper fully resolves. The complexity analysis, while encouraging, is also run entirely on hardware within a single research lab rather than across geographically distributed institutions with real network latency between them, so real world communication delays could look different from what is reported here. Finally, the paper’s data availability statement notes that the authors do not have permission to share the underlying data, and no code repository link accompanies this particular paper, which means independent reproduction will depend on institutions building their own implementation from the architecture described rather than starting from a public release.

Conclusion

The core contribution here is showing, with real benchmark numbers rather than just a plausible sounding architecture diagram, that weakly supervised video anomaly detection does not have to choose between institutional privacy and shared learning. By keeping CLIP’s encoders frozen and training only a handful of lightweight modules, the framework keeps communication costs low enough to be practical, and by replacing plain federated averaging with three purpose built aggregation strategies, it addresses the specific ways surveillance data tends to diverge across institutions rather than treating federated learning as a one size fits all recipe.

The more interesting conceptual point is what the three aggregation strategies reveal about the problem itself. Client drift in federated surveillance is not one problem, it is at least three different problems wearing the same name, uneven client skill, genuinely different data distributions, and lopsided anomaly proportions, and the paper’s approach of building a distinct tool for each rather than one universal fix is a more honest way to handle heterogeneity than hoping a single averaging trick covers every case.

Whether these specific strategies generalize cleanly beyond video anomaly detection to other federated computer vision tasks is an open question the paper does not attempt to answer, and any team adapting this approach to a different domain should treat that as a genuine unknown rather than an assumed transfer.

The remaining gaps, a persistent though narrow centralized advantage, degraded performance under more realistic non independent and identically distributed splits, and complexity numbers measured on a single research cluster rather than a true multi institution network, all point toward reasonable next steps rather than settled conclusions. None of them undercut the paper’s central result, they simply mark where the next round of research needs to look.

What is worth carrying away from this paper is less any single accuracy number and more the demonstration that privacy and performance are not automatically in tension here. A federated system that comes within roughly half a point of a centralized upper bound on one of its benchmarks is a genuinely useful existence proof for anyone who has been told that protecting data privacy necessarily means accepting a materially worse model.

A runnable sketch of the client network and the three aggregation strategies

The following is a simplified but complete and runnable implementation of the paper’s client network and its three server aggregation strategies, including the multiple instance ranking loss, the Top-K instance selection, the regularization term, and a smoke test on synthetic frame features so you can see a full federated round run end to end.

Running this trains three separate client networks for one local step each on random synthetic features, then merges their weights three different ways, printing the resulting norm of the aggregated image prompt under each strategy so you can see the three aggregation rules genuinely produce different global weights from the same client updates, exactly the behavior the paper is arguing matters once client data actually diverges.

Frequently asked questions

Related reading