The Four Secrets Behind Video Vision Transformers Explained

What makes this survey worth a careful read

Vision transformer surveys are common enough that a new one needs a clear organizing idea to stand out. This paper leans on a memorable framing, four secrets, and mostly earns it. Rather than organizing around a chronological list of named models the way many surveys do, it organizes around the four design decisions every video transformer has to make regardless of what the model is called, then places named models as examples underneath each decision. That structure makes the paper genuinely useful as a reference, especially the attention mechanism section, which tracks twenty three separate mechanisms along a real publication timeline from March 2021 through February 2026 and groups them into five functional categories.

The paper is also refreshingly explicit about its own lineage. Reference twenty five points to an earlier piece by an overlapping set of authors covering the same four secrets framework for plain image vision transformers, published in Information Fusion in 2024. A meaningful share of this paper’s structure, particularly the section mapping developments from ViT to ViViT and back, is really an extension exercise, take the image side taxonomy and ask what changes when a time axis gets added. That is not a criticism. It is a sensible way to build a second survey, and the paper says so itself rather than pretending the framework arrived fully formed for video.

How ViViT actually differs from ViT under the hood

The starting point is simple. ViT takes a static image, divides it into fixed size non overlapping patches, and turns each patch into a token through a linear projection. ViViT has to do the same job for a clip, and the paper describes two baseline strategies for that. The first processes each frame independently the way ViT would, then concatenates all the resulting tokens into one long sequence. The second samples three dimensional tubes directly across height, width and time, folding spatial and temporal information together at the moment of division rather than afterward.

From there the pipeline mirrors ViT closely. Learnable position embeddings get added so the model can tell tokens apart by where and when they sit in the clip. A class token gets inserted for classification tasks. The resulting sequence passes through a standard transformer encoder, layer norm, multi head self attention, another layer norm, a multilayer perceptron, with residual connections around each block. None of that machinery is new, and the paper is right not to spend much space on it. The interesting engineering happens in the four secrets layered on top.

Secret one, how to divide the patch

The paper groups patch division into four families. Non overlapping patch or tube division is the baseline described above, simple to implement but rigid, since a fixed spatio temporal window can cut straight through a coherent object or action and fragment its semantic meaning. Non overlapping division with multi resolution runs two or more branches at different patch sizes in parallel, which the paper credits with better handling of objects at very different scales, at the cost of a harder fusion problem when the branches need to be recombined. Frame wise patch division applies multiple division strategies within a single frame, either multiple patch shapes or multiple direction cuts, trading a need for hand designed prior knowledge for richer spatial context. The fourth family, labeled other patch division, covers overlapping patches that preserve semantic continuity across patch borders, and feature map patch division, where a convolutional network extracts features first and the resulting feature map gets divided rather than the raw pixels, borrowing the inductive bias CNNs bring to spatial locality at the cost of no longer being a pure transformer.

Two recent systems the paper cites as evidence this taxonomy still applies to current work are VideoMamba, which uses straightforward frame wise non overlapping division for efficient action recognition, and V-JEPA 2, which uses tube based division to capture spatial appearance and temporal dynamics together at the patch level. Naming currently deployed systems next to a five family taxonomy is a good way to check that the categories still describe what people are actually building, and here they do.

Secret two, how to select which tokens survive

Video multiplies the token count problem that already exists for images, since every additional frame adds another full set of spatial tokens, and consecutive frames are often highly redundant, a penguin waddling across three consecutive frames does not need three near identical background regions each fully represented. The paper organizes token selection into three families with a combined total of fourteen named techniques.

Pruning simply evaluates token importance and discards the ones that fail to clear a threshold, seven separate pruning strategies get covered, ranging from spatial temporal separation, which independently ranks frames in time and regions in space, through keyframe centric pruning, which keeps every token from a designated keyframe and uses it to guide which non keyframe tokens survive. Merging takes a gentler approach, combining similar tokens rather than deleting them outright, five strategies are covered here including a progressive multi granularity approach that first merges within a frame and then across clips in two separate stages. Masking is the most conservative option, it skips the computation for low importance tokens while still preserving their position in the sequence, which the paper notes makes it a better fit for tasks like multi person action localization where knowing exactly where something was, even if its content got skipped, still matters.

Secret three, how position gets encoded across space and time

Absolute position encoding, numbering each token from one to the sequence length, is the ViT default, and the paper explains why it becomes a weaker fit once time joins the mix. A high resolution, long duration clip needs a position scheme flexible enough to represent three axes at once without the encoding length exploding. Five mechanisms get covered. Residual position encoding folds the difference between consecutive frames into the position embedding itself, concentrating information where the video actually changes. Relative position encoding measures the distance between token pairs using a piecewise function and euclidean distance rather than assigning each token a fixed absolute slot, which the paper credits with better translational invariance for variable length clips. Dynamic position encoding uses a three dimensional depth wise convolution with zero padding so tokens near the edges of a clip can infer their position by querying their neighbors. Temporal and spatial position encoding assigns each of the three dimensions, time, height and width, its own independent sine and cosine encoding at different frequencies, a fairly direct three dimensional extension of the original transformer’s position scheme. Rotary position encoding, the newest of the five, injects relative position through a rotational phase applied to each token’s coordinate, and the paper cites a 2025 refinement built specifically to fix temporal allocation problems in earlier rotary variants.

Secret four, how attention actually gets calculated

This is where the paper does its most ambitious cataloging. Full global spatio temporal attention, comparing every token against every other token across the whole clip, has quadratic complexity in the token count, written as roughly the square of the frame count times the square of the per frame token count. That cost is the reason essentially every mechanism that follows exists, each one is some way of avoiding a full pairwise comparison while still capturing enough of the relevant structure.

The twenty three mechanisms get grouped into five functional categories. Attention based on spatio temporal separation decouples the full problem into a lower dimensional temporal pass and a lower dimensional spatial pass computed one after another, the divided space time attention from 2021 is the clearest early example and the paper credits it with genuinely low training and inference cost. Attention based on local areas or windows restricts comparisons to a nearby temporal and spatial neighborhood, the 3D shifted window approach adapted from the Swin Transformer family is the best known example here. Attention based on sparse sampling only computes attention for the most relevant key value pairs for a given query, K nearest neighbor attention and several 2025 and 2026 additions built for video diffusion transformers fall in this bucket. Attention based on dynamic content perception lets the receptive field of attention change based on what is actually in the frame, trajectory attention and deformable attention are the representative examples. Attention based on hybrid models borrows machinery from outside the transformer family entirely, Mamba attention and CNN based spatio temporal attention both appear here, introducing an inductive bias the pure transformer lacks in exchange for a harder architecture to design well.

Several of the most recent entries on the timeline, Video Monarch Attention and Monarch for Real Time Attention among them, come from work on video diffusion transformers rather than classification or detection models, both dated within the past several months of the paper’s own publication window. That is a useful signal about where attention efficiency research has been concentrating lately, generation and real time synthesis rather than the recognition tasks that dominated the earlier years on the timeline.

What the comparison tables do not tell you

Here is the honest gap in an otherwise thorough paper. Every one of the four comparison tables, for patch division, token selection, position encoding and attention, evaluates its options along the same four qualitative axes, idea, advantages, disadvantages and selection guidelines. The attention section additionally reports asymptotic complexity in Big O notation for each of the twenty three mechanisms, which is genuinely useful for reasoning about how a method will scale. What none of the tables include is a shared benchmark, no side by side accuracy, FLOPs or throughput numbers pulled from a common dataset and evaluation protocol across competing methods.

That absence is understandable given the scope, twenty three attention mechanisms alone were not all evaluated on the same benchmark by their original authors, so the survey cannot manufacture a fair comparison that does not exist in the literature. But it does mean a reader cannot use this paper alone to decide, for instance, whether 3D deformable attention actually outperforms K nearest neighbor attention on a real video recognition task, only that they were designed to solve related problems in structurally different ways. The complexity notation tells you which method will be cheaper to run at a given input size. It does not tell you which one will be more accurate. Readers who need that second answer will have to chase the twenty three individual citations back to their original benchmark tables, which somewhat undercuts the promise that a single survey can settle the question of which technique to reach for.

The two way bridge between ViT and ViViT

The paper’s final analytical section is arguably its most original piece of synthesis, since it is not simply cataloging existing work but drawing an explicit before and after map. Six examples trace how image side ViT techniques became video side ViViT techniques, two dimensional patch division becoming three dimensional tube division, spatial top k token selection becoming a two stage spatial and temporal top k selection, and windowed two dimensional attention becoming windowed three dimensional attention among them.

The reverse direction is framed as forward looking rather than already established, five directions the authors expect video side innovations to feed back into image models. Frame wise multi shape patch division suggests an image side multi shape division. Key frame guided token selection in video suggests key token guided selection in still images, where certain regions of a single image could play the same anchoring role a keyframe plays in a clip. Whether these five predictions hold up is obviously not something the survey itself can prove, and the authors are careful to label this section as future development directions rather than established results, which is the right way to present a forecast.

Honest limitations of this survey and this coverage of it

The paper is, by design, a taxonomy rather than an experimental study, and its value depends on how completely and fairly the taxonomy covers the field. On breadth, it does well, twenty three attention mechanisms tracked by publication date is a substantial cataloging effort, and the recent references, several of them from late 2025 and early 2026, suggest the authors kept the timeline current right up to submission. On depth of evidence, as covered above, the comparison tables are qualitative rather than benchmarked, so treat every advantage and disadvantage listed as the original authors’ framing of their own method rather than an independently verified ranking.

It is also worth flagging that a meaningful share of this paper’s structure descends from the authors’ own earlier ViT survey, which is disclosed but easy to miss on a first read. That lineage is a reasonable way to build a follow up paper, not a flaw, but readers interested in the full picture of how the four secrets framework was first developed for images may want to look at the 2024 predecessor alongside this one.

Conclusion

What this survey does well is impose real structure on a research area that could easily read as an unordered list of acronyms. Organizing around four consistent design questions, rather than a chronological parade of named models, makes the paper genuinely useful as something to return to when a specific design decision needs a menu of options, particularly the attention mechanism section with its publication timeline and five way functional grouping.

The four secrets framing also holds up conceptually. Every video transformer, no matter what it calls itself, does have to decide how to cut the clip into pieces, which pieces matter enough to keep computing, how each piece knows where and when it sits, and which pairs of pieces get compared during attention. Reducing the field to those four questions is a genuinely clarifying move, and the paper’s own conclusion is right that the four mechanisms share a single underlying subject, the video patch itself, viewed from four different angles.

Where the paper is weaker is in the depth of evidence behind its own comparisons. Idea, advantages, disadvantages and selection guidelines is a useful shape for a table, but it is not a substitute for shared benchmark numbers, and the complexity notation in the attention section tells a reader about computational cost without saying anything about accuracy. A reader who wants to actually choose between, say, local global sparse attention and 3D axial attention for a specific project will still need to open both original papers and find out how each one performed on a dataset that matters for that project.

The ideas here transfer well beyond video specifically. The tension between pruning, merging and masking as three philosophies for reducing token count applies just as much to long document transformers in natural language processing as it does to video clips. The position encoding survey, particularly the rotary encoding discussion, is directly relevant to anyone working on long context sequence models generally, video or otherwise. And the two way bridge between ViT and ViViT is a useful reminder that innovation in this field rarely flows in only one direction, ideas built for images keep getting adapted for video, and increasingly the reverse is starting to happen too.

For now, the practical guidance is to use this survey as a map rather than a verdict. It will tell a reader what options exist for each of the four secrets and roughly what tradeoff each option makes. It will not tell them which specific combination will perform best on their specific task, and that answer still has to come from running the actual benchmarks the original papers report, or better, running new ones on the exact data and hardware a project actually needs.

Frequently asked questions

Related reading