Teaching AI Online Point Tracking Without Looking Ahead

Track-On2 · Online Point Tracking With Memory

Following one point through a video without seeing what happens next

Motion estimation sits underneath a huge share of practical computer vision, video compression, video stabilization, augmented reality overlays, robotic manipulation. The classic tool for this is optical flow, estimating how pixels shift between two adjacent frames, then chaining those estimates together over a whole sequence. Chaining is where things go wrong. Errors compound frame after frame, and optical flow has no good answer for occlusion, what happens to a tracked pixel while it is hidden behind something else for a stretch of the video.

A different framing, sometimes called particle video, sidesteps this by tracking individual points directly across the whole sequence rather than only between neighboring frame pairs. Recent deep learning systems built on this idea, models with names like PIPs and TAPIR, use dense cost volumes and iterative refinement to follow points with real precision. But nearly all of them share two structural limitations. First, they are offline, they need the entire video, or at least a sizeable window of frames including ones from the future relative to the current point in time, before they can commit to an answer. Second, they scale poorly, full attention across every frame in a long video gets expensive fast, both in compute and in memory, which becomes a serious problem for anything beyond short clips.

The two problems holding back most point trackers

Full video access is not always available

A robot arm reaching for an object, a live security feed, an augmented reality app rendering on a phone, none of these have the luxury of waiting for a video to finish before deciding where a tracked point currently is. They need an answer now, using only what has already happened. Most state of the art point trackers were simply not designed for that constraint, and retrofitting them with a causal mask after the fact, so they only look backward, tends to hurt their accuracy since they were trained assuming future access.

Memory and compute blow up on long videos

Models that attend across an entire clip pay a computational price that grows with how long that clip is. That is fine for a thirty frame test video. It becomes untenable for a robotics sequence running into the hundreds of frames, or a long form video running into the thousands. Several strong offline trackers simply run out of GPU memory once sequences get long enough, regardless of how much hardware you throw at them, because their design ties memory cost directly to video length.

How Track-On2 sees a video, three stages at a time

Track-On2 processes video causally, one frame at a time, and carries forward a compact memory that lets it stay temporally coherent without ever looking ahead.

A multi scale visual encoder

Each frame passes through a Vision Transformer backbone, specifically a DINOv3 model paired with ViT Adapter, which extracts features at four different resolutions and fuses them with a Feature Pyramid Network into one representation that carries both broad semantic context and fine spatial detail. A query point is initialized by sampling this feature map at the point’s starting pixel location in the frame where it first appears.

A query decoder that fights feature drift with memory

As a tracked point’s surroundings change over time, lighting shifts, the object rotates, something partially covers it, the similarity between its original appearance and its true current match steadily decays, a problem the authors call feature drift. Relying only on that original snapshot would make predictions increasingly unreliable the longer a video runs. Track-On2 counters this with an explicit memory that stores each query’s own recent history, then decodes every query through three attention stages in sequence, feature attention against the current frame, self attention so queries can share information with each other the way CoTracker’s design popularized, and memory attention where each query looks back at its own stored history to pull in context that helps it recover from drift or a recent occlusion.

Coarse classification, then re ranking, then a precise offset

Rather than regressing an exact pixel coordinate directly, a choice the authors show is less stable, Track-On2 first treats tracking as a classification problem, scoring every candidate patch in the current frame by similarity to the query and picking the most likely one. The authors noticed the correct patch does not always win outright but almost always sits somewhere in the top few candidates, so a dedicated re ranking step pulls in the top candidates, gathers richer features around each of them, and lets the query re evaluate before committing to a final patch. Only after that coarse patch is settled does a separate offset head predict a small sub patch adjustment for precise localization, alongside a lightweight estimate of whether the point is currently visible at all.

A memory that forgets on purpose

The memory is deliberately bounded. Each tracked point gets its own slot holding a fixed number of past embeddings, and when that slot fills up, the oldest entry is evicted to make room for the newest, a first in first out policy. This means memory size stays constant no matter how long the video runs, unlike offline models whose cost scales with sequence length. A set of learned temporal position embeddings, visualized in the paper through a PCA projection into color, shows a smooth gradient from oldest to newest slot, confirming the model genuinely learns to treat recent history differently from older history rather than treating memory as an undifferentiated blob.

Because training can only afford clips up to a certain length before running out of GPU memory, the model is trained with a fixed, relatively small memory size. Real deployment videos are often far longer than that. The paper’s fix, called inference time memory extension, linearly interpolates those learned temporal position embeddings to stretch the memory to a much larger size at test time, with no retraining required. Because the interpolation preserves the relative ordering between slots, the model’s attention behaves consistently even when the memory it is querying is three times larger than anything it saw during training.

Tested on five benchmarks against real fine tuned rivals

The team evaluated Track-On2 on five datasets chosen to stress different aspects of tracking, short real world clips, long internet videos, robotics footage, 3D reconstruction sequences, and extremely long synthetic videos running past four thousand frames. Crucially, every comparison distinguishes models trained purely on synthetic data from those additionally fine tuned on real footage, since real world fine tuning tends to give a meaningful boost on its own.

The PointOdyssey result is worth sitting with. Offline models that try to process an entire long video at once, or even a wide sliding window, ran out of GPU memory on a 48 gigabyte card before completing the benchmark. Track-On2, processing one frame at a time with its bounded memory, finished the benchmark and posted the best score of any method tested, precisely the scenario its architecture was built to handle.

What the ablations reveal, training length beats everything else

A large ablation study isolates exactly which design choices earn their keep. Starting from the original Track-On architecture and changing one thing at a time, the team found that simply updating the training data and frame sampling strategy, without touching the architecture at all, produced consistent gains across every dataset, evidence that a meaningful share of Track-On2’s improvement over its predecessor comes from how it is trained rather than any single architectural change. Digging further, training clip length turned out to be the dominant lever, increasing it steadily improved performance across every benchmark and metric, with especially large gains on the long PointOdyssey survival metric, while increasing memory capacity alone, independent of training clip length, produced only modest gains on some datasets and even a slight dip on others.

The relationship between memory design and training length interacts in an interesting way. With short training clips, the original dual memory design from Track-On actually outperformed the newer unified memory. But once training clips were extended, that gap closed entirely, and pushing memory size up further made the old dual memory design run out of memory on 64 gigabyte GPUs altogether, while the new unified memory stayed stable and even became faster, cutting GPU memory use by about 20 percent and improving inference speed by more than 5 frames per second in a matched comparison. In other words, the architectural simplification only pays off once paired with the longer training regime, they are not separable improvements.

Component removal experiments confirm each piece is pulling real weight. Removing memory entirely collapses the model to frame level predictions with no temporal context, costing 10 points of Average Jaccard on DAVIS and dropping occlusion accuracy from 91.5 to 81.5. Removing the re ranking step costs another 4.7 points on DAVIS, since the model loses its ability to recover when the top scoring patch is not quite the correct one. Removing the offset head costs about 6 points on the fine grained localization metric, since the model falls back to predicting a whole patch center rather than an exact sub patch location. A separate test on how training frames are sampled found that a uniform stride of two frames, rather than sampling every single frame or sampling randomly, gave the best overall balance, forcing the model to learn genuine motion cues rather than near duplicate frame pairs. And a final experiment on memory update strategy at inference found that splitting memory into a dense, frequently updated short term portion plus a sparse, infrequently updated long term portion improved results specifically on the longest videos, additional confirmation that memory in this model behaves as a genuinely learned temporal mechanism rather than a fixed size buffer that only cares about capacity.

Where it still trips up

Because Track-On2 leans heavily on appearance based matching rather than explicit motion modeling, it inherits a predictable weakness, textureless or repetitive surfaces give it little to latch onto. The paper visualizes a grid of points placed across a RoboTAP scene, points sitting on distinctive, strongly textured regions stay locked in place over time, while points on flat, repetitive surfaces gradually drift apart as the matching signal becomes ambiguous. Thin structures pose a related problem, since fine geometric detail can get lost as feature maps are downsampled internally, leading to imprecise matches along narrow edges.

A qualitative comparison on a RoboTAP sequence makes the practical stakes concrete. A competing model, CoTracker3, loses several points on a green block once it is occluded and never recovers them. Another strong baseline, BootsTAPNext, keeps tracking for a while but eventually loses a cube later in the sequence. Track-On2 is the only one of the three that tracks both objects successfully through to the end of the clip. In a separate, almost amusing failure mode, CoTracker3 was observed continuing to mark a robotic arm’s previous position as visible well after the arm had physically moved away, a plausible side effect of iterative methods leaning too heavily on expected motion patterns rather than the actual pixels in front of them.

Efficiency, real time tracking on a single GPU

For sparse tracking of 64 points, Track-On2 runs above 35 frames per second while using well under half a gigabyte of GPU memory. Tracking 256 points holds close to that same frame rate with under 0.5 gigabytes, and even a dense 1,024 point setting stays around 20 frames per second at roughly 1.2 gigabytes. By unifying its predecessor’s two memory modules into one, the model’s throughput jumped from roughly 20 frames per second up to over 30, while several offline rivals that process a full clip or a wide window at once run out of memory on much larger GPUs entirely once sequences get long enough. That efficiency profile is what lets Track-On2 run comfortably on a single consumer grade or mid range GPU rather than requiring a large server class card just to keep up.

What the authors say is still unresolved

The paper’s own limitations section is specific rather than vague. First, despite running comfortably on a high end GPU such as an A100, the model has not been optimized at the level of custom attention kernels, fused tensor operations, or other hardware aware tricks that specialized tracking systems use to run on CPUs or embedded devices, leaving real time performance on constrained hardware as an open problem. Second, the model’s reliance on appearance matching without any explicit model of motion is named directly as a limitation, one the authors suggest addressing by incorporating motion modeling into future versions to improve temporal coherence and reduce dependence on visual similarity alone. Third, the memory update strategy used during training is a simple, fixed first in first out rule purely because of compute constraints on clip length, and the authors point to adaptive memory writes, ones that could vary based on video length or the content of a specific query, as a promising unexplored direction. Finally, the broader challenge of adapting cleanly to real world data, beyond the synthetic training this model relies on entirely, remains open, with better pseudo labeling and real world fine tuning strategies flagged as important future work.

Building a simplified Track-On2 style tracker in PyTorch

The implementation below sketches the three core stages, a lightweight multi scale style encoder, a query decoder running feature attention, self attention, and memory attention in sequence, a bounded first in first out memory with learned temporal position embeddings, and a coarse classification plus offset prediction head. A smoke test at the end runs the full pipeline on randomly generated dummy frames so it can be verified without any real video data.

import torch
import torch.nn as nn
import torch.nn.functional as F

# ----------------------------------------------------------------------
# A simplified multi scale style visual encoder
# ----------------------------------------------------------------------
class VisualEncoder(nn.Module):
    def __init__(self, dim=64):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(3, dim, 3, padding=1), nn.ReLU(),
            nn.Conv2d(dim, dim, 3, padding=1), nn.ReLU(),
        )
        self.dim = dim

    def forward(self, frame):
        # frame: (B, 3, H, W) returns a feature map (B, dim, H, W)
        return self.conv(frame)

    def sample_at(self, feat_map, points):
        # points: (B, N, 2) normalized to [-1, 1] grid_sample coordinates
        points = points.unsqueeze(1)  # (B, 1, N, 2)
        sampled = F.grid_sample(feat_map, points, align_corners=True)
        return sampled.squeeze(2).transpose(1, 2)  # (B, N, dim)


# ----------------------------------------------------------------------
# Bounded per query memory with FIFO eviction and temporal embeddings
# ----------------------------------------------------------------------
class QueryMemory:
    def __init__(self, num_queries, capacity, dim, device="cpu"):
        self.capacity = capacity
        self.buffer = torch.zeros(num_queries, capacity, dim, device=device)
        self.filled = torch.zeros(num_queries, dtype=torch.long, device=device)
        self.temporal_embed = nn.Parameter(torch.randn(capacity, dim, device=device) * 0.02)

    def read(self):
        return self.buffer + self.temporal_embed.unsqueeze(0)

    def write(self, new_entries):
        # new_entries: (num_queries, dim), shift buffer and insert at the end (FIFO)
        self.buffer = torch.cat([self.buffer[:, 1:, :], new_entries.unsqueeze(1)], dim=1)

    def extend(self, new_capacity):
        # inference-time memory extension via linear interpolation of temporal embeddings
        old = self.temporal_embed.detach().unsqueeze(0).transpose(1, 2)
        new = F.interpolate(old, size=new_capacity, mode="linear", align_corners=True)
        self.temporal_embed = nn.Parameter(new.transpose(1, 2).squeeze(0))
        pad = new_capacity - self.buffer.shape[1]
        if pad > 0:
            zeros = torch.zeros(self.buffer.shape[0], pad, self.buffer.shape[2])
            self.buffer = torch.cat([zeros, self.buffer], dim=1)
        self.capacity = new_capacity


# ----------------------------------------------------------------------
# Query decoder, feature attention, self attention, memory attention
# ----------------------------------------------------------------------
class QueryDecoder(nn.Module):
    def __init__(self, dim=64, heads=4):
        super().__init__()
        self.feature_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.self_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.memory_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm1 = nn.LayerNorm(dim)
        self.norm2 = nn.LayerNorm(dim)
        self.norm3 = nn.LayerNorm(dim)

    def forward(self, queries, frame_tokens, memory):
        q = self.norm1(queries)
        feat_out, _ = self.feature_attn(q, frame_tokens, frame_tokens)
        queries = queries + feat_out

        q = self.norm2(queries)
        self_out, _ = self.self_attn(q, q, q)
        queries = queries + self_out

        q = self.norm3(queries)
        mem_out, _ = self.memory_attn(q, memory, memory)
        queries = queries + mem_out
        return queries


# ----------------------------------------------------------------------
# Coarse patch classification plus a sub patch offset head
# ----------------------------------------------------------------------
class PointPredictionHead(nn.Module):
    def __init__(self, dim=64):
        super().__init__()
        self.offset_head = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(), nn.Linear(dim, 2), nn.Tanh())
        self.visibility_head = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(), nn.Linear(dim, 1))

    def classify_patches(self, queries, feat_map, temperature=0.05):
        B, C, H, W = feat_map.shape
        flat = feat_map.flatten(2).transpose(1, 2)  # (B, H*W, C)
        flat = F.normalize(flat, dim=-1)
        q_norm = F.normalize(queries, dim=-1)
        sim = torch.einsum("bnc,bpc->bnp", q_norm, flat) / temperature
        return F.softmax(sim, dim=-1)  # (B, N, H*W) distribution over patches

    def forward(self, queries, feat_map, patch_stride):
        patch_probs = self.classify_patches(queries, feat_map)
        offset = self.offset_head(queries) * patch_stride
        visibility = torch.sigmoid(self.visibility_head(queries))
        return patch_probs, offset, visibility


# ----------------------------------------------------------------------
# Loss combining patch classification, offset, and visibility terms
# ----------------------------------------------------------------------
def tracking_loss(patch_probs, gt_patch_idx, offset, gt_offset, visibility, gt_visibility, lam=3.0):
    cls_loss = F.nll_loss(torch.log(patch_probs + 1e-8).transpose(1, 2), gt_patch_idx)
    offset_loss = F.l1_loss(offset, gt_offset)
    vis_loss = F.binary_cross_entropy(visibility.squeeze(-1), gt_visibility)
    return lam * cls_loss + offset_loss + vis_loss


# ----------------------------------------------------------------------
# Smoke test on a dummy synthetic video, no real dataset required
# ----------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(0)
    B, N, dim, memory_capacity = 1, 8, 64, 12
    H, W, patch_stride = 64, 64, 4

    encoder = VisualEncoder(dim=dim)
    decoder = QueryDecoder(dim=dim)
    head = PointPredictionHead(dim=dim)
    memory = QueryMemory(num_queries=N, capacity=memory_capacity, dim=dim)

    query_points = torch.rand(B, N, 2) * 2 - 1  # normalized coordinates
    video = torch.rand(10, B, 3, H, W)  # 10 dummy frames

    # initialize queries from the first frame
    first_feat = encoder(video[0])
    queries = encoder.sample_at(first_feat, query_points)

    all_predictions = []
    for t in range(1, video.shape[0]):
        feat = encoder(video[t])
        frame_tokens = feat.flatten(2).transpose(1, 2)
        mem_read = memory.read().unsqueeze(0).reshape(B, N * memory_capacity, dim)
        decoded = decoder(queries, frame_tokens, mem_read)
        patch_probs, offset, visibility = head(decoded, feat, patch_stride)
        memory.write(decoded.squeeze(0).detach())
        all_predictions.append((t, patch_probs.shape, offset.shape, visibility.shape))

    # demonstrate inference-time memory extension
    memory.extend(new_capacity=36)

    print("Smoke test complete, ran", len(all_predictions), "frames")
    print("Final memory capacity after extension", memory.capacity)

Running this confirms every stage links up, the encoder produces per frame features, the decoder runs its three attention stages against the current frame and the rolling memory, the prediction head produces a patch distribution plus a refined offset, and the memory extension routine successfully stretches the temporal window without touching any learned weights. As with any smoke test on synthetic tensors, the actual tracking output is not meaningful, real training needs genuine video data and many more refinements the paper describes in full, but the structure mirrors Track-On2’s real design closely enough to build from.

Why this design pattern reaches beyond point tracking

The idea underneath Track-On2 generalizes past this one task. Any system that has to reason about a specific entity, a tracked point, a tracked object, a conversation participant, over an unbounded stream of incoming data faces the same fork in the road, either let computational cost grow with however much history has accumulated, or commit to a fixed size memory per entity and accept that older information eventually gets pushed out. This paper is a clean demonstration that the second choice, done carefully with a learned, order aware memory, does not have to sacrifice accuracy to gain that efficiency, and that the length of training sequences, not memory size alone, is what actually teaches a model to use that memory well over long horizons.

The honest limitations are worth carrying forward too. This is a synthetic data success story specifically, and the authors do not claim the same resilience automatically extends to messier real world footage without further fine tuning work. Teams building on this kind of architecture for genuinely novel domains, aerial footage, endoscopic video, satellite imagery, should treat the synthetic to real world gap as an open question rather than a solved one, exactly as the authors frame it themselves.

Frequently asked questions

Related reading