MC-DeTra Sharpens Driving Forecasts at No Inference Cost

Why splitting perception from prediction costs you

An autonomous vehicle must perceive the actors around it and predict how they will move. The classical way to build this uses three modules in a chain, a 3D object detector, then a tracker that links detections across time, then a separate motion forecaster that guesses future paths. That factorization is convenient but lossy. The forecaster is cut off from the raw sensor context the detector saw, and any error in detection or tracking compounds as it flows downstream. By the time the forecaster runs, it is reasoning about a cleaned up abstraction rather than the messy evidence.

A newer line of work fuses these tasks into a single network that consumes LiDAR and a high definition map and emits, for each actor, both a current bounding box and a set of possible future trajectories. The strongest such model to the authors’ knowledge is called DeTra, which frames the whole thing as one trajectory refinement problem, where the first pose of a trajectory is the current detection and the remaining poses are multimodal future waypoints, all refined together over a shared representation. It is a clean idea. This line connects to broader work on both halves of the problem, from 3D object detection over LiDAR point clouds to joint trajectory prediction for driving.

Two obstacles motivate this paper. First, DeTra has no public implementation, which makes it hard to reproduce or extend, so the authors, from the Moscow Institute of Physics and Technology and the Artificial Intelligence Research Institute, built and released their own documented reimplementation. Second, and more interesting, forecasting the actors that are actually moving is much harder than detecting them, and static parked cars dominate urban LiDAR scenes. Those static actors inflate aggregate forecasting scores, which makes a model look good on average while it quietly underperforms on exactly the moving vehicles that matter most for safety.

Three free signals hiding in the annotations

The central idea of MC-DeTra is that the annotations used to train the future prediction already contain other useful signals that DeTra never exploits, and that each can be turned into a training only auxiliary objective. Because these objectives attach to representations the network already computes and are removed at inference, the deployed model runs exactly as fast as the plain reproduction. They shape how the model learns without costing anything when it runs.

The first signal is the observed past. The network already estimates where each actor is now, so it can also be asked to reconstruct where that actor just came from. This Past Reconstruction objective decodes each matched actor’s recent history from the same current frame query the detection head uses, through a small recurrent decoder, and penalizes the error against the cached ground truth history.

The intuition is that reconstructing where an actor came from forces the current query to encode short term dynamics, which are exactly the dynamics that also help predict where it is going. The second signal is the surrounding traffic itself. A dormant bird eye view head predicts an occupancy and flow field over the shared grid, supervised by rasterizing the ground truth boxes over a window of past and future frames. This Occupancy Auxiliary grounds the shared representation in where the scene’s mass actually is and how it moves, which is the social context an actor’s behavior depends on. The same instinct animates other multi actor perception work, such as grounding a model in the full layout of a scene rather than one object at a time.

The third signal is different in kind. Heading Consistency does not add new supervision from the data at all. It is a constraint between two of the model’s own outputs. For a moving actor, the oriented box should point the same way the model predicts the actor will travel, yet DeTra never couples the box orientation to the forecast. Heading Consistency ties them together.

where the set is the matched moving actors whose predicted future travels more than half a meter, the first term is the predicted box yaw, and the second is the direction of the predicted best trajectory. Because the target is the model’s own forecast rather than a ground truth heading, this is a self consistency term, not a second supervision.

Measuring which signal actually pushes

Here is where the paper gets genuinely useful. When several losses share one backbone, their gradients compete, and naive unit weights are misleading. A term can look conceptually important yet contribute almost nothing to the shared representation, or a small looking term can quietly dominate. So the authors measure each objective’s weighted gradient norm at the shared trunk and report it as a fraction of the forecast gradient.

This diagnostic is the quiet centerpiece, because it turns a vague question, which auxiliary matters, into a number. The answer is clear and a little surprising.

Occupancy is the heavyweight, pushing on the shared representation with about a third of the strength of the forecast loss itself, which is why it earns the largest auxiliary weight. Past reconstruction is a modest contributor. And heading consistency, despite a respectable nominal weight of 0.25, is starved to two hundredths of a percent. That number reframes what looked like an inert term. It is not that heading consistency does nothing conceptually, it is that the forecast waypoints are pinned so firmly by the main trajectory loss that the heading term has almost no gradient to give, so it ends up being met almost entirely by gently rotating the box yaw toward the predicted motion. The diagnostic also warns against over weighting a starved term, since pushing heading consistency up to a large weight became the worst configuration in the sweep. The authors even implemented a fourth term, a trajectory occupancy consistency, found it gradient starved at every stable weight, and reported it as a negative result rather than burying it.

What the numbers say

Everything is evaluated on the Waymo Open Dataset with a single base checkpoint that initializes every variant, so the rows differ only in which losses are enabled. The first thing to check is that the auxiliaries did not quietly damage detection, and they did not.

Every variant lands within a rounding error of the no auxiliary control on detection, so there is no detection versus forecasting trade off here. The interesting movement is in forecasting, and specifically in the dynamic split, the moving vehicles that the aggregate score tends to hide.

The full model takes the best final displacement error, average displacement error, and brier score on moving vehicles, improving dynamic minFDE by about 2.4 percent over the initializer and 2.9 percent over plain fine tuning. Two patterns stand out. Continued fine tuning alone actually made forecasting slightly worse, so the gain is coming from the auxiliaries, not just more training. And heading consistency on its own, at a heavy weight, was the worst row, which is exactly what the gradient diagnostic predicted for an over weighted starved term. The paper is careful to say the margin over the past reconstruction plus occupancy pairing is small, and it presents the full model as the best calibrated configuration rather than a decisive win.

Because the improvement is small, the authors look at the per actor distribution rather than just the mean. Across 135,000 dynamic vehicles, MC-DeTra improved the forecast for 54.6 percent and worsened it for 45.4 percent, with near balanced tails and a mean shift of about six centimeters. That is an honest picture of a reliable small gain, not a dramatic one.

Free at inference, and it stays that way

The whole design rests on the auxiliaries being removable, and the latency table confirms the payoff. Because the helper heads are deleted after training, the deployed model runs at the same speed as the plain reproduction.

Even on a laptop GPU the model stays inside the 500 millisecond budget for a 2 hertz prediction cycle, and peak memory is a modest few gigabytes. This is the practical case for train only auxiliaries. You get whatever representational benefit they provide during learning, and you pay nothing for it when the car is driving. A method that improved forecasting by adding an occupancy head that had to run at inference would be a much harder sell.

Where it falls short

The authors are unusually forthright about limits, which is a point in the paper’s favor. The gains are modest and concentrated on dynamic actors, and the lead over the simpler past reconstruction plus occupancy pairing is small enough that they decline to call it a decisive win. The per actor improvements are broadly distributed with near balanced tails, a reliable small step rather than a leap.

There are honest reproduction caveats too. Because DeTra released no code, this reimplementation re derived several components and made documented approximations, including a lighter map encoder and a simpler attention implementation, and it trained for far fewer steps than the original. As a result the reproduction sits below the published DeTra numbers on both detection and forecasting error. The comparisons in the paper are all internal, from one shared initializer under one protocol, which is the right way to make the auxiliary deltas meaningful, but it means these are not leaderboard numbers against the original.

Two more caveats matter for anyone building on this. The results are single run ablations, so the paper explicitly does not claim statistical significance and notes that multi seed training and confidence intervals would be needed to pin down uncertainty on gains this small. And the auxiliary weights were selected on the same validation protocol used for reporting, which the authors acknowledge may be mildly optimistic, since a separate held out tuning split would be cleaner. The study also covers vehicles only, not pedestrians or cyclists. None of these sink the contribution, but they right size it.

Why the approach travels

The transferable lesson is methodological, and it outlasts this particular model. In any network where several objectives share a backbone, conceptual importance is not the same as actual influence, and the honest way to tell them apart is to measure the fraction of the trunk gradient each term contributes. That single diagnostic caught a signal that looked useful doing almost nothing, and it justified weighting the occupancy term heaviest. Anyone training a multi task model on a shared representation could run the same check before hand tuning weights blindly.

The second portable idea is the value of train only supervision. When a useful signal can be recovered from data you already have, or from a consistency between outputs you already produce, adding it as a removable auxiliary is close to a free lunch, because it shapes the representation during learning and vanishes at deployment. That pattern fits far beyond driving, anywhere a model has a shared trunk and a latency budget. It sits alongside the broader push in autonomous perception toward models that reason about whole scenes and multiple actors at once, a theme running through work from rethinking multi object tracking to unified detection and prediction. MC-DeTra’s contribution is less a new architecture than a disciplined way to decide what to feed one.

Reference implementation in PyTorch

The code below is a runnable reconstruction of MC-DeTra’s core ideas, the three train only auxiliary heads and the gradient norm diagnostic, based on the paper’s equations. A small stub stands in for the shared bird eye view trunk and the detection and forecast heads so the file runs without LiDAR data. It includes the Past Reconstruction recurrent decoder, an Occupancy Auxiliary head, the Heading Consistency self alignment loss, a gradient norm calibrator that measures each term’s pull on the shared trunk, and a training step that drops every auxiliary at inference. A smoke test runs it on dummy tensors. Swap in the real trunk and Waymo data for actual experiments.

# mc_detra_reference.py
# Train only auxiliaries and gradient norm calibration for a joint detector forecaster.
# Replace SharedTrunkStub with the DeTra style BEV backbone for real runs.

import torch
import torch.nn as nn
import torch.nn.functional as F

D = 128          # shared feature dim
K, T = 6, 11     # 6 modes, current frame plus 10 future waypoints
H = 6            # history steps for past reconstruction


class SharedTrunkStub(nn.Module):
    """Stand in for the shared BEV trunk. Emits per query features + a BEV map."""
    def __init__(self, n_query=64):
        super().__init__()
        self.bev = nn.Sequential(nn.Conv2d(32, D, 3, 1, 1), nn.ReLU(inplace=True))
        self.query = nn.Linear(D, D)

    def forward(self, bev_in, q_seed):
        bev = self.bev(bev_in)                    # (B, D, Hc, Wc) shared map
        q = self.query(q_seed)                    # (B, N, D) actor queries
        return bev, q


class DetForecastHeads(nn.Module):
    """Detection box plus K mode, T slot trajectory volume."""
    def __init__(self):
        super().__init__()
        self.box = nn.Linear(D, 5)                # cx, cy, w, l, yaw
        self.traj = nn.Linear(D, K * T * 2)       # future waypoints per mode
        self.mode = nn.Linear(D, K)

    def forward(self, q):
        B, N, _ = q.shape
        box = self.box(q)
        traj = self.traj(q).view(B, N, K, T, 2)
        mode = self.mode(q).softmax(-1)
        return box, traj, mode


class PastReconstruction(nn.Module):
    """Train only. Decode the observed past from the current query, Eq 1."""
    def __init__(self):
        super().__init__()
        self.gru = nn.GRU(D, D, batch_first=True)
        self.out = nn.Linear(D, 2)
        self.seed = nn.Parameter(torch.randn(H - 1, D))

    def forward(self, q, past_gt, mask):
        B, N, _ = q.shape
        h0 = q.reshape(1, B * N, D)
        seed = self.seed.unsqueeze(0).expand(B * N, -1, -1)
        pred = self.out(self.gru(seed, h0)[0]).view(B, N, H - 1, 2)
        err = F.smooth_l1_loss(pred, past_gt, reduction="none").sum(-1)
        return (mask * err).sum() / mask.sum().clamp(min=1)


class OccupancyAux(nn.Module):
    """Train only. Predict a BEV occupancy field on the shared map."""
    def __init__(self):
        super().__init__()
        self.head = nn.Conv2d(D, 1, 1)

    def forward(self, bev, occ_gt):
        logit = self.head(bev)
        return torchvision_sigmoid_focal(logit, occ_gt)   # focal, alpha 0.75 gamma 2


def torchvision_sigmoid_focal(logit, target, alpha=0.75, gamma=2.0):
    p = torch.sigmoid(logit)
    ce = F.binary_cross_entropy_with_logits(logit, target, reduction="none")
    pt = p * target + (1 - p) * (1 - target)
    w = alpha * target + (1 - alpha) * (1 - target)
    return (w * (1 - pt) ** gamma * ce).mean()


def heading_consistency(box, traj, mode, tau_v=0.5):
    """Train only. Align box yaw with predicted best mode motion, Eq 2."""
    best = mode.argmax(-1)                         # (B, N)
    bt = torch.gather(traj, 2, best[..., None, None, None].expand(-1, -1, 1, T, 2))
    bt = bt.squeeze(2)
    disp = bt[:, :, -1] - bt[:, :, 0]              # net displacement
    moving = disp.norm(dim=-1) > tau_v
    ang = torch.atan2(disp[..., 1], disp[..., 0])
    yaw = box[..., 4]
    term = 1 - torch.cos(yaw - ang)
    return (term * moving).sum() / moving.sum().clamp(min=1)


def trunk_grad_norm(loss, trunk_params, weight):
    """g_k = w_k * || grad of L_k wrt trunk ||, the calibration diagnostic, Eq 3."""
    g = torch.autograd.grad(loss, trunk_params, retain_graph=True,
                            allow_unused=True, create_graph=False)
    flat = torch.cat([x.reshape(-1) for x in g if x is not None])
    return (weight * flat.norm()).item()


if __name__ == "__main__":
    B, N = 2, 64
    trunk, heads = SharedTrunkStub(), DetForecastHeads()
    pr, oa = PastReconstruction(), OccupancyAux()
    bev_in = torch.rand(B, 32, 48, 48)
    q_seed = torch.rand(B, N, D)
    bev, q = trunk(bev_in, q_seed)
    box, traj, mode = heads(q)

    past_gt = torch.rand(B, N, H - 1, 2)
    mask = (torch.rand(B, N, H - 1) > 0.2).float()
    occ_gt = (torch.rand(B, 1, 48, 48) > 0.9).float()

    L_pr = pr(q, past_gt, mask)
    L_oa = oa(bev, occ_gt)
    L_hc = heading_consistency(box, traj, mode)
    trunk_p = list(trunk.parameters())
    print("PR trunk grad %", round(trunk_grad_norm(L_pr, trunk_p, 0.05), 5))
    print("OA trunk grad %", round(trunk_grad_norm(L_oa, trunk_p, 0.30), 5))
    print("HC trunk grad %", round(trunk_grad_norm(L_hc, trunk_p, 0.25), 5))
    # at inference: build only trunk + heads, skip pr, oa, hc entirely

Conclusion

The core achievement of MC-DeTra is twofold, and both halves are useful even though the accuracy gain is modest. It gives the community an openly released, documented reimplementation of the strongest joint detection and forecasting model, which had no public code, and it shows that three signals hiding in the existing annotations and outputs can shape a shared representation to forecast moving vehicles a little better, at no cost when the model runs. On the Waymo Open Dataset it posted the best displacement errors on dynamic actors while leaving detection untouched.

The conceptual contribution that will outlast the specific numbers is the gradient norm view of a shared backbone. When many objectives compete at one trunk, the right question is not which one sounds important but which one actually moves the representation, and measuring the fraction of the trunk gradient each term contributes answers it directly. That diagnostic caught the heading term contributing two hundredths of a percent while occupancy carried a third of the forecast gradient, and it explained both why heading looked inert and why over weighting it made things worse.

The design also points at a broadly useful pattern. Train only auxiliaries are close to free, because they shape learning and then disappear, so the honest cost accounting is entirely on the training side. When a helpful signal can be recovered from data you already have, or from a consistency between outputs you already produce, adding it as a removable head is a low risk way to nudge a shared representation, whether the task is driving or anything else with a latency budget.

The honest limitations keep the result in proportion. This is a preprint whose gains are small and concentrated on moving actors, whose reproduction trails the original model it rebuilds, whose ablations are single run without significance testing, and whose auxiliary weights were tuned on the reporting split. The authors say all of this plainly, and they report a failed fourth auxiliary as a negative result rather than hiding it, which is the kind of candor that makes the positive claims easier to trust.

For anyone building multi task perception, the practical takeaway is compact. Before hand tuning loss weights, measure how hard each term actually pulls on the shared trunk, because your intuition about importance will sometimes be wrong. And when you find a useful signal, ask whether it can live entirely in training, so it improves the model without ever slowing it down. MC-DeTra is a preprint with public code, and the reference above is a place to start testing both ideas on a model of your own.

Frequently asked questions

What is joint detection and trajectory forecasting?

It is the approach of using a single network to both detect the vehicles around a self driving car and predict their future paths, instead of chaining a separate detector, tracker, and forecaster. Fusing the tasks lets the forecaster use the same sensor context the detector saw and avoids errors compounding across module handoffs.

What does MC-DeTra add to the base model?

It adds three training only auxiliary objectives that shape the shared bird eye view representation and are removed before inference. They reconstruct each actor’s observed past, predict the occupancy of surrounding traffic, and align a vehicle’s box heading with its predicted direction of motion. Because they are removed at test time, they add no inference latency.

Why measure gradient norms at the shared trunk?

When several losses share one backbone their gradients compete, and a term can look important yet contribute almost nothing, or a small looking term can dominate. Measuring each objective’s weighted gradient norm at the trunk, as a fraction of the forecast gradient, shows which signal actually shapes the representation. It revealed that occupancy carried about 35 percent while heading consistency was starved to 0.02 percent.

How much did MC-DeTra improve forecasting?

On moving vehicles it posted the best final and average displacement errors and the best brier score, improving dynamic minFDE from 2.0407 to 1.9915, about 2.4 percent over the initializer, while detection accuracy stayed flat. Across 135,000 dynamic vehicles it improved 54.6 percent of forecasts and worsened 45.4 percent, a reliable but small gain.

Do the extra objectives slow the model down?

No. The auxiliary heads are used only during training and are deleted for deployment, so the running model matches the plain reproduction. It stays within the 500 millisecond budget for a 2 hertz prediction cycle even on a laptop GPU, at 360.7 milliseconds, and uses only a few gigabytes of memory.

What are the main limitations?

The gains are modest and concentrated on dynamic actors, and the lead over a simpler two auxiliary pairing is small. It is an unreviewed preprint whose reimplementation trails the original DeTra, its ablations are single run without significance testing, its auxiliary weights were tuned on the reporting split, and it covers vehicles only rather than pedestrians or cyclists.