DeepPro: Revolutionizing Infrared Small Target Detection

DeepPro · Probing the Temporal Profile

A target that is both tiny and nearly invisible

Infrared small target detection sits behind a wide range of practical uses, some civilian, some defense related, spanning maritime search and rescue, early wildfire warning, vehicle and drone monitoring, reconnaissance, and precision guidance, exactly as the researchers themselves describe the field’s applications. What makes the underlying detection problem so stubborn is a combination of two things happening at once. The targets themselves are frequently tiny, often occupying nine pixels by nine pixels or less, and dim, with a signal barely three times stronger than the surrounding noise floor. At the same time, the background is rarely calm. Real infrared scenes are full of clutter, moving clouds, camera vibration, sensor artifacts, much of which fluctuates in ways that look deceptively similar to an actual target passing through.

Layer motion on top of that and the problem compounds further. A real target’s brightness and its immediate surroundings change continuously as it moves, so whatever signature a detector learns to recognize cannot stay fixed either. And because these systems are meant to process live video streams, any solution also has to run fast enough for real time use, which rules out simply throwing more computation at the problem.

Why more information hasn’t been the answer

Single frame methods run out of things to look at

A large family of existing methods works entirely within a single frame, using increasingly elaborate network designs to tease out local contrast, saliency, or texture differences between a target and its background. The trouble is structural. Once a target is dim enough, there is simply not enough spatial signal left in one frame to reliably tell it apart from clutter, no matter how sophisticated the network analyzing that frame becomes.

Short clip methods still miss the bigger picture

A second family looks at motion across a handful of adjacent frames, hoping the target’s movement pattern gives it away. This helps in some cases, but the paper’s central argument is that a few nearby frames are still too short a window. Both families also share a practical cost, chasing more spatial or short term temporal detail tends to expand how much of the image a network has to consider at once, which increases compute substantially and works directly against the real time processing these systems need.

A different place to look, the temporal profile

Rather than expanding outward in space or across a short clip, the authors zoomed into a single fixed pixel location and tracked its brightness across the entire length of a video. They call this trace the temporal profile, and its shape turns out to be remarkably informative. As a target moves through a given pixel’s position, that pixel’s brightness rises, peaks, and falls in one clean bump, wider for bigger or slower targets, narrower for smaller or faster ones, taller for brighter targets. Crucially, this shape survives even when the target is so faint it is invisible in the raw image, since the temporal profile is looking at the same handful of pixels’ worth of signal, just organized along time instead of space.

The more decisive finding is about correlation, not just shape. The team mixed simulated target signals with different intensities of noise and clutter and measured how correlated the combined signal was with itself at different time delays. A genuine target signal turns out to be strongly self correlated, its up down bump follows a predictable, autocorrelated pattern, while noise and clutter signals at that same location remain essentially uncorrelated with the target’s pattern, even when the noise or clutter itself is individually strong.

Proving it with a purpose built attribution tool

A theoretical argument is one thing. The team wanted direct evidence that real, well trained detection networks were actually relying on this signal, rather than assuming it. To get that evidence, they built the first prediction attribution tool designed specifically for infrared small target detection networks, adapting a technique called integrated gradients that was originally developed to explain image classifiers.

Adapting it was not trivial. Integrated gradients was built for networks that output a single class label, while detection networks here output a full per pixel prediction map. The team designed a converter that sums the predicted scores across all the pixels belonging to one target, turning a dense segmentation output back into a single number that attribution could meaningfully explain, and carefully chose a baseline reference value rather than the usual all zero baseline, since an all zero baseline fails to properly highlight faint, dark targets.

Running this tool across many trained networks and random seeds surfaced two consistent findings the authors call Phenomenon 1 and Phenomenon 2. The first is that the pixels a network’s prediction actually depends on are not spread across a wide area following where the target physically moved, as the design of most existing methods would suggest they should be. Instead, the influential pixels cluster tightly in a narrow, almost cylindrical zone at the target’s fixed spatial location, stretched out along the time axis. The second finding is more surprising still. Plotting how much each of nineteen reference frames influenced a prediction produced a U shaped curve rather than a steadily declining one, meaning frames from well in the past or future mattered nearly as much as the most recent frames, directly contradicting the common assumption that only nearby frames carry useful signal.

Redesigning detection as a one dimensional anomaly problem

Guided by both the theory and the attribution evidence, the authors reframed the entire detection task. Instead of asking whether an image patch looks like a target, DeepPro asks whether one pixel’s brightness trace over time looks like a statistical outlier against ordinary background fluctuation, a one dimensional anomaly detection problem running independently at every pixel location.

The core mechanism enabling this is what the authors call a temporal probe, TPro for short. Input features are split into several groups along the channel dimension, and for each pixel, the network extracts that pixel’s complete temporal trace across every frame. A small, learnable correlation matrix, sized to match the number of frames in the window, is then applied to that trace to surface temporal correlation features, essentially learning the same kind of self correlation versus cross correlation distinction the theoretical analysis identified by hand. Multiple such matrices run in parallel, each free to learn a different statistical pattern, and their outputs are fused with a lightweight pointwise convolution.

DeepPro’s full architecture runs three parallel processing levels at different spatial resolutions, purely to give the network multiple views of how quickly a target crosses a location, but every one of the actual feature extraction operations inside those levels, the temporal convolutions and the temporal probe mechanism, computes exclusively along the time axis. No spatial convolution ever touches the core feature path. The authors describe this as the first infrared small target detector where every multiplication and addition in feature extraction happens strictly in the time dimension.

Tested against everything, and it wins on both accuracy and speed

The team evaluated DeepPro on the NUDT-MIRSDT dataset, split into an especially difficult low signal subset and an easier higher signal subset, the IRDST-simulation dataset with over one hundred thousand frames, and a custom stress test dataset called NUDT-MIRSDT-HiNo built by adding strong noise on top of the original data specifically to push detectors to their limit. Comparisons spanned traditional filtering and tensor based methods, deep learning single frame detectors, and deep learning multi frame detectors, evaluated on detection probability, false alarm rate, and area under the ROC curve, alongside parameter count, computational cost, and inference speed.

A version with a touch of space, DeepPro-Plus

A natural question follows from a method built entirely around temporal information, would adding back even a small amount of spatial context help further. The team built DeepPro-Plus to test exactly this, replacing the pure temporal convolutions with spatial temporal convolutions while collapsing the three parallel resolution levels down to just one, keeping the network’s spatial field of view theoretically as narrow as nineteen pixels by nineteen pixels. In every scenario tested, DeepPro-Plus edged out even the original DeepPro, with detection probability improving by more than 20 percentage points over prior state of the art methods in the noisiest scenes specifically, at only a negligible increase in computational cost.

A follow up ablation confirms why one lean level is enough. Comparing DeepPro-Plus against a variant that kept all three levels but added the same spatial temporal convolutions throughout showed almost no accuracy difference between them, while the three level version demanded substantially more compute for that non existent benefit. The lesson lines up neatly with everything else in the paper, a wide spatial receptive field and elaborate feature extraction are not the ingredients that matter here, a small amount of spatial context paired with a strong temporal signal is enough to do the job.

What the ablations confirm

Removing the temporal probe mechanism entirely and replacing it with ordinary convolutions of the same size costs 3.4 and 5.96 percentage points of detection probability on the two toughest test settings, plus a clear rise in false alarms, direct confirmation that the learnable correlation matrices are doing real, irreplaceable work rather than just adding parameters. Testing different window lengths, from as short as five frames up to eighty, showed detection accuracy climbing steadily as the window grew, then plateauing past forty frames while computational cost kept rising, marking forty frames as the practical sweet spot. A shorter five frame window degraded performance sharply, and visualizing the learned correlation matrices explained why, past a length of forty frames those matrices settle into a symmetric pattern reflecting that a signal’s importance depends on its relative distance from the point being predicted rather than its absolute position in time, a property the network simply cannot learn reliably from a shorter window.

Running several correlation matrices side by side rather than just one meaningfully improved detection probability, similar in effect to running several attention heads in parallel, though returns diminished quickly beyond four matrices without any added benefit. The entire mechanism remains remarkably cheap regardless, a single correlation matrix costs only about 6.56 thousand parameters, and the rest of DeepPro’s feature extraction totals just over one hundred thousand parameters. Finally, a direct robustness test across a range of injected noise intensities showed every compared method’s accuracy declining somewhat as noise increased, but DeepPro’s decline stayed noticeably gentler than its strongest rivals, some of which showed sharp jumps in false alarms under the heaviest noise conditions while DeepPro’s false alarm rate stayed comparatively flat.

Where it actually fails

The authors report their failure cases from real world testing directly rather than only showcasing successes. Targets that are partially obscured, a pedestrian walking behind trees in one example, produce an incomplete, unusually narrow spike in their temporal profile, and DeepPro sometimes misses these, either due to the broken signal itself or simply because training data with similarly obscured examples was scarce. The second and more fundamental failure mode is stationary targets. Because DeepPro’s entire premise rests on detecting the rise and fall bump a moving target leaves behind at a fixed pixel location, an object that does not move produces no such bump at all, its brightness simply holds steady for the whole window, indistinguishable from an unchanging background. The authors state plainly that DeepPro is built for moving infrared small targets specifically, not stationary ones, a scoping choice worth understanding clearly before applying this approach to a use case involving parked vehicles or loitering objects.

Building a simplified temporal probe in PyTorch

The implementation below sketches DeepPro’s central idea, a temporal probe module that extracts each pixel’s full time trace and runs it through learnable correlation matrices, a temporal difference residual block inspired by the paper’s design, and a small multi level network that fuses everything into a per pixel, per frame confidence map. A smoke test at the end runs the pipeline on a randomly generated dummy infrared style video so it can be verified without any real dataset.

import torch
import torch.nn as nn
import torch.nn.functional as F

# ----------------------------------------------------------------------
# Temporal Probe mechanism (TPro), calculations strictly along time
# ----------------------------------------------------------------------
class TemporalProbe(nn.Module):
    def __init__(self, channels, time_len, num_groups=4):
        super().__init__()
        assert channels % num_groups == 0
        self.num_groups = num_groups
        self.group_channels = channels // num_groups
        self.time_len = time_len
        # one learnable T x T correlation matrix (SCorM) per group
        self.corr_matrices = nn.ParameterList([
            nn.Parameter(torch.randn(time_len, time_len) * 0.02) for _ in range(num_groups)
        ])
        self.fuse = nn.Sequential(
            nn.Conv3d(channels, channels, kernel_size=1),
            nn.BatchNorm3d(channels),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        # x: (B, C, T, H, W)
        B, C, T, H, W = x.shape
        groups = torch.split(x, self.group_channels, dim=1)
        corr_outputs = []
        for g, W_corr in zip(groups, self.corr_matrices):
            # reshape so the time axis is last, then apply the T x T correlation matrix
            g_flat = g.permute(0, 1, 3, 4, 2).reshape(B, self.group_channels, H * W, T)
            g_corr = torch.matmul(g_flat, W_corr)  # apply correlation along time
            g_corr = g_corr.reshape(B, self.group_channels, H, W, T).permute(0, 1, 4, 2, 3)
            corr_outputs.append(g_corr)
        fused_input = torch.cat(corr_outputs, dim=1)
        return self.fuse(fused_input)


# ----------------------------------------------------------------------
# Temporal Difference residual block, no spatial convolution involved
# ----------------------------------------------------------------------
class TDResBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.td_conv = nn.Conv3d(channels, channels, kernel_size=(3, 1, 1), padding=(1, 0, 0), dilation=(2, 1, 1))
        self.t_conv = nn.Conv3d(channels, channels, kernel_size=(3, 1, 1), padding=(1, 0, 0))
        self.p_conv = nn.Conv3d(channels, channels, kernel_size=1)
        self.norm = nn.BatchNorm3d(channels)

    def forward(self, x):
        residual = x
        out = F.relu(self.td_conv(x))
        out = F.relu(self.t_conv(out))
        out = self.p_conv(out)
        out = self.norm(out + residual)
        return F.relu(out)


# ----------------------------------------------------------------------
# A simplified single level DeepPro style detector
# ----------------------------------------------------------------------
class SimpleDeepPro(nn.Module):
    def __init__(self, in_channels=1, base_channels=16, time_len=40, num_groups=4):
        super().__init__()
        self.stem = nn.Conv3d(in_channels, base_channels, kernel_size=(5, 1, 1), padding=(2, 0, 0))
        self.td_block = TDResBlock(base_channels)
        self.tpro = TemporalProbe(base_channels, time_len, num_groups)
        self.head = nn.Conv3d(base_channels, 1, kernel_size=1)

    def forward(self, x):
        # x: (B, 1, T, H, W), a stack of grayscale infrared frames
        feat = F.relu(self.stem(x))
        feat = self.td_block(feat)
        feat = self.tpro(feat)
        confidence = torch.sigmoid(self.head(feat))
        return confidence.squeeze(1)  # (B, T, H, W)


# ----------------------------------------------------------------------
# Soft IoU style loss for sparse target segmentation
# ----------------------------------------------------------------------
def soft_iou_loss(pred, target, eps=1e-6):
    intersection = (pred * target).sum(dim=[1, 2, 3])
    union = (pred + target - pred * target).sum(dim=[1, 2, 3])
    iou = (intersection + eps) / (union + eps)
    return (1.0 - iou).mean()


# ----------------------------------------------------------------------
# Smoke test on a dummy infrared style video, no real dataset required
# ----------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(0)
    B, T, H, W = 2, 40, 64, 64

    # synthetic dim background plus a faint moving target bump at one pixel
    video = torch.rand(B, 1, T, H, W) * 0.1
    target_track = torch.zeros(T)
    bump_center, bump_width = T // 2, 6
    for t in range(T):
        target_track[t] = torch.exp(torch.tensor(-((t - bump_center) ** 2) / (2 * bump_width ** 2))) * 0.3
    video[:, 0, :, 32, 32] += target_track

    gt_mask = torch.zeros(B, T, H, W)
    gt_mask[:, bump_center - 3:bump_center + 3, 32, 32] = 1.0

    model = SimpleDeepPro(time_len=T)
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

    for step in range(20):
        optimizer.zero_grad()
        pred = model(video)
        loss = soft_iou_loss(pred, gt_mask)
        loss.backward()
        optimizer.step()

    print("Smoke test complete, final loss", round(loss.item(), 4))
    print("Predicted confidence at the target pixel over time")
    print(pred[0, :, 32, 32].detach().round(decimals=3))

Running this trains the tiny model for a handful of steps on a synthetic clip with one faint, deliberately embedded target bump, and prints the network’s learned confidence at that exact pixel across time, which should rise noticeably around the bump’s center after training. As with any smoke test on synthetic data, this is meant to verify the plumbing rather than to demonstrate real detection performance, genuine training needs the actual benchmark datasets and the full three level architecture the paper describes, but the core temporal probe mechanism here mirrors DeepPro’s real design.

Why looking at time, not space, travels beyond infrared sensors

The pattern underneath this paper generalizes well past infrared cameras specifically. Any sensor that repeatedly measures the same fixed location over time, a security camera pixel, a seismic monitoring station, a single channel in an industrial sensor array, can in principle be treated the same way, as a one dimensional time series worth running anomaly detection on directly, rather than only ever being analyzed alongside its spatial neighbors. The attribution methodology is arguably just as exportable as the architecture. Building a dedicated tool to check what a trained detection network is actually paying attention to, rather than assuming it matches the intuition baked into the network’s design, is a habit worth adopting well beyond this one narrow task.

The honest scoping in this paper matters for anyone considering the approach for a new setting. DeepPro’s entire design is built around the assumption that a genuine target produces a distinctive rise and fall pattern by physically moving through a fixed sensor location over time. That assumption breaks down cleanly and predictably for stationary objects and for targets obscured long enough to leave an incomplete signal, and the authors say so directly rather than leaving it to be discovered later. Anyone adapting this idea to a new domain, tracking a different kind of faint, moving signal against a noisy fixed background, should treat that motion assumption as the load bearing wall of the whole method.

Frequently asked questions

Related reading