RA-SOD Detects Salient Objects When Sensors Fail

Why two sensors are not automatically better than one

RGB and thermal salient object detection, usually shortened to RGB-T SOD, asks a model to find the visually prominent objects in a scene using two aligned inputs, an ordinary color image and a thermal infrared image. The appeal is complementarity. Color carries texture and fine detail, thermal carries heat signatures that survive darkness and glare. Put them together and you should beat either one alone, especially in low light and cluttered backgrounds.

The problem runs deeper than it first appears. In the real world the quality of each feed is unstable. RGB degrades under low illumination, motion blur, and heavy noise. Thermal suffers from contrast compression, suppressed temperature gradients, and block like artifacts, including the hot block effect where a warm surface floods a region with false signal. The paper groups all of this under one name, modality degradation, and the trouble is not just that a degraded feed is less useful. It is that a degraded feed provides misleading evidence, and misleading evidence actively corrupts fusion.

Here is the part most methods miss. They assume the features pulled out by the encoder are reliable and that trouble only shows up later, when the two modalities are combined. In practice, degradation appears early and spreads. A standard backbone applies the same fixed transformation to every input, so it cannot adapt when a signal is distorted. Noise gets propagated up the hierarchy and amplified, and by the time the fusion module runs, it is working on already corrupted representations. Trying to separate real structure from sensor noise at that stage is close to hopeless.

The idea, reliability as a first class citizen

RA-SOD, from a team at Harbin Institute of Technology and the University of Sydney, makes one commitment and follows it through the whole network. Reliability is not an afterthought or a late stage weighting trick. It is a core quantity that gets modeled and propagated across three stages, representation, refinement, and fusion. Each stage has its own mechanism for asking the same question, how much should I trust this signal right here.

At the encoder, a reliability conditioned backbone adaptively compensates degraded features while keeping the stable structure the pretrained backbone already knows. In the decoder, an uncertainty guided dual stream scheme predicts where each modality is unreliable and uses that to regulate how the two modalities correct each other. At fusion, a pixel wise competition step lets RGB and thermal contend for influence at every single location. The name to keep in mind is that the whole design is about not letting a local failure become a global one. This is the same instinct behind related work we have covered on routing each feature point by a learned confidence score, pushed here into a full reliability pipeline.

Stage one, a backbone that adapts to bad inputs

The first component is the Reliability-Conditioned Backbone Representation, or RCBR. It starts from a frozen pretrained backbone, which produces a stable base feature at each stage. Freezing it is deliberate. That backbone carries a dependable structural prior learned from clean data, and you do not want degradation to wreck it. So rather than fine tune the backbone, RA-SOD adds a parallel residual branch that learns to correct the base feature when a modality is degraded.

The correction is routed. From the base feature, a lightweight gate predicts routing weights that sum to one.

Why route on the base feature? Because degradation shifts the distribution of that feature in a telltale way, so conditioning the routing decision on it lets the model implicitly sense which modality is reliable without ever being told directly. The residual itself comes from a mixture of experts, a bank of shared transformations that each capture a different adaptation pattern, plus a small modality specific low rank adapter.

The design is a clean split of labor. The frozen backbone holds the stable prior, and the routed residual supplies adaptive correction only where it is needed. The ablations later show this pairing beats both the frozen backbone alone and the residual alone. The mixture of experts idea here echoes other multimodal work such as expert routing that lets modalities cancel out each other’s noise, applied at the representation stage rather than at fusion.

Stage two, letting the network say what it is unsure about

The second component is Uncertainty-Guided Dual-Stream Decoding, or UGDD. RGB and thermal each get their own top down decoding stream, and the twist is that each stream predicts its own uncertainty map as it goes.

That uncertainty map is the interesting part, because it controls how the two modalities correct each other. When one stream refines the other, the correction is weighted by uncertainty.

Read that carefully, because the logic is elegant. The RGB stream pulls in help from the thermal stream, but only in proportion to how uncertain the RGB stream is about itself. Where RGB is confident, it ignores the other modality. Where RGB is shaky, it leans on thermal. The same holds in reverse. Correction flows toward doubt, not away from it, which is exactly what you want when the whole point is to shore up the weak spots without letting a strong modality get overwritten.

Does the uncertainty map actually track degradation, or is it just noise dressed up as confidence? The paper checks. Averaged across a corrupted benchmark, the RGB uncertainty gate sits near 0.32 on clean inputs, then climbs to 0.57 under low light, 0.48 under noise, and 0.51 under blur, the three RGB specific corruptions. The thermal gate stays flat through those, then rises to 0.55 under contrast loss and 0.53 under the hot block artifact, the two thermal corruptions. The gate rises only for the modality that actually failed. That is a genuinely reassuring result.

Stage three, a competition at every pixel

The third component runs a separate fusion stream alongside the two modality streams, and it settles the question of which modality wins where. Instead of a fixed blend, RA-SOD runs a pixel wise modality competition. At each spatial location the model predicts a pair of weights for RGB and thermal.

The weights compete through a softmax, so at any given pixel the two modalities split a fixed budget of influence. A location where thermal is clean and RGB is blurred tilts toward thermal, and a neighboring location can tilt the other way. Fixed fusion weights cannot do this. They apply one compromise to the whole image, which is wrong wherever the reliability of the two feeds diverges within a single frame, which under real degradation is almost everywhere.

All three streams, RGB, thermal, and fusion, produce their own saliency prediction through a lightweight head, and the three are summed into the final map. Training applies deep supervision to all three with a combination of binary cross entropy and an intersection over union loss, which keeps any single modality from dominating and yields complementary cues.

What the numbers say

RA-SOD was trained on 2500 image pairs from VT5000 and evaluated on the rest of VT5000 plus the full VT821 and VT1000 benchmarks, with the harder VT-IMAG set held for stress testing. The backbone is an ImageNet pretrained Res2Net-50, inputs are resized to 352 by 352, and training ran for 100 epochs on a single RTX 3090. Five standard metrics are reported, the structure measure, mean F measure, weighted F measure, mean enhanced alignment measure, and mean absolute error, where lower is better only for the last one.

On the largest and most diverse benchmark, VT5000, RA-SOD takes the top result on every metric.

The gains over the strong ConTriNet baseline are modest on clean data, roughly a point of mean F measure and weighted F measure, which is about what you expect near the top of a mature benchmark. On the smaller VT821 and VT1000 sets RA-SOD stays within the top two on every metric. The clean benchmark story is that RA-SOD is competitive with the best, not that it rewrites the leaderboard. The real argument is elsewhere.

That argument is degradation. When the authors deliberately corrupted one modality at a time on VT1000, the ranking held up in a way it did not for others.

The spread widens exactly where it should. Against the strongest competitor RA-SOD lifts the weighted F measure from 0.895 to 0.907 under degradation, a bigger margin than on clean data, because the competitors have no mechanism to notice a feed has gone bad. On the deliberately brutal VT-IMAG set, which piles on thermal crossover, bad weather, and strong noise, RA-SOD leads on four of five metrics and improves mean F measure over ConTriNet by 0.020. This is the pattern that matters. A method built for reliability should pull ahead precisely when reliability is under attack, and it does.

Which parts pull weight, and the cost

The ablations trace the accuracy to specific parts. Removing the uncertainty guided decoding drops the weighted F measure from 0.913 to 0.908 on VT1000 and, more tellingly, from 0.907 to 0.897 on the averaged degradation setting, so its value grows under stress. Replacing the pixel wise competition with a fixed attention module lowers scores across all three benchmarks. Building the reliability conditioned backbone up piece by piece, adding the low rank adapter and then the expert bank, improves results at each step. The authors also swept the number of experts and found three to be the sweet spot, since fewer limit transformation diversity and more bring redundancy and harder optimization.

None of this is free, but the cost is reasonable. RA-SOD uses 46.72 million parameters and 60.13 billion floating point operations, running at 11.24 frames per second. That is heavier than the lean ConTriNet at 34.78 million parameters, but it is the second lightest model in the comparison and faster than several competitors. The reliability machinery adds overhead, yet the model stays in practical range rather than ballooning. The design also transferred cleanly to a Swin-B transformer backbone, where it again took the top spot on VT5000, which suggests the reliability idea is not tied to one kind of encoder.

Where it still falls short

The honest caveats start with the nature of the evidence. This is a preprint, and while the code and models are public, the degradation experiments lean on synthetic corruptions applied to clean benchmarks rather than a large corpus of genuinely degraded field data. Synthetic low light and simulated hot block artifacts are reasonable stand ins, but they are not the same as the messy, correlated failures a real sensor rig produces on a bad day. The VT-IMAG results push toward realism, yet the strongest claim would still need validation on data captured under actual adverse conditions.

There is also a quieter assumption worth naming. The whole scheme presumes the two modalities are well aligned spatially, so that a pixel wise competition and a cross modal correction even make sense location by location. Real RGB and thermal rigs have calibration drift and parallax, and the paper does not stress test what misalignment does to the uncertainty maps. And while the uncertainty gate responds cleanly to the single modality corruptions the authors designed, it is less clear how it behaves when both feeds degrade at once, or when a degradation mimics a real object, such as a genuine hot spot that looks like a thermal artifact.

Why the approach travels

Strip away the salient object detection specifics and RA-SOD is a template for any system that fuses inputs of uneven, changing quality. The pattern is worth holding onto. Do not assume your inputs are clean. Predict where each one is unreliable, let correction flow toward the uncertain regions rather than away from them, and resolve conflicts locally rather than with one global weight. That recipe would fit multi sensor robotics, medical scans of varying quality, and any perception stack that must survive a sensor having a bad moment.

It also sits in a clear lineage. The routing by reliability recalls availability aware fusion that decides participation before it fuses, and the broader move toward robust multimodal perception shows up across recent work on fusing RGB and thermal for segmentation. RA-SOD’s specific contribution is to make reliability the organizing principle from the first layer to the last, rather than a correction bolted on at the end.

Reference implementation in PyTorch

The code below is a runnable reconstruction of the RA-SOD idea based on the equations in the paper. A compact convolutional encoder stands in for the frozen Res2Net-50 backbone so the file runs without external weights. It includes the reliability conditioned backbone with feature conditioned routing, a mixture of experts residual and a low rank adapter, the uncertainty guided dual stream cross modal correction, the pixel wise modality competition, the tri stream heads, the combined binary cross entropy and intersection over union loss, a training step, and a smoke test on dummy tensors. Swap in the real backbone and dataset for actual experiments.

# ra_sod_reference.py
# Runnable reconstruction of the RA-SOD reliability pipeline.
# Replace FrozenStub with a frozen Res2Net-50 for real experiments.

import torch
import torch.nn as nn
import torch.nn.functional as F


class FrozenStub(nn.Module):
    """Stand in for the frozen pretrained backbone. Two scale features."""
    def __init__(self, in_ch=3, dims=(64, 128)):
        super().__init__()
        self.s1 = nn.Sequential(nn.Conv2d(in_ch, dims[0], 3, 2, 1),
                                nn.BatchNorm2d(dims[0]), nn.ReLU(inplace=True))
        self.s2 = nn.Sequential(nn.Conv2d(dims[0], dims[1], 3, 2, 1),
                                nn.BatchNorm2d(dims[1]), nn.ReLU(inplace=True))
        for p in self.parameters():
            p.requires_grad = False                    # frozen prior

    def forward(self, x):
        f1 = self.s1(x)
        f2 = self.s2(f1)
        return [f1, f2]


class RCBRStage(nn.Module):
    """Routing plus mixture of experts residual plus low rank adapter."""
    def __init__(self, ch, k=3, rank=8):
        super().__init__()
        self.k = k
        self.gate = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(ch, k))          # routing weights
        self.experts = nn.ModuleList(
            [nn.Conv2d(ch, ch, 3, 1, 1) for _ in range(k)])
        # modality specific low rank adapter, down then up projection
        self.a_down = nn.Conv2d(ch, rank, 1)
        self.a_up = nn.Conv2d(rank, ch, 1)

    def forward(self, base, prev):
        alpha = F.softmax(self.gate(base), dim=1)         # (B, K)
        res = 0
        for i, e in enumerate(self.experts):
            w = alpha[:, i].view(-1, 1, 1, 1)
            res = res + w * e(prev)
        res = res + self.a_up(self.a_down(prev))            # low rank adapter
        return base + res                               # reliability conditioned feature


class Encoder(nn.Module):
    def __init__(self, dims=(64, 128)):
        super().__init__()
        self.backbone = FrozenStub(3, dims)
        self.rcbr = nn.ModuleList([RCBRStage(d) for d in dims])

    def forward(self, x):
        base = self.backbone(x)
        prev = [x] + base[:-1]
        feats = []
        for i, (b, p) in enumerate(zip(base, prev)):
            # prev projected to match channels for the residual path
            if p.shape[1] != b.shape[1]:
                p = F.interpolate(b, size=b.shape[-2:])
            feats.append(self.rcbr[i](b, b))
        return feats


class UncertaintyDecoder(nn.Module):
    """Per modality decoder head with an uncertainty predictor."""
    def __init__(self, ch):
        super().__init__()
        self.dec = nn.Conv2d(ch, ch, 3, 1, 1)
        self.unc = nn.Conv2d(ch, 1, 3, 1, 1)      # phi
        self.align = nn.Conv2d(ch, ch, 1)              # channel alignment T

    def forward(self, f):
        d_hat = F.relu(self.dec(f))
        u = torch.sigmoid(self.unc(d_hat))             # U = sigma(phi(D_hat))
        return d_hat, u


def cross_modal_correct(d_r, u_r, d_t, u_t, align):
    # correction flows toward the uncertain modality
    dr = d_r + u_r * align(d_t)
    dt = d_t + u_t * align(d_r)
    return dr, dt


class PixelCompetition(nn.Module):
    """Pixel wise modality competition for the fusion stream."""
    def __init__(self, ch):
        super().__init__()
        self.psi = nn.Conv2d(2 * ch, 2, 3, 1, 1)   # two logits per pixel

    def forward(self, d_r, d_t):
        logits = self.psi(torch.cat([d_r, d_t], dim=1))
        w = F.softmax(logits, dim=1)               # W_r, W_t compete
        h = w[:, 0:1] * d_r + w[:, 1:2] * d_t
        return h


class RASOD(nn.Module):
    def __init__(self, dims=(64, 128)):
        super().__init__()
        self.enc_r = Encoder(dims)
        self.enc_t = Encoder(dims)
        top = dims[-1]
        self.dec_r = UncertaintyDecoder(top)
        self.dec_t = UncertaintyDecoder(top)
        self.pmc = PixelCompetition(top)
        self.head_r = nn.Conv2d(top, 1, 1)
        self.head_t = nn.Conv2d(top, 1, 1)
        self.head_f = nn.Conv2d(top, 1, 1)

    def forward(self, rgb, thermal):
        fr = self.enc_r(rgb)[-1]
        ft = self.enc_t(thermal)[-1]
        d_r, u_r = self.dec_r(fr)
        d_t, u_t = self.dec_t(ft)
        d_r, d_t = cross_modal_correct(d_r, u_r, d_t, u_t, self.dec_r.align)
        h = self.pmc(d_r, d_t)
        def up(x): return F.interpolate(x, size=rgb.shape[-2:],
                                      mode="bilinear", align_corners=False)
        s_r = torch.sigmoid(up(self.head_r(d_r)))
        s_t = torch.sigmoid(up(self.head_t(d_t)))
        s_f = torch.sigmoid(up(self.head_f(h)))
        return s_r, s_t, s_f, (s_r + s_t + s_f) / 3


def bce_iou(pred, gt, l1=1.0, l2=1.0):
    bce = F.binary_cross_entropy(pred, gt)
    inter = (pred * gt).sum((2, 3))
    union = (pred + gt).sum((2, 3)) - inter
    iou = 1 - ((inter + 1) / (union + 1)).mean()
    return l1 * bce + l2 * iou


def train_step(model, opt, batch):
    model.train()
    rgb, thermal, gt = batch
    s_r, s_t, s_f, _ = model(rgb, thermal)
    loss = bce_iou(s_r, gt) + bce_iou(s_t, gt) + bce_iou(s_f, gt)  # deep supervision
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()


if __name__ == "__main__":
    model = RASOD()
    trainable = [p for p in model.parameters() if p.requires_grad]
    opt = torch.optim.Adam(trainable, lr=5e-5)
    B, H, W = 2, 352, 352
    rgb = torch.rand(B, 3, H, W)
    thermal = torch.rand(B, 3, H, W)
    gt = (torch.rand(B, 1, H, W) > 0.5).float()
    for step in range(3):
        print("step", step, "loss", round(train_step(model, opt, (rgb, thermal, gt)), 4))
    n = sum(p.numel() for p in model.parameters())
    print("parameters", round(n / 1e6, 2), "M")

Conclusion

The core achievement of RA-SOD is a working answer to a question the field had been stepping around. What happens to RGB and thermal fusion when one sensor is lying? The usual answer, a fixed weighting or a late fusion tweak, treats the symptom. RA-SOD treats the cause by making reliability something the network estimates and carries from the first encoder layer to the final fusion step. On VT5000 it leads on every metric, and under deliberate degradation its lead widens rather than shrinks, which is the behavior a reliability method is supposed to show.

The conceptual shift is the part worth remembering. For years multimodal work framed fusion as combination, as if the task were to average two views into a better one. RA-SOD reframes it as adjudication. The two modalities are witnesses of uneven trustworthiness, and the job is to weigh their testimony location by location, moment by moment. The uncertainty gate that rises only for the failed modality, and the pixel wise competition that lets a clean thermal patch outvote a blurred color one, are both expressions of that single idea.

The design also transfers. The routed residual on a frozen backbone, the uncertainty weighted cross correction, and the pixel wise competition are not specific to salient object detection or to thermal imaging. Any perception system that fuses inputs of changing quality, from multi sensor robots to variable quality medical scans, could borrow the same three moves. The clean jump from a Res2Net-50 backbone to a Swin-B transformer, with the top result following along, hints that the reliability principle is what carries the performance, not the particular encoder.

The honest limitations keep the result in proportion. This is a preprint whose degradation evidence rests largely on synthetic corruptions, it assumes well aligned modalities, and it has not been stressed against simultaneous failure of both feeds or against degradations that mimic real objects. Those are real gaps, and the strongest version of this work would close them with field captured adverse data. None of that undoes the contribution. It marks the road ahead.

For anyone building fusion systems, the practical message is compact. Stop assuming your inputs are clean. Estimate where each one is unreliable, route correction toward the doubt rather than away from it, and settle conflicts pixel by pixel instead of with one global compromise. RA-SOD shows the combination holds up exactly when a naive fusion model would break, and the public code plus the reference above are a place to start testing the idea on a sensor stack of your own.

Frequently asked questions

What is RGB-T salient object detection?

It is the task of finding the visually prominent objects in a scene using two aligned inputs, an ordinary color image and a thermal infrared image. The two modalities are complementary, since color carries texture and detail while thermal carries heat signatures that survive darkness and glare, so fusing them should improve robustness over either one alone.

What does modality degradation mean here?

It means one of the two sensor feeds loses quality. RGB can degrade under low illumination, motion blur, and noise, while thermal can suffer contrast compression and block like artifacts such as the hot block effect. Degraded feeds provide misleading evidence, which corrupts naive fusion, and this is the failure mode RA-SOD is built to handle.

How does RA-SOD know which modality to trust?

It models reliability at three stages. A routing branch at the encoder senses distributional shifts caused by degradation, each decoder stream predicts its own uncertainty map, and a pixel wise competition step assigns weights to RGB and thermal at every location. In tests the uncertainty gate rose only for the modality that was actually corrupted, for example to 0.57 for RGB under low light and to 0.55 for thermal under contrast loss.

How well does RA-SOD perform?

On the large VT5000 benchmark it ranks first on all five standard metrics, with a structure measure of 0.897 and a mean absolute error of 0.029. Under simulated single modality degradation on VT1000 it improved the weighted F measure from 0.895 to 0.907 against the strongest competitor, and its advantage grew under harder degradation.

Is RA-SOD efficient enough to use?

It uses 46.72 million parameters and 60.13 billion floating point operations at about 11.24 frames per second. That is heavier than the leanest baseline but still the second lightest model in the comparison and faster than several competitors, so the reliability machinery adds moderate overhead rather than an impractical cost.

Can I try RA-SOD myself?

Yes. The authors released code and models publicly, and a runnable reference implementation of the reliability pipeline is included in this article. The source paper is a preprint, so results should be read as strong early evidence rather than a finished production benchmark.