DiffFuseNet: Why Feature-Space Diffusion Beats Image-Space Diffusion for Infrared-Visible Fusion

The Real Problem With Fusing Infrared and Visible Images

Infrared sensors see heat. Visible sensors see texture and color under adequate light. Combine them well and you get an image where a person standing in shadow is still clearly a person, where a license plate is readable at night, where a building’s outline is visible through fog. Combine them badly and you get a muddy compromise that loses what each sensor was good at.

The paper from Wang et al., published in Pattern Recognition (2026), frames the persistent difficulty in four parts. Encoders struggle to extract clean modality specific features under low light or sensor noise. Naive combination strategies, simple addition or concatenation, let one modality dominate the other. Explicit separation of what the two modalities share versus what is unique to each is rarely done properly, which limits how adaptively a network can fuse them. And training is inherently unstable because there is no ground truth fused image to supervise against, since no camera produces a perfectly fused infrared and visible pair to learn from directly.

That last point matters more than it sounds. Most fusion networks are trained with reconstruction losses, structural similarity, and hand designed objectives because there is nothing else to anchor on. Generative adversarial approaches have tried to fill that gap but tend to suffer mode collapse, where the generator settles into producing a narrow range of outputs regardless of input variation. Spatial transformer networks assume the misalignment between modalities can be modeled with parametric warping, which breaks down when the physical discrepancy between thermal and optical imaging is not rigid (a person’s outline in infrared does not deform the same way as their outline in visible light under perspective shift).

Why Feature Space Diffusion Beats Image Space Diffusion

This is the paper’s central design decision, and it is worth understanding precisely because it explains both the quality gains and the tenfold speed advantage over the most comparable prior method.

Diffusion models work by learning to reverse a noise corruption process. You take clean data, add Gaussian noise across many steps until it becomes pure noise, then train a network to predict and remove that noise step by step. Applied directly to images, as the existing method Diff-IF does, this means iterative denoising across the full pixel grid, which is expensive. The paper reports Diff-IF at 637.65 GFLOPs and 0.78 seconds per image pair.

DiffFuseNet’s D²FE module instead applies the noise injection and denoising to features produced by a shallow shared encoder consisting of just two convolutional layers with GroupNorm and GELU activation. These are not deep semantic features. They retain substantial spatial locality and have not yet abstracted away from the pixel grid into high level concepts. Running diffusion on this representation rather than on raw pixels means the lightweight conditional UNet doing the denoising has far less to process, and you only need it to operate well enough to regularize the features toward a shared modality invariant manifold rather than to synthesize realistic image content from scratch.

Here F superscript zero is the initial shallow feature from either modality, and the noise schedule progressively corrupts it over T steps until it approaches pure Gaussian noise at the final step. A conditional UNet is trained to predict the injected noise at each timestep, conditioned on a sinusoidal positional embedding of the current step.

There is a subtlety in how the denoised features get back into the pipeline that is worth flagging because the authors address a real concern about it directly. After denoising, the enhanced feature map is fused and added back to the original image as a residual correction, not used to generate new pixel content outright.

The authors justify this by noting the shallow encoder has limited receptive fields and has not encoded highly abstract semantic concepts, so adding this feature map back functions as a learned, spatially adaptive contrast and detail enhancement, conceptually similar to ResNet’s identity mapping that lets a network refine an input incrementally rather than hallucinate new content. This is a reasonable design justification and it sidesteps a real failure mode of naive diffusion in the loop fusion, where deep abstract features get projected back into pixel space and introduce semantic artifacts that do not correspond to anything physically present in the scene.

Explicit Decoupling Tells Shared and Unique Information Apart

Once features are enhanced and aligned via diffusion, the EDFD module performs the second core operation. It explicitly splits what the two modalities have in common from what each one uniquely contributes. This sounds obvious but most prior fusion networks do not do it cleanly. A shared encoder learns a single representation and downstream fusion has to implicitly figure out which parts came from where.

The shared feature captures scene layout common to both sensors, things like where objects are positioned in the frame, the rough geometry of the scene. Modality specific features are then obtained by literally subtracting the shared component from each modality’s denoised features and refining the residual.

The visible specific feature ends up encoding textures and colors, the things a thermal camera physically cannot see. The infrared specific feature captures thermal saliency and edge information that survives in low light where the visible sensor struggles. This separation gives the network something concrete to control during fusion rather than hoping a single mixed representation sorts itself out implicitly.

Frequency Decomposition With Haar Wavelets and Invertible Networks

The third piece addresses a different axis of the problem, separating global structure from local texture, which is largely orthogonal to the shared versus modality specific split. A single level Haar wavelet decomposition losslessly splits each enhanced feature into four subbands, a low frequency component capturing global contour and energy, and three high frequency components capturing horizontal, vertical, and diagonal detail.

What makes this more than a standard wavelet fusion approach is the use of invertible neural networks to refine the subbands before fusion. The forward transformation in the invertible block splits the input into two halves and applies coupling operations.

An important detail here, and one that affects how you should think about this module’s actual cost, is that only the forward pass is used during both training and inference. The invertibility property is never exploited to run the network backward, which the authors note preserves the lossless information property of the architecture without the additional computational overhead that running the inverse transformation would require.

What the Ablation Study Actually Tells You

The ablation results are the most useful part of this paper for anyone deciding whether to adapt pieces of this architecture rather than the whole thing. Six variants were tested, each removing or replacing one component while holding the rest fixed.

Two findings deserve more attention than the paper gives them in passing. First, removing bidirectional attention and replacing it with simple element wise addition creates the largest drop in spatial frequency (12.80 versus 13.91), meaning the attention mechanism is doing real work in preserving fine detail, not just marginal refinement. Second, the gap between standard wavelet and INN refined wavelet decomposition is the smallest of all six ablations on most metrics, which raises a fair question about whether the INN refinement earns its computational cost compared to the other three modules. The Qabf gap (0.61 versus 0.63) is real but modest, and a practitioner with tight compute budgets might reasonably trade that small loss for a simpler wavelet pipeline.

Benchmark Results Show Where DiffFuseNet Actually Wins

Across TNO, RoadScene, and M3FD, the pattern is consistent without being a clean sweep. The model leads or ties for the lead on entropy (EN) and spatial frequency (SF) on TNO, leads on the perceptual Qabf metric on RoadScene, and leads on both VIF and Qabf on M3FD. It does not win every metric on every dataset. CDDFuse posts a higher standard deviation on both TNO and RoadScene, and the authors address this directly rather than ignoring it.

The explanation the authors give for the CDDFuse standard deviation gap is worth taking seriously rather than dismissing as post-hoc justification. CDDFuse tends to amplify high-frequency components aggressively to boost contrast, which inflates SD but risks halo artifacts and excess enhancement in flat regions. DiffFuseNet’s bidirectional cross frequency attention imposes a more conservative high frequency integration, prioritizing texture authenticity over raw contrast stretching. Whether you find that argument convincing depends partly on whether you trust VIF and Qabf as better proxies for perceptual quality than SD, which the broader image fusion literature generally does treat as the case, since SD measures pixel value dispersion rather than anything about structural or perceptual fidelity.

Downstream Task Performance Is Where the Argument Gets Stronger

Benchmark metrics on fusion quality are useful but somewhat self referential, since the metrics were largely designed around what fusion researchers think matters. Downstream task performance is a better test of whether the fused image is actually useful.

On the LLVIP object detection benchmark using YOLOv5, DiffFuseNet achieves 0.745 recall, second only to SwinFusion’s 0.766, with an mAP@.5 of 0.852, the second-best score behind SwinFusion’s 0.867. On semantic segmentation using SegFormer-B0, the result is more clearly favorable. DiffFuseNet achieves the highest mIoU among all ten compared methods at 0.7300, with particularly strong gains in challenging categories like bicycles, car stops, and guardrails, categories where fine edge preservation matters most for correct pixel classification.

This is a meaningful signal. A fusion method that scores well on EN and SF but does not translate to better downstream detection or segmentation would be optimizing for the wrong thing. The consistency between DiffFuseNet’s perceptual metrics and its downstream task gains suggests the architecture is preserving information that actually matters for machine vision tasks, not just information that happens to inflate one particular metric that uses no reference image.

The Computational Efficiency Story

This is arguably the paper’s strongest practical claim, and the numbers back it up clearly.

The FLOPs reduction from 637.65G to 30.75G against Diff-IF is roughly twenty fold, and the wall clock inference time drops from 0.78 seconds to 0.074 seconds, slightly more than tenfold. This is the practical payoff of keeping diffusion confined to shallow features rather than running it across the full image. Notably, DiffFuseNet’s inference time is also faster than CDDFuse and dramatically faster than SwinFusion, despite being a more architecturally complex pipeline than either, which speaks to how much the choice of where to apply expensive operations matters more than the raw parameter count.

It is worth being precise about what this comparison does and does not establish. DiffFuseNet is not the fastest method in the comparison set overall. ReCoNet and HaarFuse, both pure CNN approaches, are faster still. The honest framing, which the authors themselves use, is that DiffFuseNet offers a favorable position on the tradeoff between speed and quality rather than dominating on raw speed. If your deployment constraint is strictly inference latency on embedded hardware, a lightweight CNN based fusion network remains the better starting point.

This tradeoff is the same one we examined in our coverage of diffusion guided knowledge distillation, where the lesson was also that the location in the pipeline where you apply an expensive technique matters more than whether you apply it at all. Confining diffusion or attribution computation to a smaller, earlier representation rather than the full output consistently buys efficiency without giving up the benefit of the technique.

The Medical Imaging Results Deserve a Careful Read

The paper extends its evaluation to MRI-CT, MRI-PET, and MRI-SPECT fusion as a secondary domain. It is worth being precise about what this section does and does not claim. The paper reports quantitative metrics (entropy, spatial frequency, mutual information, VIF, Qabf) on public medical fusion benchmarks, and DiffFuseNet performs competitively, achieving the best EN, SF, and Qabf scores on the MRI-CT dataset and the best MI, SCD, and Qabf on MRI-PET.

What the paper does not claim, and what this article will not imply, is that these results constitute a validated clinical tool. This is architecture level benchmark evaluation using standard public datasets, not a clinical trial or a diagnostic accuracy study. The interesting technical observation here is that both diffusion based methods in the comparison, Diff-IF and DiffFuseNet, consistently rank in the top tier across the medical fusion metrics, which the authors interpret as evidence that diffusion guided feature alignment generalizes well to a different imaging domain with different noise characteristics and structural properties than the infrared and visible setting it was originally designed for. That is a reasonable architectural observation. It is not a clinical readiness claim, and any application of this or related architectures in a real diagnostic pipeline would require dedicated clinical validation well beyond what a Pattern Recognition methods paper provides.

Honest Limitations

A Reproducible PyTorch Sketch of the D²FE Module

Below is a focused implementation of the feature space diffusion module, the architectural piece most likely to be useful in isolation for other fusion or cross modal alignment work. It includes the shallow encoder, the forward noise process, the conditional UNet denoiser, the fusion network, and a smoke test on dummy paired images.

Where This Fits in the Broader Landscape of Diffusion for Vision

The paper situates itself against two categories of prior diffusion work, feature caching acceleration methods like TaylorSeer and Spectrum, and unified multimodal generative frameworks like Transfusion and Show-o. Both categories are doing something fundamentally different from what DiffFuseNet does. Feature caching methods speed up single modal generation by reusing computation across denoising steps, but they were not designed to align two different sensor modalities at all. Unified multimodal frameworks generate fused content through iterative denoising in image space, applying the same diffusion process to both modalities homogeneously, which the authors argue fails to model the feature level discrepancies that actually distinguish infrared from visible imaging.

DiffFuseNet’s contribution sits in a narrower, more specific niche than either of those directions. It does not try to be a general purpose multimodal generator. It uses diffusion as a targeted regularization tool for a specific two modality alignment problem, and that narrower scope is exactly what allows the architecture to be lightweight enough to run at 0.074 seconds per pair. If you are exploring where diffusion models add genuine value versus where they are an expensive default, our analysis of sample level adaptive distillation for video models makes a related point from a different angle. Matching the cost of an expensive technique to the part of the problem that actually needs it, rather than applying it uniformly, is consistently where the practical gains show up.

Frequently Asked Questions

Closing Thoughts

DiffFuseNet is a well built answer to a specific, well posed question. Can diffusion models contribute to image fusion without the computational cost that has limited their practical deployment in this domain. The answer the paper gives is yes, provided you are willing to confine the diffusion process to a narrow, well defined regularization role rather than asking it to do generative heavy lifting.

The architecture combines that insight with two other solid ideas, explicit decoupling of shared and modality specific information, and frequency aware fusion via wavelets and invertible networks, both of which the ablation study supports as meaningful contributors rather than decorative additions. The bidirectional cross frequency attention component in particular earns its place in the architecture based on the ablation evidence.

Where the paper is appropriately cautious is in acknowledging what remains unresolved. The alignment assumption, the untested extreme weather robustness, and the relatively thin case for the INN cost versus benefit tradeoff are all real gaps the authors name rather than gloss over. That honesty is worth noting, particularly in a subfield where overclaiming about diffusion model benefits has become common.

The most transferable lesson here extends beyond infrared and visible fusion specifically. When considering whether to add diffusion based regularization to any cross modal alignment problem, the relevant design question is not whether diffusion helps, since in some form it usually does, but where in the pipeline you can apply it cheaply enough that the benefit outweighs the cost. DiffFuseNet’s answer, push it as early and as shallow as the architecture will tolerate, is a template worth testing in other multimodal settings beyond image fusion.

Related Articles