REHDR: Event Guided Multimodal Fusion for HDR Video Reconstruction

Why Traditional Multi-Exposure Fusion and Inverse Tone Mapping Break Down

The problem runs deeper than it first appears. Standard digital camera sensors capture a narrow slice of real-world luminance, typically limited to around 60 decibels of dynamic range, whereas natural outdoor scenes regularly span more than 100 decibels between direct sunlight and cast shadow. When scene irradiance exceeds the saturation capacity of a pixel well, highlights clip to 255 in 8-bit quantization; when irradiance falls below the read-noise floor, shadows collapse to zero.

For decades, computational photography tried to bypass sensor clipping using two main strategies, both of which carry severe trade-offs in dynamic video scenes:

• Multi-Exposure Bracketing and Fusion: Methods pioneered by Debevec and Malik, Mertens et al., and Kalantari et al. capture three or more Low Dynamic Range (LDR) frames at alternating shutter speeds and merge them via optical flow, attention networks (such as AHDRNet by Yan et al. or SGARN by Tang et al.), or conditional diffusion models. In fast-moving video, however, camera shake and foreground object motion between exposures create severe spatial misalignment and ghosting artifacts. Worse, capturing long exposures to see into dark shadows directly caps the maximum frame rate of the video stream.
• Single-Image Inverse Tone Mapping (iTMO): Networks by Banterle et al., Eilertsen et al., Endo et al., and Liu et al. attempt to hallucinate HDR radiance from a single LDR exposure. While single-frame inference avoids multi-exposure ghosting, it remains fundamentally ill-posed whenever a large region of a building or sky is completely blown out, because zero spatial gradient remains inside the clipped patch from which a convolutional kernel can extrapolate texture.

Bio-inspired neuromorphic event cameras offer a physical escape hatch from this trade-off. Instead of integrating photons over a fixed global shutter interval, each pixel on a dynamic vision sensor independently monitors changes in logarithmic light intensity \(\log L_{(x,y)}^t\) in continuous time. Whenever the logarithmic change crosses a hardware contrast threshold \(C_{th}\), the pixel fires an asynchronous microsecond packet \(e_i(x_i, y_i, t_i, p_i)\) recording the spatial coordinate \((x_i, y_i)\), timestamp \(t_i\), and polarity \(p_i \in \{-1, +1\}\) (an ON event for rising brightness, an OFF event for falling brightness). Because the photodiode operates logarithmically, it achieves a dynamic range above 120 dB without saturating in bright sun or stalling in dark rooms.

Why Prior Event-Guided HDR Pipelines Struggled With Real-World Video

If event cameras capture over 120 dB of dynamic range, why not reconstruct HDR video purely from event streams using recurrent networks like E2VID by Rebecq et al. or EventHDR by Zou et al.? Because event sensors only fire when luminance changes. Static regions produce zero events, and monochrome event streams encode differential log-irradiance rather than absolute chrominance, leaving pure event reconstructions washed out, grayscale, or color-distorted.

Combining an event stream with an RGB frame camera gives you the best of both worlds—absolute color and static texture from the RGB sensor, plus high-speed highlight and shadow gradients from the event sensor—a multimodal philosophy that aligns closely with the cross-modal architectures in our computer vision research hub. Yet early hybrid event-RGB HDR systems such as NeurImg-HDR+ by Han et al. and HDRev-Net by Yang et al. ran into four practical roadblocks:

1. Reliance on Low-Resolution Simulators: Most prior models trained on synthetic events generated by ESIM or low-resolution monochrome DAVIS346 sensors (\(346 \times 260\)), which fail to model real sensor shot noise, refractory periods, and high-definition spatial details.
2. Cumbersome Multi-Stage Training: Pipelines like Han et al. first pre-train an event-to-intensity reconstruction network and then train a second refinement network to merge that synthetic intensity map with the LDR frame, multiplying compute overhead and propagating stage-one stripe artifacts into the final image.
3. Rigid Handcrafted Exposure Masks: Analytic fusion methods (such as Pan et al.) weight LDR pixels using fixed hat-shaped functions like \(w_i = \frac{0.5 – \max(|L – 0.5|, \tau – 0.5)}{1 – \tau}\) with \(\tau = 0.2\). In real scenes with extreme 1-to-3-to-9 exposure ratios, fixed thresholding zeroes out vast swaths of valid pixels and introduces severe color blotches.
4. Memory-Heavy Fixed-Order Video Loading: Loading entire uncompressed video sequences sequentially into RAM during training restricts spatial data augmentation, prevents flexible batch sizing, and triggers out-of-memory crashes on high-resolution datasets.

In their Expert Systems With Applications paper, Guo and colleagues resolve all four bottlenecks through the REHDR architecture, the SSASDL data loader, and the physical RealHDR dataset.

Mathematical Formulations of Event-Guided HDR Reconstruction

To follow how REHDR bridges asynchronous microsecond events and synchronous 8-bit RGB frames, let us trace the mathematical pipeline defined in Section 3 of the paper. First, an event camera monitors continuous log-intensity at pixel \((x, y)\) and triggers an event polarity \(E_{(x,y)}^t \in \{-1, 0, +1\}\) whenever the temporal change over interval \(\Delta t\) crosses threshold \(C_{th}\) (with small constant \(\epsilon\) preventing \(\log(0)\)).

Meanwhile, a conventional RGB camera exposed for duration \(\Delta t\) under scene irradiance \(R\) forms a true linear HDR radiance map \(I_{HDR} = R \cdot \Delta t\), which gets clipped to \([0, 1]\), passed through a nonlinear Camera Response Function \(f_{CRF}\), and quantized to 8-bit integers to produce the observed frame \(I_{LDR}\).

Because convolutional layers require dense grid tensors rather than sparse tuples \(e_i(x_i, y_i, t_i, p_i)\), the asynchronous events occurring within frame exposure window \(\Delta T\) starting at \(t_0\) are discretized into an Event Spike Tensor across \(B\) temporal bins using bilinear temporal voting (Equations 7 and 8).

Given the linearized LDR frame \(L\) and the encoded event spike tensor \(\text{Enc}(E)\), the reconstruction network \(\mathcal{R}\) with learnable weights \(\theta_{\mathcal{R}}\) predicts the linear HDR frame \(\hat{Y} = \mathcal{R}(L, \text{Enc}(E); \theta_{\mathcal{R}})\). Because linear HDR radiance values span several orders of magnitude, computing loss directly in the linear domain allows bright highlight pixels to dominate gradient updates. REHDR therefore applies the differentiable \(\mu\)-law tone-mapping operator \(\mathcal{T}(\cdot)\) with \(\mu = 5000\) before evaluating the training loss against ground truth \(Y\) (Equations 5 and 6).

Inside the REHDR Architecture: Dual Encoders, AFMF, and ConvLSTM

Rather than converting events into intermediate grayscale images first, REHDR feeds an event sequence \(\mathbb{E} = \{E_i\}_{i=1}^T\) and a linearized LDR sequence \(\mathbb{L} = \{L_i\}_{i=1}^T\) of length \(T = 8\) into an end-to-end dual-encoder U-Net with base channel width \(C = 32\). Each encoder stage uses a Residual Block (two \(3 \times 3\) convolutions with LeakyReLU of negative slope \(0.2\) and a \(1 \times 1\) residual shortcut) followed by a strided convolution (`SConv` with kernel size 4) for \(2\times\) spatial downsampling across two scales (\(\frac{H}{2} \times \frac{W}{2}\) at \(2C\) channels, and \(\frac{H}{4} \times \frac{W}{4}\) at \(4C\) channels), while the decoder restores resolution via transposed convolutions (`TConv` with kernel size 5) and skip connections.

Adaptive Feature Modulation Fusion (AFMF) and the Dynamic Range Mask

Let \(F_E = \text{ResBlock}(E_i)\) and \(F_L = \text{ResBlock}(L_i)\) denote the preliminary event and LDR features at a given encoder scale (Equations 10 and 11). First, the raw LDR input frame \(L_i\) is downsampled via \(\text{Down}(L_i)\) to match the current spatial resolution \(L_i\downarrow\). As formulated in Equations 12, 13, and 14, the Dynamic Range Mask module adds a \(3 \times 3\) convolved projection of \(L_i\downarrow\) to \(\text{Conv}_{3\times 3}(F_L)\), passes the sum through another \(3 \times 3\) convolution and a Sigmoid gate to generate pixel-wise reliability weights \(W_{pixel}\), and modulates \(F_L\) via element-wise multiplication \(\odot\) and a residual addition.

Once the clipped regions in \(F_L\) have been suppressed to form \(F_L’\), the module concatenates \(F_L’\) and \(F_E\) along the channel axis, applies a \(3 \times 3\) convolution and ReLU activation to produce joint feature \(f_{in}\), and pools global spatial context via Global Average Pooling (\(\text{GAP}\)). Two parallel \(1 \times 1\) convolutions project the pooled vector into modality logits \(w_1\) and \(w_2\), which are normalized across the two branches via Softmax to yield adaptive modality weights \(w_1’\) and \(w_2’\) (Equations 15, 16, and 17).

Finally, in Equation 18, the weighted sum \(w_1′ F_L’ + w_2′ F_E\) passes through a Squeeze-and-Excitation Channel Attention (\(\text{CA}\)) block (Hu et al.) and a \(1 \times 1\) projection layer, selectively amplifying the most informative feature channels.

ConvLSTM Recurrent Feature Extraction and Total Training Loss

Because event firing rates fluctuate wildly with camera motion speed, frame-by-frame networks suffer from severe temporal brightness flicker. REHDR places a ConvLSTM module directly after the AFMF block in the encoder. Initialized with hidden state \(h_1 = 0\) at the start of a sequence, the ConvLSTM propagates hidden states \(h_i\) forward across \(T = 8\) consecutive time steps loaded by the SSASDL sampler during training (and across unlimited sequence lengths during inference).

Over a sequence of \(T\) frames with network predictions \(O_i\) and ground-truth HDR frames \(G_i\), the network minimizes a composite objective combining tone-mapped \(L_1\) reconstruction error (\(\lambda_1 = 10.0\)) and a perceptual + Gram-matrix style loss (\(\lambda_2 = 0.5\)) computed at layer `relu3_3` of an ImageNet pre-trained VGG-16 backbone (Equations 19, 20, 21, and 22).

Building the RealHDR Dataset With a Synchronized Coaxial Beam-Splitter Rig

One of the most valuable contributions of this study is hardware-level dataset engineering. To eliminate the sim-to-real gap that plagued earlier ESIM-trained models—a domain gap challenge we also examined in our analysis of physics-guided synthetic data vs real sensor acquisition—the authors constructed a coaxial optical rig (Figure 6 of the paper):

• Event Sensor: A Prophesee EVK4 HD neuromorphic camera capturing \(1280 \times 720\) asynchronous events.
• RGB Sensor: A high-speed Basler acA800-510uc global-shutter RGB camera capturing \(800 \times 600\) frames at up to 510 FPS.
• Matched Optics & Beam Splitter: Identical MVL-MF1628M-8MP lenses mounted behind a Thorlabs CCM1-BS013 non-polarizing 50:50 cube beam splitter (\(400\text{ to }700\text{ nm}\)) so both sensors share a parallax-free optical axis.
• Microcontroller Hardware Triggering: An Arduino UNO R3 signal generator dispatches synchronized 5V square-wave pulses simultaneously to both cameras. The Basler RGB camera starts exposure on the rising edge and ends on the falling edge; modulating the pulse width creates alternating exposures that are merged via Debevec and Malik’s algorithm to form ground-truth HDR frames while exact trigger timestamps are stamped directly into the EVK4 event stream.

The resulting public RealHDR dataset contains 19 real-world video sequences totaling 5,510 aligned \((E, L, H)\) triplets at \(800 \times 600\) resolution across 12 outdoor and 7 indoor scenes, split into 4,210 training pairs and 1,300 evaluation pairs.

Experimental Benchmarks and Ablation Studies on RealHDR

All models were trained and evaluated under identical settings on an NVIDIA GeForce RTX 3090 GPU (100,000 iterations, batch size 4, \(256 \times 256\) random crops, cosine annealing learning rate from \(2 \times 10^{-4}\) down to \(2 \times 10^{-7}\), taking 26 hours). Across the 1,300 test frames of RealHDR, the authors benchmarked REHDR against multi-exposure frame fusion (Li et al., 2020; AHDRNet by Yan et al., 2019), pure event reconstruction (E2VID by Rebecq et al., 2019), and state-of-the-art event-guided multimodal fusion (HDRev-Net by Yang et al., 2023; NeurImg-HDR+ by Han et al., 2023).

Look at the performance jump in Table 1 above. Pure event reconstruction with E2VID scores just 12.07 dB PSNR and 0.316 SSIM because events alone lack static chrominance and baseline luminance. Multi-stage NeurImg-HDR+ reaches 16.37 dB PSNR (0.71 seconds runtime) and shared-latent HDRev-Net reaches 18.85 dB PSNR (1.01 seconds runtime), struggling to recover large blown-out building facades and sky regions. REHDR leaps by +6.16 dB over HDRev-Net and +8.64 dB over NeurImg-HDR+ to reach 25.01 dB PSNR and 0.878 SSIM, while achieving the best perceptual visibility score (64.369 HDR-VDP-2), the lowest video flickering distortion (0.329 HDR-VQM), and the fastest inference runtime at 0.34 seconds per \(800 \times 600\) frame.

“Unlike simply assigning weights, our dynamic range mask is computed from the feature maps of both inputs. The learnable parameters can be optimized through end-to-end training to find the appropriate weights for different input images. This enables the correct image features and event features to be fused through attention-based weighting.”

Guangsha Guo, Ming Sun, Shilong Jing, et al., Expert Systems With Applications (2026)

Systematic Ablation Studies Across Fusion, Recurrence, and Data Loading

To verify the exact contribution of each architectural component on RealHDR, the authors ran three controlled ablation suites (Tables 2, 3, and 4 of the paper):

Every single component in Table 2 earns its keep. Replacing the AFMF module with standard concatenation and a \(3 \times 3\) convolution (`REHDR w/o AFMF`) causes PSNR to drop by 2.93 dB down to 22.08 dB, leaving overexposed building windows washed out. Stripping out just the learnable Dynamic Range Mask (`w/o Mask_DR`) costs 0.76 dB in PSNR and 0.022 in SSIM. Removing the ConvLSTM costs 0.029 in SSIM (dropping from 0.878 to 0.849), while swapping ConvLSTM for ConvGRU degrades PSNR by 2.17 dB. Finally, training with the randomized SSASDL loader adds +0.17 dB PSNR over traditional fixed-order video loading while preventing RAM bottlenecks.

Honest Limitations and Open Challenges in Neuromorphic HDR Imaging

In Section 6 of the paper, Guo and colleagues provide an unusually candid analysis of three physical and algorithmic limitations that practitioners must consider before deploying event-guided HDR in the wild:

1. Motion Blur During Ground-Truth Dataset Acquisition: Because an event camera only fires when brightness changes across pixels, capturing dense texture information in static scenes requires physically moving or shaking the camera rig. However, during the longer exposure intervals required to capture shadow frames for multi-exposure ground-truth synthesis on the RGB camera, camera motion introduces motion blur into the reference frames and complicates sub-pixel alignment between events and RGB pixels.
2. Chrominance Recovery in Deeply Saturated Regions: The Prophesee EVK4 HD sensor records monochromatic logarithmic luminance changes, not RGB color channels. When a large region of an LDR frame is completely clipped to pure white (\(255, 255, 255\)), the event stream supplies sharp structural edges, but the network must infer chrominance from surrounding unclipped context. Integrating semantic colorization priors or lightweight foundation adapters—similar to the parameter-efficient strategies we explored in Rein++ vision foundation model adaptation—could prevent color desaturation in massive highlight zones.
3. Uniform Time-Step Assumption in ConvLSTM vs. Asynchronous Events: While ConvLSTM reduces video flicker, its recurrent gates assume uniform temporal intervals between inputs. Real event streams are highly non-uniform in time, firing furiously during fast motion and going silent during pauses. As the authors specifically point out in Section 6.3, replacing ConvLSTM with continuous-time Neural ODEs or Selective State Space Models (Mamba)—a temporal state-space paradigm we also analyzed in TEAMS Mamba Snake—is a natural next step to model variable-rate event dynamics.

Conclusion and Future Directions

REHDR marks a decisive shift in how neuromorphic event streams and conventional RGB frames are combined for High Dynamic Range video reconstruction. By abandoning brittle two-stage event-to-image translation pipelines and training a single recurrent dual-encoder network from scratch on real hardware-synchronized data, Guo et al. eliminate intermediate stripe artifacts while cutting inference latency to a fraction of a second.

The core engineering lesson of REHDR lies in its Adaptive Feature Modulation Fusion (AFMF) block. Standard concatenation treats every RGB pixel as equally trustworthy, allowing clipped highlights and crushed shadows to poison the joint representation. Computing a learned pixel-wise Dynamic Range Mask directly from multi-scale LDR features before applying global Softmax modality weighting and channel attention allows the network to silence saturated RGB channels locally while leaning on the 120 dB event stream for edge recovery.

This multimodal gating blueprint transfers cleanly far beyond computational photography. Autonomous driving perception stacks operating in night-time headlight glare, surgical endoscopes encountering specular tissue reflections, and high-speed industrial inspection rigs can all use AFMF-style dynamic range masking to fuse complementary sensors whenever one modality saturates.

Paired with the public release of the 5,510-pair RealHDR dataset and the memory-efficient SSASDL sequence loader, REHDR sets a strong, reproducible baseline for the next generation of neuromorphic video systems. Future extensions uniting joint motion deblurring, semantic chrominance priors, and Mamba selective state-space recurrence will bring artifact-free, human-eye dynamic range to every high-speed camera platform.

Complete PyTorch Implementation of REHDR

Below is a complete, self-contained, runnable PyTorch implementation of the proposed REHDR framework matching Section 3 and Section 5.1 of the paper, including Event Spike Tensor discretization (Equations 7 and 8), differentiable \(\mu\)-law tone mapping (\(\mu = 5000\), Equation 6), the Residual Block (`ResBlock`), the Dynamic Range Mask (\(Mask_{DR}\), Equations 12–14), the Adaptive Feature Modulation Fusion (`AFMF`) module with Global Average Pooling Softmax weighting and Squeeze-and-Excitation Channel Attention (Equations 15–18), the `ConvLSTM` recurrent cell, the Dual-Encoder U-Net architecture (`REHDR`, \(C=32\)), the SSASDL dataset sampler, and a runnable smoke test across an 8-frame video sequence.

import random
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

# =====================================================================
# 1. EVENT SPIKE TENSOR (EQS. 7 & 8) & MU-LAW TONE MAPPING (EQ. 6)
# =====================================================================

def events_to_spike_tensor(
    x: torch.Tensor, y: torch.Tensor, t: torch.Tensor, p: torch.Tensor,
    H: int, W: int, num_bins: int = 5
) -> torch.Tensor:
    """
    Implements Section 3.2 (Eqs. 7 & 8):
    Converts asynchronous events (x, y, t, p) into a fixed grid Event Spike Tensor
    of shape [2 * num_bins, H, W] with bilinear temporal interpolation.
    """
    grid = torch.zeros((2, num_bins, H, W), dtype=torch.float32, device=t.device)
    if t.numel() == 0:
        return grid.view(2 * num_bins, H, W)

    t0, dt = t.min(), (t.max() - t.min()).clamp(min=1e-6)
    t_star = (num_bins - 1) * (t - t0) / dt  # Eq. (8)

    pol_idx = torch.where(p > 0, torch.zeros_like(p, dtype=torch.long), torch.ones_like(p, dtype=torch.long))
    x_idx = x.long().clamp(0, W - 1)
    y_idx = y.long().clamp(0, H - 1)

    for b in range(num_bins):
        weights = torch.clamp(1.0 - torch.abs(b - t_star), min=0.0)  # Eq. (7)
        grid.index_put_((pol_idx, torch.full_like(x_idx, b), y_idx, x_idx), weights, accumulate=True)

    return grid.view(2 * num_bins, H, W)


def mu_law_tonemap(y: torch.Tensor, mu: float = 5000.0) -> torch.Tensor:
    """Implements Equation (6): Differentiable mu-law tone mapping T(Y)."""
    y_pos = torch.clamp(y, min=0.0)
    return torch.log(1.0 + mu * y_pos) / torch.log(torch.tensor(1.0 + mu, device=y.device, dtype=y.dtype))


# =====================================================================
# 2. RESBLOCK, DYNAMIC RANGE MASK & AFMF MODULE (SECTION 3.5)
# =====================================================================

class ResBlock(nn.Module):
    """Fig. 3(b): Two 3x3 Conv + LeakyReLU(0.2) layers with 1x1 shortcut."""
    def __init__(self, in_ch: int, out_ch: int):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(out_ch, out_ch, kernel_size=3, padding=1),
            nn.LeakyReLU(0.2, inplace=True),
        )
        self.shortcut = nn.Conv2d(in_ch, out_ch, kernel_size=1) if in_ch != out_ch else nn.Identity()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.body(x) + self.shortcut(x)


class ChannelAttention(nn.Module):
    """Squeeze-and-Excitation Channel Attention (Hu et al., 2018) used in Eq. (18)."""
    def __init__(self, channels: int, reduction: int = 8):
        super().__init__()
        mid = max(4, channels // reduction)
        self.se = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Conv2d(channels, mid, kernel_size=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(mid, channels, kernel_size=1),
            nn.Sigmoid(),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return x * self.se(x)


class DynamicRangeMask(nn.Module):
    """Implements Fig. 3(a) & Equations (12), (13), and (14): Mask_DR."""
    def __init__(self, feat_ch: int, img_ch: int = 3):
        super().__init__()
        self.conv_img = nn.Conv2d(img_ch, feat_ch, kernel_size=3, padding=1)
        self.conv_fl_gate = nn.Conv2d(feat_ch, feat_ch, kernel_size=3, padding=1)
        self.conv_w = nn.Conv2d(feat_ch, feat_ch, kernel_size=3, padding=1)
        self.conv_fl_branch = nn.Conv2d(feat_ch, feat_ch, kernel_size=3, padding=1)

    def forward(self, F_L: torch.Tensor, L_raw: torch.Tensor) -> torch.Tensor:
        H, W = F_L.shape[-2:]
        L_down = F.interpolate(L_raw, size=(H, W), mode='bilinear', align_corners=False)  # Eq. (12)
        w_pixel = torch.sigmoid(self.conv_w(self.conv_img(L_down) + self.conv_fl_gate(F_L)))  # Eq. (13)
        mask_dr = w_pixel * self.conv_fl_branch(F_L) + F_L  # Eq. (14)
        return mask_dr


class AFMF(nn.Module):
    """
    Implements Fig. 3(c) & Equations (15) to (18):
    Adaptive Feature Modulation Fusion combining Mask_DR, GAP Softmax weighting, and Channel Attention.
    """
    def __init__(self, channels: int, img_ch: int = 3):
        super().__init__()
        self.dr_mask = DynamicRangeMask(channels, img_ch)
        self.conv_in = nn.Sequential(
            nn.Conv2d(channels * 2, channels, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
        )
        self.conv_w1 = nn.Conv2d(channels, channels, kernel_size=1)
        self.conv_w2 = nn.Conv2d(channels, channels, kernel_size=1)
        self.ca = ChannelAttention(channels)
        self.conv_out = nn.Conv2d(channels, channels, kernel_size=1)

    def forward(self, F_L: torch.Tensor, F_E: torch.Tensor, L_raw: torch.Tensor) -> torch.Tensor:
        F_L_prime = self.dr_mask(F_L, L_raw)
        f_in = self.conv_in(torch.cat([F_L_prime, F_E], dim=1))  # Eq. (15)

        gap = F.adaptive_avg_pool2d(f_in, 1)
        w1 = self.conv_w1(gap)  # Eq. (16)
        w2 = self.conv_w2(gap)

        # Softmax across modality dimension (Eq. 17)
        weights = F.softmax(torch.stack([w1, w2], dim=1), dim=1)
        w1_prime, w2_prime = weights[:, 0], weights[:, 1]

        fused = w1_prime * F_L_prime + w2_prime * F_E
        return self.conv_out(self.ca(fused))  # Eq. (18)


# =====================================================================
# 3. CONVLSTM CELL & DUAL-ENCODER REHDR NETWORK (SECTIONS 3.3 & 3.6)
# =====================================================================

class ConvLSTMCell(nn.Module):
    """Convolutional LSTM (Shi et al., 2015) maintaining temporal hidden states h_i, c_i."""
    def __init__(self, channels: int):
        super().__init__()
        self.channels = channels
        self.conv_gates = nn.Conv2d(channels * 2, channels * 4, kernel_size=3, padding=1)

    def forward(self, x: torch.Tensor, state=None):
        B, C, H, W = x.shape
        if state is None:
            h_prev = torch.zeros((B, C, H, W), dtype=x.dtype, device=x.device)
            c_prev = torch.zeros((B, C, H, W), dtype=x.dtype, device=x.device)
        else:
            h_prev, c_prev = state

        gates = self.conv_gates(torch.cat([x, h_prev], dim=1))
        i, f, o, g = torch.chunk(gates, 4, dim=1)
        i = torch.sigmoid(i)
        f = torch.sigmoid(f)
        o = torch.sigmoid(o)
        g = torch.tanh(g)

        c_next = f * c_prev + i * g
        h_next = o * torch.tanh(c_next)
        return h_next, (h_next, c_next)


class REHDR(nn.Module):
    """
    Complete Recurrent Event-Guided Multimodal Fusion Network (Fig. 3):
    Dual-encoder U-Net with SConv (k=4, s=2), TConv (k=5, s=2, p=2, op=1),
    AFMF fusion at each scale, and ConvLSTM temporal state propagation.
    """
    def __init__(self, event_bins: int = 10, img_ch: int = 3, base_ch: int = 32):
        super().__init__()
        C = base_ch
        self.head_e = nn.Conv2d(event_bins, C, kernel_size=3, padding=1)
        self.head_l = nn.Conv2d(img_ch, C, kernel_size=3, padding=1)

        # --- Event Encoder Branch (ResBlock + SConv k=4, s=2, p=1) ---
        self.e_res0 = ResBlock(C, C)
        self.e_sconv0 = nn.Conv2d(C, C, kernel_size=3, padding=1)

        self.e_res1 = ResBlock(C, 2 * C)
        self.e_sconv1 = nn.Conv2d(2 * C, 2 * C, kernel_size=4, stride=2, padding=1)

        self.e_res2 = ResBlock(2 * C, 4 * C)
        self.e_sconv2 = nn.Conv2d(4 * C, 4 * C, kernel_size=4, stride=2, padding=1)

        # --- Image + Fusion + ConvLSTM Encoder Branch ---
        self.l_res0 = ResBlock(C, C)
        self.l_sconv0 = nn.Conv2d(C, C, kernel_size=3, padding=1)
        self.afmf0 = AFMF(C, img_ch)
        self.lstm0 = ConvLSTMCell(C)

        self.l_res1 = ResBlock(C, 2 * C)
        self.l_sconv1 = nn.Conv2d(2 * C, 2 * C, kernel_size=4, stride=2, padding=1)
        self.afmf1 = AFMF(2 * C, img_ch)
        self.lstm1 = ConvLSTMCell(2 * C)

        self.l_res2 = ResBlock(2 * C, 4 * C)
        self.l_sconv2 = nn.Conv2d(4 * C, 4 * C, kernel_size=4, stride=2, padding=1)
        self.afmf2 = AFMF(4 * C, img_ch)
        self.lstm2 = ConvLSTMCell(4 * C)

        # --- Two Intermediate Residual Bottleneck Modules (Section 3.3) ---
        self.bottleneck = nn.Sequential(
            ResBlock(4 * C, 4 * C),
            ResBlock(4 * C, 4 * C),
        )

        # --- Decoder Branch (ResBlock + TConv k=5, s=2, p=2, output_padding=1) ---
        self.dec2_res = ResBlock(4 * C, 4 * C)
        self.dec2_tconv = nn.ConvTranspose2d(4 * C, 2 * C, kernel_size=5, stride=2, padding=2, output_padding=1)

        self.dec1_res = ResBlock(4 * C, 2 * C)
        self.dec1_tconv = nn.ConvTranspose2d(2 * C, C, kernel_size=5, stride=2, padding=2, output_padding=1)

        self.dec0_res = ResBlock(2 * C, C)
        self.out_conv = nn.Sequential(
            nn.Conv2d(2 * C, img_ch, kernel_size=3, padding=1),
            nn.Sigmoid(),
        )

    def forward_step(self, E_t: torch.Tensor, L_t: torch.Tensor, states=None):
        if states is None:
            states = [None, None, None]

        fe_in = self.head_e(E_t)
        fl_in = self.head_l(L_t)

        # Scale 0 (H x W)
        e0 = self.e_sconv0(self.e_res0(fe_in))
        l0 = self.l_sconv0(self.l_res0(fl_in))
        m0 = self.afmf0(l0, e0, L_t)
        h0, s0 = self.lstm0(m0, states[0])

        # Scale 1 (H/2 x W/2)
        e1 = self.e_sconv1(self.e_res1(e0))
        l1 = self.l_sconv1(self.l_res1(h0))
        m1 = self.afmf1(l1, e1, L_t)
        h1, s1 = self.lstm1(m1, states[1])

        # Scale 2 (H/4 x W/4)
        e2 = self.e_sconv2(self.e_res2(e1))
        l2 = self.l_sconv2(self.l_res2(h1))
        m2 = self.afmf2(l2, e2, L_t)
        h2, s2 = self.lstm2(m2, states[2])

        # Bottleneck & Decoder with Skip Connections
        b = self.bottleneck(h2)
        u2 = self.dec2_tconv(self.dec2_res(b))
        u1 = self.dec1_tconv(self.dec1_res(torch.cat([u2, h1], dim=1)))
        u0 = self.dec0_res(torch.cat([u1, h0], dim=1))

        hdr_pred = self.out_conv(torch.cat([u0, fl_in], dim=1))
        return hdr_pred, [s0, s1, s2]

    def forward(self, E_seq: torch.Tensor, L_seq: torch.Tensor):
        """Processes sequence [B, T, C, H, W] and returns reconstructed HDR sequence."""
        T = L_seq.size(1)
        states = None
        outputs = []
        for t in range(T):
            out_t, states = self.forward_step(E_seq[:, t], L_seq[:, t], states)
            outputs.append(out_t)
        return torch.stack(outputs, dim=1)


# =====================================================================
# 4. SSASDL STREAMING SEQUENTIAL DATA LOADER (SECTION 3.4) & SMOKE TEST
# =====================================================================

class SSASDLVideoDataset(Dataset):
    """
    Implements Section 3.4 (SSASDL):
    Dynamically samples a random starting index within long video streams,
    loads only a fixed window of length TS (8 frames), and applies synchronized flips.
    """
    def __init__(self, num_sequences=4, seq_len=20, ts_window=8, event_bins=10, H=64, W=64):
        self.num_sequences = num_sequences
        self.seq_len = seq_len
        self.ts = ts_window
        self.events = torch.randn(num_sequences, seq_len, event_bins, H, W)
        self.ldr = torch.rand(num_sequences, seq_len, 3, H, W)
        self.hdr_gt = torch.rand(num_sequences, seq_len, 3, H, W)

    def __len__(self):
        return self.num_sequences

    def __getitem__(self, idx):
        start_t = random.randint(0, self.seq_len - self.ts)
        e_clip = self.events[idx, start_t:start_t + self.ts]
        l_clip = self.ldr[idx, start_t:start_t + self.ts]
        g_clip = self.hdr_gt[idx, start_t:start_t + self.ts]

        # Synchronized random horizontal flip augmentation across sequence
        if random.random() > 0.5:
            e_clip = torch.flip(e_clip, dims=[-1])
            l_clip = torch.flip(l_clip, dims=[-1])
            g_clip = torch.flip(g_clip, dims=[-1])
        return e_clip, l_clip, g_clip


if __name__ == "__main__":
    torch.manual_seed(42)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # 1. Verify Event Spike Tensor Discretization (Eqs. 7 & 8)
    dummy_x = torch.randint(0, 64, (500,), device=device).float()
    dummy_y = torch.randint(0, 64, (500,), device=device).float()
    dummy_t = torch.linspace(0, 1000, 500, device=device)
    dummy_p = torch.where(torch.rand(500, device=device) > 0.5, 1.0, -1.0)
    spike_tensor = events_to_spike_tensor(dummy_x, dummy_y, dummy_t, dummy_p, H=64, W=64, num_bins=5)

    # 2. Initialize SSASDL Loader (TS=8) and REHDR Model (C=32)
    dataset = SSASDLVideoDataset(num_sequences=4, seq_len=20, ts_window=8, event_bins=10, H=64, W=64)
    loader = DataLoader(dataset, batch_size=2, shuffle=True)

    model = REHDR(event_bins=10, img_ch=3, base_ch=32).to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=2e-4, betas=(0.9, 0.999))

    # 3. Run Sequence Forward & Backward Optimization (Eqs. 19 - 22)
    e_seq, l_seq, g_seq = next(iter(loader))
    e_seq, l_seq, g_seq = e_seq.to(device), l_seq.to(device), g_seq.to(device)

    model.train()
    optimizer.zero_grad()
    o_seq = model(e_seq, l_seq)

    # Compute tone-mapped L1 + spatial gradient perceptual proxy loss
    t_o = mu_law_tonemap(o_seq, mu=5000.0)
    t_g = mu_law_tonemap(g_seq, mu=5000.0)
    loss_l1 = F.l1_loss(t_o, t_g)
    loss_perc = F.mse_loss(t_o[..., 1:, :] - t_o[..., :-1, :], t_g[..., 1:, :] - t_g[..., :-1, :])
    total_loss = 10.0 * loss_l1 + 0.5 * loss_perc

    total_loss.backward()
    optimizer.step()

    params_m = sum(p.numel() for p in model.parameters()) / 1e6
    print(f"Spike Tensor Shape: {tuple(spike_tensor.shape)} | Video Output Shape: {tuple(o_seq.shape)}")
    print(f"Smoke Test Total Loss: {total_loss.item():.4f} | Model Parameters: {params_m:.2f}M")

Frequently Asked Questions

Related Research Analyses on aitrendblend.com