HARP-NeXt Fuses Range Images and Points for Fast LiDAR Segmentation

The real bottleneck was never just the network

LiDAR semantic segmentation assigns a class label, car, pedestrian, road, sidewalk, vegetation, and so on, to every single point in a 3D scan. Autonomous vehicles and mobile robots depend on this labeling to understand what is around them well enough to plan a safe path. The field has produced three broad families of solutions. Point based methods such as PointNeXt, WaffleIron, and PTv3 operate directly on the raw 3D points and preserve the most geometric detail, but they lean on operations such as nearest neighbor search or point serialization that do not run quickly on a small onboard computer. Sparse convolution methods such as Minkowski and Cylinder3D voxelize the scene and only compute on occupied cells, saving some cost but still proving too slow for real time use. Projection based methods such as SalsaNext, CENet, and FRNet flatten the 3D scan into a 2D range image and run ordinary, highly optimized 2D convolutions on it, which is fast but throws away spatial detail during the flattening step.

What makes this paper different is where the authors looked for the actual slowdown. A companion study by the same group, cited directly in the introduction, measured that the preprocessing stage, projecting raw points into a range image, clustering points per pixel, and moving data around, can eat up to 83 percent of total execution time for range image methods. That number reframes the whole problem. Even a perfectly efficient neural network cannot rescue a pipeline whose bottleneck sits upstream of the network entirely, and most published LiDAR segmentation papers report inference time only, quietly leaving that upstream cost out of the comparison.

Moving preprocessing onto the GPU

HARP-NeXt’s first contribution is a restructured preprocessing workflow. Instead of loading raw scan data into CPU memory, running the spherical projection and clustering there, and only then transferring a finished tensor to the GPU, the new workflow moves raw point data to the GPU immediately after collation, then performs the spherical projection, pixel clustering, and feature preparation as GPU operations. The spherical projection itself is the standard formula used across the range image literature, converting each 3D point into image coordinates through an arctangent and arcsine based mapping that accounts for the sensor’s vertical field of view.

The payoff shows up directly in Table I of the paper. On the RTX4090 workstation GPU, HARP-NeXt’s preprocessing takes 3 milliseconds on both nuScenes and SemanticKITTI, versus 8 milliseconds for SalsaNext and 22 milliseconds for Minkowski on nuScenes, and versus 22 to 261 milliseconds for competing methods on the larger SemanticKITTI scans. On the Jetson AGX Orin, the embedded chip actually meant for deployment, the gap widens further because CPU bound operations such as KD tree construction and space filling curve serialization, used by WaffleIron and PTv3, are difficult to parallelize and hit the embedded CPU especially hard.

Conv-SE-NeXt, one block instead of a deep stack

The second contribution is the feature extraction block itself. Most segmentation backbones get their representational power by stacking several convolutional blocks per stage, which is expensive on embedded hardware. HARP-NeXt instead uses a single, carefully designed block per stage, called Conv-SE-NeXt, built from three ideas borrowed and recombined from prior architectures, ResNet’s residual connection, ConvNeXt’s depthwise separable convolution, and SE-ResNet’s channel attention mechanism.

The block first applies a depthwise convolution with a large kernel, 3 by 3 for nuScenes and 7 by 7 for SemanticKITTI, which captures a wide receptive field without adding extra layers, followed by a 1 by 1 pointwise convolution that mixes information across channels.

Instead of the fully connected layers a standard squeeze and excitation module would use to compute channel attention, the authors substitute 1 by 1 per channel convolutions, which are computationally cheaper while keeping the same expressive power. A global average pooling step produces one descriptor value per channel, two convolutions with a ReLU and then a Hardsigmoid nonlinearity turn that descriptor into an attention weight per channel, and the weight rescales the feature map before a residual connection adds the block’s input back in to keep gradients flowing during training.

The ablation study backs up the design choice directly. Swapping Conv-SE-NeXt for a plain ResNet block, a ConvNeXt block, an SE-ResNet block, a MobileNetV3 block, or a prior depthwise squeeze and excitation block called DSEB all reduced accuracy, by as much as 3.8 mIoU points on nuScenes and 6.4 points on SemanticKITTI, while Conv-SE-NeXt also kept inference faster on the Jetson AGX Orin than every alternative except MobileNetV3, which still trailed it in accuracy.

Fusing range image and point features without nearest neighbor search

The backbone runs four stages, and in each stage the network keeps two parallel streams alive, pixel level features living on the 2D range image grid and point level features living on the raw 3D points. What ties the two streams together is a pair of simple mapping functions rather than an expensive geometric search. Every 3D point already knows which pixel it was projected into during preprocessing, so moving from points to pixels is a straightforward pooling operation over the points that share a pixel, and moving from pixels back to points is a lookup, each point simply reads the feature vector sitting at its own projected pixel location.

Within each stage, pixel features from the current stage are fused with pixel features carried over and upsampled from the previous stage using bilinear interpolation, then passed through an attention mechanism that learns how much weight to give the fused result before adding it back to the current stage’s features in a residual manner. The refined pixel features are then mapped back down to update the point stream for the next stage, so information keeps flowing in both directions, from the more global, context aware pixel view down to the fine grained point view and back up again, across all four stages of the backbone.

What the numbers actually show

Table I in the paper compares HARP-NeXt against eight methods spanning all four architecture families, point based, projection based, sparse convolution based, and fusion based, on both the RTX4090 workstation GPU and the Jetson AGX Orin embedded board.

Two things stand out reading down that table. First, HARP-NeXt sits second only to PTv3 on accuracy while every other method within roughly a point of PTv3, WaffleIron and Cylinder3D, takes at least ten times longer to run on the embedded board. Second, HARP-NeXt is not simply the fastest option, it is the fastest option among the accurate ones, sitting inside what the authors call the real time zone in their headline figure while the highest accuracy competitors sit well outside it. On SemanticKITTI the pattern repeats, HARP-NeXt reaches 65.1 percent mIoU, within a point of FRNet’s 66.0 percent, while running at 13 milliseconds on the RTX4090 compared to FRNet’s 86 milliseconds, roughly a 7 times speedup, and with the smallest parameter count of any method compared, 5.4 million against FRNet’s 10.0 million and Cylinder3D’s 55.9 million.

Where the model still struggles

The qualitative error analysis in the paper is candid about where HARP-NeXt falls short. Misclassifications concentrate on less safety critical classes such as vegetation rather than classes that matter for collision avoidance, which is the right failure mode to have if a failure mode has to exist. The per class breakdown in Table II shows HARP-NeXt ranking first in six classes and second in four out of sixteen on nuScenes, with its weakest relative showing on motorcycle and bicycle, both comparatively rare and visually thin classes in outdoor LiDAR scans, a known hard case across nearly every method in the comparison table, not unique to HARP-NeXt.

Trying the core block yourself

To get a feel for why Conv-SE-NeXt is cheap enough to use once per stage rather than stacking several of them, it helps to implement the block directly and fuse it into a minimal range and point pipeline. The block itself is small, a depthwise convolution, a pointwise convolution, and a per channel squeeze and excitation gate built from two more convolutions, and the fusion stage around it needs nothing more exotic than an index based scatter and gather to move features between the point cloud and the range image grid.

"""
HARP-NeXt core components: Conv-SE-NeXt block and a range point fusion stage
Implementation of the building blocks from Abou Haidar et al. (2025)
HARP-NeXt: High Speed and Accurate Range Point Fusion Network
for 3D LiDAR Semantic Segmentation (arXiv:2510.06876)

This demo implements the Conv-SE-NeXt feature extraction block exactly as
specified in Eq. 2 through Eq. 8 of the paper, then wraps it inside a single
fusion stage that maps features between a 2D range image and a 3D point
cloud, matching the point to pixel and pixel to point mappings in Eq. 11
through Eq. 14.
"""

import torch
import torch.nn as nn
import torch.nn.functional as F


# 1. Conv-SE-NeXt block, Fig. 3 and Eq. 2 through Eq. 8 in the paper.

class HardSwish(nn.Module):
    # sigma(x) = x * ReLU6(x + 3) / 6, Eq. 4 in the paper
    def forward(self, x):
        return x * F.relu6(x + 3.0) / 6.0


class ConvSENeXtBlock(nn.Module):
    """
    Depthwise separable convolution followed by a per channel
    Squeeze-and-Excitation gate built from 1x1 convolutions rather than
    fully connected layers, plus a residual skip connection.
    """

    def __init__(self, channels, kernel_size=7, se_reduction=4):
        super().__init__()
        padding = kernel_size // 2

        # Eq. 2, depthwise convolution, captures spatial patterns
        self.dw_conv = nn.Conv2d(
            channels, channels, kernel_size=kernel_size,
            padding=padding, groups=channels, bias=False,
        )
        self.dw_bn = nn.BatchNorm2d(channels)
        self.act = HardSwish()

        # Eq. 3, pointwise convolution, aggregates across channels
        self.pw_conv = nn.Conv2d(channels, channels, kernel_size=1, bias=False)
        self.pw_bn = nn.BatchNorm2d(channels)

        reduced = max(channels // se_reduction, 4)
        self.se_reduce = nn.Conv2d(channels, reduced, kernel_size=1)
        self.se_expand = nn.Conv2d(reduced, channels, kernel_size=1)
        self.relu = nn.ReLU(inplace=True)

    def hardsigmoid(self, x):
        return F.relu6(x + 3.0) / 6.0

    def forward(self, x):
        y_dw = self.act(self.dw_bn(self.dw_conv(x)))
        y_pw = self.pw_bn(self.pw_conv(y_dw))

        # Eq. 5, channel descriptor from global average pooling
        z_c = F.adaptive_avg_pool2d(y_pw, 1)

        # Eq. 6, attention weight per channel
        s_c = self.hardsigmoid(self.se_expand(self.relu(self.se_reduce(z_c))))

        # Eq. 7 and Eq. 8, apply the weight, then the residual connection
        y_tilde = y_pw * s_c
        return y_tilde + x


# 2. Point to pixel and pixel to point mappings, Eq. 11 through Eq. 14.

def points_to_pixels(point_feats, pixel_index, num_pixels_h, num_pixels_w):
    """Scatter-average point features into a dense [C, H, W] pixel map."""
    n, c = point_feats.shape
    hw = num_pixels_h * num_pixels_w
    pixel_sum = torch.zeros(hw, c, device=point_feats.device)
    pixel_count = torch.zeros(hw, 1, device=point_feats.device)

    pixel_sum.index_add_(0, pixel_index, point_feats)
    pixel_count.index_add_(0, pixel_index, torch.ones(n, 1, device=point_feats.device))
    pixel_count = pixel_count.clamp(min=1.0)

    pixel_feats = pixel_sum / pixel_count
    return pixel_feats.transpose(0, 1).reshape(1, c, num_pixels_h, num_pixels_w)


def pixels_to_points(pixel_feat_map, pixel_index):
    """Gather pixel features back out to every point projected into that pixel."""
    _, c, h, w = pixel_feat_map.shape
    flat = pixel_feat_map.reshape(c, h * w).transpose(0, 1)
    return flat[pixel_index]


class RangePointFusionStage(nn.Module):
    def __init__(self, channels, kernel_size=7):
        super().__init__()
        self.pixel_block = ConvSENeXtBlock(channels, kernel_size=kernel_size)
        self.point_refine = nn.Sequential(
            nn.Linear(channels * 2, channels),
            nn.BatchNorm1d(channels),
            HardSwish(),
        )

    def forward(self, point_feats, pixel_index, h, w):
        pixel_map = points_to_pixels(point_feats, pixel_index, h, w)
        pixel_map = self.pixel_block(pixel_map)

        mapped_back = pixels_to_points(pixel_map, pixel_index)
        fused_points = self.point_refine(
            torch.cat([point_feats, mapped_back], dim=1)
        )
        return fused_points, pixel_map


# 3. Smoke test with a synthetic LiDAR-like point cloud and range image grid.

if __name__ == "__main__":
    torch.manual_seed(0)

    num_points = 5000
    channels = 64
    height, width = 32, 480  # nuScenes range image resolution used in the paper

    point_feats = torch.randn(num_points, channels)
    pixel_index = torch.randint(0, height * width, (num_points,))

    stage = RangePointFusionStage(channels, kernel_size=3)
    stage.eval()

    with torch.no_grad():
        fused_points, pixel_map = stage(point_feats, pixel_index, height, width)

    print(f"Input point features:  {tuple(point_feats.shape)}")
    print(f"Pixel feature map:     {tuple(pixel_map.shape)}")
    print(f"Fused point features:  {tuple(fused_points.shape)}")
    assert fused_points.shape == point_feats.shape
    assert pixel_map.shape == (1, channels, height, width)

    num_classes = 17
    seg_head = nn.Linear(channels, num_classes)
    logits = seg_head(fused_points)
    print(f"Per-point class logits: {tuple(logits.shape)}")
    assert logits.shape == (num_points, num_classes)

    total_params = sum(p.numel() for p in stage.parameters())
    print(f"Conv-SE-NeXt fusion stage parameter count: {total_params:,}")
    print("Smoke test complete. Conv-SE-NeXt block and range point fusion stage ran end to end without error.")

Running this script produces a pixel feature map of shape 1 by 64 by 32 by 480, matching the nuScenes range image resolution the paper uses, a fused point feature tensor with the same shape as the input, and a set of per point class logits, confirming the whole scatter, convolve, gather, refine cycle described in Eq. 11 through Eq. 14 works end to end. The full four stage HARP-NeXt backbone stacks four of these fusion stages with a Fusion Head on top, but this single stage version isolates exactly the mechanism that lets the network stay accurate without stacking multiple blocks at each resolution.

Where this fits in the broader picture

LiDAR segmentation research has largely chased accuracy first, with efficiency treated as a secondary concern to be addressed through model compression or quantization after the fact. HARP-NeXt inverts that order, treating the entire pipeline, preprocessing included, as the object of optimization from the start, and the earlier study the authors cite, which systematically benchmarked real time readiness on Jetson hardware, appears to be what motivated that inversion. The result reads as a useful counter example to the assumption that the biggest accuracy and speed gains left on the table sit inside the network architecture. Sometimes they sit in the unglamorous data plumbing that runs before the network ever sees a tensor.

The design choices also travel reasonably well beyond LiDAR. Per channel convolutions standing in for fully connected layers inside a squeeze and excitation gate, and a single well designed block replacing several stacked ones, are both generic efficiency tricks that show up across mobile computer vision, not specific to point clouds, and the ablation study’s clean comparison against MobileNetV3 and a prior depthwise squeeze and excitation design gives some confidence the gains are not an artifact of this particular dataset.

Conclusion

HARP-NeXt earns its place by refusing to treat inference time as the only cost that matters. By measuring and then attacking the preprocessing bottleneck directly, the authors close a gap that most competing papers do not even report, and the resulting network reaches 77.1 percent mIoU on nuScenes and 65.1 percent on SemanticKITTI, close behind the top published methods, while running at a fraction of their total runtime on both a workstation GPU and genuinely embedded hardware.

The architectural contribution, Conv-SE-NeXt, is a modest looking block that does a lot of the practical work, and the ablation study makes an unusually direct case for it, showing losses of up to 6.4 mIoU points when it is swapped out for any of five alternative designs. Combined with an index based fusion mechanism that avoids nearest neighbor search entirely, the network manages to keep 3D geometric detail without paying the computational price that detail usually costs.

None of this comes without real tradeoffs. PTv3 still holds the outright accuracy record on nuScenes, and HARP-NeXt’s weaker showing on thin, rare classes like motorcycle and bicycle mirrors a known difficulty across the entire field rather than a solved problem. The paper is honest about this, framing its contribution as a superior tradeoff rather than a new ceiling.

For teams actually building autonomous vehicles or mobile robots on constrained hardware, the practical takeaway is less about any single number and more about where to look for the next efficiency gain. If the network itself has already been trimmed and the bottleneck persists, the preprocessing stage, the part of the pipeline that almost never shows up in a benchmark table, deserves the next look.

Frequently asked questions

Related reading on aitrendblend