Why HSS-Net Splits Echo Video Segmentation Into Two Speeds

A metric that looks reassuring but hides the problem that matters

Ejection fraction is one of the most commonly used numbers in cardiology, a single percentage describing how much blood the left ventricle pumps out with each heartbeat. Getting it right depends on accurately tracing the boundary of the left ventricle, the endocardium, at two specific moments in the cardiac cycle, end diastole when the chamber is at its fullest and end systole when it has just finished contracting, then feeding those two outlines into a volume formula. Automating that tracing step with deep learning has been an active research area for years, and by the usual yardstick, the Dice coefficient that measures how closely a predicted region overlaps a ground truth region, many existing methods already look quite good, often scoring in the low to mid 90s out of 100.

The paper’s opening figure makes an argument that should reframe how anyone reads those Dice scores. It shows two segmentation results from the same model, one with a mean Dice of 95.51 and a resulting ejection fraction bias of 2.88 percentage points against ground truth, and another with a mean Dice of 94.72, barely a point lower, but a resulting bias of 11.18 percentage points, nearly four times worse. The difference between the two cases is not how much of the ventricle got mislabeled overall, it is precisely where the mistakes landed. The second case had small errors concentrated at the base and apex of the ventricle, the two ends of the chamber that carry outsized weight in the standard Simpson’s method of disks used to convert a 2D outline into an estimated volume.

Why the base and apex punch above their weight

The Simpson’s method of disks, the formula both this paper and standard clinical practice use to turn a traced ventricle outline into a volume estimate, works by stacking a series of thin disks along the long axis of the chamber and summing their volumes. The disks near the base and apex sit at the ends of that stack, where the chamber is narrowing to a point, and small errors in exactly where the boundary sits at those narrow ends can shift the estimated radius of several disks at once, propagating into a disproportionately large change in the total volume estimate. A segmentation model trained purely to maximize overall Dice overlap has no particular incentive to prioritize accuracy in those specific end regions over accuracy anywhere else in the chamber, which is exactly the mismatch the paper sets out to correct.

Splitting the network into a fast local half and a slow global half

HSS-Net, short for Hierarchical Spatio-temporal Segmentation Network, is built as a symmetric encoder decoder with four stages on each side. The first two stages of the encoder process each video frame individually using separable convolution blocks, adapted from the inverted separable convolution design in MobileNetV2, whose job is capturing fine grained, single frame detail such as edges and local texture. The third and fourth stages switch to a fundamentally different mechanism, Spatio Temporal Mamba blocks built around a state space model, whose job is capturing how the ventricle boundary moves and deforms across the whole sequence of frames making up one heartbeat.

This is a deliberate response to a trade off the authors observed in prior work. Methods that only look at single frames, treating end diastole and end systole as two unrelated still images, can nail fine local detail but have no mechanism for enforcing that the boundary moves in a temporally coherent way from one frame to the next, so small per frame errors can accumulate independently without correcting each other. Methods that lean heavily on multi frame or video level processing gain temporal consistency but can lose track of fine spatial detail in the process, since attending broadly across many frames tends to smooth over the kind of sharp, localized boundary precision that the base and apex regions specifically need. Putting convolution first and Mamba second, rather than choosing one architecture throughout, is the paper’s attempt to get both properties without sacrificing either.

Why Mamba instead of a Transformer for the temporal half

The paper’s related work section notes that Transformer based architectures with self attention have already been applied to video segmentation, including echocardiography, and that adding specialized modules on top of self attention can capture temporal information. But modeling long sequences with self attention scales expensively, since attention cost grows quadratically with sequence length, which becomes a real constraint once you are processing several frames of a full heartbeat cycle rather than a single still image. Mamba style state space models offer a way to model long range dependencies across a sequence with cost that scales linearly rather than quadratically, which is part of why the compute numbers in the results table turn out to favor HSS-Net so clearly despite it modeling motion across ten frames rather than working from isolated stills.

The Spatio Temporal Cross Scan module, and why it scans diagonally

State space models like Mamba were originally built for one dimensional sequences such as text, so applying one to a spatio temporal video volume requires first flattening the multi dimensional data into a single long sequence the model can scan through. How you choose to flatten that data, in what order the model walks through frames and spatial positions, turns out to matter a great deal for what kinds of dependencies the model can easily pick up on.

The paper’s Spatio Temporal Cross Scan module, STCS for short, defines four distinct ways to walk through the video volume, each paired with both a forward and a backward pass. A temporal scan walks along the sequence of frames at a fixed spatial position, forward and backward, to capture how a single point on the ventricle boundary moves over time. A spatial scan walks across spatial positions within a shared time reference, capturing consistency at the same moment across nearby locations. The more unusual pair, a spatio temporal diagonal scan and an anti diagonal scan, rearranges the order in which spatial positions get visited into a diagonal zigzag pattern before scanning forward and backward through time, deliberately breaking up the neighboring, locally adjacent order that the temporal and spatial scans preserve.

The rationale the authors give for the diagonal scans is specific and grounded in cardiac physiology rather than being an arbitrary architectural flourish. A heartbeat is not a perfectly synchronized, uniform contraction, different regions of the ventricle wall move with somewhat different timing and magnitude, so relationships worth capturing, such as how apex motion changes relate to lateral wall contraction elsewhere in the chamber, are not necessarily between spatially neighboring points. A pure sequential scan, always moving to the next adjacent position, has an implicit bias toward picking up local, nearby correlations most strongly. The diagonal scanning pattern deliberately disrupts that local adjacency, forcing the state space model’s scan to connect more distant, less obviously related positions, which the ablation results below suggest genuinely helps rather than just adding complexity for its own sake.

Does the hierarchical split actually earn its keep

The core architectural claim gets tested directly with two controlled ablations. An Image-level variant replaces every stage of the network with the convolutional separable blocks, discarding the Mamba stages entirely and processing frames independently throughout. A Video-level variant does the opposite, using spatio temporal Mamba blocks at every stage instead of convolution anywhere. Both variants underperformed the full hierarchical HSS-Net, and the gap shows up most clearly in the correlation metric, arguably the single number that best captures whether the model’s ejection fraction estimates actually track the ground truth across a whole population of patients rather than just matching pixel outlines on average.

Neither single architecture approach came close to the hierarchical combination, and interestingly, the two ablated variants score fairly close to each other despite representing opposite architectural philosophies, roughly 81 to 83 percent correlation on CAMUS and 75 to 78 percent on EchoNet-Dynamic. That pattern is a reasonably clean signal that neither convolution alone nor Mamba alone was the bottleneck holding earlier approaches back, it was the absence of both operating together at the stage they are each suited for.

Each of the four scan directions is pulling its own weight

A second ablation strips out each of the four STCS scanning modes individually while keeping the rest of the architecture intact. Every removal cost some accuracy, though not by equal amounts. Removing the diagonal and anti diagonal scans hurt less than removing the temporal or spatial scans on the correlation metric, dropping CAMUS correlation to 86.69 and 88.09 respectively compared with drops to 83.73 and 80.44 for removing the temporal and spatial scans. That ordering makes intuitive sense, temporal and spatial coherence are the more fundamental signal, with the diagonal scans acting as a refinement on top rather than a replacement for either. But every single scan mode still contributed measurably to the full model’s final score of 90.47, and removing any one of the four left a gap the full four way combination closed.

How HSS-Net stacks up against prior methods

The comparison set is broad, spanning image based segmentation architectures originally built for general medical imaging, such as UNet++, TransUNet, SegFormer, and H2Former, purpose built echocardiography models such as SSCF and PKEchoNet, and two other video focused state space model approaches, VideoMamba and Vivim, giving a reasonably comprehensive picture of where the field currently stands.

Two things stand out beyond the headline correlation number. First, HSS-Net used the least compute of any method in the table by a wide margin, 5.6 gigaflops per frame against 38.6 for TransUNet and around 20 to 22 for the two video Mamba baselines, while still posting the best Dice, the best HD95 boundary distance, and the best correlation. Second, TransUNet actually posted a slightly tighter mean bias, 1.72 against HSS-Net’s 2.43, but with a noticeably worse correlation and more than three times the parameter count, which suggests TransUNet’s bias number may partly reflect its predictions clustering in a way that happens to average out well across the dataset rather than tracking the ground truth consistently case by case, a distinction the correlation metric is better suited to catch than a mean bias figure alone.

The pediatric dataset results make the case for the approach even more concretely, since pediatric echocardiography is generally regarded as a harder imaging setting due to smaller chamber sizes and often more challenging acoustic windows.

The mean bias drop here is the most dramatic number in the whole paper, from a range of roughly 5.3 to 7.6 percentage points for every prior method down to 1.29 for HSS-Net, a reduction of well over 70 percent relative to the closest competitor. Given that a several point swing in ejection fraction can be the difference between a reading read as normal cardiac function and one flagged as reduced, a bias reduction of that scale is not a marginal statistical improvement, it is the kind of gap that plausibly changes how many borderline cases would be classified correctly.

Where the approach still has real limits

A few things are worth being direct about, since the paper itself is largely focused on demonstrating the method works rather than dwelling on its boundaries.

Two of the three benchmark datasets, EchoNet-Pediatric and EchoNet-Dynamic, only provide ground truth annotations for the end diastolic and end systolic frames, leaving every intermediate frame in the ten frame clips the model processes without direct supervision. HSS-Net’s temporal modeling has to learn coherent motion across those unlabeled intermediate frames indirectly, guided only by the two labeled endpoints and whatever consistency the architecture itself enforces, which is a real constraint on how much the model can be checked against ground truth during training even though it is exactly the frame continuity problem the architecture is designed to exploit.

The evaluation also depends entirely on Simpson’s method of disks as the ground truth generating formula, both for the labels the model trains against and for how its predictions get scored. That formula itself carries known simplifying assumptions in real clinical use, treating the ventricle as built from stacked disks along a single long axis, and a 2D single or biplane view inherently misses out of plane geometry that true 3D imaging would capture. None of that is a flaw specific to this paper, it is the same convention essentially all comparable echocardiography segmentation research and, at least historically, a great deal of everyday clinical practice both rely on, but it does mean the ejection fraction numbers reported throughout the paper inherit whatever systematic limitations that formula has relative to a true 3D ground truth, a gap this study does not attempt to quantify or close.

All three datasets are established public research benchmarks rather than a genuinely external, held out clinical site collected specifically to test generalization beyond what the model saw representative examples of during development. CAMUS comes from a single French hospital, and while EchoNet-Pediatric and EchoNet-Dynamic each come from Stanford affiliated hospitals, the standard train, validation, and test splits used here are still drawn from the same overall data collection efforts the field has repeatedly benchmarked against, so genuinely prospective, multi institution validation on data collected after and independent of this study remains an open step before any claim of real world clinical generalization would be warranted.

Where this fits in the broader echocardiography AI landscape

The paper’s related work section traces a reasonably clear arc from earlier post processing approaches, such as work that enforces temporal consistency on top of an already generated segmentation using cardiac anatomical priors, through pseudo label based collaborative learning of segmentation and tracking, up to the current wave of architectures built to model temporal information more directly inside the network itself. What distinguishes HSS-Net within that arc is less the use of Mamba specifically, since two of the baselines it compares against, VideoMamba and Vivim, already apply state space models to video segmentation, and more the explicit hierarchical separation of which architecture handles which job, plus the STCS module’s deliberately non local diagonal scanning pattern designed around the specific physiological premise that cardiac motion is not spatially uniform.

It is also worth noting where HSS-Net did not win outright. The paper is candid that image based methods remain competitive on certain datasets and conditions, particularly where inter frame differences are minimal or dynamic changes are subtle enough that a strong single frame model captures most of what matters anyway. That is a reasonable, physically grounded caveat rather than a hedge, since a hierarchical architecture built to balance detail against motion should logically show its biggest advantage precisely in cases where both genuinely compete for attention, and offer less differential advantage in cases where one clearly dominates.

A simplified PyTorch implementation of the STCS module

The snippet below reimplements the core mechanics of the Spatio Temporal Cross Scan module described in Section 2.2 and Figure 3 of the paper, showing how a spatio temporal feature volume gets reshaped into four different scan orders, each processed by a bidirectional state space style scan, before being recombined. It uses a lightweight recurrent scan as a stand in for a full selective state space model implementation, which is enough to demonstrate the scanning logic without requiring a specialized Mamba kernel.

Two caveats before adapting this. First, the real STCS module uses a proper selective state space (S6) scan, the core Mamba mechanism, which has data dependent parameters that let it selectively propagate or forget information along the sequence, a property this GRU stand in only loosely approximates for demonstration purposes. Second, the paper’s actual encoder interleaves this module between the two convolutional stages and the decoder’s up sampling path inside a full four stage encoder decoder, whereas this snippet isolates just the STCS mechanism itself to keep the example focused and runnable without the rest of the architecture around it.

The bigger picture

The most transferable idea in this paper is not really about echocardiography specifically, it is a caution about metric mismatch that shows up constantly in medical image segmentation more broadly. Dice score, and similar region overlap metrics, are attractive because they are simple, well understood, and easy to optimize directly, but they treat every pixel of error as roughly equally costly, when the actual clinical quantity downstream of the segmentation, here ejection fraction, can be extremely sensitive to errors in specific, geometrically important regions while being comparatively tolerant of errors elsewhere. The paper’s opening figure, two segmentation maps with nearly identical Dice scores and wildly different ejection fraction bias, is really the whole argument in miniature, and it is a pattern worth watching for in any medical imaging task where the segmentation itself is a means to a downstream clinical calculation rather than the end goal in its own right.

The specific architectural response here, splitting convolution and Mamba across a hierarchy rather than picking one paradigm throughout, and designing the scanning pattern around a physiological premise about non uniform cardiac motion rather than an arbitrary engineering choice, is a reasonably disciplined way of acting on that observation rather than just bolting on more capacity and hoping accuracy improves generally. The ablation results support that discipline paid off, both the hierarchical split and each individual scan direction measurably moved the correlation and bias numbers in the right direction.

What remains open is how far these gains hold up outside the specific benchmark datasets used here, given that genuinely independent, prospective multi site validation was not part of this study, and how sensitive the approach is to the particular choice of Simpson’s method of disks as the underlying volume formula both training and evaluation depend on. Those are the natural next steps for this line of work rather than flaws in what has been shown so far, and the size of the reported gains, particularly the bias reduction on the pediatric dataset, is large enough to make that next round of validation worth watching for.

Honest limitations, summarized

• Two of the three benchmark datasets only annotate the end diastolic and end systolic frames, leaving intermediate frames in each video clip without direct ground truth supervision during training.
• Ground truth ejection fraction throughout the study is generated using Simpson’s method of disks from 2D or biplane views, which carries known simplifying geometric assumptions relative to true 3D ventricle volume and is a limitation shared with most comparable prior work rather than unique to this paper.
• All evaluation happens on established public research benchmarks using standard splits, rather than on a genuinely independent, prospectively collected external clinical dataset, so claims about generalization to new hospitals and new scanner types remain to be tested directly.
• The paper acknowledges that purely image based methods remain competitive on certain datasets or conditions with minimal inter frame variation, meaning the hierarchical approach’s advantage is not uniform across every possible imaging scenario.

Frequently asked questions

Related reading