Sam2Rad Explained: Teaching SAM2 to Prompt Itself on Ultrasound

Why bones in ultrasound are a hard problem for a foundation model

The Segment Anything Model, usually shortened to SAM, was trained on more than a billion masks drawn from natural photographs. It learned an extraordinarily general sense of what an object boundary looks like, and that generality is exactly what makes it useful across so many domains without retraining. But generality has a cost. SAM decides where an object is based on a prompt someone gives it, a box, a click, or a rough mask, and it was trained on the kind of crisp, high contrast boundaries you see in everyday photos. Bone in an ultrasound frame does not look like that at all. Ultrasound forms an image from reflected sound waves, and bone shows up as a bright line with a dark shadow behind it, surrounded by soft tissue that can look deceptively similar in texture and brightness. The paper points to speckle noise specifically, a grainy interference pattern that appears when the distance between adjacent reflecting structures falls below the resolution of the scanner, as one of the artifacts that throws foundation models trained on natural images off balance.

The authors ran a small side experiment that makes the scale of the problem concrete. They tested MedSAM, a version of SAM specifically fine tuned on a large curated set of medical images, against the original SAM and the newer SAM2, all given the same ground truth bounding boxes on the same three ultrasound datasets. MedSAM, despite its medical fine tuning, came in worst of all the models tested, scoring a Dice coefficient of just 14.5 percent on hip and 13.1 percent on shoulder. SAM2, trained purely on natural images with no medical fine tuning at all, comfortably beat it. That is a strange and useful result. It suggests that fine tuning a foundation model on medical images broadly does not automatically transfer to a specific, unusual modality like musculoskeletal ultrasound, and that the choice of base architecture and its zero shot generalization habits can matter more than domain specific fine tuning that was aimed at a different kind of medical image entirely.

The core idea, let the model draw its own prompts

Existing attempts to automate SAM’s prompting generally fall into two camps, and the paper’s Figure 3 lays them out clearly. One camp, including methods like PerSAM and Matcher, uses a learning free approach built on cosine similarity between a reference image and a test image, comparing patches to guess where the object of interest sits. That approach assumes foreground and background patches look different enough to tell apart by simple similarity, which the authors note breaks down in ultrasound where foreground and background textures are often genuinely hard to distinguish even by eye. The other camp, including GroundedSAM and AutoSAM, bolts on a separate, standalone image encoder or object detector to generate the prompts, which works reasonably well but adds a second full sized neural network’s worth of compute and does not share any of SAM’s own learned representations.

Sam2Rad takes a third path. Rather than building a new encoder from scratch or relying on patch similarity, it reuses the feature maps SAM’s own frozen image encoder already computed for the segmentation task, and trains a lightweight cross attention module called the Prompt Predictor Network, or PPN, to turn those existing features into prompts. The image encoder never gets updated. Only the small PPN, a fraction of the parameter count of the encoder it sits on top of, gets trained. That design choice does two things at once. It keeps the broad, general purpose knowledge baked into SAM’s pretraining fully intact, since nothing about the encoder changes, and it keeps the compute and data requirements for adapting to a new domain very low, since the PPN is the only thing being fit to the ultrasound data.

How the Prompt Predictor Network actually works

Learnable queries meet SAM’s own features

The PPN starts with a set of learnable object queries, a matrix Q of shape N by C, where N is the number of prompts to generate and C matches SAM’s internal embedding dimension of 256. These queries have no meaning at initialization, they are just trainable vectors, and the whole point of the network is to refine them through attention until each one represents something useful about where the target object sits. SAM’s image encoder extracts hierarchical feature maps from the input image, with SAM2 producing three levels at different resolutions, moving from coarse, high level semantic detail at the top of the hierarchy down to fine, low level spatial detail at the bottom. Before anything else happens, sinusoidal positional encodings are added to each of these feature levels, which is a standard trick that lets an attention mechanism reason about where a feature sits spatially rather than only what the feature represents.

At each hierarchical level, the queries and the current image features pass through what the paper calls a two way cross attention module. The name reflects the fact that information flows in both directions at once, the object queries look at the image features to update themselves, and the image features look at the object queries to update themselves too. In the paper’s own notation, this operation is written

where the tilde over z denotes the feature map after the positional encoding has been added. The updated features from one level then get added back into the raw features of the next level down, after interpolating to match spatial resolution and passing through a small convolution to keep the channel count consistent. That is a deliberate echo of how Feature Pyramid Networks combine coarse and fine information, letting semantic context from the low resolution level inform the higher resolution level rather than treating each scale in isolation. The process repeats through all three hierarchical levels for SAM2, or just once for the original single scale SAM, with the queries accumulating more refined, object specific information at every step.

Turning refined queries into prompts SAM already understands

Once the queries have been through all the cross attention rounds, the final refined set gets split into pieces with different jobs. The first two queries, a small slice of shape 2 by C, get passed through a linear layer to directly predict the four bounding box coordinates, the minimum and maximum x and y positions of the target object. The remaining N minus 2 queries are kept as high dimensional prompts, essentially abstract, learned representations that carry richer information than a box or a point ever could, since they exist in the same 256 dimensional space SAM’s own prompt encoder operates in. Separately, the final refined image features from the lowest, most detailed level of the hierarchy pass through a small convolutional block to produce a coarse mask prompt at one quarter resolution.

All three of these outputs, the predicted box, the high dimensional embeddings, and the mask prompt, get handed to SAM’s completely unmodified mask decoder exactly the way a human drawn box or mask would be. From the mask decoder’s point of view, nothing about the input pipeline looks unusual, it is simply receiving prompts, it just happens that those prompts were generated by a small trained network instead of a person with a mouse. That compatibility with SAM’s original interface is what lets Sam2Rad avoid needing any changes to the frozen backbone at all, and the same PPN design works whether it sits on top of SAM, SAM2, or in principle any future SAM variant with the same prompt encoder interface.

For tasks with more than one class to segment, such as distinguishing multiple bone structures in a single frame, the paper simply assigns a separate set of learnable queries to each class, so scaling to more targets means adding more query sets rather than redesigning the network.

Teaching the network what a good prompt looks like

Training the PPN requires a loss function that can supervise three different kinds of output at once, the mask, the box, and whether the queried object is even present in the frame at all, since some training frames may not contain the anatomical structure being queried. The total loss combines all three

with the weighting terms set to 10, 1, and 1 respectively in the experiments, meaning the mask term dominates the overall gradient signal. The mask loss itself is a focal loss, chosen because it downweights easy, already well classified pixels and concentrates gradient updates on the harder boundary regions, applied to both the intermediate mask prompt and the final predicted mask. The box loss combines a straightforward L1 distance between predicted and ground truth coordinates with a generalized IoU loss, a formulation that behaves better than plain L1 when predicted and true boxes do not overlap at all, which matters early in training when the PPN’s guesses can be wildly off. The objectness loss is a standard binary cross entropy term trained to predict whether the queried class is actually present, and at inference time, if that objectness head outputs a value below zero, the segmentation output for that query gets discarded entirely rather than forcing a mask onto an object that is not there.

What the results actually show

Zero shot performance is uneven across anatomy

Before introducing Sam2Rad’s own results, the paper benchmarks a range of SAM and SAM2 variants using ground truth bounding boxes as prompts, which represents something close to a best case scenario for prompt quality. The pattern that emerges matters for understanding why Sam2Rad exists in the first place.

Every model, without exception, does noticeably worse on shoulder than on hip or wrist. The best zero shot shoulder score across every model tested tops out at 25.3 percent Dice, roughly a third of what the same models achieve on hip. The paper attributes this directly to the shoulder bursa dataset’s greater anatomical variability and its more challenging imaging artifacts, and this is the single most important limitation to hold onto when reading the rest of the paper’s results, since it shapes everything that follows.

Learned prompts beat human drawn boxes, even without touching SAM2’s weights

The most striking single result in the paper is a comparison between bounding box prompts and Sam2Rad’s learned prompts, both fed into a completely frozen SAM2 with none of its own parameters updated.

Notice that the box column here uses ground truth boxes, meaning a human or an automated system with perfect knowledge of the true object location drew them. The learned prompts still win by a wide margin, especially on wrist, where every SAM2 variant jumps from the high 40s or low 60s into the low 80s. This tells us something worth pausing on. A perfectly placed bounding box only tells the mask decoder roughly where an object sits, it says nothing about the object’s shape, texture, or the fine boundary details that separate bone from surrounding tissue. The high dimensional learned prompts appear to be encoding some of that extra information, effectively giving the frozen mask decoder cues that a simple geometric box cannot represent no matter how accurately it is drawn.

The full picture across models and anatomy

The paper’s main results table combines learned prompts alone against learned prompts stacked with a predicted bounding box, tested across five different SAM and SAM2 backbones and all three anatomical datasets.

Adding the predicted bounding box on top of the learned prompts helps in every single row of this table, which makes sense given that a box grounds the mask decoder with an explicit spatial constraint even when the high dimensional prompts already carry rich information. The shoulder numbers, even at their best with SAM2Rad Hiera-Tiny reaching 77.6 percent Dice, still sit well below the roughly 85 percent achieved on hip and wrist. Sam2Rad clearly narrows the gap on shoulder dramatically compared to the 25.3 percent zero shot ceiling, but it does not erase the underlying anatomical difficulty entirely, and that residual gap is worth remembering before assuming this technique solves segmentation uniformly across every possible ultrasound target.

The ablation study, what each prompt type actually contributes

The authors ran a component study using the smallest SAM2 variant, frozen entirely, trained in a few shot setting with only 10 labeled wrist images, specifically to isolate how much each type of predicted prompt contributes on its own.

This table is the clearest evidence in the whole paper for why the high dimensional prompts matter so much. A predicted bounding box on its own only reaches 45.1 percent Dice, barely more than half of what the high dimensional prompts achieve entirely on their own at 79.8 percent. Combining prompt types keeps helping incrementally, topping out at 83.3 percent with box, mask, and high dimensional prompts together, and adding the hierarchical multiscale features on top nudges that up only slightly further to 83.7 percent, a gain the authors themselves describe as marginal. Worth noting too, Sam2Rad’s best configuration here edges out both UNet and UNet++, two well established, purpose built medical segmentation architectures trained directly on this task, while retaining the flexibility of sitting on top of a general purpose foundation model rather than being a specialized network trained from scratch for this one job.

Few shot performance and label noise

To test how little labeled data the approach actually needs, the authors trained Sam2Rad using only 10 randomly selected images from the hip training set and evaluated on the entire held out test set of 4849 hip images. Across every SAM2 variant tested, Dice scores landed around 0.74 to 0.78 with this tiny training set, a result broadly consistent with the ablation study’s finding that only 10 wrist images were also enough to demonstrate the value of combined prompts. The paper is candid that the 10 images were chosen at random rather than selected for difficulty or representativeness, so a more deliberate active learning style selection could plausibly do better, something the authors leave as an open direction rather than a claim they tested.

A separate qualitative observation concerns label quality. The ground truth masks used for training reportedly contained inconsistencies, including partial masks that did not fully outline the true bone boundary. The paper shows examples in its Figure 6 where Sam2Rad’s predicted segmentation appears more complete than the imperfect ground truth label it was trained against, which the authors interpret as evidence the model learned an underlying, generalizable notion of bone boundaries rather than simply memorizing the specific, sometimes flawed labels in its training set. That is a reasonable and honest interpretation of the qualitative evidence shown, though it rests on a small number of illustrated examples rather than a systematic audit of label quality across the full dataset.

Clinical translation gap

There is meaningful distance between these benchmark numbers and a tool ready for a radiology department. All three datasets in this study came from institutional ultrasound archives collected under a single ethics approval, split into training and test sets by subject rather than by hospital, scanner, or operator. That means the test set, however large, still reflects the same imaging equipment, the same acquisition protocols, and likely a similar patient population as the training set. Real world deployment would involve different ultrasound machine manufacturers, different probe frequencies, different operator technique, and patient populations with different body compositions, none of which this evaluation directly measures. The paper’s own future work section acknowledges the model is currently limited to two dimensional images only, while a meaningful share of real musculoskeletal ultrasound examinations, particularly for fracture assessment, involve scanning through a full volume or sweeping across multiple planes that a purely 2D model cannot natively reason about together. Any clinical use of a system like this would also need prospective validation against radiologist interpretation as a reference standard, formal regulatory clearance appropriate to its intended use, and a clear protocol for how the objectness head’s uncertainty gets communicated to whoever is using the tool, none of which this research paper attempts to establish.

Honest limitations

Beyond the generalization question already raised, a few other constraints deserve to be stated plainly. The shoulder dataset consistently underperforms hip and wrist across every configuration tested in the paper, including the best Sam2Rad results, and the authors do not fully resolve why shoulder ultrasound remains harder beyond attributing it to greater anatomical variability, which leaves an open question about whether further architectural changes or simply more training data would close that gap. The paper also reports that including hierarchical, multiscale features from SAM2’s encoder produced only a marginal 0.4 to 0.6 percentage point improvement, a smaller gain than the added complexity might suggest, and the authors themselves note this could matter more for smaller anatomical targets that this study did not specifically test.

No inference latency or runtime benchmarks are reported anywhere in the paper, so it is not possible to state how much additional computation the PPN adds on top of a frozen SAM2 forward pass, information that would matter for anyone considering real time deployment during a live ultrasound scan. The training loss weighting was fixed at 10 for mask, 1 for box, and 1 for objectness across all experiments, and the paper does not report a sensitivity analysis showing how much these specific values matter versus alternative weightings. Finally, the paper’s own future work section is explicit that the current framework only handles two dimensional images and has not yet been extended to CT, MRI, or X-ray, so any generalization claims about the technique should be scoped specifically to two dimensional ultrasound of the anatomical sites actually tested.

Where this points next

The most transferable idea here has little to do with ultrasound or bone specifically. Freezing a large pretrained foundation model entirely and training only a small attention based network to generate its inputs is a pattern that could extend to any domain where a general purpose model’s zero shot behavior is close but not quite right for a specialized use case, and where labeled data for full fine tuning is scarce. The authors’ own stated priorities, extending to CT, MRI, and X-ray, adapting to full three dimensional volumes rather than individual 2D frames, and distilling the trained system into something small enough for real time use, are all sensible next steps that would meaningfully strengthen the case for clinical relevance if they pan out. Anyone building on this work should pay particular attention to testing across multiple ultrasound vendors and clinical sites before treating the reported Dice improvements as representative of performance in a new hospital system.

Complete PyTorch implementation

The following is a full, runnable reproduction of the core Prompt Predictor Network idea described in the paper, including the two way cross attention module, the hierarchical feature pyramid style refinement, the box, mask, and objectness prediction heads, the combined focal, L1 plus GIoU, and binary cross entropy loss, a training loop, an evaluation function, and a smoke test on random dummy data so you can confirm the shapes work end to end before connecting a real frozen SAM2 checkpoint.

# sam2rad_ppn.py
# A runnable reproduction of the Prompt Predictor Network (PPN) core idea from
# Wahd, Felfeliyan, Zhou, Ghosh, McArthur, Zhang, Jaremko, Hareendranathan,
# "Sam2Rad", Computers in Biology and Medicine, 2025.
# This module is designed to sit on top of a frozen SAM or SAM2 image encoder.
# It does not reimplement SAM itself, only the prompt generation network.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader


# ---------------------------------------------------------------------
# Sinusoidal positional encoding for 2D feature maps
# ---------------------------------------------------------------------
def build_2d_sincos_pos_embed(H, W, C, device):
    assert C % 4 == 0, "channel count should be divisible by 4 for 2D sincos encodings"
    dim_quarter = C // 4
    omega = torch.arange(dim_quarter, device=device).float() / dim_quarter
    omega = 1.0 / (10000 ** omega)

    y_pos = torch.arange(H, device=device).float()
    x_pos = torch.arange(W, device=device).float()

    out_y = torch.einsum("m,d->md", y_pos, omega)
    out_x = torch.einsum("m,d->md", x_pos, omega)

    pos_y = torch.cat([out_y.sin(), out_y.cos()], dim=1)   # H x C/2
    pos_x = torch.cat([out_x.sin(), out_x.cos()], dim=1)   # W x C/2

    pos_y = pos_y[:, None, :].expand(H, W, C // 2)
    pos_x = pos_x[None, :, :].expand(H, W, C // 2)
    pos = torch.cat([pos_y, pos_x], dim=-1)                # H x W x C
    return pos.permute(2, 0, 1).unsqueeze(0)              # 1 x C x H x W


# ---------------------------------------------------------------------
# Two way cross attention, queries update from features and features
# update from queries in the same block
# ---------------------------------------------------------------------
class TwoWayCrossAttention(nn.Module):
    def __init__(self, dim=256, num_heads=8, mlp_ratio=4):
        super().__init__()
        self.q_to_f = nn.MultiheadAttention(dim, num_heads, batch_first=True)
        self.f_to_q = nn.MultiheadAttention(dim, num_heads, batch_first=True)
        self.norm_q1 = nn.LayerNorm(dim)
        self.norm_f1 = nn.LayerNorm(dim)
        self.norm_q2 = nn.LayerNorm(dim)
        self.mlp_q = nn.Sequential(
            nn.Linear(dim, dim * mlp_ratio), nn.GELU(), nn.Linear(dim * mlp_ratio, dim)
        )

    def forward(self, queries, features):
        # queries: B x N x C, features: B x L x C  (L = H*W, flattened spatial features)
        q_updated, _ = self.q_to_f(queries, features, features)
        queries = self.norm_q1(queries + q_updated)

        f_updated, _ = self.f_to_q(features, queries, queries)
        features = self.norm_f1(features + f_updated)

        queries = self.norm_q2(queries + self.mlp_q(queries))
        return queries, features


# ---------------------------------------------------------------------
# The Prompt Predictor Network
# ---------------------------------------------------------------------
class PromptPredictorNetwork(nn.Module):
    """
    Takes hierarchical frozen SAM/SAM2 image features and a set of
    learnable object queries, refines the queries through n levels of
    two-way cross attention (n=3 for SAM2 hierarchical features, n=1
    for plain SAM), and emits a bounding box, a mask prompt, and
    high-dimensional prompt embeddings ready for SAM's mask decoder.
    """
    def __init__(self, num_queries=8, embed_dim=256, num_levels=3, num_classes=1):
        super().__init__()
        self.num_queries = num_queries
        self.embed_dim = embed_dim
        self.num_levels = num_levels
        self.num_classes = num_classes

        # one learnable query set per class, shape C x N x embed_dim
        self.object_queries = nn.Parameter(torch.randn(num_classes, num_queries, embed_dim) * 0.02)

        self.cross_attn_blocks = nn.ModuleList([
            TwoWayCrossAttention(dim=embed_dim) for _ in range(num_levels)
        ])
        # 1x1 conv to keep channel dimension consistent when merging levels
        self.level_merge = nn.ModuleList([
            nn.Conv2d(embed_dim, embed_dim, kernel_size=1) for _ in range(num_levels - 1)
        ])

        self.box_head = nn.Linear(embed_dim, 4)
        self.objectness_head = nn.Linear(embed_dim, 1)
        self.mask_head = nn.Sequential(
            nn.ConvTranspose2d(embed_dim, 64, kernel_size=2, stride=2),
            nn.GELU(),
            nn.Conv2d(64, 1, kernel_size=1),
        )

    def forward(self, feature_levels, class_idx=0):
        """
        feature_levels: list of tensors, coarse to fine, each B x C x H x W,
        length must equal self.num_levels (use the last level repeated for
        plain SAM which only has one level).
        """
        B = feature_levels[0].shape[0]
        device = feature_levels[0].device
        Q = self.object_queries[class_idx].unsqueeze(0).expand(B, -1, -1).contiguous()

        running_feat = None
        for i, feat in enumerate(feature_levels):
            _, C, H, W = feat.shape
            pos = build_2d_sincos_pos_embed(H, W, C, device)
            feat_with_pos = feat + pos

            if running_feat is not None:
                # fold in the refined lower resolution feature from the previous level
                prev_up = F.interpolate(running_feat, size=(H, W), mode="bilinear", align_corners=False)
                prev_up = self.level_merge[i - 1](prev_up)
                feat_with_pos = feat_with_pos + prev_up

            flat_feat = feat_with_pos.flatten(2).permute(0, 2, 1)   # B x (H*W) x C
            Q, refined_flat = self.cross_attn_blocks[i](Q, flat_feat)
            running_feat = refined_flat.permute(0, 2, 1).view(B, C, H, W)

        box_query = Q[:, 0, :]                       # first query predicts the box
        obj_query = Q[:, 1, :]                       # second query predicts objectness
        hd_prompts = Q[:, 2:, :]                     # remaining queries are high-dimensional prompts

        pred_box = self.box_head(box_query).sigmoid()      # normalized xmin,ymin,xmax,ymax
        pred_objectness = self.objectness_head(obj_query)  # raw logit, threshold at 0 at inference
        pred_mask = self.mask_head(running_feat)            # B x 1 x 2H x 2W, quarter resolution mask prompt

        return {
            "box": pred_box,
            "objectness": pred_objectness,
            "mask_prompt": pred_mask,
            "high_dim_prompts": hd_prompts,
        }


# ---------------------------------------------------------------------
# Loss functions matching the paper, focal for masks, L1 + GIoU for boxes,
# BCE for objectness
# ---------------------------------------------------------------------
def focal_loss(pred_logits, target, gamma=3.0, alpha=0.7):
    pred = torch.sigmoid(pred_logits)
    pred = pred.clamp(1e-6, 1 - 1e-6)
    pos_term = -target * (1 - pred).pow(gamma) * torch.log(pred)
    neg_term = -alpha * (1 - target) * pred.pow(gamma) * torch.log(1 - pred)
    return (pos_term + neg_term).mean()


def box_iou_giou(pred_box, target_box):
    # boxes are xmin, ymin, xmax, ymax, normalized 0 to 1
    px1, py1, px2, py2 = pred_box.unbind(-1)
    tx1, ty1, tx2, ty2 = target_box.unbind(-1)

    inter_x1 = torch.max(px1, tx1)
    inter_y1 = torch.max(py1, ty1)
    inter_x2 = torch.min(px2, tx2)
    inter_y2 = torch.min(py2, ty2)
    inter_area = (inter_x2 - inter_x1).clamp(0) * (inter_y2 - inter_y1).clamp(0)

    pred_area = (px2 - px1).clamp(0) * (py2 - py1).clamp(0)
    target_area = (tx2 - tx1).clamp(0) * (ty2 - ty1).clamp(0)
    union = pred_area + target_area - inter_area + 1e-7
    iou = inter_area / union

    enc_x1 = torch.min(px1, tx1)
    enc_y1 = torch.min(py1, ty1)
    enc_x2 = torch.max(px2, tx2)
    enc_y2 = torch.max(py2, ty2)
    enc_area = (enc_x2 - enc_x1).clamp(0) * (enc_y2 - enc_y1).clamp(0) + 1e-7

    giou = iou - (enc_area - union) / enc_area
    return 1 - giou.mean()


def sam2rad_loss(outputs, targets, lambda_mask=10.0, lambda_box=1.0, lambda_obj=1.0):
    mask_pred = F.interpolate(outputs["mask_prompt"], size=targets["mask"].shape[-2:], mode="bilinear", align_corners=False)
    loss_mask = focal_loss(mask_pred, targets["mask"])

    loss_box_l1 = F.l1_loss(outputs["box"], targets["box"])
    loss_box_giou = box_iou_giou(outputs["box"], targets["box"])
    loss_box = loss_box_l1 + loss_box_giou

    loss_obj = F.binary_cross_entropy_with_logits(
        outputs["objectness"].squeeze(-1), targets["objectness"]
    )

    total = lambda_mask * loss_mask + lambda_box * loss_box + lambda_obj * loss_obj
    return total, {"mask": loss_mask.item(), "box": loss_box.item(), "objectness": loss_obj.item()}


# ---------------------------------------------------------------------
# Dummy dataset standing in for frozen SAM2 encoder feature maps
# ---------------------------------------------------------------------
class DummyFrozenFeatureDataset(Dataset):
    """
    In a real pipeline these feature maps come from a frozen SAM2 image
    encoder run once per image. Here we simulate three hierarchical
    levels with random tensors purely to validate the PPN's shapes and
    training loop end to end.
    """
    def __init__(self, n_samples=16, embed_dim=256):
        self.n_samples = n_samples
        self.embed_dim = embed_dim
        self.level_sizes = [(16, 16), (32, 32), (64, 64)]   # coarse to fine

    def __len__(self):
        return self.n_samples

    def __getitem__(self, idx):
        feats = [torch.randn(self.embed_dim, h, w) for (h, w) in self.level_sizes]
        target_mask = (torch.rand(1, 64, 64) > 0.7).float()
        target_box = torch.sort(torch.rand(4))[0]   # ensures xmin
        target_objectness = torch.tensor(1.0 if target_mask.sum() > 0 else 0.0)
        return feats, target_mask, target_box, target_objectness


def collate_features(batch):
    feats_batched = []
    n_levels = len(batch[0][0])
    for lvl in range(n_levels):
        feats_batched.append(torch.stack([item[0][lvl] for item in batch], dim=0))
    masks = torch.stack([item[1] for item in batch], dim=0)
    boxes = torch.stack([item[2] for item in batch], dim=0)
    objectness = torch.stack([item[3] for item in batch], dim=0)
    return feats_batched, masks, boxes, objectness


# ---------------------------------------------------------------------
# Training loop
# ---------------------------------------------------------------------
def train_one_epoch(model, loader, optimizer, device):
    model.train()
    running_loss = 0.0
    for feats, masks, boxes, objectness in loader:
        feats = [f.to(device) for f in feats]
        masks, boxes, objectness = masks.to(device), boxes.to(device), objectness.to(device)

        optimizer.zero_grad()
        outputs = model(feats, class_idx=0)
        targets = {"mask": masks, "box": boxes, "objectness": objectness}
        loss, _ = sam2rad_loss(outputs, targets)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * masks.size(0)
    return running_loss / len(loader.dataset)


# ---------------------------------------------------------------------
# Evaluation function, reports Dice on the predicted mask prompt
# ---------------------------------------------------------------------
def evaluate(model, loader, device):
    model.eval()
    dice_scores = []
    with torch.no_grad():
        for feats, masks, boxes, objectness in loader:
            feats = [f.to(device) for f in feats]
            masks = masks.to(device)
            outputs = model(feats, class_idx=0)
            pred = F.interpolate(outputs["mask_prompt"], size=masks.shape[-2:], mode="bilinear", align_corners=False)
            pred = (torch.sigmoid(pred) > 0.5).float()
            inter = (pred * masks).sum(dim=(1, 2, 3))
            denom = pred.sum(dim=(1, 2, 3)) + masks.sum(dim=(1, 2, 3)) + 1e-7
            dice = (2 * inter / denom)
            dice_scores.append(dice.mean().item())
    return sum(dice_scores) / len(dice_scores)


# ---------------------------------------------------------------------
# Smoke test, confirms the PPN produces the right shapes end to end
# using random data standing in for frozen SAM2 features
# ---------------------------------------------------------------------
if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model = PromptPredictorNetwork(num_queries=8, embed_dim=256, num_levels=3, num_classes=1).to(device)

    train_ds = DummyFrozenFeatureDataset(n_samples=16)
    val_ds = DummyFrozenFeatureDataset(n_samples=8)
    train_loader = DataLoader(train_ds, batch_size=4, shuffle=True, collate_fn=collate_features)
    val_loader = DataLoader(val_ds, batch_size=4, collate_fn=collate_features)

    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.1)

    print("Running smoke test on random dummy feature maps")
    train_loss = train_one_epoch(model, train_loader, optimizer, device)
    val_dice = evaluate(model, val_loader, device)

    print(f"train loss {train_loss:.4f}")
    print(f"val dice {val_dice:.4f}")
    print("Smoke test finished, shapes are consistent through the full forward and backward pass.")

A few honest notes on this implementation. The real Sam2Rad connects to a frozen SAM2 Hiera image encoder and a frozen SAM2 mask decoder, both of which come from Meta’s official SAM2 release rather than being reimplemented here, since the paper’s own contribution is specifically the PPN and its training scheme, not the base SAM2 architecture. The dummy dataset above stands in for the feature maps a real frozen encoder would output, and the mask head’s exact output resolution and the precise cross attention implementation details around SAM’s specific two way transformer block may differ in small ways from the authors’ original code, which is publicly released and linked below for anyone who wants the exact reference implementation.

Conclusion

What Sam2Rad demonstrates most convincingly is that the gap between a foundation model’s raw capability and its usefulness on a new domain is sometimes not a modeling problem at all, it is an interface problem. SAM2 already contains features rich enough to segment ultrasound bone accurately, the paper’s own ablation table proves that with high dimensional prompts alone beating specialized architectures, the missing piece was simply a way to translate those internal features into the specific prompt format the frozen mask decoder expects. That is a narrower, more tractable problem than retraining a segmentation network from scratch, and solving it with a small trained network rather than a hand engineered heuristic turns out to work remarkably well.

The conceptual shift worth carrying forward is the finding that learned prompts beat ground truth boxes fed into the same frozen model. It would be easy to assume a perfectly accurate human drawn box represents a ceiling on what prompt based guidance can achieve, since it tells the model exactly where the object sits with no ambiguity. The results here say that assumption is wrong, at least for this task. A box only encodes location. The high dimensional prompts the PPN learns appear to encode something closer to what the object actually looks like, and that extra information matters more than positional precision alone.

On transferability, nothing about the two way cross attention design or the frozen encoder strategy is specific to bone or to ultrasound. Any medical imaging task where a general purpose foundation model shows promising but imperfect zero shot behavior, and where full fine tuning is impractical because labeled data is scarce, is a reasonable candidate for the same pattern, train a small prompt generating network, leave the foundation model’s weights untouched, and let the two systems communicate through the prompt interface the foundation model was already designed to accept. Histopathology, endoscopy, and other ultrasound applications beyond musculoskeletal imaging all seem like plausible extensions worth testing.

The honest remaining limitations are substantial enough to matter. Every dataset in this study came from a single set of institutions and imaging protocols, shoulder segmentation lags meaningfully behind hip and wrist even with the full Sam2Rad pipeline applied, and the framework has not yet been extended past two dimensional frames despite volumetric scanning being common in real musculoskeletal ultrasound practice. None of that diminishes the core architectural contribution, but it does mean the reported Dice improvements describe performance within this specific study’s data, not a general guarantee across every ultrasound machine and clinical site a future user might encounter.

Where this leaves a reader in mid 2026 is with a clean, well tested idea about how to make foundation models more useful in specialized domains without the cost and risk of full fine tuning, backed by an open source implementation anyone can inspect and extend. The gap between this research result and a tool ready for a radiology department is still real, and closing it will take the kind of multi site, multi vendor validation the authors themselves flag as future work. This paper is a solid foundation for that next stage rather than a finished clinical product.

Frequently asked questions

For more coverage in this area, see our AI for medical imaging and healthcare hub and our related piece on how the Segment Anything Model works.

Related reading