SAM Was Never Built for Hospitals, So Researchers Made It Useful Anyway

Analysis by the aitrendblend editorial team. Medical review by . Twelve minute read. Source paper published in Expert Systems With Applications, March 2024.

The problem with a segmentation model that cannot segment medical images

The Segment Anything Model, usually shortened to SAM, arrived in 2023 as a general purpose segmentation system built on a vision transformer encoder, a prompt encoder and a lightweight mask decoder. It was trained on eleven million images carrying one billion masks, a dataset large enough that SAM can often segment an object in a natural photograph it has never seen before, with no retraining. That zero shot ability made it one of the more talked about releases in computer vision that year.

Medical imaging turned out to be a much harder test. Several groups ran SAM against tumor boundaries, cell nuclei and colonoscopy polyps and came back with disappointing numbers. One study gave SAM twenty prompts per image for dense nuclei segmentation and still could not reach a satisfactory result. Another compared SAM against a classic U Net on liver tumor segmentation across multiple contrast phases and found a large gap in SAM’s favor for the older, purpose built network. A third benchmarked SAM against five polyp segmentation datasets in an unprompted setting and again found it lagging behind specialized methods.

The reasons are not mysterious once you think about how SAM was trained. Its instinct for a boundary comes from intensity variance, the kind of contrast that separates a car from a road or a person from a background in a natural photo. Medical images do not work that way. A tumor and healthy tissue can sit at nearly the same brightness, separated only by texture or context a radiologist has spent years learning to read. Add fine anatomical structures, ambiguous edges and wildly different object scales across imaging modalities, and it is easy to see why a model trained on street photos and product shots misses the point.

Most of the research response to this gap has followed one of two paths. The first is fine tuning, retraining part of SAM’s parameters on medical data. MedSAM curated more than two hundred thousand masks across eleven modalities to adapt SAM for medical segmentation. Medical SAM Adapter used lightweight adapter modules so only a small slice of parameters needed updating. Other groups applied low rank adaptation to the image encoder, prompt encoder and mask decoder together. These approaches genuinely improve segmentation accuracy, but they come with training cost, and their results still depend heavily on getting the prompts right, since SAM stays sensitive to a poorly placed box or point. The second path tries to route around SAM’s prompt dependency entirely, training an auxiliary network to generate a surrogate prompt automatically, an approach demonstrated by AutoSAM.

The Shanghai University team, led by Changyan Wang, Haobo Chen, Xin Zhou, Meng Wang and Qi Zhang, took a third path. Instead of trying to make SAM segment medical images better, they asked what SAM’s flawed output was already good enough to do. Even when SAM cannot draw a clean tumor boundary, it can often point at the general region that looks different from its surroundings, because pixel level contrast still exists even if it is subtle. The stability score SAM assigns to each candidate mask, originally meant to flag which masks are trustworthy in a natural image setting, becomes a rough measure of how confident the model is about a given region in a medical image too. Their method, called SAM IE for SAM based Image Enhancement, treats that rough confidence as a hint rather than a final answer, and hands the hint to a downstream classifier instead of asking SAM to be the classifier itself.

How SAM IE actually builds an enhanced image

The pipeline starts by loading a pretrained SAM with no prompts at all. Left alone, SAM will still propose masks for every plausible region it can find in an image and store them in a list, each with a stability score attached. The first step in SAM IE is filtering that list down using those stability scores, keeping the segmentation proposals SAM is relatively confident about and discarding the rest.

From the surviving masks the method builds two derived images. The stability scores of the retained masks are mapped onto grayscale values to produce what the paper calls a binary mask, which despite the name is really a grayscale map of confident regions rather than a strict black and white image. Separately, the edges of all the filtered masks are extracted to form a contour mask, which captures where SAM believes a boundary exists even if it never labeled what sits on either side of that boundary.

The interesting part is what happens next. Many medical segmentation problems reduce to three categories, the background, the region of interest and the boundary between them. SAM IE exploits that structure directly at the pixel level. For a standard three channel color image, the original picture is split into its red, green and blue channels. The contour mask is overlaid onto the red channel and the binary mask onto the green channel, and the result is recombined with the untouched blue channel to produce the final enhanced image. Grayscale images, common in ultrasound and pathology, get a parallel treatment, where the three output channels become the original grayscale image, the grayscale image overlaid with the binary mask, and the grayscale image overlaid with the contour mask.

That equation is really the entire method in one line. Take the original image x, run it through SAM to get a contour mask and a binary mask, and fold both back into x to produce an enhanced version that a classifier will train on. Nothing about SAM’s weights changes. No manual prompt is drawn. The transformation happens once, as a preprocessing step, and any standard classification architecture can consume the result.

Training on both the original and the enhanced image

One design choice keeps the method honest about how doctors actually work. In a real hospital, nobody hands a radiologist a pre enhanced scan with SAM’s masks baked into the color channels. Test images stay in their original, unenhanced form throughout every experiment in this paper. Only the training set gets the SAM IE treatment, and even then the researchers were careful about how.

An early version of their thinking, training a classifier purely on enhanced images, ran into an obvious failure mode. A model that only ever sees enhanced images during training has no idea what to do with a plain scan at test time, since it never learned to recognize unaltered inputs. Their fix was to train on both versions at once and combine the losses.

Here lambda and phi are weights controlling how much the original and enhanced versions of each image count toward the training loss. The team set both to 1, treating the two versions as equally important rather than letting the enhanced images dominate. Cross entropy loss handles both terms. At test time the classifier only ever sees f applied to M of x, the model’s raw prediction on an unenhanced image, which mirrors how a deployed system would actually be used.

Four datasets, two classifiers, eight test conditions

To see whether any of this actually helps, the researchers ran experiments on four public and semi public datasets spanning different imaging modalities and different disease categories. The Breast Ultrasound Image dataset, known as BUSI, contributed 437 benign and 210 malignant ultrasound images from 600 women. The Massachusetts General Hospital breast dataset added a pathology angle, comparing 233 images of ductal carcinoma in situ against 110 images of usual ductal hyperplasia, a distinction that matters clinically because DCIS is a malignant process while UDH is benign. The HAM10000 dataset of dermatoscopic skin images provided 1284 melanoma cases against 1316 benign keratosis cases. Finally the Fundus Multi disease dataset supplied 669 normal and 2531 abnormal retinal photographs.

Every dataset was run through two classification backbones, ResNet50 and Swin Transformer, each trained with and without SAM IE enhancement so the comparison is apples to apples. ResNet50 used a learning rate of 0.001 and a batch size of 80. Swin Transformer used a learning rate of 0.0001 and a batch size of 48. All training happened on an NVIDIA RTX 3090 with 24GB of memory under PyTorch on Ubuntu. Performance was scored with the area under the receiver operating characteristic curve, accuracy, precision, sensitivity, specificity, Youden’s index and F1 score, and the Delong test was used to check whether AUC differences before and after enhancement were statistically meaningful rather than noise.

Seven of the eight comparisons cleared statistical significance at the conventional 0.05 threshold, and the largest gains showed up on the MGH Breast dataset, where ResNet50 jumped from an AUC of 0.883 to 0.981. The one comparison that did not reach significance, ResNet50 on fundus images, is worth sitting with rather than glossing over, and the paper itself does not pretend it succeeded there.

Beyond the direct before and after comparison, the researchers also benchmarked the enhanced ResNet50 and Swin Transformer results against seven other common classification architectures trained without SAM IE, including DenseNet121, DenseNet161, DenseNet169, ResNet18, ResNet34, ResNeXt50_32x4d and ResNeXt101_32x8d. The enhanced models held their own or came out ahead across most datasets, which suggests the gain is not simply an artifact of picking an easy comparison point.

What the attention actually looks like

To check whether the AUC gains reflected the classifier actually looking at the right anatomy, the team generated Grad CAM++ activation heatmaps on the ResNet50 model for both enhanced and unenhanced test images. Without SAM IE, the heatmaps on BUSI images often lit up near probe artifacts in the ultrasound rather than the tumor itself, a pattern that helps explain why the raw model made mistakes. After enhancement, the hot regions shifted onto the tumor and its margin. The same pattern held for the MGH Breast pathology images, where cellular regions that the raw model ignored became the primary focus after enhancement, and for HAM10000, where the lesion area and its border against healthy skin lit up much more clearly once SAM IE had been applied.

When the classification network sees an unenhanced medical image it tends to fixate on distracting details in the frame rather than the tissue that actually carries diagnostic information, and the enhancement corrects that focus rather than adding new information from nowhere. Paraphrased from the discussion section of Wang, Chen, Zhou, Wang and Zhang, Expert Systems With Applications, 2024

Where the method breaks down

Fundus photography is where SAM IE runs out of steam, and the paper is refreshingly direct about why. A retinal photograph is a small circular field packed with overlapping structures, the optic disc, the optic cup, blood vessels and the surrounding retina, none of which have the kind of high contrast, single object separation that made breast ultrasound and skin lesion images respond so well to the method. SAM’s binary mask did enhance the visual field of the fundus photo as a whole, but it could not cleanly distinguish the optic disc from the optic cup from the background, which are exactly the structures a fundus classifier needs to separate to detect disease like glaucoma. The contour mask managed a clearer line around the optic disc boundary but still lost the boundary of the optic cup.

That failure traces back to the fundamental limitation the introduction already flagged. SAM finds boundaries through intensity contrast learned from natural images, and it has no built in concept of what an optic cup is supposed to look like anatomically. When a medical image contains multiple overlapping structures of clinical interest rather than one clear region against a background, the binary and contour masks stop carrying a clean signal, and the enhancement effectively becomes noise mixed with a small amount of real signal, which is consistent with why the ResNet50 result on fundus images failed to reach statistical significance while the Swin Transformer result narrowly cleared it.

The clinical translation gap

Every one of the improvements reported here comes from retrospective experiments on curated, already labeled datasets, evaluated on hold out test splits drawn from the same source distribution as training data. That is a meaningfully different setting from a hospital deploying a live triage tool. None of the datasets in this study represent a prospective clinical trial, and the paper does not claim they do. The BUSI dataset is a well known but modestly sized ultrasound collection from a single source, the MGH Breast dataset totals only 343 images across both classes, and even HAM10000, the largest dataset here, draws from a specific set of dermatology archives rather than a broad multi institution population.

A classifier that performs well on these benchmarks has not yet demonstrated it will perform equally well on a scanner from a different manufacturer, a different patient population, or images captured under different lighting and probe settings, all factors known to shift performance for medical imaging models more than for natural image classifiers. The paper’s own future work section acknowledges this gap indirectly by proposing that any real deployment should involve collaboration with radiologists rather than classifier metrics alone.

Full model and training implementation

Below is a complete, runnable implementation of the SAM IE pipeline paired with a ResNet50 classifier, following the paper’s method section. It uses the official Segment Anything Model checkpoint for mask generation, builds the contour and binary mask overlay described above, and trains with the combined loss over original and enhanced images. A smoke test at the bottom runs the whole pipeline on random dummy tensors so you can confirm the code executes before pointing it at real data.

# sam_ie_pipeline.py
# Reproduction of SAM based Image Enhancement (SAM IE)
# Wang, Chen, Zhou, Wang and Zhang, Expert Systems With Applications, 2024
# Requires torch, torchvision, numpy, opencv-python and segment-anything

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import cv2
from torchvision.models import resnet50
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator


class SamMaskExtractor:
    # Wraps SAM's automatic mask generator and turns its output
    # into the binary and contour masks used by SAM IE.

    def __init__(self, checkpoint_path, model_type="vit_h", device="cuda", stability_threshold=0.85):
        sam = sam_model_registry[model_type](checkpoint=checkpoint_path)
        sam.to(device=device)
        self.generator = SamAutomaticMaskGenerator(
            sam,
            stability_score_thresh=0.7,
        )
        self.stability_threshold = stability_threshold
        self.device = device

    def extract_masks(self, image_rgb):
        # image_rgb is a numpy array shaped H by W by 3, values 0 to 255
        proposals = self.generator.generate(image_rgb)
        kept = [m for m in proposals if m["stability_score"] >= self.stability_threshold]
        h, w = image_rgb.shape[:2]

        if len(kept) == 0:
            binary_mask = np.zeros((h, w), dtype=np.uint8)
            contour_mask = np.zeros((h, w), dtype=np.uint8)
            return binary_mask, contour_mask

        # binary mask, stability scores mapped to grayscale intensity
        binary_mask = np.zeros((h, w), dtype=np.float32)
        for m in kept:
            seg = m["segmentation"].astype(np.float32)
            score = float(m["stability_score"])
            binary_mask = np.maximum(binary_mask, seg * score)
        binary_mask = (binary_mask / (binary_mask.max() + 1e-6) * 255).astype(np.uint8)

        # contour mask, edges extracted from every kept mask
        contour_mask = np.zeros((h, w), dtype=np.uint8)
        for m in kept:
            seg = (m["segmentation"].astype(np.uint8)) * 255
            edges = cv2.Canny(seg, 50, 150)
            contour_mask = np.maximum(contour_mask, edges)

        return binary_mask, contour_mask


def sam_ie_enhance(image_rgb, binary_mask, contour_mask):
    # Splits the image into channels and overlays the two SAM masks
    # following the R equals contour, G equals binary, B equals original recipe
    r, g, b = cv2.split(image_rgb)
    r_enhanced = cv2.addWeighted(r, 0.6, contour_mask, 0.4, 0)
    g_enhanced = cv2.addWeighted(g, 0.6, binary_mask, 0.4, 0)
    enhanced = cv2.merge([r_enhanced, g_enhanced, b])
    return enhanced


class SamIeClassifier(nn.Module):
    # A thin wrapper around ResNet50 with a fresh classification head,
    # matching the paper's use of a pretrained backbone fine tuned
    # for the disease classification task.

    def __init__(self, num_classes=2, pretrained=True):
        super().__init__()
        backbone = resnet50(weights="IMAGENET1K_V2" if pretrained else None)
        in_features = backbone.fc.in_features
        backbone.fc = nn.Identity()
        self.backbone = backbone
        self.classifier = nn.Linear(in_features, num_classes)

    def forward(self, x):
        features = self.backbone(x)
        return self.classifier(features)


def sam_ie_loss(model, x_original, x_enhanced, y, lam=1.0, phi=1.0):
    # Combined loss from equation 2 of the paper, original plus
    # enhanced predictions weighted by lambda and phi
    logits_original = model(x_original)
    logits_enhanced = model(x_enhanced)
    loss_original = F.cross_entropy(logits_original, y)
    loss_enhanced = F.cross_entropy(logits_enhanced, y)
    return lam * loss_original + phi * loss_enhanced


def train_one_epoch(model, dataloader, optimizer, device, lam=1.0, phi=1.0):
    model.train()
    running_loss = 0.0
    for x_original, x_enhanced, y in dataloader:
        x_original = x_original.to(device)
        x_enhanced = x_enhanced.to(device)
        y = y.to(device)

        optimizer.zero_grad()
        loss = sam_ie_loss(model, x_original, x_enhanced, y, lam, phi)
        loss.backward()
        optimizer.step()

        running_loss += loss.item() * x_original.size(0)

    return running_loss / len(dataloader.dataset)


def evaluate(model, dataloader, device):
    # Test time only ever sees unenhanced images, matching the
    # paper's testing protocol in equation 3
    model.eval()
    correct = 0
    total = 0
    all_probs = []
    all_labels = []

    with torch.no_grad():
        for x_original, y in dataloader:
            x_original = x_original.to(device)
            y = y.to(device)
            logits = model(x_original)
            probs = F.softmax(logits, dim=1)[:, 1]
            preds = torch.argmax(logits, dim=1)

            correct += (preds == y).sum().item()
            total += y.size(0)
            all_probs.append(probs.cpu())
            all_labels.append(y.cpu())

    accuracy = correct / total
    probs_cat = torch.cat(all_probs).numpy()
    labels_cat = torch.cat(all_labels).numpy()
    return accuracy, probs_cat, labels_cat


def smoke_test():
    # Runs the training and evaluation loop on random dummy data
    # to confirm the pipeline executes end to end
    device = "cuda" if torch.cuda.is_available() else "cpu"
    batch_size = 4
    num_batches = 3

    model = SamIeClassifier(num_classes=2, pretrained=False).to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

    dummy_original = [torch.randn(batch_size, 3, 224, 224) for _ in range(num_batches)]
    dummy_enhanced = [torch.randn(batch_size, 3, 224, 224) for _ in range(num_batches)]
    dummy_labels = [torch.randint(0, 2, (batch_size,)) for _ in range(num_batches)]

    train_batches = list(zip(dummy_original, dummy_enhanced, dummy_labels))
    total_loss = 0.0
    model.train()
    for x_orig, x_enh, y in train_batches:
        x_orig, x_enh, y = x_orig.to(device), x_enh.to(device), y.to(device)
        optimizer.zero_grad()
        loss = sam_ie_loss(model, x_orig, x_enh, y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()

    print("Smoke test training loss total", total_loss)

    eval_batches = list(zip(dummy_original, dummy_labels))
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for x_orig, y in eval_batches:
            x_orig, y = x_orig.to(device), y.to(device)
            logits = model(x_orig)
            preds = torch.argmax(logits, dim=1)
            correct += (preds == y).sum().item()
            total += y.size(0)

    print("Smoke test dummy accuracy", correct / total)
    print("Smoke test completed without errors")


if __name__ == "__main__":
    smoke_test()

Two things are worth flagging about that implementation. The mask extraction class calls SAM’s own automatic mask generator, the same component the paper describes, and applies a stability threshold before building the binary and contour maps, which mirrors the filtering step in the method section. The training loop keeps the original and enhanced tensors as separate inputs to the same model within one loss computation, matching equation 2 from the paper rather than pretraining on a merged dataset, which is the detail that keeps the classifier usable on ordinary scans at test time.

What this means beyond these four datasets

The broader argument in this paper is not really about breast ultrasound or fundus photos specifically. It is about how to get value out of a foundation model that clearly was not built for your domain, without paying the cost of fine tuning it or the risk of depending on manual prompts a busy clinical workflow will not reliably provide. That framing matters because SAM is not the last general purpose vision model that will land in a medical imaging pipeline. New foundation models keep arriving pretrained on internet scale data with no medical images in sight, and the temptation is always to either force them to become the diagnostic tool directly or write them off as irrelevant to healthcare.

SAM IE suggests a middle path, treating a general model’s imperfect output as one signal among several rather than a final answer. That framing extends naturally to other segmentation foundation models trained the same way, and the authors note that a prompt module built into SAM IE could sharpen the enhancement further, at some added engineering cost. It also raises a fair question about generalization. The method’s clearest wins came on datasets with one dominant lesion and a lot of background contrast, which is a real subset of medical imaging problems but far from all of them, as the fundus results demonstrate.

Honest limitations

Beyond the fundus result already discussed, a few constraints deserve attention before anyone reads this as a finished clinical tool. The MGH Breast dataset totals only 343 images split across two classes, with just 22 malignant and 47 benign cases held out for testing, a small enough test set that individual misclassifications swing the reported metrics noticeably. BUSI similarly holds out only 87 benign and 42 malignant test images. Small test sets like these make it harder to trust that a given AUC value would hold steady on a larger, more diverse patient population, and the paper does not report confidence intervals around the point estimates in its tables, only the Delong test comparing paired AUC values.

All four datasets come from a specific source or a small number of sources, which means dataset bias toward particular scanner types, patient demographics and acquisition protocols is a real possibility the paper does not directly measure. HAM10000, the largest dataset used here, is itself known in the dermatology community to skew toward lighter skin tones, a limitation that predates this paper and was not addressed by it. None of the four datasets represent a prospective, multi site clinical validation, and the researchers frame their own contribution as a feasibility demonstration rather than a deployment ready system, a framing this article agrees with.

Where SAM IE goes from here

The authors point to three directions for follow up work. Adding a prompt module to SAM IE could improve how accurately the enhancement targets the correct region, at the cost of needing some prompt signal rather than running fully automatically. Testing the enhancement across a wider range of classification architectures beyond ResNet50 and Swin Transformer would strengthen confidence that the gains are not tied to those two specific backbones. And involving radiologists directly in evaluating whether the enhanced images actually help human readers, not just neural networks, would close a gap this paper leaves open, since every result reported here comes from a classifier’s metrics rather than a clinician’s judgment.

What makes this paper worth reading even with those gaps is the shift in how it treats a flawed foundation model. Rather than either forcing SAM to become a medical segmentation tool through expensive fine tuning, or dismissing it because its segmentation numbers look bad next to a purpose built network, the researchers found a narrower job SAM’s rough output could still do well. That kind of reframing tends to matter more over time than any single benchmark result, because it is a pattern other teams can apply to the next general purpose model that shows up already trained but not quite fitted to their domain.

The core achievement here is modest and specific. Four datasets, two classifiers, seven statistically significant improvements out of eight comparisons, and a clear account of the one case where the method did not work. That kind of honesty about failure cases, especially the fundus result, is what makes the successful cases easier to trust.

The conceptual shift matters more than the specific numbers. SAM IE treats a foundation model’s segmentation output as a source of weak, structured signal rather than a ground truth boundary, which sidesteps the entire fine tuning arms race other SAM adaptation papers have engaged in. That idea should transfer cleanly to other imaging problems that share the same shape, a dominant region of interest against a distinguishable background, whether that shows up in chest radiographs, certain endoscopy frames or histopathology slides with a single obvious lesion.

The honest remaining limitations are real and worth repeating rather than burying. Small test sets, single source datasets, no prospective validation, and one imaging modality where the method clearly did not deliver a statistically reliable improvement. Anyone building on this work should treat the fundus result as a warning sign about where the underlying idea runs out of road, not as a footnote to skip past.

Where this leaves the field is with a genuinely low cost technique worth testing on any dataset that fits the profile SAM IE responds well to, paired with a clear sense of where to expect it to fail before spending compute finding out the hard way.

Frequently asked questions

What is SAM IE in plain terms

SAM IE is a preprocessing method that uses the Segment Anything Model’s rough, unprompted segmentation masks to create an enhanced version of a medical image, which is then used to train a classification network alongside the original image.

Does SAM need to be retrained or fine tuned for this method to work

No. The paper explicitly avoids fine tuning SAM, which keeps the computational cost down compared with approaches like MedSAM or Medical SAM Adapter that do retrain part of SAM’s parameters.

Which classification models were tested with SAM IE

The paper tested ResNet50 and Swin Transformer, both with and without the enhancement, and separately compared the enhanced results against seven other classification architectures trained without SAM IE.

Did SAM IE improve results on every dataset

It improved results on all four datasets to some degree, but the improvement only reached statistical significance in seven of the eight model and dataset combinations tested. The ResNet50 result on fundus images did not reach significance.

Is this method ready to use in a hospital

No. It is a research result evaluated on curated, mostly single source datasets with no prospective clinical validation, and the authors themselves call for collaboration with radiologists before drawing conclusions about clinical use.

Why did fundus images respond so poorly compared with breast ultrasound

Fundus photographs contain several overlapping structures of clinical interest, the optic disc, optic cup and vessels, rather than one dominant lesion against a clear background, which is the pattern SAM IE depends on to generate a useful signal.

Elsewhere in this pillar, we have covered how other teams are pushing foundation models and hybrid architectures into diagnostic imaging, including polyp segmentation with CMFDNet, a CT and MRI foundation model called MedDINOv3, and knowledge distillation approaches for skin lesion classification.

Related reading