A Top Scoring Skin Cancer AI Missed Most Basal Cell Cases

The number the abstract leads with, and the one it does not

Skin cancer is common enough, and dangerous enough when caught late, that the stakes for this kind of research are genuinely high. Melanoma’s ten year survival rate falls below 39 percent when it is detected at an advanced stage, according to the staging data the paper cites from Balch and colleagues, compared with over 93 percent when it is caught early. That gap is the entire reason computer aided diagnosis for skin lesions is such an active research area, and it is why a paper claiming state of the art performance on a widely used benchmark deserves a careful read rather than a quick nod at the topline number.

The paper in question, from Asli Nur Omeroglu, Hussein Mohammed, Emin Argun Oral, and Serdar Aydin at Ataturk University in Turkey, published in Engineering Applications of Artificial Intelligence in 2023, proposes a multi branch deep learning framework for multi label skin lesion classification. It reports an average accuracy of 83.04 percent, which the authors say improves on the strongest prior published method on the same benchmark by more than two percentage points. That is a real, meaningful gain in a field where incremental improvements are the norm.

But average accuracy, here, is an average across eight separate classification tasks, seven visual criteria used by dermatologists plus a final diagnosis label with five possible categories. One of those five diagnosis categories is basal cell carcinoma, the most common cancer in humans worldwide, and it is also the rarest category in this particular dataset. Table 5 of the paper reports sensitivity, specificity, precision, and area under the curve for every category separately, and it is there, not in the abstract, that the basal cell carcinoma result actually shows up. The model caught 17.6 percent of true basal cell carcinoma cases in the test set. The paper’s own discussion section acknowledges this number directly, noting that the remaining 82.4 percent of the corresponding test samples were misclassified, though it treats this as a minor footnote to an otherwise strong result rather than as the headline it arguably deserves to be.

Why skin cancer diagnosis is a multi modal problem in the first place

Dermatologists rarely make a diagnosis from a single image. They look at a clinical photograph, taken with an ordinary camera, which captures the lesion’s overall shape, elevation, border, and color in context with the surrounding skin. They also look at a dermoscopic image, taken through a specialized magnifying device using polarized light or liquid immersion to cut down surface reflection, which reveals finer structures inside the lesion such as pigment networks, dots, and vascular patterns invisible to the naked eye. On top of both images, a clinician weighs the patient’s history, including family history, ethnicity, sun exposure, and the lesion’s location on the body.

One widely used formal framework for combining these visual cues is the seven point checklist, developed by Argenziano and colleagues, which scores seven specific dermoscopic features, things like an atypical pigment network or irregular streaks, and flags a lesion as likely melanoma once the weighted score crosses a threshold of three points. It is a structured, interpretable alternative to a dermatologist’s unaided visual judgment, and it is also exactly what this paper’s benchmark dataset is built around.

Most earlier machine learning work in this space used only one image type, typically dermoscopy, missing the complementary information a clinical photo or a patient’s history can add. The paper positions itself against that limitation directly, building a model with three separate processing branches, one for dermoscopy images, one for clinical images, and one that fuses features from both together with the patient’s structured metadata.

How the three branch, soft attention model is built

The backbone for both the dermoscopy and clinical image branches is a modified version of Xception, a convolutional architecture introduced by Chollet in 2017 that replaces standard convolutions with depthwise separable convolutions, a design choice that lowers computational cost while typically improving accuracy compared to earlier Inception style networks. The authors trim the original Xception’s middle flow from 24 separable convolution layers down to 21, and remove the global average pooling and two separable convolution layers from the exit flow, changes they justify by pointing out that their dataset is small enough that some of that original capacity is unnecessary and even counterproductive.

The distinguishing architectural piece is what the authors call a soft attention module, adapted from earlier work by Shaikh and colleagues on writer verification. A feature map pulled from partway through the Xception backbone is passed through a small set of three dimensional convolutions, normalized with a softmax function so that spatially important regions receive proportionally more weight, and then used to rescale the original feature map before it continues through the network. In effect, the module learns to amplify the parts of an image that matter for classification and suppress everything else, hair, veins, and other visual clutter that carries no diagnostic value.

Features from the dermoscopy and clinical branches are concatenated and passed into a third branch the authors call the hybrid meta branch, which further processes that combined feature map alongside the patient’s metadata through another soft attention module before producing its own prediction. Each of the three branches, dermoscopy, clinical, and hybrid meta, ends in its own softmax classifier, and the final prediction is obtained by averaging the class probabilities across all three, an approach the authors call average late fusion. They argue this beats simple majority voting because it preserves the confidence information in each branch’s prediction rather than collapsing it down to a single hard label before combining.

“Use of soft attention module enables better feature representation for all modalities as it focuses on relevant areas of images.” From the paper’s discussion of the soft attention ablation, Section 5.2

What the headline numbers actually show

The authors ran two full sets of experiments, one without the soft attention module and one with it, each tested across five configurations, clinical images alone, dermoscopy images alone, the hybrid metadata branch alone, a plain fusion of the two image types, and the full three branch model. Without soft attention, the full fusion model reached 79.68 percent average accuracy. Adding soft attention across all branches lifted that to 83.04 percent, a 4.2 percentage point gain the authors attribute specifically to the attention mechanism’s ability to focus each branch on diagnostically relevant image regions rather than background skin, hair, or imaging artifacts.

Comparing single modality results is informative too. Dermoscopy images alone reached 79.72 percent average accuracy, notably higher than clinical images alone at 70.57 percent, consistent with the established clinical view that dermoscopy captures more diagnostically useful detail. Combining both image types pushed accuracy higher still, and folding in the patient metadata through the hybrid branch added another meaningful gain on top of that. Each additional modality helped, which is the paper’s central claim and, on the aggregate numbers, a well supported one.

Against other published methods evaluated on the identical dataset and identical train, validation, and test split, the proposed model posted the highest accuracy on every individual seven point checklist label except the overall diagnosis category, where it landed a fraction behind the best prior result. Table 4 in the paper lines up nine different methods, ranging from an early multimodal approach by Kawahara and colleagues in 2018 through more recent entries such as FusionM4Net-SS from Tang and colleagues in 2022 and a constrained classifier chain method from Wang and colleagues in 2021, and the proposed model comes out ahead on the aggregate score across all of them.

Where the aggregate number stops telling the whole story

Table 5 is where the paper breaks its results down by individual category rather than averaging them together, and this is where the basal cell carcinoma result surfaces. The table reports four separate metrics for every label, sensitivity, specificity, precision, and area under the ROC curve, for the proposed model and for the two strongest prior methods it was compared against.

That is a genuinely strange pattern once you sit with it. The proposed model has the best area under the curve of the three for basal cell carcinoma, meaning that when you look at how well it ranks true basal cell carcinoma cases above non cases across every possible decision threshold, it is actually the strongest performer. Yet at the specific threshold the model’s late fusion averaging scheme settles on, it catches fewer true basal cell carcinoma cases than either of the two models it otherwise outperforms. High AUC with low sensitivity at the chosen threshold is a recognizable pattern in machine learning, and it usually points to a model that has learned reasonable underlying structure but is making its final calls under a decision rule poorly suited to a rare class. Here that rare class happens to be a form of skin cancer.

Scale matters here too, and it is worth being precise about it rather than letting the percentage alone drive the conclusion. The test set contains only 17 basal cell carcinoma cases out of 395 total test samples, and the full dataset contains only 43 basal cell carcinoma cases out of 1011 patients total, by far the smallest of the five diagnosis categories. A sensitivity of 17.6 percent on 17 samples corresponds to roughly three correctly identified cases out of 17, which means a single additional case caught or missed would shift that percentage by nearly six points. That volatility is real and worth keeping in mind. It does not erase the finding, since the model still performs worse than both comparison baselines on this exact same small test set under the exact same conditions, but it does mean nobody should treat 17.6 percent as a precise, stable estimate of real world performance.

The authors themselves point to class imbalance as the likely explanation, noting elsewhere in the paper that no data balancing technique was applied, in contrast to most competing studies, and that categories with more even label distributions such as dots and globules and black whitish veil produced better sensitivity, specificity, and precision than the more heavily imbalanced ones. That is a reasonable diagnosis of the underlying cause. It does not change the fact that the specific number for the specific cancer subtype that this whole research area exists to help detect is the model’s single weakest result.

The clinical translation gap

It is worth being explicit about the distance between a result like this and anything that could responsibly be used in an actual clinic, because that distance is large and the paper does not claim otherwise. This is a retrospective benchmark study on a single publicly available dataset, the Seven Point Criteria dataset compiled by Kawahara and colleagues from 1011 patients at what is, as far as the dataset’s own documentation indicates, a limited set of source institutions. There is no prospective clinical trial here, no evaluation on an independent hospital cohort collected after the model was built, and no comparison against practicing dermatologists reading the same cases, all of which would be required before a result like this could inform actual clinical decision making.

There is also no mention anywhere in the paper of regulatory clearance, and none would be expected at this stage. A benchmark accuracy figure on a public research dataset is a meaningfully different thing from a medical device that has passed through the kind of review a regulator such as the FDA or a comparable body elsewhere would require before a system like this could be marketed as a diagnostic aid. Nothing in this paper suggests the authors are claiming otherwise, and their framing throughout is appropriately about advancing a research benchmark rather than a deployable product, but that distinction is easy to lose once a headline percentage travels beyond the paper itself.

Clinical limitations, in the paper’s own numbers

Several specific constraints are worth stating plainly, using the paper’s own reported figures rather than general caution. The basal cell carcinoma test set contains 17 patients. The full basal cell carcinoma sample across the entire dataset, training, validation, and test combined, is 43 patients. Several other diagnostic categories are similarly thin, seborrheic keratosis has only 44 patients total and miscellaneous conditions only 98, against 575 for the dominant nevus category. Any result on these smaller categories, in either direction, carries wide uncertainty simply from sample size, and the paper reports point estimates without confidence intervals for the per class sensitivity and precision figures in Table 5, though it does provide a formula for confidence intervals around the overall accuracy figure elsewhere.

The authors’ own conclusion section names two further limitations directly. First, that the dataset lacks chronologically sorted clinical or dermoscopy images, meaning the model cannot learn from how a lesion changes over time, an important clinical signal dermatologists actually use. Second, and worth quoting carefully because it matters for exactly the kind of deployment scenario this technology is often pitched for, that the trained network parameters are not confirmed to transfer efficiently to datasets drawn from different ethnic origins. Skin tone and lesion presentation both vary meaningfully across populations, and a model trained and tested entirely within one dataset’s demographic composition offers no evidence about how it would perform elsewhere. The authors also flag the possibility of noisy images in the source data as an open concern.

What builds the improvement, and what does not

The paper’s own ablation work is fairly clear about where the accuracy gain actually comes from. Adding the second image modality on top of a single modality model produced a meaningful jump. Adding the patient metadata through the hybrid branch added a further, smaller gain, averaging 7.36 percent across all label types by the authors’ own accounting. Adding the soft attention module on top of that fused representation produced the largest single jump, 4.2 percentage points in overall average accuracy. All three of these contributions are demonstrated by direct comparison in the paper’s tables, and none of them is disputed by the basal cell carcinoma finding above, since that finding is about a threshold and imbalance effect layered on top of an architecture that is, by the paper’s own aggregate evidence, a genuine improvement over what came before it.

The proposed soft attention module in PyTorch

Below is a complete implementation of the paper’s soft attention module and a simplified two branch fusion classifier, following the architecture in Section 2 of the paper. It uses a small convolutional stand in for the modified Xception backbone, since the paper’s distinguishing contribution is the attention and fusion mechanism rather than the backbone itself.

import torch
import torch.nn as nn
import torch.nn.functional as F

# --- Soft attention module, following Eq. 1 through Eq. 4 of the paper ---
class SoftAttentionModule(nn.Module):
    def __init__(self, in_channels, k_maps=16):
        super().__init__()
        # Stand in for the paper's 3D convolution set, implemented as a 2D conv
        # producing K attention maps from the input feature tensor
        self.conv_3d_stand_in = nn.Conv2d(in_channels, k_maps, kernel_size=3, padding=1)
        self.gamma = nn.Parameter(torch.tensor(1.0))

    def forward(self, t):
        # t has shape (batch, channels, height, width)
        b, c, h, w = t.shape
        t_prime = self.conv_3d_stand_in(t)  # (batch, K, h, w)

        # Spatial softmax over h and w for each of the K maps, Eq. 1
        t_flat = t_prime.view(b, -1, h * w)
        t_double_prime = F.softmax(t_flat, dim=-1).view_as(t_prime)

        # Sum across K maps to form the unified attention weighting, Eq. 2
        alpha = t_double_prime.sum(dim=1, keepdim=True)  # (batch, 1, h, w)

        # Scale the original feature tensor by alpha and the learnable gamma, Eq. 3
        x_sa = self.gamma * t * alpha

        # Residual connection back to the original feature tensor, Eq. 4
        o = t + x_sa
        return o

# --- Simplified two branch fusion classifier, dermoscopy and clinical ---
class SimpleBackbone(nn.Module):
    def __init__(self, out_channels=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1), nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(32, out_channels, kernel_size=3, padding=1), nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)

class MultiModalSkinClassifier(nn.Module):
    def __init__(self, num_classes=5, feat_channels=64):
        super().__init__()
        self.derm_backbone = SimpleBackbone(feat_channels)
        self.clin_backbone = SimpleBackbone(feat_channels)
        self.derm_attn = SoftAttentionModule(feat_channels)
        self.clin_attn = SoftAttentionModule(feat_channels)
        self.hybrid_attn = SoftAttentionModule(feat_channels * 2)

        self.derm_head = nn.Linear(feat_channels, num_classes)
        self.clin_head = nn.Linear(feat_channels, num_classes)
        self.hybrid_head = nn.Linear(feat_channels * 2, num_classes)

    def forward(self, derm_img, clin_img):
        h_derm = self.derm_attn(self.derm_backbone(derm_img))
        h_clin = self.clin_attn(self.clin_backbone(clin_img))

        h_hybrid = torch.cat([h_derm, h_clin], dim=1)
        h_hybrid = self.hybrid_attn(h_hybrid)

        p_derm = F.softmax(self.derm_head(F.adaptive_avg_pool2d(h_derm, 1).flatten(1)), dim=-1)
        p_clin = F.softmax(self.clin_head(F.adaptive_avg_pool2d(h_clin, 1).flatten(1)), dim=-1)
        p_hybrid = F.softmax(self.hybrid_head(F.adaptive_avg_pool2d(h_hybrid, 1).flatten(1)), dim=-1)

        # Average late fusion, Eq. 5 of the paper
        p_final = (p_derm + p_clin + p_hybrid) / 3.0
        return p_final, p_derm, p_clin, p_hybrid

# --- Smoke test on dummy data, standing in for a batch of paired images ---
if __name__ == "__main__":
    torch.manual_seed(0)
    batch_size = 8
    num_classes = 5

    model = MultiModalSkinClassifier(num_classes=num_classes)
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
    criterion = nn.NLLLoss()

    derm_img = torch.randn(batch_size, 3, 64, 64)
    clin_img = torch.randn(batch_size, 3, 64, 64)
    labels = torch.randint(0, num_classes, (batch_size,))

    losses = []
    for step in range(50):
        p_final, _, _, _ = model(derm_img, clin_img)
        loss = criterion(torch.log(p_final + 1e-8), labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(loss.item())

    assert losses[-1] < losses[0], "loss did not decrease, smoke test failed"
    print(f"Smoke test passed. Loss went from {losses[0]:.4f} to {losses[-1]:.4f}")

A team wanting to responsibly extend this architecture for a rare class problem like basal cell carcinoma would want to look at the loss function first. Weighting the cross entropy loss inversely by class frequency, or training with a focal loss that down weights easy majority class examples, are both standard tools for exactly the kind of imbalance this paper’s own results point to, and neither is present in the model as described.

Conclusion

This paper makes a solid, well evidenced case that combining dermoscopy images, clinical images, and patient metadata through a soft attention mechanism improves skin lesion classification over single modality baselines, and its aggregate benchmark result genuinely does beat prior published work on the same dataset and the same evaluation split. That core architectural contribution stands on its own merits regardless of what follows.

The conceptual shift worth taking from this paper is not really about attention mechanisms or multi branch fusion, useful as those are. It is a reminder that a single average accuracy number, however impressive, is a summary statistic that can conceal a serious failure in exactly the subgroup a reader would most want to know about. The basal cell carcinoma result here did not require any special statistical sleuthing to find. It was sitting in the paper’s own Table 5, correctly reported by the authors, and even flagged in a sentence of the discussion section. It simply was not the number that made it into the abstract.

Whether this pattern generalizes beyond skin lesion classification is worth thinking about directly. Rare disease categories, severe class imbalance, and a late fusion scheme that averages probabilities across branches without any explicit rebalancing are not unique to this paper or to dermatology. Any multi label medical classification system trained on a real world clinical dataset is likely to face some version of this same tension between an appealing aggregate number and a weak result on its least common, and possibly most clinically urgent, category.

The honest limitations here are real. A test set of 17 basal cell carcinoma patients is genuinely small, and a single institution’s dataset, whatever its scale, cannot answer questions about performance across different populations or imaging equipment. The authors’ own stated future direction, incorporating a graph convolutional network to model label dependencies, would not by itself fix the class imbalance issue at the center of this piece, since it addresses relationships between labels rather than the frequency imbalance within any single label.

For anyone building or evaluating a medical classification system, the actionable lesson is not to distrust multi modal fusion or soft attention, both of which this paper demonstrates work well. It is to insist on seeing the per class breakdown for every clinically important category before accepting an aggregate accuracy figure as a meaningful endorsement, particularly for the categories with the fewest examples and the highest real world stakes.

Frequently asked questions

Related reading