How EFAM-Net Reads Skin Lesions With ConvNeXt Attention Blocks

Analysis by the aitrendblend editorial team, filed under AI for Medical Imaging and Healthcare

about a seventeen minute read

The specific problem this paper is trying to solve

Skin cancer is common enough that in the United States alone it affects roughly three million people every year, and melanoma, one of its three primary forms alongside basal cell carcinoma and squamous cell carcinoma, is particularly dangerous because it arises from mutated melanocytes and can spread quickly if caught late. Catching it early still matters more than almost any other factor in patient survival, which is exactly why so much effort has gone into automating the visual read of a lesion image rather than relying purely on a clinician’s trained eye and the time that requires.

Dermoscopy helps by magnifying a lesion up to ten times under non reflective, non invasive imaging, giving both human experts and algorithms a cleaner view than a plain photograph. But the authors point to three specific reasons this remains a hard computer vision problem even with a good image in hand. Normal skin and lesion tissue often differ by only a small margin in a dermoscopic image, and artifacts like hair or ruler marks can sit right on top of the region that matters. Several distinct lesion classes look remarkably similar in color, texture, and shape to each other. And some individual classes show enormous variation from one patient’s lesion to the next, so a model cannot just memorize one canonical appearance per class and expect it to generalize.

What came before EFAM-Net

Convolutional networks have already done meaningful work here. Gilani et al. built a spiking neural network based on the older VGG-13 architecture that swaps traditional neurons for impulse based ones, reaching 89.57 percent accuracy and a 90.07 percent F1 score on the ISIC 2019 dataset. Villa-Pulgarin et al. ran four separate experiments comparing DenseNet-201, Inception-ResNet-V2, and Inception-V3 on skin lesion data, largely to demonstrate that transfer learning genuinely helps in this domain rather than to push a new architecture. Zeng et al. took a different angle entirely, building a dual stage knowledge distillation framework they call DSP-KD that transfers information between a larger teacher network and a smaller student network, landing strong accuracy with a noticeably lighter final model, though the authors of the current paper note this line of work tends to sacrifice interpretability, since a distilled student network is even harder to inspect than the teacher it learned from.

A second research thread focuses on combining features from multiple networks rather than deepening a single one. Maurya et al. extracted features from several CNNs and then used an XGBoost based selection step to prune the useless ones before fusing what remained, reporting a real accuracy gain from that selection and fusion combination. Mahbod et al. fused EfficientNet B0, EfficientNet B1, and SeResNeXt-50 across six different image crop scales, a strategy that works but multiplies the number of backbones you need to train and run at inference time. Maqsood and Damasevicius took yet another route, pairing modified Xception, ResNet-50, ResNet-101, and VGG16 with a sparse decomposition algorithm before handing everything to a support vector machine classifier. The EFAM-Net authors argue that when you extract features from several CNNs of similar depth, those features end up close to each other anyway, which inflates the parameter count without buying much real diversity.

A third thread leans on attention. Lan et al. combined a capsule network with the Convolutional Block Attention Module to address spatial information loss, and reported strong results for early skin cancer detection. Karthik et al. swapped the Squeeze and Excitation block inside EfficientNetV2 for the lighter Efficient Channel Attention block, reaching 84.70 percent accuracy across four skin lesion classes including acne and psoriasis. Liu et al. combined a linear bottleneck inverted residual block with local cross channel attention. The tradeoff the EFAM-Net authors flag here is a fairly intuitive one, more attention machinery usually means more parameters to learn, and more parameters on a small training set is a fast route to overfitting rather than better generalization.

How EFAM-Net is actually built

The backbone is ConvNeXt, the 2022 architecture from Liu et al. that reintroduced several transformer style design choices into a purely convolutional network. EFAM-Net keeps that four stage structure, with stage 2 holding nine blocks and the other three stages holding three blocks each, but it swaps out two of those stages’ original blocks for new designs, and it adds an entirely new fusion layer between the backbone and the final classifier.

The ARLC block handles the shallow, low level features

In stage 0, the original ConvNeXt block is replaced with a newly designed Attention Residual Learning ConvNeXt block, shortened to ARLC and inspired by earlier attention residual work from Zhang et al. on skin lesion classification. The idea is to give the shallow part of the network, the part responsible for basic colors, edges, and textures, a built in attention mechanism without paying the usual price in extra learnable parameters or instability. ARLC splits its input feature map into three branches. The first is a plain identity mapping. The second runs a depthwise separable convolution, a layer normalization, another convolution, a GELU activation, and one more convolution. The third branch takes that second branch’s output, applies a layer normalization across channels, and then runs a softmax function to generate an attention mask, which gets multiplied element wise against the original input and scaled by a single learnable weight the authors call alpha.

That learnable alpha weight is doing quieter work than it might look like at first glance. Its job is to stop noise from an early, still rough feature map from leaking into the deeper representation before the network has learned to trust it, which is exactly the kind of gradient explosion and overfitting risk the paper’s related work section flags as the usual cost of bolting attention onto a small dataset. The three branches are then summed with an element wise addition to produce the block’s output.

The PCNXt block handles the deep, semantic features

Stage 3, the deepest stage in the backbone, gets a different replacement called the Parallel ConvNeXt block, or PCNXt. Depth in a network is what lets it capture abstract, high level structure, but adding more depth or more parameters is expensive and prone to diminishing returns. PCNXt tries to get more out of the same depth by running a parallel branch alongside the usual residual path, one that applies a global average pooling operation followed by a layer normalization. Pooling at this stage compresses spatial detail into a more general, semantic summary, which the authors say helps the model recognize background tissue and locate where a lesion actually sits rather than just what it locally looks like.

Worth noting, the authors specifically tried replacing the ConvNeXt blocks in the other stages with PCNXt as well and found it only added model complexity without a meaningful accuracy gain, which is why PCNXt appears only in stage 3 in the final design. That is a small but honest detail, since it would have been easy to claim the block helps everywhere and just not show the negative result.

The MEAFF block ties the scales together

The Multi-scale Efficient Attention Feature Fusion block, MEAFF, sits after the backbone and pulls in feature maps from stages 1 through 3. Shallow features carry positional information, roughly where things are in the image, while deep features carry semantic information, roughly what those things are. MEAFF’s job is to combine both without letting one wash out the other. It first runs each stage’s feature map through an Efficient Channel Attention style weighting step, generating per channel importance scores through a cross channel convolution sized according to the number of channels in that particular feature map.

Because the three stages produce feature maps with different channel counts due to the network’s down sampling operations, MEAFF runs a global average pooling step on each weighted feature map to bring them to a common size, then merges the three results with a simple element wise addition, producing one feature map that the authors describe as carrying strong semantic content while still preserving detail from the earlier, more localized feature maps.

Turning features into a prediction

The fused feature vector passes through a linear layer to produce logits, and a standard softmax function converts those logits into a probability distribution across the lesion classes.

Training uses ordinary multi class cross entropy loss between the predicted probabilities and the true one hot labels, optimized with the AdamW variant of Adam that adds decoupled weight decay.

The three datasets, and how thin some of the classes actually are

EFAM-Net was tested on ISIC 2019, HAM10000, and a private dataset the authors collected through Peking Union Medical College Hospital. ISIC 2019 is the largest of the three, holding 25331 labeled images across eight classes, but it is also badly imbalanced, with 12875 melanocytic nevus images against only 239 dermatofibroma images and 253 vascular lesion images. HAM10000 holds 10015 images across seven classes and, according to the authors, shows large variation within a class alongside relatively small differences between classes, which is its own particular difficulty. The private dataset is the smallest by far at 2900 images across six classes specific to this cohort, acne, discoid lupus erythematosus, melasma, Ota nevus, rosacea, and seborrheic dermatitis, and the authors note it was reviewed by multiple experienced dermatologists to catch labeling errors before use.

All three were split 80 percent training and 20 percent testing. That split ratio matters more than it might seem for the rarest classes. Dermatofibroma in ISIC 2019 ends up with only 47 images in the entire test set. Vascular lesions get 50. In the private dataset, discoid lupus erythematosus has just 25 test images and seborrheic dermatitis has 34. A handful of misclassified images in a test set that small can swing a class level accuracy figure by several percentage points, which is a detail worth holding onto once you get to the results tables below.

What the reported numbers actually show

Against a standard ConvNeXt baseline and three other well known backbones, ResNet-101, DenseNet-201, and EfficientNet B0, EFAM-Net came out ahead on overall accuracy across all three datasets, at 92.30 percent on ISIC 2019, 93.95 percent on HAM10000, and 94.31 percent on the private dataset. The one exception across the whole evaluation is average precision on HAM10000, where ResNet-101 edges ahead at 94.99 percent against EFAM-Net’s 90.23 percent, a gap the paper does not dwell on but is worth flagging rather than glossing over.

What the ablation study reveals about which block matters most

The paper’s ablation study on ISIC 2019 is genuinely informative here. Adding ARLC alone to the ConvNeXt baseline nudged overall accuracy from 90.91 to 91.11 percent. PCNXt alone brought it to 91.18. MEAFF alone, on its own, brought the biggest single block gain to 91.88 percent and also produced the highest average recall of any configuration tested at 89.80 percent, even edging out the full three block model on that particular metric. When the authors paired blocks, PCNXt combined with MEAFF won on three of five metrics, and ARLC combined with MEAFF came in close behind, which the authors read as evidence that pairing an attention block with the fusion block captures feature correlations more effectively than any single block alone. Only when all three blocks were combined did the model hit its peak overall accuracy of 92.30 percent, average precision of 88.27 percent, and average F1 of 88.52 percent, though notably the full combination’s average recall of 88.86 percent actually trails the MEAFF only configuration.

How EFAM-Net compares against other published models

Against the wider published literature on ISIC 2019, EFAM-Net’s 92.30 percent overall accuracy beats every model in the comparison table by at least 0.37 percentage points, with the closest competitor being a hybrid model at 91.93 percent. The one place it loses ground is average precision, where a Cubic SVM based approach reaches 92.04 percent against EFAM-Net’s 88.27 percent, putting EFAM-Net in second place on that single metric. On HAM10000, EFAM-Net was reported as the outright best performer against every comparison model and every metric the authors tracked, including RegNetY, an enhanced ShuffleNet variant, and a fused CNN ensemble.

The parameter count is arguably the most interesting number in the whole comparison table. EFAM-Net carries 28.8 million parameters against the ConvNeXt baseline’s 27.8 million, an increase of roughly 3.6 percent for three entirely new block designs. That is a genuinely modest cost, and it lines up with the authors’ stated goal of adding attention and fusion capacity without paying the usual overfitting tax that showed up repeatedly in the related work they cite.

What the Grad-CAM visualizations add, and what they do not prove

The authors also ran Gradient Weighted Class Activation Mapping, a well established explainability technique, comparing where EFAM-Net focuses its attention against where the plain ConvNeXt baseline focuses. On sample images from actinic keratosis and vascular lesion classes, the baseline model’s heatmap lands partly on background noise rather than the lesion itself, while EFAM-Net’s heatmap concentrates more tightly on the actual lesion boundary. On melanoma and melanocytic nevus samples, the baseline sometimes attends to normal surrounding skin, where EFAM-Net again shows tighter focus on the lesion region.

That is a useful qualitative signal, and it is consistent with the block designs doing what they were meant to do. But it comes from one randomly selected sample per class out of eight classes, not a systematic, quantified interpretability metric across the full test set. A single illustrative image per class is a reasonable way to show a reader what attention looks like, but it is not evidence about how often the model’s attention is well placed across the thousands of images it was actually tested on.

Clinical translation gap

There is a real distance between what this paper demonstrates and what a dermatology clinic would need before trusting a tool like this in an actual care pathway. ISIC 2019 and HAM10000 are both curated, dermoscopic image collections assembled specifically for research challenges, captured with dedicated dermoscopy equipment under fairly consistent conditions. A phone camera photo taken by a patient at home, or even a standard clinical photograph without dermoscopic magnification, looks meaningfully different in lighting, focus, and scale, and the paper does not test on that kind of image at all.

The private dataset does at least represent something closer to routine clinical photography, since it came directly from Peking Union Medical College Hospital and was reviewed by practicing dermatologists rather than pulled from a public research archive. That is a genuine strength relative to relying on public benchmarks alone. But it still comes from a single hospital in one country, which means the model has not been shown to generalize across different patient populations, different skin tones, different cameras, or different clinical settings, any of which can shift how a lesion appears in an image. The paper also reports a single train and test split for each dataset rather than repeated cross validation runs with confidence intervals, so the exact accuracy figures carry more uncertainty than a bare percentage suggests, particularly for the classes with only a few dozen test images.

None of this means the underlying architecture is not a genuine contribution. It means the distance between a strong classification benchmark result and a validated clinical decision support tool is still substantial, and that distance is exactly where regulatory review, prospective clinical studies, and multi site validation normally live before something like this reaches an actual patient encounter.

Clinical and technical limitations worth sitting with

A few specific weaknesses deserve to be named plainly rather than buried in a caveat at the end. Several classes across all three datasets rest on genuinely small test sets, 47 images for dermatofibroma in ISIC 2019, 23 for the same class in HAM10000, and 25 for discoid lupus erythematosus in the private dataset, which means the reported per class accuracy for those specific categories carries a wide, unstated margin of error. ISIC 2019’s severe class imbalance, with over 12000 melanocytic nevus images against under 300 dermatofibroma images, means the overall accuracy figure is disproportionately influenced by how well the model handles the common classes rather than the rare, often more clinically concerning ones.

The private dataset, while a welcome addition over relying on public benchmarks alone, comes from a single hospital and was not validated against an external, independently collected cohort. The paper reports one train and test split per dataset rather than repeated runs with confidence intervals, so small differences between models in the comparison tables, some as narrow as a few tenths of a percentage point, should be read cautiously rather than treated as settled. The ethical statement notes that formal ethical review was waived because the data involved only anonymous images with informed consent, which is a reasonable basis for a retrospective imaging study but is a different bar than the prospective clinical evaluation a real diagnostic aid would eventually need.

Why this still matters despite the caveats

Strip away the clinical translation question for a moment and look at this purely as an architecture design exercise, and there is a genuinely useful lesson here for anyone building attention heavy vision models on modest sized medical datasets. The consistent theme across the paper’s own related work section is that attention and multi network fusion tend to buy accuracy at the cost of parameters, and more parameters on a few thousand training images is a well known route to overfitting rather than improvement. EFAM-Net’s answer, add attention and fusion through lightweight, carefully scoped blocks rather than through brute force scale, and its own ablation numbers back up that the approach worked reasonably well, at under a four percent parameter increase over the baseline it started from.

Full conclusion

The core achievement here is a set of three compact blocks, ARLC for shallow attention, PCNXt for deep parallel features, and MEAFF for multi scale fusion, that together push a ConvNeXt backbone to meaningfully higher accuracy across three separate datasets while adding under four percent to the parameter count. That is a legitimate, well documented engineering result, and the paper’s own ablation table gives an honest, granular account of which piece contributes what, rather than presenting the full model as an undifferentiated black box improvement.

The conceptual shift worth remembering is narrower than the headline accuracy numbers suggest. This is not a claim that attention mechanisms are now solved for small medical datasets in general. It is a demonstration that a specific set of lightweight, carefully placed attention and fusion operations can capture more of what the related work in this exact subfield has struggled with, minor visual differences between lesion types, close resemblance between distinct classes, and high variation within a single class, without triggering the overfitting problems the authors observed in prior attention heavy approaches.

Whether this transfers to other domains is a fair question, and there is reason for cautious optimism. The same three failure modes, subtle intra class variation, close inter class resemblance, and thin data for rare categories, show up constantly in other medical imaging tasks, from diabetic retinopathy grading to histopathology slide classification. A block design this parameter efficient is worth testing in those settings, though nothing in this paper itself tests that transfer, so it remains a reasonable hypothesis rather than a demonstrated result.

The honest remaining limitations are the ones already laid out above and should not be softened. Several clinically important classes rest on test sets small enough that individual misclassified images meaningfully move the reported accuracy. The private dataset, while a real strength relative to public benchmarks alone, comes from one hospital and has not been externally validated. A single train and test split per dataset, without repeated cross validation, means small gaps between competing models in the comparison tables deserve more skepticism than a bare percentage conveys.

Put simply, EFAM-Net is a solid, carefully ablated piece of classification architecture work, and it is not, and does not claim to be, a validated diagnostic tool. Reading it as the former, a genuinely useful contribution to how attention and fusion can be added cheaply to a strong backbone, and not the latter, is the accurate way to hold both the paper’s real strengths and its real limits at the same time.

Reference implementation in PyTorch

The code below is an independent, simplified implementation of the ARLC, PCNXt, and MEAFF blocks described in Section III of the paper, wired onto a small convolutional stand in backbone so the whole pipeline can run end to end as a smoke test on dummy data. It is not the authors’ original code, which the paper does not publish, and it uses a lightweight backbone rather than a full ConvNeXt implementation to keep the example runnable and focused on the three block designs that are the actual contribution of this paper.

# efam_net_reference.py
# Independent reference implementation of the ARLC, PCNXt, and MEAFF
# blocks from Section III of the EFAM-Net paper, wired onto a compact
# convolutional backbone in place of a full ConvNeXt implementation.
# Every block below mirrors the paper's Equations 1 through 8.

import torch
import torch.nn as nn
import torch.nn.functional as F


class DepthwiseSeparable(nn.Module):
    """A depthwise convolution followed by layer norm and a pointwise
    convolution, the DWSC building block referenced throughout the paper."""

    def __init__(self, channels):
        super().__init__()
        self.depthwise = nn.Conv2d(channels, channels, kernel_size=7,
                                    padding=3, groups=channels)
        self.norm = nn.GroupNorm(1, channels)  # channel wise layer norm stand in
        self.pointwise = nn.Conv2d(channels, channels, kernel_size=1)

    def forward(self, x):
        x = self.depthwise(x)
        x = self.norm(x)
        x = self.pointwise(x)
        return x


class ARLCBlock(nn.Module):
    """Attention Residual Learning ConvNeXt block, Equation 1 and 2.
    Three branches, identity, a depthwise separable path with GELU,
    and an attention path gated by a learnable scalar alpha."""

    def __init__(self, channels):
        super().__init__()
        self.branch2_dwsc = DepthwiseSeparable(channels)
        self.branch2_conv_in = nn.Conv2d(channels, channels, kernel_size=1)
        self.branch2_conv_out = nn.Conv2d(channels, channels, kernel_size=1)
        self.branch3_norm = nn.GroupNorm(1, channels)
        self.alpha = nn.Parameter(torch.tensor(0.1))

    def forward(self, x):
        branch1 = x  # identity mapping, l = 1

        # l = 2, depthwise separable convolution then two 1x1 convs with GELU
        b2 = self.branch2_dwsc(x)
        b2 = self.branch2_conv_in(b2)
        b2 = F.gelu(b2)
        b2 = self.branch2_conv_out(b2)
        branch2 = b2

        # l = 3, layer norm on branch 2, softmax attention mask, gated by alpha
        b3_norm = self.branch3_norm(branch2)
        b, c, h, w = b3_norm.shape
        attn = F.softmax(b3_norm.view(b, c, -1), dim=-1).view(b, c, h, w)
        branch3 = x * attn * self.alpha

        return branch1 + branch2 + branch3


class PCNXtBlock(nn.Module):
    """Parallel ConvNeXt block, Equation 3. A residual depthwise
    separable path plus a global average pooling path, summed with
    the identity map."""

    def __init__(self, channels):
        super().__init__()
        self.dwsc = DepthwiseSeparable(channels)
        self.conv_in = nn.Conv2d(channels, channels, kernel_size=1)
        self.conv_out = nn.Conv2d(channels, channels, kernel_size=1)
        self.global_norm = nn.GroupNorm(1, channels)

    def forward(self, x):
        residual = self.dwsc(x)
        residual = self.conv_in(residual)
        residual = F.gelu(residual)
        residual = self.conv_out(residual)

        pooled = F.adaptive_avg_pool2d(x, 1)
        pooled = self.global_norm(pooled)
        pooled = pooled.expand_as(x)

        return x + residual + pooled


class MEAFFBlock(nn.Module):
    """Multi-scale Efficient Attention Feature Fusion block,
    Equations 4 through 6. Weights each stage's feature map with an
    ECA style channel attention, pools every stage to a common size,
    then fuses with an element wise addition."""

    def __init__(self, channel_list, eca_kernel=3):
        super().__init__()
        self.eca_convs = nn.ModuleList([
            nn.Conv1d(1, 1, kernel_size=eca_kernel,
                      padding=eca_kernel // 2, bias=False)
            for _ in channel_list
        ])

    def forward(self, feature_maps):
        # feature_maps is a list of tensors from stages 1 to 3
        pooled_outputs = []
        for x, eca in zip(feature_maps, self.eca_convs):
            gap = F.adaptive_avg_pool2d(x, 1).squeeze(-1).squeeze(-1)
            gap = gap.unsqueeze(1)  # shape batch, 1, channels for the 1d conv
            weights = torch.sigmoid(eca(gap)).squeeze(1).unsqueeze(-1).unsqueeze(-1)
            weighted = x * weights
            pooled = F.adaptive_avg_pool2d(weighted, 1).flatten(1)
            pooled_outputs.append(pooled)

        # bring every stage to the same channel size before the final addition
        target_dim = min(p.shape[1] for p in pooled_outputs)
        aligned = [p[:, :target_dim] for p in pooled_outputs]
        fused = torch.stack(aligned, dim=0).sum(dim=0)
        return fused


class EFAMNet(nn.Module):
    """A compact stand in for the full EFAM-Net pipeline. Uses a small
    convolutional backbone with four stages, ARLC in stage 0, PCNXt
    in stage 3, and MEAFF fusing the outputs of stages 1 through 3
    before the final linear classifier."""

    def __init__(self, num_classes, base_channels=32):
        super().__init__()
        c0, c1, c2, c3 = base_channels, base_channels * 2, base_channels * 4, base_channels * 8

        self.stem = nn.Conv2d(3, c0, kernel_size=4, stride=4)
        self.stage0 = ARLCBlock(c0)

        self.down1 = nn.Conv2d(c0, c1, kernel_size=2, stride=2)
        self.stage1 = nn.Sequential(DepthwiseSeparable(c1), nn.GELU())

        self.down2 = nn.Conv2d(c1, c2, kernel_size=2, stride=2)
        self.stage2 = nn.Sequential(DepthwiseSeparable(c2), nn.GELU())

        self.down3 = nn.Conv2d(c2, c3, kernel_size=2, stride=2)
        self.stage3 = PCNXtBlock(c3)

        self.meaff = MEAFFBlock(channel_list=[c1, c2, c3])
        fusion_dim = min(c1, c2, c3)
        self.classifier = nn.Linear(fusion_dim, num_classes)

    def forward(self, x):
        x = self.stem(x)
        x0 = self.stage0(x)

        x1 = self.down1(x0)
        x1 = self.stage1(x1)

        x2 = self.down2(x1)
        x2 = self.stage2(x2)

        x3 = self.down3(x2)
        x3 = self.stage3(x3)

        fused = self.meaff([x1, x2, x3])
        logits = self.classifier(fused)
        return logits


def cross_entropy_loss(logits, labels):
    """Multi class cross entropy loss, Equation 8 in the paper, using
    the built in log softmax plus negative log likelihood combination
    that PyTorch already provides for numerical stability."""
    log_probs = F.log_softmax(logits, dim=-1)
    return F.nll_loss(log_probs, labels)


def evaluate_accuracy(logits, labels):
    """Simple overall accuracy, matching Equation 9 in aggregate form,
    for a quick smoke test rather than a full metrics suite."""
    preds = torch.argmax(logits, dim=-1)
    correct = (preds == labels).float().sum()
    return (correct / labels.shape[0]).item()


def run_smoke_test():
    """Runs a short training loop on random dummy images shaped like
    the paper's 224 by 224 resized inputs, to confirm the model, loss,
    and evaluation function all execute correctly end to end."""
    torch.manual_seed(0)
    num_classes = 8  # matches the eight ISIC 2019 classes
    batch_size = 8

    model = EFAMNet(num_classes=num_classes)
    optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, weight_decay=1e-4)

    for epoch in range(3):
        images = torch.randn(batch_size, 3, 224, 224)
        labels = torch.randint(0, num_classes, (batch_size,))

        logits = model(images)
        loss = cross_entropy_loss(logits, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        acc = evaluate_accuracy(logits, labels)
        print(f"epoch {epoch} loss {loss.item():.4f} batch accuracy {acc:.4f}")

    assert logits.shape == (batch_size, num_classes), "unexpected output shape"
    print("smoke test passed")


if __name__ == "__main__":
    run_smoke_test()

Frequently asked questions

Related reading in this pillar