How a Mixture of Experts Network Learns to Balance Skin Images and Patient Metadata

MHDF-Net, a mixture of experts approach to fusing skin images and patient metadata

The diagnostic problem doctors actually face

Skin cancer is common enough that the World Health Organization attributes roughly one in three cancer diagnoses worldwide to it, and melanoma is the deadliest form. Caught early, the ten year survival rate for melanoma patients can exceed 93 percent. Caught late, it falls to around 39 percent. That gap is the entire argument for computer aided diagnosis tools. There simply are not enough experienced dermatologists everywhere a suspicious mole shows up, and the earlier a lesion gets flagged, the better the odds.

Most automated systems built so far look at one thing, the image. Models such as EfficientNet-B6 style classifiers and transformer hybrids have pushed image only accuracy quite far. The trouble is that image only models hit a wall precisely in the cases that matter most, the ones where a melanoma and a harmless nevus share almost the same color palette, the same rough border, the same size. A dermatologist in that situation does not just stare harder at the photo. They ask how old the patient is, where on the body the lesion sits, whether it has grown, whether there is a family history. That context, captured as structured metadata rather than pixels, is exactly what many prior fusion models have struggled to use well.

What earlier fusion attempts got wrong

Combining images with metadata is not a new idea. MetaBlock and MetaNet, two influential earlier systems, learned to reweight image features using metadata as a kind of importance signal, but neither let the metadata actively steer where the image network looked during feature extraction. MLDF-Net later added a dynamic feature selection step so metadata could guide which image features mattered at different stages, which was a real step forward. Other lines of work, including CAFNet and CMACF, brought in attention based fusion and measured cross modal agreement through similarity matrices or mutual information, and SGMC Net used a self contrastive approach to sharpen shallow features before fusing them.

What almost all of these share, according to the authors, is a fixed fusion strategy. Weighted averaging, a static attention map, a single learned gate applied the same way to every sample. That works fine when a case is easy. It breaks down for the visually ambiguous cases and for lesions with unusual shapes, because the model has no mechanism to say this particular sample needs more from the metadata branch, or this particular region of the image needs a different scale of processing than the last one did. The authors frame this as an imbalance problem. Discriminative information is unevenly spread across modalities and across spatial scales, and a fixed fusion rule cannot track that unevenness from one patient to the next.

Inside MHDF-Net, the architecture

MHDF-Net answers that imbalance problem by borrowing the mixture of experts idea from large language model training, where a gating network learns to activate only the most relevant sub networks for a given input, and adapting it to run at two different levels of the pipeline. A VGG-16 backbone extracts image features at four scales as the image shrinks and its channel depth grows. At every one of those four scales, a local fusion module runs before the next scale is computed, and after all four local modules have run, their outputs are handed to a second, global fusion module that decides how much each scale should count toward the final diagnosis.

Local Cross Modal Mixture of Experts and Top-k attention

The first component, LC-MoE, treats the image branch and the metadata branch as two experts that talk to each other through a mechanism the authors call TKCA, or Top-k cross attention. In the image branch, the projected image vector acts as the query while the concatenation of the image and metadata vectors serves as the key and value. The metadata branch mirrors that setup in reverse. This is a fairly standard cross attention setup, except for one deliberate constraint. Instead of letting every position attend to every other position with some nonzero weight, the Top-k operator keeps only the k highest scoring connections in each row of the attention matrix and zeroes out the rest before the softmax normalization runs. The idea, borrowed from a sparse attention technique originally built for image deraining, is that dense attention lets small, mostly irrelevant similarities dilute the signal, while sparse attention forces the network to commit to the handful of connections that actually matter.

Once the image expert and the metadata expert have each produced an enhanced feature vector, a small gating network called LC-Gate decides how much weight each one deserves for this particular sample. It multiplies the concatenated features by a learnable weight matrix, keeps only the top scoring paths, and normalizes the result with a softmax, which is the same keep the strongest, drop the rest philosophy applied one level higher.

The gated output then feeds back into the image branch through an elementwise multiplication with the original image features, so the enhanced features actually change what the next VGG-16 layer sees, rather than sitting off to the side as an auxiliary signal that gets bolted on only at the very end.

Global Multi Scale Mixture of Experts

Once all four local modules have produced their scale specific fusion features, GM-MoE takes over. It concatenates the four outputs, runs each through a small convolutional expert, and uses a second gating network, GM-Gate, to weight the four expert outputs before summing them into a single global representation that gets passed to the classifier.

The reasoning here is about lesion morphology rather than modality balance. A lesion with a sharp, regular border and a small footprint might carry most of its diagnostic value in fine surface texture, which shows up at the shallower, higher resolution scales. A lesion with a ragged border and irregular size might need the coarser, more global scales to capture its overall shape. A fixed weighted average across scales cannot tell those two cases apart. A learned gate, in principle, can.

What the results actually show

The authors tested MHDF-Net against five other fusion strategies, a plain concatenation baseline, MetaNet, MetaBlock, MLDF-Net and SGMC Net, plugging each one into three different backbones, VGG-16, ResNet50 and MobileNet-v2, on two public datasets.

PAD-UFES-20 contains 2298 smartphone photographed clinical images along with 21 metadata fields, covering six diagnostic categories including actinic keratosis, basal cell carcinoma, melanoma, nevus, squamous cell carcinoma and seborrheic keratosis. After excluding 358 samples that were missing metadata, the working dataset held 1940 cases, split into 1557 for training and 383 for testing. Derm7pt is smaller, 1011 cases with a clinical image, a dermoscopic image and 14 metadata fields each, though this study used only the clinical image and metadata, split roughly 60 percent training and 40 percent testing across five diagnostic categories.

On PAD-UFES-20 with the VGG-16 backbone, MHDF-Net beat the concatenation baseline by 9.3 percentage points on accuracy and 10.0 points on balanced accuracy, a metric that matters a lot here because several disease categories, melanoma in particular with only 43 total cases, are heavily underrepresented. On Derm7pt, the improvement over the strongest rival, MetaBlock, was 5.3 points on accuracy and 6.1 points on balanced accuracy. The gains held up, though somewhat more modestly, when the backbone was swapped to ResNet50 or MobileNet-v2, which is a reasonable signal that the fusion strategy itself is doing the work rather than some VGG specific quirk.

The authors also ran statistical tests rather than just reporting point estimates, which is worth flagging because plenty of papers in this space do not. A Kruskal Wallis test across the six PAD-UFES-20 models returned a p value of 0.019, and follow up Mann Whitney U tests found MHDF-Net significantly ahead of the concatenation baseline, MetaBlock, MLDF-Net and SGMC Net, all at p values below 0.01. The one exception was MetaNet, where the difference did not reach significance, at p equals 0.053. That is a useful bit of honesty baked into the paper itself, and this article is repeating it rather than smoothing it over.

Confusion matrix analysis on PAD-UFES-20 showed MHDF-Net winning outright on three of six disease categories, with squamous cell carcinoma classification improving by 5 percentage points over the next best model. On Derm7pt, MHDF-Net led on four of five categories, including a clean win on melanoma itself, the category where a missed case carries the highest cost.

Clinical translation gap

A benchmark accuracy of roughly 85 percent on a curated, metadata complete dataset is not the same thing as a tool ready for a clinic waiting room. PAD-UFES-20’s own authors built it from smartphone photos specifically to approximate real world image quality, which is a point in this study’s favor, but the dataset still required excluding 358 cases for missing metadata, and real clinical intake is messier than that. Melanoma made up only 43 of the 1940 usable PAD-UFES-20 cases, roughly 2 percent, so even a well validated balanced accuracy number rests on a small number of positive examples for the disease category where errors matter most. None of the reported numbers here have been through a prospective clinical trial, a regulatory review, or testing against a patient population outside the two datasets used. That gap between a strong benchmark result and a deployable diagnostic aid is the single most important thing to keep in mind before this kind of research gets described as clinic ready.

Regulatory and safety notes

Nothing in this paper describes a system cleared or approved by any medical regulator, and the authors do not claim otherwise. A computer aided diagnosis tool intended for actual clinical use in most jurisdictions would need to go through a formal regulatory pathway, generally involving a defined intended use, evidence from a representative patient population, and ongoing post market monitoring. Research accuracy figures like the ones reported here are an early input into that process, not a substitute for it. Readers evaluating this or any similar dermatology AI paper for potential clinical adoption should look for that regulatory evidence separately, not infer it from a benchmark table.

A model that treats every patient’s case with the same fixed formula for combining a photo and a chart is, in a real sense, refusing to individualize its own reasoning. Editorial framing, aitrendblend, on the core argument of the MHDF-Net paper

Where this fits in the bigger picture

Mixture of experts architectures have mostly made their name in large language models, where systems such as Switch Transformers scaled to trillion parameter counts by activating only a sparse subset of experts per token, and Vision MoE brought the same conditional computation idea to image transformers. What this paper does is take that same divide and conquer principle, built for scaling capacity efficiently, and repurpose it for a much smaller, much more targeted problem, deciding case by case how two very different data types should be weighted against each other. That repurposing is the paper’s real contribution. It is less about beating a benchmark by a few points and more about showing that sparse gating is a genuinely useful tool for multimodal medical fusion generally, not just for scaling giant models.

The ablation study, what each piece actually contributes

Ablation tables are where a lot of architecture papers get honest about which of their proposed components are actually doing work, and this one is no exception. Starting from the full model at 84.95 percent accuracy and 0.961 AUC on PAD-UFES-20 with the VGG-16 backbone, the authors removed one component at a time.

Two things stand out here. First, metadata alone accounts for the largest single chunk of the improvement, which is a healthy reminder that fancy fusion mechanics matter less than simply having the right second data source in the first place. Second, LC-MoE and GM-MoE contribute roughly similar sized gains to each other, around 4 to 5 accuracy points apiece, suggesting the two stage design is genuinely additive rather than one module quietly doing all the work while the other rides along.

Honest limitations

The paper is unusually direct about its own weak points, and it is worth reporting those plainly rather than glossing over them. Balanced accuracy carried a noticeably higher standard deviation across the five random seed runs than the other metrics, which the authors attribute to class imbalance and the small number of minority class samples, melanoma at 43 cases in PAD-UFES-20 being the clearest example. A metric that swings more under different random seeds is a metric you should trust a little less as a precise number and more as a rough range.

On computational cost, the authors report that MHDF-Net actually trains faster and uses fewer parameters than every comparison method, 130.45 seconds and 24.365 million parameters against, for example, 146.66 seconds and 144.519 million parameters for SGMC Net. But it also carries higher FLOPs and peak memory usage than several rivals, including MLDF-Net, which the authors flag as a real barrier to running the model on resource constrained edge devices such as a phone in a rural clinic. That is a meaningful, practical limitation for a tool whose whole appeal is supposed to be expanding access where specialist dermatologists are scarce.

The deepest limitation, and the authors say this themselves in their closing discussion, is that the study deliberately excludes dermoscopic images, the specialized magnified images taken with a dermatoscope that real dermatologists rely on heavily alongside ordinary clinical photos. The Derm7pt dataset actually contains dermoscopic images for every case, and the authors chose not to use them here, reasoning that a model working from standard clinical photos alone better matches primary care and mobile screening scenarios where a dermatoscope may not be available. That is a defensible research choice, but it also means none of the reported numbers reflect how the model would perform if dermoscopic detail were added back in, which the authors list as their own next step.

Finally, both datasets come from specific geographic and clinical settings, PAD-UFES-20 from Brazil and Derm7pt compiled for a seven point checklist study, and neither paper claims the accuracy figures would transfer unchanged to a different population with different skin tones, imaging equipment, or disease prevalence. Generalization across populations is an open question this study does not attempt to answer.

A working PyTorch implementation

The block below is an original, independently written implementation that follows the architecture described in the paper, including the TKCA sparse cross attention, the LC-Gate and GM-Gate gating networks, and the four scale local to global fusion pipeline. It is meant as a learning reference and a starting point for experimentation, not a copy of the authors’ own repository. The official code is linked at the end of this article.

Conclusion

MHDF-Net’s core achievement is fairly narrow and fairly clear at once. It shows that a two level sparse gating architecture, one gate operating within each image scale to balance image and metadata, a second gate operating across scales to balance fine and coarse spatial information, produces measurably better skin disease classification than fixed fusion rules across two independent public datasets and three different backbone networks. The 9 to 10 percentage point gains over simple concatenation, and the roughly 5 to 6 point gains over the strongest prior fusion method the authors tested, held up under formal statistical testing rather than resting on a single lucky run.

The conceptual shift worth remembering is smaller than the accuracy numbers suggest. Mixture of experts architectures were built to let enormous language models scale efficiently by activating only a fraction of their parameters per input. This paper’s contribution is recognizing that the same sparse, input dependent routing is just as useful in a completely different setting, a comparatively small medical fusion model that needs to decide, case by case, which of two very different information sources to trust more. That is a transferable insight, not a dermatology specific trick, and it is reasonable to expect similar gating ideas to show up in other clinical fusion problems, radiology paired with lab values, cardiology imaging paired with electronic health records, wherever a fixed fusion rule currently papers over case by case variation.

Transferability aside, the honest remaining gaps are real and the authors do not hide them. No dermoscopic imaging was included, despite dermoscopic images being available in one of the two datasets used and being a standard tool in actual dermatology practice. FLOPs and peak memory sit above several lighter competing methods, which the authors themselves flag as a barrier to edge deployment on the kind of low cost mobile hardware that would matter most for expanding access in underserved regions. Balanced accuracy showed meaningfully more run to run variance than other metrics, tied to the small number of melanoma cases available for training and testing.

None of that undercuts the paper’s central claim, that adaptive, sample specific gating beats fixed fusion rules for this task. It does mean the distance between this result and a tool a primary care clinic could actually deploy tomorrow is still measured in additional validation studies, additional modalities, and additional engineering for efficiency, not in a single next paper.

For readers building their own multimodal fusion systems, medical or otherwise, the practical lesson is to test whether a fixed fusion rule is quietly costing accuracy on the hardest subset of your data before reaching for a heavier architecture. The ablation numbers here suggest that simply having the second modality in the first place accounts for most of the gain, and that the sparse gating machinery, while genuinely useful, is refining an already strong foundation rather than rescuing a weak one.

Frequently asked questions