AFME Teaches Knowledge Graphs Which Modality To Trust

The unglamorous problem hiding inside every multimodal knowledge graph

Knowledge graph completion sounds abstract until you picture the actual data sitting in one of these graphs. An entity might have a blurry thumbnail, a two sentence description that was scraped and never edited, and a handful of relations that were extracted automatically and are only sometimes right. Add video or audio, as the TIVA and KVC16K datasets used in this paper both do, and the quality gap between modalities gets even wider. A model that treats a low resolution image the same way it treats a clean structural embedding is going to get pulled around by noise it should be ignoring.

That is the specific failure this paper by Zenglong Wang, Xuan Liu, Zheng Liu, Yu Weng, and Chaomurilige is trying to fix. Their framework, AFME, stands for Adaptive Fusion and Modality Information Enhancement, and the name is a fair description of what it actually does rather than marketing. It does not propose a new scoring function from scratch. It proposes a way to clean, weigh, and if necessary regenerate modality features before they ever reach a fairly conventional rotation based scoring function, on the argument that fusion quality, not scoring function novelty, is what has been holding this task back.

Why concatenation was never going to be enough

Most prior multimodal completion methods fall into a predictable pattern. IKRL added visual features with attention over which modality to trust. TBKGC and TransAE combined modal representations with structural embeddings using autoencoders or fixed weighting. RSME and OTKGE went further with sparse coding and optimal transport respectively, both trying to find a smarter static way to combine modalities. What none of them do, according to this paper, is let the fusion weight of a modality change based on what relationship is actually being predicted right now, or aggressively suppress the parts of a modality that have nothing to do with that relationship before fusion happens.

There is a second pattern the paper points at, this time in the GAN literature. KBGAN used adversarial training to generate harder negative samples, and MMKRL extended a similar idea to multimodal embeddings. Both treat the generator as a way to make training data more difficult, not as a way to actively repair a specific missing or noisy modality feature using the graph’s own structure as a guide. AFME’s generator does something closer to the second thing, using the structural embedding of an entity as a conditioning signal for regenerating its weaker modalities.

Two modules, one doing triage and one doing repair

AFME is built from two pieces that run in sequence. The Modal Information Fusion module, MoIFu, cleans and weighs the modalities that already exist. The Modal Information Enhancement module, MoIEn, uses a GAN to produce higher quality versions of the weaker ones. Both sit on top of a fairly standard setup, pretrained encoders per modality (BERT for text, a convolutional network for images, established encoders for video and audio), each projected into a shared embedding space, alongside trainable structural embeddings for the graph’s own entities and relations.

MoIFu, denoise first, then decide how much each modality is worth

Before any fusion happens, each modality feature passes through a gate conditioned on the relationship of the current triple. The relationship embedding interacts with the modality feature through an elementwise product, gets pushed through a sigmoid, and the result multiplies back into the original feature.

Read plainly, this means the model learns, for a given relationship, which parts of a given modality actually matter and which parts are noise or irrelevant detail, and it turns down the volume on the latter before that modality gets anywhere near the fusion step. That is a meaningfully different design choice from denoising a modality in isolation, since the same image feature could be treated as mostly signal for one relationship and mostly noise for another, depending on what the relationship actually needs from it.

Once each modality has been cleaned, AFME still has to decide how much weight it deserves relative to the others. It computes a confidence score for each modality from the L2 norm of its cleaned feature, on the reasoning that a feature vector with more energy in it is carrying more information, then folds that confidence into a softmax that is also conditioned on a relationship specific temperature term.

The relationship specific temperature is the detail worth sitting with. A fixed softmax temperature applies the same sharpness to every relationship in the graph, whether that relationship depends heavily on a photograph or barely needs one. Letting the temperature vary by relationship means the model can be decisive about modality weighting for relations where one modality clearly dominates, and more balanced for relations where several modalities genuinely contribute.

MoIEn, let the graph structure repair what is weak or missing

Cleaning and weighting only goes so far when a modality is not noisy but simply absent, which happens constantly in real multimodal graphs. This is where the GAN comes in, and the specific design choice worth noting is what conditions the generator. Instead of starting from noise alone, the way a typical GAN generator would, AFME concatenates the entity’s structural embedding with a random noise vector, on the argument that the graph’s own relational structure carries information a purely visual or textual generator has no access to.

That initial guided feature then goes through a multi layer self attention block to sharpen internal consistency within the modality, followed by an inter modal interaction step that lets each modality’s representation absorb something from every other modality, weighted by a learned attention score between them.

A discriminator then checks the generated feature against a real one, concatenating both and scoring their alignment, while training uses the Wasserstein GAN objective with a gradient penalty term borrowed from WGAN-GP rather than the original minimax loss, specifically to avoid the vanishing gradients and mode collapse that plain adversarial training is known to produce.

The full training objective sums the knowledge graph completion loss, a margin based negative sampling loss over the fused embeddings, together with the adversarial loss and the gradient penalty term, weighted by two coefficients the authors tune from a small grid.

What the benchmarks actually show

The evaluation runs across four datasets that differ a lot in shape. MKG-W and MKG-Y, drawn from Wikidata and YAGO, cover text and images only, at roughly fifteen thousand entities each. TIVA and KVC16K go further, adding video and audio, with KVC16K in particular covering four full modalities across sixteen thousand entities pulled from a short video encyclopedia. That range matters because a fusion method that only works when two clean modalities are available is not proving much about the harder, messier case.

The gain over NATIVE, the previous best performing method and itself an adversarial adaptive fusion approach, is the number that matters most here, since it is the comparison against the toughest opponent rather than against older translation based models. It comes out to about one point seven percent MRR on MKG-W, one point six percent on MKG-Y, under one percent on TIVA, and two point seven percent on KVC16K. The KVC16K result stands out because that is the dataset with all four modalities and the lowest baseline scores across the board, which is roughly where you would expect a better denoising and enhancement strategy to matter most, since there is more noisy or missing modality information for it to clean up.

A second comparison the authors highlight is more specific and, to their credit, more honest about what it does and does not prove. AFME still beats IKRL and VBKGC, two methods that use only structural, text, and image information, even when AFME is run on the same limited modality set. That rules out a mundane explanation for the headline numbers, namely that AFME is simply winning because it has access to more modalities than its older comparison points.

Taking the two modules apart

The ablation study removes each module in turn on the MKG-Y and TIVA datasets. Removing MoIFu and replacing relationship gated, confidence weighted fusion with plain concatenation costs the model measurable MRR on both datasets. Removing MoIEn and letting the generator produce features with no self attention refinement and no structural guidance costs even more, which is a useful signal about where the bulk of the performance is actually coming from.

On both datasets, losing MoIEn hurts more than losing MoIFu, and the gap between the two removed variants is not tiny. That fits a fairly intuitive story. Fusing modalities better still leaves you stuck if a modality is simply missing or badly degraded, while the enhancement module is the piece actually responsible for producing a usable substitute in that case. Neither module carries the whole result on its own, and the paper’s own framing of a collaborative interaction between the two is a reasonable read of the numbers rather than an overstatement.

significantly improves multi modal feature utilization and knowledge reasoning capabilities Wang, Liu, Liu, Weng, and Chaomurilige, Neural Networks, 2025

A generalization test that is more convincing than another leaderboard row

The most persuasive experiment in the paper is arguably not the main results table at all. The authors took the MoIEn module out of AFME entirely and bolted it onto three unrelated baseline architectures, TBKGC, MMKRL, and QEB, then measured whether those baselines improved on TIVA. All three did. That is a different and frankly more useful kind of evidence than beating a leaderboard, because it suggests the enhancement mechanism is not just tuned to work well specifically inside AFME’s own architecture, it is doing something generally useful for modality completion that a different model’s fusion strategy can also benefit from.

The hyperparameters that actually moved the needle

Three sensitivity experiments are worth knowing if you are trying to reproduce or adapt this work. The denoising intensity parameter, which controls how strongly the relationship gate suppresses raw modality features versus purified ones, degrades performance at both extremes, too little denoising leaves noise in, too much strips out genuinely useful detail along with it. The margin value in the negative sampling loss peaked at four within a tested range of two to six, balancing discriminative power against training stability. The number of self attention layers in the enhancement module peaked at four as well, with both fewer layers failing to capture cross modal dependencies and more layers adding computational cost without a matching gain. None of these are surprising shapes for a sensitivity curve to take, but having the actual peak values reported saves anyone reproducing this architecture a real amount of tuning time.

Where the paper is candid about its limits

It is also worth noting plainly what the paper does not claim. It does not claim to beat every baseline on every single metric on every dataset, the Hits@10 columns on MKG-W and MKG-Y are left blank in the source table rather than reported, and the improvement margins range from under one percent to nearly three percent depending on the benchmark, which is a real but not uniform advantage.

A reference implementation of the core mechanism

No code repository is linked from the paper, so the block below is an independent PyTorch implementation of the mechanism it describes, written to make the moving pieces concrete. It covers the relationship gated denoising step, the confidence weighted dynamic fusion softmax, a structure guided generator with self attention and inter modal interaction, a discriminator, the knowledge graph completion loss, the WGAN style adversarial loss with a gradient penalty, a training loop, an evaluation function, and a smoke test on random data.

Two caveats on the code above, in the interest of accuracy. The full paper batches this process across thousands of entities and triples at once, while the smoke test above runs on a single entity’s worth of features for clarity. And the inter modal interaction step here assumes every modality is present, whereas a real deployment needs to handle the case where a modality is absent entirely for a given entity, which the paper’s enhancement module is specifically designed to address.

What this adds up to for anyone building on multimodal graphs

The most transferable idea in this paper is not really the GAN, GANs have been tried in knowledge graph completion before. It is the decision to condition both the denoising step and the fusion weighting on the relationship being predicted, rather than treating modality quality as a fixed, entity level property. A photograph that is highly informative for a physical location relation may be nearly useless for a causal relation between two historical events, and a fusion mechanism that cannot tell the difference is leaving performance on the table regardless of how sophisticated its generator is.

The generalization experiment, where the enhancement module improved three unrelated baseline architectures, is the strongest piece of evidence in the paper that this idea is not narrowly tied to AFME’s specific scoring function. That makes MoIEn a reasonable candidate for anyone maintaining an existing multimodal completion pipeline who wants an incremental upgrade rather than a full architecture rewrite.

Conclusion

AFME’s contribution is a genuinely two part answer to a problem that most prior work treated as one part. Cleaning and weighting modality features by relationship context, through MoIFu, and regenerating weak or missing modality features using the graph’s own structural signal, through MoIEn, turn out to be complementary rather than redundant, based on both the ablation study and the fact that losing either one costs real performance on MKG-Y and TIVA.

The conceptual shift worth remembering is treating modality quality as something that depends on the relationship being asked about, not as a fixed property of the modality itself. Once fusion weights are allowed to move per relationship rather than staying static across the whole graph, a framework gets room to be decisive where one modality clearly dominates and balanced where several genuinely contribute, which a single global weighting scheme cannot do.

Where this could go next follows fairly directly from the paper’s own stated gaps. Testing on graphs with heavier long tail entity and rare relation coverage, and on harder domains such as medical visual question answering or social network reasoning, would tell us whether the relationship conditioned gating and the structure guided generator hold up outside the relatively well populated benchmarks used here.

The honest limitations are worth repeating rather than glossing over. Blank Hits@10 entries for two of the four benchmark datasets in the source table, a generalization experiment run on a single dataset with three baseline models, and an explicit admission that rare entities and relation types were not specifically evaluated, all mark where the next version of this framework needs to go rather than undercutting what has already been shown.

What stays with you after reading this paper is not the percentage improvement over NATIVE. It is the gating equation itself, the quiet idea that a model can be taught to ask, for this specific relationship, how much do I actually trust this specific piece of evidence, before it ever tries to fuse anything together.

Frequently asked questions

Related reading on this site