A Contrast Free Tumor AI Owes Little to Its Text Model

The problem contrast agents were supposed to solve, and their cost

Contrast enhanced imaging is genuinely useful, which is exactly why it became standard practice. In MRI, gadolinium based agents make the difference in signal between tumor tissue and normal tissue much easier to see, and in CT, iodinated contrast media do something similar for vascular and tissue visualization. For tumors in the brain, liver, and kidney, all three of which this paper studies, the boundary between tumor and healthy tissue is often genuinely ambiguous on a plain scan, and contrast makes that boundary sharper, which matters directly for surgical planning and treatment decisions.

The cost side of that tradeoff is real too. Gadolinium injection is associated with nephrogenic systemic fibrosis, a serious complication particularly relevant for patients with impaired kidney function, a group that overlaps meaningfully with people who need a kidney tumor scanned in the first place. Contrast injection also adds time, cost, and a degree of risk that is simply unnecessary for many patients if an equally accurate reading could be obtained from the scan they would get anyway. That gap, between what non contrast imaging can show on its own and what a contrast enhanced scan reveals, is the entire reason a field of research around AI contrast agent generation exists.

What EGTA-KD actually does

The core idea is a form of knowledge distillation, a technique where a smaller or more constrained student model learns by imitating a larger or more capable teacher model rather than learning from raw labels alone. Here, the teacher network is trained on contrast enhanced images, which are easier to segment accurately because the tumor boundary is visually clearer. The student network is trained on the corresponding non contrast images from the same patients, and it learns not just from the ground truth tumor labels but also from the teacher’s internal representations, effectively absorbing some of what contrast made visible even though the student never sees a contrast enhanced scan itself.

That basic distillation idea is not new. What the paper adds is described across three components. The first, the tumor aware cross modal synchronizer, or TACMS, feeds medical text describing the scan and the tumor type into the model and fuses that text with image features at the pixel level, so the model has semantic knowledge of what it is looking for baked into its feature representation. The second, the uncertainty quantified evidence unit, or UQEU, replaces the ordinary softmax classification layer with a Dirichlet distribution based system grounded in Dempster Shafer evidence theory, allowing the model to express how confident it is about each pixel’s classification rather than producing an overconfident probability by default. The third, dual level knowledge distillation, or DLKD, is the mechanism that actually transfers knowledge from teacher to student, matching both the pixel level tumor score maps and the evidence distributions between the two networks.

The paper is explicit that the text branch is only used during the teacher’s pretraining stage. Once a set of text embeddings has been learned and stored, the student network at inference time uses only images, with no live text processing required, which keeps the deployed system lightweight, around 25 million parameters, even though the full training pipeline runs closer to 113 million parameters because of the CLIP text encoder involved in pretraining.

“No work has attempted to integrate multi modal image and text information for tumor segmentation on non contrast medical images.” From the paper’s framing of its contribution, Section 2.2

Where the paper puts its emphasis

It is worth being precise about how the paper presents itself, because that framing shapes how a reader is likely to remember it. The title leads with evidence modeling meeting knowledge distillation. The abstract’s second listed limitation of prior work, right after the uncertainty quantification gap, is that existing methods use vision information only and ignore rich semantic features in medical language. TACMS gets its own detailed subsection with a dedicated figure comparing it favorably against classic text embedding and cross attention approaches, described as bridging a gap that no prior work had addressed. Context aware prompt learning, the mechanism that turns raw text like a brain MRI with tumor into a learnable embedding, gets its own subsection too, with three different prompt template versions tested for robustness.

That is a reasonable amount of space to spend on a genuinely novel piece of engineering, and the cross modal framing is clearly the part the authors consider their most original contribution, distinct from the evidence modeling and distillation ideas which build more directly on existing techniques like Dempster Shafer theory and Hinton style knowledge distillation. The question worth asking is whether the paper’s own results back up that emphasis.

What the ablation table actually shows

Table 4 in the paper reports Dice similarity coefficient results for the full EGTA-KD model against four ablated versions, each missing one component, across all three organs. This is the paper’s own data, and it tells a fairly clear story once the four ablations are lined up side by side.

Line these up and the pattern holds across every single organ tested. DLKD and UQEU, the two components that involve no text or language processing at all, each cost roughly two to three percentage points of Dice score when removed. TACMS and CAPL, the two components built around the paper’s headline cross modal, text driven innovation, cost roughly half to less than two percentage points when removed, consistently the smaller effect of the four in every column of the table. On the kidney dataset specifically, removing the entire text branch, TACMS and CAPL combined, costs a maximum of about 1.4 percentage points, while removing UQEU alone costs 3.1 points, more than double.

None of this means the text based components do nothing. They are consistently positive contributors, and the paper reports statistically significant p values below 0.05 for every ablation comparison, meaning these are real, measurable effects rather than noise. But a reader who took the abstract and introduction at face value would reasonably expect the vision language fusion idea to be the load bearing piece of this architecture, given how much space and how much first of its kind language surrounds it. The evidence in the paper’s own table points the other way. The quieter, less novel sounding evidence modeling and distillation machinery is doing more of the actual work.

How the full model performed against other methods

Setting the internal ablation aside, EGTA-KD’s headline comparison against five other published approaches is a strong result. The comparison used nnU-Net as a baseline, Rg-GAN as a GAN based non contrast segmentation method, CLIP-UniMo as a CLIP driven universal segmentation model, and two prior knowledge distillation approaches, KD-Net and PKL, both of which also use contrast enhanced images as a privileged teacher modality.

Lower Hausdorff Distance means the predicted tumor boundary sits closer to the true boundary, so EGTA-KD’s meaningfully lower numbers across all three organs represent a real improvement in boundary accuracy, not just overlap. The paper reports paired t-tests for every comparison, all significant at p below 0.05, which is a reasonable standard of evidence for this kind of benchmark study. The gap between EGTA-KD and the next best method, PKL, is largest in relative terms on brain tumors and smallest on kidney tumors, which lines up with the paper’s own explanation that kidney tumors are harder to segment in general because the dataset provides only single modality CT rather than the richer multi modality MRI available for brain and liver.

The paper also tested robustness to a realistic distribution shift, using the fact that the BraTS 2021 brain dataset combines data from two centers, the University of California San Francisco with 382 subjects and the University of Pennsylvania with 511 subjects. Training on anywhere from 5 percent to 95 percent UCSF data and testing on the remainder, the average Dice score stayed within a narrow band, from 0.907 to 0.919, which is a genuinely reassuring sign that the model is not simply overfitting to one center’s scanner characteristics. It is worth being clear that this particular robustness test only exists for the brain dataset, since it is the only one of the three with a documented multi center split. No equivalent cross center test is reported for the liver or kidney data.

The clinical translation gap

It is worth walking through, plainly, how far this result sits from anything usable in an actual clinic today. This is a retrospective benchmark evaluation using 5 fold cross validation on three specific datasets, one of them, BraTS 2021, a large and well established public benchmark with 1251 subjects, but the liver MRI dataset has 238 subjects and the kidney CT dataset has 113, both modest by the standards of a system that would need to generalize across the full range of tumor presentations, scanner types, and patient populations found in real practice. There is no prospective clinical trial here, no comparison against a panel of practicing radiologists reading the same non contrast scans, and no evidence about performance at institutions or on scanner hardware outside the datasets used.

There is also an aspect of the clinical story that is easy to miss on a first read. Training this system to be contrast agent free at inference time still requires a substantial number of patients who underwent contrast agent injection during data collection, since the teacher network learns specifically from paired contrast enhanced and non contrast scans of the same patients with tumor labels attached. The benefit this technology offers is to future patients scanned after such a model has been trained and validated, not a reduction in contrast agent use for the patients whose data built the system in the first place. That is not a flaw in the research design, since there is no other way to build the necessary paired training data, but it is a nuance worth keeping in mind whenever a contrast agent free framing is used to describe this kind of work.

No regulatory clearance is mentioned anywhere in the paper, and none would be expected for a result at this stage. A benchmark accuracy figure on three research datasets is a different thing entirely from a cleared diagnostic tool, and nothing in the paper suggests the authors are claiming otherwise. Their own framing throughout positions this as a step toward a contrast agent free tool to assist clinical diagnosis, appropriately forward looking language rather than a claim of present day readiness.

Clinical limitations, in the paper’s own numbers

A few specific constraints deserve to be stated using the paper’s actual figures. The kidney CT dataset contains 113 subjects total, and kidney tumor segmentation is also the task with the lowest reported Dice score, 0.763, against 0.908 for brain and 0.905 for liver, a gap the authors attribute to the kidney dataset providing only single modality CT rather than multi modality imaging, and to kidney tumors generally being smaller and harder to delineate. The liver dataset’s 238 subjects were split into two clinical presentations, 110 hemangioma cases and 128 hepatocellular carcinoma cases, and its ground truth segmentation labels were produced by two radiologists reconciling disagreements between them, a reasonable annotation protocol but one that still reflects the judgment of two specific readers rather than a broader consensus process.

The robustness test described above, showing stable performance across the UCSF and UPENN patient splits, is a genuine strength of the evidence provided for the brain tumor task specifically. It does not extend to liver or kidney segmentation, where no multi center or multi scanner robustness data is reported at all, and where the smaller sample sizes leave open questions about how performance would hold up on tumors or imaging protocols not represented in these particular 238 and 113 patient cohorts.

The proposed evidence and distillation modules in PyTorch

Below is a complete implementation of the paper’s uncertainty quantified evidence unit and dual level distillation losses, the two components the ablation study points to as the most load bearing, along with a simplified tumor score map calculation standing in for the full TACMS module.

import torch
import torch.nn as nn
import torch.nn.functional as F

# --- Tumor score map, Eq. 2 of the paper, simplified pixel wise image text matching ---
def tumor_score_map(image_feats, text_feats):
    # image_feats: (batch, H, W, C), text_feats: (K, C)
    x_norm = F.normalize(image_feats, dim=-1)
    t_norm = F.normalize(text_feats, dim=-1)
    # pixel wise dot product against each of the K class text embeddings
    score_map = torch.einsum('bhwc,kc->bhwk', x_norm, t_norm)
    return score_map  # (batch, H, W, K)

# --- Uncertainty Quantified Evidence Unit, Eq. 3 through Eq. 6 of the paper ---
class UQEU(nn.Module):
    def __init__(self, in_channels, num_classes):
        super().__init__()
        self.evidence_layer = nn.Conv2d(in_channels, num_classes, kernel_size=1)

    def forward(self, fused_feats):
        # fused_feats: (batch, C, H, W), the concatenation of tumor score map and vision features
        evidence = F.softplus(self.evidence_layer(fused_feats))  # e >= 0, Eq. before Eq. 3
        alpha = evidence + 1.0  # Dirichlet parameters, Eq. 3
        total_strength = alpha.sum(dim=1, keepdim=True)  # D, Eq. 4

        belief = (alpha - 1.0) / total_strength  # belief mass b^k, Eq. 5
        num_classes = alpha.shape[1]
        uncertainty = num_classes / total_strength  # uncertainty mass u, Eq. 5

        prob_map = alpha / total_strength  # predicted probability, Section 3.3
        return prob_map, uncertainty, alpha, belief

# --- Evidential cross entropy loss, Eq. 10, using the digamma function ---
def evidential_cross_entropy(alpha, target_one_hot):
    # alpha, target_one_hot: (batch, K, H, W)
    S = alpha.sum(dim=1, keepdim=True)
    loss = target_one_hot * (torch.digamma(S) - torch.digamma(alpha))
    return loss.sum(dim=1).mean()

# --- Dual Level Knowledge Distillation, Eq. 7 and Eq. 8 of the paper ---
def dlkd_loss(score_map_teacher, score_map_student, prob_teacher, prob_student, temperature=1.0):
    tsm_loss = F.mse_loss(score_map_student, score_map_teacher.detach())

    # KL divergence between teacher and student evidence probabilities, temperature scaled
    p_t = F.softmax(prob_teacher.detach() / temperature, dim=1)
    p_s = F.log_softmax(prob_student / temperature, dim=1)
    belief_loss = (temperature ** 2) * F.kl_div(p_s, p_t, reduction='batchmean')

    return tsm_loss, belief_loss

# --- Smoke test on dummy data, standing in for a batch of paired teacher and student features ---
if __name__ == "__main__":
    torch.manual_seed(0)
    batch_size = 2
    num_classes = 3  # whole tumor, tumor core, enhancing tumor
    h, w = 16, 16
    channels = 32

    teacher_uqeu = UQEU(channels + num_classes, num_classes)
    student_uqeu = UQEU(channels + num_classes, num_classes)
    optimizer = torch.optim.Adam(student_uqeu.parameters(), lr=1e-3)

    image_feats_teacher = torch.randn(batch_size, h, w, channels)
    image_feats_student = torch.randn(batch_size, h, w, channels)
    text_feats = torch.randn(num_classes, channels)
    target = torch.randint(0, num_classes, (batch_size, h, w))
    target_one_hot = F.one_hot(target, num_classes).permute(0, 3, 1, 2).float()

    losses = []
    for step in range(50):
        score_teacher = tumor_score_map(image_feats_teacher, text_feats).permute(0, 3, 1, 2)
        score_student = tumor_score_map(image_feats_student, text_feats).permute(0, 3, 1, 2)

        fused_teacher = torch.cat([score_teacher, image_feats_teacher.permute(0, 3, 1, 2)[:, :num_classes]], dim=1)
        fused_student = torch.cat([score_student, image_feats_student.permute(0, 3, 1, 2)[:, :num_classes]], dim=1)

        prob_t, unc_t, alpha_t, _ = teacher_uqeu(fused_teacher)
        prob_s, unc_s, alpha_s, _ = student_uqeu(fused_student)

        ce_loss = evidential_cross_entropy(alpha_s, target_one_hot)
        tsm_loss, belief_loss = dlkd_loss(score_teacher, score_student, prob_t, prob_s)

        total_loss = ce_loss + tsm_loss + belief_loss
        optimizer.zero_grad()
        total_loss.backward()
        optimizer.step()
        losses.append(total_loss.item())

    assert losses[-1] < losses[0], "loss did not decrease, smoke test failed"
    print(f"Smoke test passed. Loss went from {losses[0]:.4f} to {losses[-1]:.4f}")

A team wanting to test the paper’s own ablation finding directly could start by training a version of this student network with only the UQEU and DLKD losses active, dropping the tumor score map’s dependence on text features entirely and replacing it with a plain image only feature map. The ablation table suggests that version would retain most of the full model’s accuracy while removing the CLIP text encoder dependency altogether.

Conclusion

EGTA-KD is a solid piece of engineering wrapped in a narrative that oversells one of its three ideas at the expense of the other two. The evidence modeling component, grounded in Dempster Shafer theory and Dirichlet distributions, gives the model a principled way to say how confident it is at every pixel, addressing a real weakness of standard softmax based segmentation where overconfident wrong answers are indistinguishable from confident correct ones. The dual level distillation strategy gives the student network a genuinely useful training signal drawn from what the teacher learned on contrast enhanced images. Both hold up well under the paper’s own ablation testing.

The conceptual shift worth taking away is not really about vision language models in medical imaging, interesting as that direction is. It is that a paper’s framing and its evidence do not always point the same direction, and a careful reader benefits from checking the ablation table before accepting which component a paper wants credited as its central contribution. Here, the more novel sounding idea, cross modal text fusion, was consistently the weaker performer of the four tested components across three separate organs, a pattern too systematic to be a fluke of one dataset.

Whether that pattern would hold in other cross modal medical imaging work is worth watching for. Text conditioned segmentation and CLIP style vision language pretraining are active, fast moving research directions in medical AI right now, and it would be worth checking future work in this space for the same kind of ablation transparency this paper provides, since a flashy multi modal architecture is not automatically the part of a system doing the most work.

The honest limitations here are real. The liver and kidney datasets are modest in size, 238 and 113 subjects respectively, robustness testing across imaging centers exists only for the brain task, and building the training data for a contrast agent free tool still requires patients who received contrast agent injections during data collection. None of that erases the paper’s central technical achievement, a working distillation framework that meaningfully improves non contrast tumor segmentation with statistically significant gains over five compared methods.

For anyone building on this work, the actionable lesson is to weigh engineering effort by the ablation numbers rather than by the abstract’s emphasis, and for anyone reading papers like this more generally, it is worth remembering that the component a paper spends the most words describing is not always the component doing the most work.

Frequently asked questions

Related reading