M3Surv Cancer Survival Prediction With Missing Data

Analysis by the aitrendblend editorial team · Source paper published October 2025 · AI for medical imaging and healthcare

• cancer survival prediction
• whole slide images
• multiomics
• hypergraph learning
• missing modality
• computational pathology

Why one slide and one data type are not enough

Predicting how long a patient with cancer is likely to survive helps clinicians stratify risk and think about treatment intensity. The modern recipe combines a whole slide image, a gigapixel scan of stained tissue, with molecular data such as gene expression. Earlier systems like MCAT, SurvPath and PIBD showed that tissue shape and molecular signals complement each other. The paper behind this analysis, published in Medical Image Analysis by teams at Harbin Institute of Technology, Tsinghua University and the University of New South Wales, argues that this recipe leaves two things on the table.

The first is the type of slide. Hematoxylin and eosin stained sections come in two common forms. Formalin fixed paraffin embedded slides, called FFPE, preserve tissue architecture and are the archival clinical standard, yet preparing them can take up to 36 hours. Fresh frozen slides, called FF, are cryopreserved mainly to protect nucleic acids for assays such as RNA sequencing, and they can be ready in about 15 minutes. The freezing process can introduce artifacts such as ice crystals that blur cellular detail, so FF images are usually used for fast intraoperative questions like surgical margins. Nearly every survival model in the literature uses FFPE only.

The second is the assumption of completeness. Fusion models that rely on cross attention need both modalities present at inference. Real clinics deal with budgets, time pressure and failed assays, so a gene panel or a slide is often simply absent. Methods based on distillation or variational autoencoders can patch partial gaps, but the authors say they struggle when an entire modality is gone. M3Surv is designed around that harder case.

What earlier work assumed

Slide level survival models began with multiple instance learning, where thousands of patches are scored and pooled. ABMIL, CLAM and TransMIL belong to that family. Graph methods treat patches as nodes and let information flow along spatial neighbors. Di and colleagues earlier introduced hypergraphs for survival analysis. A hyperedge can join more than two nodes at once, which suits tissue because a gland, a cluster of immune cells or a tumor nest involves many patches at the same time and not just pairs.

On the multimodal side the paper lists concatenation in Porpoise, optimal transport in MOTCat and cross attention in MCAT, CMTA, SurvPath and PIBD. Methods for missing modalities include multimodal variational autoencoders, Bayesian meta learning, shared and specific feature modelling and distilled prompt learning. The authors name three gaps. FF slides are underexplored, complete modalities are assumed, and evaluation leans on concordance based measures with few reports of time dependent metrics, calibration or external validation. We return to that third gap later, because the paper identifies it and then mostly shares it. If graph models are new to you, our primer on graph neural networks covers the background.

Inside M3Surv

The framework has three stages, which are feature extraction, multislide hypergraph learning and multimodal fusion, with a memory bank standing by for missing data. It extends the authors’ MICCAI paper in three ways. The multislide module now uses a divide and conquer design with adaptive weighting, the molecular side grew from transcriptomics to genomics plus proteomics, and the memory bank moved from a top K selection scheme to learned prototypes.

Turning slides into hypergraphs

Each FFPE slide is cut into 256 by 256 pixel patches at 20 times magnification, and 4096 patches are sampled at random to keep computation manageable. FF slides are smaller, around 300 to 500 patches, so all of them are used. A pretrained encoder such as ResNet50 turns every patch into a feature vector, and each patch keeps its spatial coordinates.

Within one slide, every patch becomes a vertex and hyperedges are built in two ways. Topological hyperedges group a patch with its spatial neighbors to capture local tissue structure. Structural hyperedges group a patch with patches that look similar in feature space, even if they sit far apart, which helps when staining style varies. The two sets are merged and passed through hypergraph convolution layers. The authors fix the neighborhood size at k equal to 9, so each hyperedge holds a patch and its eight nearest neighbors.

Inter slide links and adaptive weights

The distinctive step is linking the two slides of one patient. Patches from the FF slide connect to the most similar patches in the FFPE slide, and the reverse. The idea is that both slides come from the same tumor, so they should share biological structure even though one is blurrier. A small weighting module then learns how much each slide should count, and the weights feed into the hypergraph convolution. Finally a cross attention step refines the FF and FFPE features using the inter slide features as shared guidance, and the refined features are concatenated into one pathology representation.

Three omics layers

Genomic data, meaning RNA sequencing and copy number variation, is grouped into six functional categories such as tumor suppression, oncogenesis and protein kinases. Transcriptomic data is organized into 331 biological pathways, of which 281 come from Reactome with 1,577 genes and 50 from MSigDB Hallmark sets with 4,241 genes, covering 4,999 gene attributes in total. Proteomic data keeps the top 100 proteins by expression, and each protein gets an embedding from the ESM protein language model combined with its expression value. SNN encoders produce a fixed size vector for each gene group and pathway. The paper expands SNN as spiking neural network in one place yet cites the self normalizing network paper for its SNN baseline, and our code assumes the self normalizing version.

Fusion and the survival loss

Pathology and omics then talk to each other through mutual cross attention. Pathology queries the omics and the omics queries pathology. The two enhanced representations are concatenated and passed to a feed forward network that estimates hazard and survival probability. Training minimizes a negative log likelihood that handles both observed events and censored patients, meaning patients who were still alive or lost to follow up when data collection ended.

The prototype memory bank

This is the part of the paper most relevant to real hospitals. During training the model collects paired pathology and omics features for every patient and runs a k means clustering to find representative groups. Each cluster yields a pathology prototype and an omics prototype, which are the averages over the patients in that cluster. The prototypes are refreshed every epoch with a momentum rule, so the bank changes smoothly instead of lurching with every mini batch.

At inference, if one modality is absent, the model computes a prototype from the modality it does have, finds the most similar stored entries and returns a similarity weighted blend of their prototypes for the missing side. The authors argue this is steadier than imputing from one patient’s partial data, because a prototype stands for a subgroup of patients and not an individual. The best setting in their sweep was a momentum of 0.75 and a retrieval count of one, since pulling in more prototypes appeared to add noise from less related patients. The bank adds about 4.3 percent to training time and about 2.2 percent to inference time.

“Robustly impute information from the most similar prototype.” Qu et al., 2025, describing the memory bank at inference

The mathematics in plain view

Here \(\zeta\) is a patch coordinate, \(\delta\) a distance threshold and \(\gamma\) a similarity threshold. The first set gathers physical neighbors and the second gathers look alike patches. In practice the paper sets both through the neighbor count \(k=9\).

The first expression is a softmax that decides how much each slide contributes. The second refines the FF features \(P_{ff}\) by attending to the inter slide features \(P_s\), where \(\phi\) is the softmax and \(W_q\), \(W_k\), \(W_v\) are learned projections. The FFPE features are refined in the same way.

The first term averages the pathology features in cluster \(l\), and the omics prototypes are computed the same way. The second blends new prototypes into the bank with momentum \(\theta\). The third rebuilds the missing modality \(\hat{M}_m\) from the \(\mu\) entries whose available side best matches the query.

Data and evaluation protocol

The experiments use five TCGA cohorts. These are bladder urothelial carcinoma (BLCA, 384 patients), breast invasive carcinoma (BRCA, 968), colon and rectum adenocarcinoma (CRAD, 298), head and neck squamous cell carcinoma (HNSC, 392) and stomach adenocarcinoma (STAD, 317). Each TCGA patient contributes one FF and one FFPE slide, which gives 2359 slides of each type. A sixth dataset, APH LC, is an in house liver cancer cohort of 302 patients from Anhui Provincial Hospital, with expert region annotations and matched omics.

Performance is the concordance index, which measures how often the model ranks the patient who died earlier as higher risk. A value of 0.5 is chance and 1.0 is perfect. Results are mean and standard deviation over five fold cross validation, with the Adam optimizer, a learning rate of 0.0001, a weight decay of 0.00001 and 30 epochs. For missing modality tests the model is trained on complete data and the missingness is introduced only at inference, at ratios of 30, 50, 70 and 100 percent.

What the numbers actually say

Accuracy against other methods

The headline claim of a 2.2 percent improvement deserves a careful reading. It is 2.2 absolute points of concordance index, which is 70.7 against PIBD at 68.5. It is not a 2.2 percent relative gain, and against the authors’ own earlier M2Surv the margin is 0.6 points. That is still an advance on the field, but the incremental step from the conference version is modest.

Look at the cohort columns. M3Surv beats M2Surv in four of five cohorts, but by only 0.3 to 0.4 points in BLCA, HNSC and STAD, and it loses in BRCA, the largest cohort, by 1.2 points. Its biggest gain over M2Surv is 3.1 points in CRAD. Against PIBD the standout is HNSC, where M3Surv reaches 66.4 and PIBD reaches 60.7. The fold standard deviations run from 1.7 to 7.4 points, so most of these gaps sit well inside one standard deviation. The paper does not report significance tests against the competing methods.

When a whole modality disappears

The degradation is gentle. With no pathology at all, M3Surv scores 66.9, which beats the genomics only baselines MLP at 57.6 and SNNTrans at 62.2. With no omics at all, it matches or beats pathology only baselines. The curve is also odd. Performance at 100 percent missing pathology, 66.9, exceeds performance at 70 percent missing, 66.5. The authors explain that a single complete modality gives the bank a clean query, which is plausible. A simpler explanation is fold to fold noise of the size seen in the standard deviations above, and the paper offers no confidence intervals to separate the two stories.

One more inconsistency is worth flagging. The text reports 66.6 for the complete absence of omics, while Table 1 and Table 5 report 68.5 for pathology only. The paper does not reconcile the two, so treat the omics missing result as somewhere between 66.6 and 68.5.

The authors also tested the bank as a plug in. They attached it to MCAT and SurvPath, which cannot handle missing inputs natively, froze everything else and used 6 prototypes for genomics, 8 for transcriptomics and 16 for FFPE slides. They report performance on incomplete data comparable to those models on full data, though the text gives no numbers. Against replacing the bank with a general purpose generative model called LDCVAE, M3Surv scored 68.5 against 67.3 for pathology only and 66.9 against 64.6 for genomics only.

What each ingredient contributes

FF alone is clearly weaker at 62.7 than FFPE alone at 69.0, which fits the idea that freezing damages morphology. Yet adding FF to FFPE raises the mean by 1.7 points, so the cheap slide carries complementary signal. On the omics side, dropping transcriptomics costs the most, 3.8 points, dropping proteomics costs 1.8 points and dropping genomics costs 0.6 points. In individual cohorts the pattern is messier. In CRAD the version without genomics scores 79.4 against 78.8 for the full model, and in HNSC the version without proteomics ties the full model at 66.4. Averages hide a lot here.

The architecture ablations tell a similar story. For FF slides, a plain MLP aggregator averages 67.7, a graph convolution network gets 68.1, and a hypergraph network with topological hyperedges gets 69.3. Structural hyperedges alone score 67.1, below the MLP, so they pay off mainly in combination with topological ones. For multislide alignment, the weighting module lifts the mean from 68.2 to 70.7, which is the 2.5 points the paper cites. One sentence in the paper claims graph methods improve alignment by 66.8 percent, but Table 4 shows a graph convolution network at 66.8 against 66.5 for the baseline. That is a 0.3 point difference, so the word significant overstates it.

On the in house APH LC cohort, the full model reaches 66.5. Removing FFPE drops it to 61.9, removing FF to 65.0, removing transcriptomics to 64.7, genomics to 65.5 and proteomics to 65.9. The pattern echoes the TCGA results, though it comes from one cohort of 302 patients.

Kaplan Meier curves and heat maps

The authors split patients into high and low risk groups at the median predicted score and ran log rank tests. The p values are 0.00051 for BLCA, 0.0161 for BRCA, 0.00265 for CRAD, 0.00078 for HNSC and 0.00167 for STAD, all below 0.05. The paper does not spell out whether these curves use held out predictions only, and a median split is a blunt instrument. The heat maps and pathway plots for a single BRCA patient are illustrative. Reading them as evidence that the model found loss of tumor suppressive function in that patient goes further than a single example can support.

The clinical translation gap

A model that ranks patients well on a retrospective cohort is a long way from a tool a hospital can use. The gap has several layers. First, every number comes from cross validation within TCGA cohorts, so the model was never tested on a hospital that did not contribute training data. The authors themselves note in the related work that few studies report time dependent metrics, calibration or external validation, and this paper reports none of those either. Calibration matters because a clinician needs to know whether a predicted five year survival of 60 percent really means about six in ten patients, and the concordance index cannot say.

Second, the design choice at the heart of the paper creates a particular safety question. When a modality is missing, the memory bank returns a typical profile for a similar group of patients. That profile will look like real data downstream. A prediction made from an imputed omics profile therefore reflects the cohort average more than the individual, and any real tool would need to show clinicians clearly which inputs were measured and which were borrowed.

Third, the intended workflow is unclear. FF slides are valued because they arrive fast, but the paper trains and evaluates on retrospective archives where both slides already exist. Whether the model can deliver a useful estimate at the moment a frozen section is read, with real latency and real missing patterns, has not been tested. Missingness in the experiments is also created by deleting data at random ratios, while in hospitals data tends to go missing for systematic reasons such as cost, tumor size or referral pathway.

Regulatory and safety notes

Software that informs clinical decisions can fall under medical device rules in many jurisdictions, depending on its intended use and the claims made for it. That usually brings requirements for documented validation, performance monitoring and often prospective evidence. The authors acknowledge this direction in their outlook, which lists prospective clinical trials as essential future work. Until such evidence exists, outputs from a model like this belong in research settings and should not guide care decisions.

Clinical limitations in numbers

Sample size comes first. The smallest TCGA cohort, CRAD, has 298 patients, so each test fold in five fold cross validation holds only about 60 patients. The paper does not report event counts per cohort in the text, which matters because concordance depends on how many patients actually died. The CRAD standard deviation of 6.7 points and the STAD standard deviation of 7.4 points show how unstable per fold estimates can be. The best competitors reach standard deviations as high as 13.4 points in CRAD, so instability is not unique to this model.

Dataset bias is the next concern. The paper reports no demographics, tumor stage, treatment or contributing site information in the text, so there is no way to see whether the model performs differently across age groups, stages or institutions. Hyperparameters such as the momentum of 0.75, the retrieval count of one and the neighbor count of nine were chosen through ablations, and the paper does not say whether they were tuned on data separate from the test folds. If they were not, the reported results would be somewhat optimistic.

Generalization remains untested in the strict sense. The APH LC liver cancer cohort of 302 patients appears only in an ablation figure, and liver cancer is not among the five TCGA cancers, so it is not a test of whether a TCGA trained model transfers. The authors also state their own limits. Learned similarities used to link slides are sensitive to modality specific artifacts, and the memory bank depends on the diversity of the training data. A bank built from one hospital’s patients may not contain the subgroups another hospital sees.

Finally, nothing here shows the model changes decisions or outcomes. We did not rerun any experiment, and everything above reflects the numbers the paper reports. The code below is our reconstruction and not an official release.

Where this idea fits

M3Surv is a good example of a trend in medical AI toward models that expect imperfect inputs. The lesson travels beyond oncology. Whenever a pipeline trained on complete records meets incomplete deployment data, a retrieval based fallback is a sensible design to try, and it is cheap, since the bank costs about 4.3 percent more training time. Related questions about training across hospitals without pooling their records are covered in our notes on federated learning and AI privacy. For more work on medical image models, see our hub on AI for medical imaging and healthcare and our article on whole slide image analysis.

A compact PyTorch reimplementation

The listing below is our reconstruction of the main components, written from the paper’s descriptions and equations. It is not the authors’ release, and the paper text we reviewed gives no repository link. It includes the hypergraph layers, the slide weighting, cross attention, omics encoders, the memory bank, the discrete time survival loss, a concordance index and a training loop. Where the paper leaves a detail open, a comment says what was assumed. The smoke test at the bottom trains on random tensors for two epochs, which is meant to confirm that shapes line up and not to reproduce any result. It is for study only and must not be used for clinical purposes.

"""
M3Surv reference implementation (PyTorch), single patient per step.
Re-implementation written from the description in Qu et al., Medical Image
Analysis 107 (2026) 103846. It is NOT the authors' code. Where the paper leaves
a detail open, a comment says what was assumed. Research use only.
"""
import math
import torch
import torch.nn as nn
import torch.nn.functional as F


# ------------------------------------------------------------ hypergraph parts
def knn_incidence(x, k, metric="euclid"):
    """(N,N) incidence matrix. Column i is the hyperedge made of patch i and its k-1 nearest patches."""
    n = x.size(0)
    k = min(k, n)
    if metric == "euclid":                      # topological hyperedges use coordinates
        dist = torch.cdist(x, x)
    else:                                       # structural hyperedges use cosine distance
        xn = F.normalize(x, dim=1)
        dist = 1 - xn @ xn.t()
    idx = dist.topk(k, largest=False).indices   # (N,k)
    H = torch.zeros(n, n, device=x.device)
    H.scatter_(0, idx.t(), 1.0)
    H.fill_diagonal_(1.0)
    return H


def cross_incidence(xa, xb, k):
    """Inter slide hyperedges. Every patch of one slide links to its k-1 most similar patches of the other."""
    na, nb = xa.size(0), xb.size(0)
    sim = F.normalize(xa, dim=1) @ F.normalize(xb, dim=1).t()   # (na,nb)
    ka, kb = min(k - 1, nb), min(k - 1, na)
    ia = sim.topk(ka, dim=1).indices            # FFPE neighbours of each FF patch
    ib = sim.t().topk(kb, dim=1).indices        # FF neighbours of each FFPE patch
    H = torch.zeros(na + nb, na + nb, device=xa.device)
    ca = torch.arange(na, device=xa.device)
    cb = na + torch.arange(nb, device=xa.device)
    H[ca, ca] = 1.0
    H[cb, cb] = 1.0
    H[(na + ia).reshape(-1), ca.repeat_interleave(ka)] = 1.0
    H[ib.reshape(-1), cb.repeat_interleave(kb)] = 1.0
    return H


class HypergraphConv(nn.Module):
    """HGNN layer (Feng et al., 2019). Out = relu(Dv^-1/2 H W De^-1 H^T Dv^-1/2 X Theta)."""
    def __init__(self, din, dout):
        super().__init__()
        self.theta = nn.Linear(din, dout)

    def forward(self, x, H, w=None):
        w = torch.ones(H.size(1), device=x.device) if w is None else w
        dv = (H * w).sum(1).clamp(min=1e-6)
        de = H.sum(0).clamp(min=1e-6)
        x = self.theta(x) * dv.pow(-0.5)[:, None]
        e = (H.t() @ x) * (w / de)[:, None]
        return F.relu((H @ e) * dv.pow(-0.5)[:, None])


class IntraSlideHypergraph(nn.Module):
    """Union of topological and structural hyperedges, then L hypergraph convolutions.
    The paper fixes k = 9, so each hyperedge holds a patch and its 8 nearest neighbours."""
    def __init__(self, d, layers=2, k=9):
        super().__init__()
        self.k = k
        self.convs = nn.ModuleList([HypergraphConv(d, d) for _ in range(layers)])

    def forward(self, feats, xy):
        H = torch.cat([knn_incidence(xy, self.k, "euclid"),
                       knn_incidence(feats, self.k, "cosine")], dim=1)
        x = feats
        for conv in self.convs:
            x = conv(x, H)
        return x


class SlideWeighting(nn.Module):
    """Eq. 5. The paper does not say how patch scores become one slide score, so mean pooling is assumed."""
    def __init__(self, d):
        super().__init__()
        self.w_ff = nn.Linear(d, 1, bias=False)
        self.w_ffpe = nn.Linear(d, 1, bias=False)

    def forward(self, p_ff, p_ffpe):
        s = torch.stack([self.w_ff(p_ff).mean(), self.w_ffpe(p_ffpe).mean()])
        w = F.softmax(s, 0)
        return w[0], w[1]


class InterSlideHypergraph(nn.Module):
    """Hypergraph over the patches of both slides. The slide weight scales the hyperedges seeded in that slide (assumed)."""
    def __init__(self, d, layers=2, k=9):
        super().__init__()
        self.k = k
        self.convs = nn.ModuleList([HypergraphConv(d, d) for _ in range(layers)])

    def forward(self, p_ff, p_ffpe, w_ff, w_ffpe):
        H = cross_incidence(p_ff, p_ffpe, self.k)
        w = torch.cat([w_ff.expand(p_ff.size(0)), w_ffpe.expand(p_ffpe.size(0))])
        x = torch.cat([p_ff, p_ffpe], 0)
        for conv in self.convs:
            x = conv(x, H, w)
        return x


class CrossAttention(nn.Module):
    """Single head cross attention, softmax(Q K^T / sqrt(d)) V (Eqs. 6 and 7)."""
    def __init__(self, d):
        super().__init__()
        self.q = nn.Linear(d, d, bias=False)
        self.k = nn.Linear(d, d, bias=False)
        self.v = nn.Linear(d, d, bias=False)
        self.scale = d ** -0.5

    def forward(self, x, ctx):
        attn = F.softmax(self.q(x) @ self.k(ctx).t() * self.scale, dim=-1)
        return attn @ self.v(ctx)


# ------------------------------------------------------------------ omics side
class SNN(nn.Module):
    """Self normalizing network block (SELU plus alpha dropout). The paper expands SNN as spiking in one place
    and cites self normalizing networks elsewhere, the latter is assumed here."""
    def __init__(self, din, d, p=0.1):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(din, d), nn.SELU(), nn.AlphaDropout(p),
                                 nn.Linear(d, d), nn.SELU(), nn.AlphaDropout(p))

    def forward(self, x):
        return self.net(x)


class GroupEncoder(nn.Module):
    """One SNN per functional group or pathway. Input is a list of 1D tensors, output is (G,d)."""
    def __init__(self, group_sizes, d):
        super().__init__()
        self.enc = nn.ModuleList([SNN(s, d) for s in group_sizes])

    def forward(self, groups):
        return torch.stack([e(g) for e, g in zip(self.enc, groups)])


class ProteomicEncoder(nn.Module):
    """ESM sequence embedding plus expression value per protein (summation is assumed)."""
    def __init__(self, esm_dim, d):
        super().__init__()
        self.seq = nn.Linear(esm_dim, d)
        self.expr = nn.Linear(1, d)

    def forward(self, esm, expr):
        return self.seq(esm) + self.expr(expr[:, None])


class OmicsEncoder(nn.Module):
    """Genomic (6 groups in the paper), transcriptomic (331 pathways) and proteomic (100 proteins) tokens."""
    def __init__(self, genomic_sizes, pathway_sizes, esm_dim, d):
        super().__init__()
        self.gen = GroupEncoder(genomic_sizes, d)
        self.tra = GroupEncoder(pathway_sizes, d)
        self.pro = ProteomicEncoder(esm_dim, d)

    def forward(self, b):
        return torch.cat([self.gen(b["genomic"]), self.tra(b["pathway"]),
                          self.pro(b["esm"], b["expr"])], 0)


# ----------------------------------------------------------------- memory bank
class MemoryBank(nn.Module):
    """Prototype memory (Eqs. 8 to 10). k means gives paired prototypes, momentum theta refreshes them,
    and a missing modality is rebuilt from the top mu prototypes of the available one."""
    def __init__(self, d, k=8, momentum=0.75, mu=1):
        super().__init__()
        self.k, self.theta, self.mu = k, momentum, mu
        self.register_buffer("proto_p", torch.zeros(k, d))
        self.register_buffer("proto_o", torch.zeros(k, d))
        self.register_buffer("ready", torch.tensor(False))

    @torch.no_grad()
    def update(self, feats_p, feats_o, iters=10):
        joint = torch.cat([feats_p, feats_o], 1)
        assert joint.size(0) >= self.k, "need at least k patients to fit k prototypes"
        cent = joint[torch.randperm(joint.size(0))[:self.k]].clone()
        for _ in range(iters):
            assign = torch.cdist(joint, cent).argmin(1)
            for l in range(self.k):
                m = assign == l
                if m.any():
                    cent[l] = joint[m].mean(0)
        assign = torch.cdist(joint, cent).argmin(1)
        new_p, new_o = self.proto_p.clone(), self.proto_o.clone()
        for l in range(self.k):                              # Eq. 8
            m = assign == l
            if m.any():
                new_p[l], new_o[l] = feats_p[m].mean(0), feats_o[m].mean(0)
        if bool(self.ready):                                 # Eq. 9
            new_p = self.theta * new_p + (1 - self.theta) * self.proto_p
            new_o = self.theta * new_o + (1 - self.theta) * self.proto_o
        self.proto_p.copy_(new_p)
        self.proto_o.copy_(new_o)
        self.ready.fill_(True)

    @torch.no_grad()
    def impute(self, query, available):
        """Eq. 10. available is 'pathology' or 'omics'. Returns a (d,) vector for the missing side."""
        if not bool(self.ready):
            raise RuntimeError("memory bank is empty, train at least one epoch first")
        keys, vals = (self.proto_p, self.proto_o) if available == "pathology" else (self.proto_o, self.proto_p)
        sim = F.cosine_similarity(query[None], keys, dim=1)
        top = sim.topk(min(self.mu, sim.numel()))
        w = F.softmax(top.values, 0)
        return (w[:, None] * vals[top.indices]).sum(0)


# ----------------------------------------------------------------------- model
class M3Surv(nn.Module):
    def __init__(self, in_dim, d, genomic_sizes, pathway_sizes, esm_dim, n_bins,
                 k_nn=9, bank_k=8):
        super().__init__()
        self.proj = nn.Linear(in_dim, d)                      # maps ResNet style patch features to d
        self.intra_ff = IntraSlideHypergraph(d, k=k_nn)
        self.intra_ffpe = IntraSlideHypergraph(d, k=k_nn)
        self.weighting = SlideWeighting(d)
        self.inter = InterSlideHypergraph(d, k=k_nn)
        self.align_ff = CrossAttention(d)
        self.align_ffpe = CrossAttention(d)
        self.omics = OmicsEncoder(genomic_sizes, pathway_sizes, esm_dim, d)
        self.p2g = CrossAttention(d)                          # pathology queries omics
        self.g2p = CrossAttention(d)                          # omics queries pathology
        self.ffn = nn.Sequential(nn.Linear(2 * d, d), nn.ReLU(), nn.Dropout(0.25),
                                 nn.Linear(d, n_bins))
        self.bank = MemoryBank(d, bank_k)

    def encode_pathology(self, b):
        ff, ffpe = self.proj(b["ff"]), self.proj(b["ffpe"])
        p_ff = self.intra_ff(ff, b["ff_xy"])
        p_ffpe = self.intra_ffpe(ffpe, b["ffpe_xy"])
        w_ff, w_ffpe = self.weighting(p_ff, p_ffpe)
        p_s = self.inter(p_ff, p_ffpe, w_ff, w_ffpe)
        return torch.cat([self.align_ff(p_ff, p_s), self.align_ffpe(p_ffpe, p_s)], 0)

    def forward(self, b, missing=None):
        """missing is None, 'pathology' or 'omics'. Returns bin logits and pooled features."""
        path = omics = None
        if missing != "pathology":
            path = self.encode_pathology(b)
        if missing != "omics":
            omics = self.omics(b)
        if missing == "pathology":
            path = self.bank.impute(omics.mean(0), "omics")[None]     # one imputed token
        if missing == "omics":
            omics = self.bank.impute(path.mean(0), "pathology")[None]
        p_c = self.p2g(path, omics)
        o_c = self.g2p(omics, path)
        logits = self.ffn(torch.cat([p_c.mean(0), o_c.mean(0)]))
        return logits, path.mean(0), omics.mean(0)


# ------------------------------------------------------------------- loss, eval
def nll_survival_loss(logits, bin_idx, event, eps=1e-7):
    """Discrete time negative log likelihood (Eq. 1) for one patient.
    event is 1.0 when death was observed and 0.0 when censored."""
    h = torch.sigmoid(logits)
    S = torch.cumprod(1 - h, 0)
    S_prev = torch.cat([h.new_ones(1), S])                # S_prev[t] equals S(t-1)
    uncens = -(torch.log(S_prev[bin_idx].clamp(min=eps)) + torch.log(h[bin_idx].clamp(min=eps)))
    cens = -torch.log(S[bin_idx].clamp(min=eps))
    return event * uncens + (1 - event) * cens


def concordance_index(risk, time, event):
    """Fraction of comparable pairs in which the earlier event has the higher risk."""
    num, den = 0.0, 0
    for i in range(len(risk)):
        if event[i] != 1:
            continue
        for j in range(len(risk)):
            if time[j] > time[i]:
                den += 1
                num += 1.0 if risk[i] > risk[j] else (0.5 if risk[i] == risk[j] else 0.0)
    return num / den if den else float("nan")


@torch.no_grad()
def evaluate(model, data, missing=None):
    model.eval()
    risk, time, event = [], [], []
    for b in data:
        logits, _, _ = model(b, missing)
        S = torch.cumprod(1 - torch.sigmoid(logits), 0)
        risk.append(-S.sum().item())                      # larger value means higher risk
        time.append(b["time"])
        event.append(b["event"])
    return concordance_index(risk, time, event)


def train_epoch(model, data, opt):
    """One pass, one patient per step. The memory bank is refreshed once at the end of the epoch."""
    model.train()
    fp, fo = [], []
    for b in data:
        logits, p, o = model(b)
        loss = nll_survival_loss(logits, b["bin"], b["event"])
        opt.zero_grad()
        loss.backward()
        opt.step()
        fp.append(p.detach())
        fo.append(o.detach())
    model.bank.update(torch.stack(fp), torch.stack(fo))


# ------------------------------------------------------------------ smoke test
def make_dummy(n, in_dim, g_sizes, p_sizes, esm_dim, n_prot, n_bins):
    data = []
    for i in range(n):
        t_bin = int(torch.randint(0, n_bins, (1,)))
        data.append(dict(
            ffpe=torch.randn(48, in_dim), ffpe_xy=torch.rand(48, 2) * 100,
            ff=torch.randn(32, in_dim), ff_xy=torch.rand(32, 2) * 100,
            genomic=[torch.randn(s) for s in g_sizes],
            pathway=[torch.randn(s) for s in p_sizes],
            esm=torch.randn(n_prot, esm_dim), expr=torch.randn(n_prot),
            bin=t_bin, time=t_bin + float(torch.rand(1)),
            event=float(i % 3 != 0)))
    return data


if __name__ == "__main__":
    torch.manual_seed(0)
    g_sizes, p_sizes = [5, 4, 6, 3, 7, 5], [3, 4, 5, 6, 3, 4, 5, 6]
    data = make_dummy(8, 64, g_sizes, p_sizes, 16, 10, 4)
    model = M3Surv(64, 32, g_sizes, p_sizes, 16, n_bins=4, bank_k=3)
    opt = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)
    for epoch in range(2):
        train_epoch(model, data, opt)
        print(f"epoch {epoch} full C index {evaluate(model, data):.3f}")
    print("pathology missing", round(evaluate(model, data, "pathology"), 3))
    print("omics missing", round(evaluate(model, data, "omics"), 3))

Conclusion

M3Surv makes a clear case that survival models should be built for the data hospitals actually have. By combining fresh frozen and paraffin slides with three omics layers, it reaches a mean concordance index of 70.7 percent across five TCGA cancers, ahead of PIBD at 68.5 and the authors’ own M2Surv at 70.1. It also keeps working when pathology or omics is gone entirely, scoring 66.9 percent from genomics alone and between 66.6 and 68.5 percent from pathology alone.

The conceptual contribution is the pairing of two ideas. Hypergraphs let the model reason about groups of patches, within a slide and across two slides of the same tumor, and the memory bank lets it reason about groups of patients when a modality is missing. Both replace one to one thinking with group level thinking, and that is a sensible match for biology where tissue and patient subtypes matter more than any single patch or record.

The approach should transfer to other settings where records are incomplete, such as radiology paired with pathology or clinical notes paired with lab panels. The authors name radiology images and clinical notes as next modalities. That remains a hypothesis, since this paper only tests slides and omics in cancer cohorts, and each new setting would need its own validation.

The limits are as important as the gains. Several margins sit inside the fold to fold spread, the missing data curve is not monotonic, the paper reports two different numbers for the omics missing case, and it provides no calibration, time dependent metrics, significance tests or external validation. The one outside cohort has 302 patients and a different cancer type. A model that returns a typical profile for missing inputs needs especially careful presentation to avoid being mistaken for a measurement.

The authors list their next steps as additional modalities, longitudinal monitoring that updates predictions as new data arrives, prospective clinical trials and better efficiency and interpretability. We would add three. Report calibration and time dependent metrics, test on a hospital that never touched the training data, and evaluate with naturally occurring missingness patterns instead of random deletion.

A survival model earns trust not when it predicts well on the data it was built from, but when it says honestly what it does not know. M3Surv takes a useful step toward models that cope with gaps, and the next step is proving that coping is safe.

Frequently asked questions

Related posts