XPert: A Transformer That Predicts How Drugs Change Cells

What a drug does to a cell, in numbers

When a drug enters a cell, one of the clearest signs of its effect is which genes it turns up and which it turns down. That pattern of gene activity, the transcriptional response, is a fingerprint of the drug’s mechanism, and reading it tells researchers whether a compound hits the target they intended and what else it disturbs along the way. Mapping these fingerprints across many drugs and cell types is one of the engines of modern drug discovery.

The problem is that measuring them exhaustively is out of reach. The number of combinations of drug, dose, timing, and cell type explodes far beyond what any lab can test, and the measurements that matter most, from real patients rather than cell lines, are the scarcest of all. This is exactly the kind of gap where a good predictive model earns its keep, filling in the vast unmeasured space from the slice that has been measured. Our overview of how AI is reshaping the drug discovery pipeline sets the wider stage this work sits in.

The trouble is that the leading approaches had a habit of blurring the very signal they were meant to capture. Many relied on variational autoencoders, a kind of model that compresses data through a bottleneck, and that compression tends to smooth away the fine, cell specific detail that makes a response biologically meaningful. The authors call this over denoising, and much of XPert’s design is a reaction against it.

Two branches, one clean idea

The central design choice is to split the problem in two. A cell that receives a drug is really two things layered on top of each other, the cell’s own intrinsic state before anything happens, and the changes the drug then imposes. Tangle those together and a model struggles to tell whether a gene is active because of the cell’s nature or because of the drug. XPert keeps them apart with two separate branches.

The first branch, the base encoder, reads the untreated cell. Using self attention, the same mechanism behind large language models, it learns how genes relate to one another in that cell, building a picture of its baseline biology. The second branch, the perturbation encoder, uses cross attention to let the cell state and the drug interact, learning how this particular drug reshapes this particular cell. Reading a cell as a sentence of gene tokens, with a special summary token standing for the whole cell, is a trick borrowed directly from language modelling.

The payoff of the split shows up in the model’s key output. Rather than only predicting the treated cell’s gene activity, XPert predicts the difference between the treated and untreated states directly, subtracting one branch from the other. That difference, the change in expression, is the biologically interesting quantity and the hardest to get right, because it is a subtle shift riding on top of a large baseline.

To make sure the model captures the shape of the response and not just its rough magnitude, the training objective blends two goals, a squared error term that pins down the size of each change and a correlation term that rewards getting the overall pattern right across genes.

Teaching the model biology it cannot see

A drug is more than its shape. Two compounds can look chemically similar and behave very differently in a cell, and two that look different can share a mechanism. To give the model that biological sense, XPert does not feed it only the drug’s structure. It also builds a knowledge graph that links drugs to the proteins they target, to other drugs with similar effects, and to the web of interactions among proteins, and it learns drug representations from that graph before the main training even begins.

This is the biologically informed part of the name, and it rests on two simple intuitions. Drugs that act within the same protein network tend to produce similar responses, and structurally similar drugs often yield comparable effects. By pretraining on this graph, XPert places each drug in a space organized by biological effect rather than by chemistry alone, so that drugs sharing a mechanism sit near each other even when their molecules look unalike. The chemical structure itself comes in through a separate 3D molecular model, and dose and time enter as their own tokens rather than crude numbers, letting the model reason about how much and how long. The idea of learning from a graph of biological relationships echoes our look at mapping gene regulatory networks, where the connections carry as much meaning as the nodes.

What the benchmarks show

The real test of a perturbation model is not how well it fits data it has seen, but how well it generalizes to the unseen, and the authors stress the hardest version of that. They split the data three ways, a random split, a split that hides whole drugs from training, and a split that hides whole cell types. The last is the cruel one, because a cell type the model never met responds in ways it has no direct experience of.

The gap in the cold cell setting is the headline. Predicting responses in an unseen cell type is where the autoencoder based models fell apart, in some cases producing correlations that looked fine while their actual fit went negative, a sign they had captured the general shape of a response but not its true magnitude. XPert, by keeping the cell’s biology in its own branch, held up where they collapsed, and it was one of the few models to avoid that negative fit entirely. The authors trace the failure of the older models straight back to the denoising bottleneck that erases cell specific detail.

XPert also does something the single snapshot models cannot. By modelling dose and time, it can trace how a drug’s effect grows and shifts. In a case study on vorinostat, a cancer drug, it reconstructed smooth response surfaces showing how specific genes respond as the dose climbs and the hours pass, even reversing the direction of some genes at higher doses. That dynamic view is closer to real pharmacology, where timing and amount are everything.

XPert overcomes the over denoising issues inherent in dominant variational autoencoder based approaches, achieving substantially higher correlation and lower error in cold cell generalization, and provides mechanistic interpretability as evidenced by the identification of clinically validated resistance biomarkers. Guo and colleagues, Nature Machine Intelligence, 2026

From lab cells to patients

The most ambitious part of the work tries to cross the widest gap in the field, the one between tidy lab cell lines and messy real patients. Clinical data is precious and scarce, far too little to train a large model on its own. So the authors pretrained XPert on the enormous library of lab measurements, then fine tuned it on a small clinical dataset of patients whose gene activity was measured before and after treatment.

Transferring knowledge from lab screens improved the patient predictions across the board, which suggests the model learned something about drug biology general enough to carry from a dish to a person. More striking, XPert’s attention mechanism let the authors read out which genes drove a drug’s failure. In patients who did not respond to a breast cancer therapy, the model highlighted a gene already known in the literature to confer resistance, along with several others that a simple expression analysis would have missed. A model that not only predicts but points at the mechanism is far more useful to a biologist than a black box.

Clinical translation gap

This is where excitement has to yield to care, because the distance between an impressive benchmark and a tool an oncologist could trust is enormous. The clinical results, encouraging as they are, rest on a small dataset of a few hundred paired patient profiles across a handful of cancers and drugs. Improving a correlation on that data is a research milestone, not evidence that the model can guide a real treatment decision for a real person.

The gap is filled with hard questions the study does not answer. The model predicts a pattern of gene activity, not whether a patient will actually live longer or feel better, and the link between a predicted transcriptional response and a genuine clinical outcome is itself an active area of research. The clinical data came from one platform and a narrow set of conditions, and how the model behaves on the full diversity of patients, tumours, and drug regimens seen in a clinic is untested. The resistance genes it flagged are biologically plausible and some are known, but flagging a plausible gene is a hypothesis for a lab to check, not a validated diagnostic.

Turning this into something clinical would demand far more, prospective studies on many patients, validation against real outcomes rather than gene signatures, and the regulatory scrutiny any tool influencing treatment must face. The paper is honest that its clinical work is a proof of concept for transfer learning, a promising bridge rather than a finished crossing, and it should be read that way.

Where the method falls short

Beyond the clinical gap, the authors name several honest limits.

The model is computationally heavy. Handling thousands of gene tokens with attention is expensive, and the authors point to more efficient training methods and architectures as necessary future work if the approach is to scale to larger gene sets or to single cell resolution, where each measurement is one cell rather than an average of many. The current work predicts the average response of a population of cells, which smooths over the variation between individual cells that can matter a great deal.

The scope is also bounded to small molecule drugs and to the roughly one thousand landmark genes that stand in for the whole transcriptome. Extending it to other kinds of therapy, to the full genome, or to genetic and multi modal perturbations is future work that depends on data that does not yet exist at scale. And the biological knowledge graph, powerful as it is, inherits the gaps and biases of the databases it is built from, since only a small fraction of drug and protein interactions are actually known.

There is a subtler caveat too. XPert was tested on an independent lab dataset and showed it could resist the negative transfer that hurts naive models, which is reassuring, but every result still rests on the assumption that the patterns learned from measured data generalize faithfully to the unmeasured. That assumption holds well in the tests shown and could fail in corners of biology the benchmarks do not probe. A confident prediction from a model is a lead worth following, not a fact.

Why it matters

Step back and the contribution is a cleaner way to model what drugs do. By refusing to blur the cell into the drug, XPert recovers the cell specific detail that earlier models washed out, and it generalizes to the unseen cell types that are exactly where a predictive tool is most valuable, because those are the cases you have not measured. Paired with dose and time modelling, it moves closer to the dynamic, quantitative picture real pharmacology needs.

The broader significance is the bridge it builds. A model that can carry knowledge from cheap, plentiful lab screens toward scarce, precious patient data is chasing one of the central dreams of precision medicine, predicting in advance how a specific patient will respond to a specific drug. This work does not arrive at that dream, but it lays a plausible plank toward it, and it does so with a model whose attention you can inspect to ask why. As computational tools take on more of the early work of drug discovery, from the survival models in our coverage of graph attention for cancer survival to generative design in our look at optimizing peptide antibiotics, the ones that stay interpretable and honest about their limits are the ones worth building on.

A reference implementation

The code below is a compact, runnable version of the core architecture. It builds the two branches, a base encoder that runs self attention over gene tokens and a perturbation encoder that cross attends the cell to a drug representation, then predicts both the treated expression and the change from baseline, trained with the combined error and correlation loss. The authors full code and models are linked under the block.

# XPert style dual branch transformer, compact reference
# A base encoder reads the untreated cell, a perturbation encoder cross attends
# the cell to a drug, and the model predicts the change in gene expression.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)


class SelfBlock(nn.Module):
    def __init__(self, dim, heads=4):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm1, self.norm2 = nn.LayerNorm(dim), nn.LayerNorm(dim)
        self.ff = nn.Sequential(nn.Linear(dim, dim * 2), nn.GELU(), nn.Linear(dim * 2, dim))

    def forward(self, x):
        a, _ = self.attn(x, x, x)
        x = self.norm1(x + a)
        return self.norm2(x + self.ff(x))


class CrossBlock(nn.Module):
    # cell tokens are the query, drug tokens are the key and value
    def __init__(self, dim, heads=4):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm = nn.LayerNorm(dim)

    def forward(self, cell, drug):
        a, _ = self.attn(cell, drug, drug)
        return self.norm(cell + a)


class XPert(nn.Module):
    def __init__(self, n_genes, n_bins, dim=64, n_drug_tokens=8, depth=2):
        super().__init__()
        self.gene_emb = nn.Embedding(n_genes, dim)          # gene identity
        self.expr_emb = nn.Embedding(n_bins, dim)           # binned expression value
        self.cls = nn.Parameter(torch.randn(1, 1, dim))     # summary token for the cell
        self.drug_proj = nn.Linear(dim, dim)

        self.base = nn.ModuleList([SelfBlock(dim) for _ in range(depth)])
        self.cross = nn.ModuleList([CrossBlock(dim) for _ in range(depth)])
        self.pert_self = nn.ModuleList([SelfBlock(dim) for _ in range(depth)])

        self.head_pert = nn.Linear(dim, 1)                  # treated expression per gene
        self.head_deg = nn.Linear(dim, 1)                   # change from baseline per gene

    def encode_cell(self, gene_ids, expr_bins):
        b = gene_ids.shape[0]
        tok = self.gene_emb(gene_ids) + self.expr_emb(expr_bins)
        cls = self.cls.expand(b, -1, -1)
        return torch.cat([cls, tok], dim=1)                 # [b, 1 + genes, dim]

    def forward(self, gene_ids, expr_bins, drug_tokens):
        cell = self.encode_cell(gene_ids, expr_bins)
        base = cell
        for blk in self.base:
            base = blk(base)                                # unperturbed cell state

        drug = self.drug_proj(drug_tokens)
        pert = base
        for c, s in zip(self.cross, self.pert_self):
            pert = s(c(pert, drug))                         # drug reshapes the cell

        gene_base = base[:, 1:, :]                          # drop the summary token
        gene_pert = pert[:, 1:, :]
        x_pert = self.head_pert(gene_pert).squeeze(-1)
        x_deg = self.head_deg(gene_pert - gene_base).squeeze(-1)   # the branch difference
        return x_pert, x_deg


def pcc(a, b):
    a = a - a.mean(dim=-1, keepdim=True)
    b = b - b.mean(dim=-1, keepdim=True)
    return ((a * b).sum(-1) / (a.norm(dim=-1) * b.norm(dim=-1) + 1e-8)).mean()


if __name__ == "__main__":
    n_genes, n_bins, batch = 200, 10, 8
    model = XPert(n_genes, n_bins)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)

    gene_ids = torch.arange(n_genes).unsqueeze(0).expand(batch, -1)
    expr_bins = torch.randint(0, n_bins, (batch, n_genes))
    drug = torch.randn(batch, 8, 64)                        # tokenized drug
    true_deg = torch.randn(batch, n_genes)                  # target change in expression

    for step in range(200):
        _, x_deg = model(gene_ids, expr_bins, drug)
        loss = F.mse_loss(x_deg, true_deg) + (1.0 - pcc(x_deg, true_deg))
        opt.zero_grad()
        loss.backward()
        opt.step()
        if (step + 1) % 50 == 0:
            print(f"step {step + 1}  loss {loss.item():.4f}")

Conclusion

The core achievement here is a model that predicts what a drug does to a cell without smearing the answer. By encoding the untreated cell and the drug perturbation in two separate branches and predicting the difference between them, XPert recovers the cell specific detail that autoencoder based models washed away, and it generalizes to unseen cell types far better than its predecessors, cutting error sharply in exactly the setting that matters most for discovery.

The conceptual shift worth remembering is the refusal to blend the cell and the drug into one representation. That separation, paired with a knowledge graph that gives each drug a biological rather than merely chemical identity, and with dose and time modelled explicitly, produces a picture of drug action that is both more accurate and more faithful to how pharmacology actually works. The fact that the model’s attention can be read to name resistance genes turns it from a predictor into a source of hypotheses.

The approach travels because its ideas are general. The dual branch design, the biological knowledge graph, and the transfer from plentiful lab data to scarce clinical data are not tied to one dataset, and paired with an open release they offer a template others can build on. As drug discovery leans harder on computation, a model that stays interpretable and generalizes to the unmeasured is a genuinely useful addition to the toolkit.

The honest limits keep it grounded. The model is heavy to run, it predicts population averages rather than single cells, it covers small molecules and a landmark gene set rather than the whole picture, and, most importantly, its clinical results are a proof of concept on a small dataset rather than a validated tool. Predicting a gene expression pattern is not the same as predicting whether a patient will benefit, and that leap runs through prospective validation this work does not attempt.

Future directions follow naturally. Push the model toward single cell resolution and the full genome, make it lighter so it can scale, and, above all, test whether its predictions track real patient outcomes rather than gene signatures. If those steps hold up, the field moves closer to a day when the educated guess an oncologist makes before prescribing is backed by a model that has, in effect, run the experiment first.

Frequently asked questions