MatterChat Brings Multimodal LLMs to Materials Science

Why a chatbot cannot read a crystal

The properties of a material, whether it conducts electricity, whether it is magnetic, how much energy it takes to form, all trace back to how its atoms are arranged in space. Two materials with the same chemical formula can behave completely differently depending on how those atoms are stacked. This is why a chemical formula alone, or a text description, tells you only part of the story. The three dimensional structure is the story.

Large language models have no access to that. When you ask a model like GPT-4 about a material, you can only hand it words, a formula, a name, maybe a description. The model has read a great deal about materials, so it can often bluff a plausible answer, but it has never perceived the actual geometry, so on quantitative questions it fails badly. The paper shows this vividly. Asked to predict the formation energy of a newly discovered material, the ground truth was about minus 0.098 electron volts per atom. GPT-4o guessed around minus 0.75, Gemini around minus 0.28, and DeepSeek around minus 0.44, all wildly off. These are not stupid models. They simply cannot see what they are being asked about.

Meanwhile, materials scientists already have excellent models that do see structure. Graph neural networks and machine learning interatomic potentials, with names like CHGNet, MACE, and SchNet, represent a crystal as a graph of atoms and bonds and predict properties with real accuracy. But these models cannot hold a conversation. They output a number, not an explanation, and they cannot draw on the vast contextual knowledge, the literature, the synthesis recipes, the domain lore, that a language model carries. Yingheng Tang, Zhi Jackie Yao, and their colleagues saw the obvious opportunity. Marry the two. Let a structure encoder do the seeing and a language model do the talking.

The bridge between seeing and talking

The heart of MatterChat is how it connects these two very different kinds of model. It has three parts. A material processing branch takes the crystal, turns it into a graph, and feeds it through a frozen pretrained structure encoder, either CHGNet or MACE, producing rich embeddings for every atom. A language branch uses a frozen Mistral 7B language model to process the text of your question. And between them sits the piece that makes the whole thing work, a small trainable bridge.

The bridge is inspired by an idea from vision and language models called BLIP-2. It carries a set of learnable query vectors, thirty two of them, whose job is to interrogate the atom embeddings and distill them into a compact form the language model can understand. The queries pass through alternating layers of attention. In the cross attention layers they reach into the atom embeddings and pull out the relevant structural features, and in the self attention layers they refine that information among themselves. A final projection maps the result into the language model’s own embedding space, so the structure arrives as if it were a handful of extra words the language model can read.

The elegance is that only the bridge is trained. The large structure encoder and the large language model stay frozen, carrying all the knowledge they learned during their own expensive pretraining. This is what the authors mean by a plug and play design. Training is cheap because you are only fitting a small module, and you can swap the encoder or the language model without rebuilding everything. It also means the system stands on the shoulders of two foundation models rather than trying to relearn materials science and language from scratch.

MatterChat achieves strong performance by training only a lightweight bridge model, efficiently aligning material structure representations with textual understanding and maintaining high accuracy across diverse materials science tasks. Tang and colleagues, on the value of freezing the big models

How the bridge is taught

Getting a structure encoder and a language model to speak the same language takes a careful training recipe, and MatterChat uses two stages. The first stage aligns structures with text using three complementary objectives. A contrastive objective pulls the embedding of a structure toward the embedding of its matching text description and pushes it away from mismatched ones. A generative objective teaches the bridge to help produce a correct text description from a structure. And an association objective learns to judge whether a given structure and text actually belong together.

That first loss is the contrastive one, where \(q_i\) is a structure embedding, \(t_i\) is its matching text, and \(\tau\) controls how sharply the model separates matches from mismatches. The three losses together give the bridge a foundational sense of how structure and language correspond. The second stage then fine tunes the whole assembly on twelve materials tasks at once, three that describe a material, such as naming its chemical formula or space group, and nine that predict properties, such as whether it is metallic, its bandgap, or its formation energy.

The training set is substantial, about 142,899 crystal structures drawn from the Materials Project, spanning the periodic table up to plutonium. Because the bridge is the only trainable part, the whole thing trained on four graphics processors across eight nodes in about forty eight hours, modest by the standards of training a language model from scratch.

What the numbers show

The evaluation is where the approach proves itself, and it does so on two fronts, against general chatbots and against specialized physics models. Against the chatbots there is no contest on quantitative tasks. Pure language models cannot produce reliable numbers at all, which is why the authors had to exclude them from the regression comparisons. The formation energy example above tells the story. MatterChat predicted about minus 0.042 against a ground truth of minus 0.098, in the right neighborhood, while the chatbots were off by factors of several.

The more demanding comparison is against the specialized physics models, and here MatterChat holds its own and often wins. On classification tasks, deciding whether a material is metallic, stable, magnetic, or what kind of magnetic order it has, it consistently beat SchNet, CHGNet, and MACE. The margin was widest on magnetic order, where it reached an accuracy around 0.87 against roughly 0.68 for CHGNet. On the numeric regression tasks it posted the lowest error of any model tested.

The formation energy result is the one to dwell on. At 0.088 electron volts per atom, MatterChat’s error was less than half that of the strong physics baselines near 0.19. That is a large margin in a field where these models are already carefully tuned, and it suggests that giving the structure encoder a language model to reason alongside is not just a convenience for chatting but actually sharpens the prediction. The authors read this as evidence that combining graph based structure with language reasoning gives a more complete picture of a material than either alone.

Reasoning, retrieval, and a working memory

Beyond single property predictions, MatterChat can reason through more involved tasks, and this is where borrowing a full language model pays off. Shown a structure, it can generate a step by step synthesis procedure. For yttrium iron garnet it laid out a solid state reaction recipe with the right starting oxides, mixing ratios, and firing temperatures. For gallium nitride it produced a chemical vapour deposition protocol naming the correct precursors and a temperature window that matches established practice. It can also explain itself, for instance reasoning about why a particular crystal phase of silicon is unstable under normal conditions.

To keep this reasoning grounded, the authors add two mechanisms. One is a working memory scheme, where the frozen encoder extracts hard structural facts, formula, space group, crystal system, into a persistent context that the language model must anchor its answer to, so it is less likely to drift into invention. The other is retrieval augmented generation. Before answering, the system finds the two most similar structures in its training set by comparing embeddings, and uses them to sharpen the response. This retrieval step cut the numeric error by about twelve percent and nudged classification accuracy up, at the cost of only a tiny increase in latency, roughly twelve milliseconds per query.

The retrieval also speaks to a genuine worry with any language model, that it will hallucinate. By pulling in real neighboring examples and forcing the answer to reckon with them, retrieval gives the model a factual anchor. The authors are careful to present this as a partial fix rather than a solution, which is the honest framing, since the underlying language model can still let its text priors override the structural evidence.

Reproducing the bridge

The full system needs two frozen foundation models and a large materials dataset, but the core innovation, the bridge that distills a variable set of atom embeddings into a fixed set of tokens aligned with text, is compact and reproducible. The implementation below writes that bridge. It builds learnable query vectors that alternate cross attention over atom embeddings with self attention among themselves, trains them with a contrastive objective to align structures with their text, and checks that a structure can retrieve its correct description. A runnable smoke test shows the bridge learning to match structures to text far above chance, the same alignment MatterChat relies on.

# Bridge module that aligns material structure with text, the core of
# MatterChat from Tang, Xu, et al., "A multimodal large language model
# for materials science" (Nature Machine Intelligence 2026). Learnable
# queries interrogate frozen atom embeddings through alternating cross
# and self attention, in the style of BLIP-2. Real atom embeddings and
# the language model are replaced by toy tensors here.

import torch
import torch.nn as nn
import torch.nn.functional as F

D = 64            # embedding dimension
NQ = 8            # number of learnable query tokens
NCLASS = 12       # toy material families


class CrossAttn(nn.Module):
    """Queries attend into the frozen atom embeddings."""
    def __init__(self, d, heads=4):
        super().__init__()
        self.mha = nn.MultiheadAttention(d, heads, batch_first=True)
        self.ln = nn.LayerNorm(d)

    def forward(self, q, kv):
        a, _ = self.mha(q, kv, kv)
        return self.ln(q + a)


class SelfAttn(nn.Module):
    """Queries exchange information among themselves, plus a feedforward."""
    def __init__(self, d, heads=4):
        super().__init__()
        self.mha = nn.MultiheadAttention(d, heads, batch_first=True)
        self.ln = nn.LayerNorm(d)
        self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        self.ln2 = nn.LayerNorm(d)

    def forward(self, q):
        a, _ = self.mha(q, q, q)
        q = self.ln(q + a)
        return self.ln2(q + self.ff(q))


class Bridge(nn.Module):
    """Distill a variable set of atom embeddings into fixed query tokens."""
    def __init__(self, d=D, nq=NQ, layers=4):
        super().__init__()
        self.queries = nn.Parameter(torch.randn(1, nq, d) * 0.02)
        self.cross = nn.ModuleList([CrossAttn(d) for _ in range(layers)])
        self.selfa = nn.ModuleList([SelfAttn(d) for _ in range(layers)])
        self.proj = nn.Linear(d, d)          # into the LLM embedding space

    def forward(self, atom_emb):
        q = self.queries.expand(atom_emb.shape[0], -1, -1)
        for cross, selfa in zip(self.cross, self.selfa):
            q = cross(q, atom_emb)             # pull structural features
            q = selfa(q)                       # refine among queries
        return F.normalize(self.proj(q).mean(1), dim=-1)


def contrastive(struct_emb, text_emb, tau=0.1):
    """Align each structure with its matching text, push away the rest."""
    logits = struct_emb @ text_emb.t() / tau
    labels = torch.arange(struct_emb.shape[0])
    return 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.t(), labels))


def smoke_test():
    """Train the bridge, then check a structure retrieves the right text."""
    torch.manual_seed(0)
    proto_struct = torch.randn(NCLASS, D)      # family structure prototypes
    proto_text = torch.randn(NCLASS, D)        # family text prototypes
    text_head = nn.Linear(D, D)
    bridge = Bridge()

    def batch(bs):
        cls = torch.randint(0, NCLASS, (bs,))
        mats = [proto_struct[c] + 0.6 * torch.randn(int(torch.randint(4, 16, (1,))), D)
                for c in cls]
        n = max(m.shape[0] for m in mats)
        atoms = torch.zeros(bs, n, D)
        for i, m in enumerate(mats):
            atoms[i, :m.shape[0]] = m
        txt = F.normalize(text_head(proto_text[cls] + 0.3 * torch.randn(bs, D)), dim=-1)
        return atoms, txt, cls

    opt = torch.optim.Adam(list(bridge.parameters()) + list(text_head.parameters()), 1e-3)
    for _ in range(600):
        atoms, txt, _ = batch(48)
        loss = contrastive(bridge(atoms), txt)
        opt.zero_grad(); loss.backward(); opt.step()

    bridge.eval()
    with torch.no_grad():
        atoms, _, cls = batch(300)
        struct = bridge(atoms)
        proto_t = F.normalize(text_head(proto_text), dim=-1)
        retrieved = (struct @ proto_t.t()).argmax(1)
        acc = (retrieved == cls).float().mean().item()

    chance = 1.0 / NCLASS
    assert acc > 0.7, "the bridge should align structure with text"
    print("structure to text retrieval accuracy {:.3f}  (chance {:.3f})".format(acc, chance))
    print("the bridge aligned material structure with language, smoke test passed")


if __name__ == "__main__":
    smoke_test()

The honest note is that this toy aligns structure with text on invented families, whereas MatterChat aligns real atom embeddings from a pretrained encoder with the embedding space of a real language model, then fine tunes on a dozen materials tasks. The mechanism is faithful, though. Learnable queries distill a variable sized structure into a fixed set of tokens through alternating cross and self attention, and a contrastive objective teaches them to line up with language, which is exactly what lets the frozen language model read a crystal.

Honest limitations

The authors are unusually forthright about what MatterChat does not yet achieve, and they devote a section to it. The deepest concern is about what the model has really learned. Its success on property tasks may reflect learned correlations rather than a genuine grasp of structural meaning, which limits how far it can be trusted on compositional reasoning about structures it has not seen. Closing that gap, they say, will need training objectives that align representations at a deeper level, not just at the level of getting answers right.

A second limit is in the kind of interaction it supports. The current model was trained on single turn question and answer pairs, so it lacks the multistep reasoning and back and forth cross referencing that a real expert conversation demands. Moving to genuine multiturn, multimodal dialogue, where a scientist can probe and follow up, is future work. There is also a specific numeric weakness the authors name plainly. The model reliably retrieves discrete facts like the number of atoms in a cell, but shows a resolution gap on continuous quantities like volume and density, a known difficulty for language models doing high precision regression.

The third limit is the one that should temper any excitement about a chatbot for science, hallucination. Because the language model is frozen and carries strong text priors, it can still generate confident text that overrides the structural evidence in front of it. Retrieval and the working memory scheme help anchor the model, but the authors are clear that these are initial mitigations rather than a cure, and that fact checking and self revision mechanisms are needed before such a system could be trusted unsupervised. That candor is the right note, because a fluent wrong answer about a material is more dangerous than an obvious one.

What this changes for materials discovery

The practical promise is a materials assistant that can both compute and converse. A researcher could show it a candidate crystal and ask, in ordinary language, whether it is stable, what its bandgap is, and how one might synthesize it, and get answers grounded in the actual structure rather than in a formula the model half remembers from the literature. The plug and play design means the field’s steadily improving structure encoders and language models can be slotted in as they get better, without rebuilding the system each time.

The broader idea is a template for scientific multimodal models. The trick of freezing two powerful foundation models and training only a small bridge to align them is not special to materials. It is the same instinct behind multimodal systems in other fields, and MatterChat is a clean instance of it for atomic structure and language. For readers tracing that thread, it sits alongside other work on fusing modalities and stretching language models, from how state space models reshape multimodal fusion to a vision language action model for robots and the practical question of how we even benchmark prompt driven models. You can browse the wider set through the multimodal AI pillar.

Conclusion

The core achievement of MatterChat is a multimodal model that lets a language model reason about the real atomic structure of a material, joining a frozen structure encoder to a frozen language model through a small trainable bridge. It beat general chatbots decisively on quantitative materials questions they cannot really attempt, and it matched or outperformed specialized physics models on both classification and numeric prediction, cutting formation energy error to less than half that of the strong baselines.

The conceptual shift is toward giving language models eyes for structure rather than asking them to reason from text alone. By distilling a crystal into a handful of tokens the language model can read, and training only the bridge that does the distilling, the design captures what the structure encoder sees and what the language model knows in one system, cheaply and modularly. That the combination sharpened the numeric predictions, rather than merely wrapping a chat interface around them, is the result that makes the approach more than a convenience.

The ideas extend well beyond crystals. The same freeze two foundation models and train a bridge recipe applies wherever a specialized encoder sees something a language model cannot, from molecules to proteins to sensor data, and the retrieval and working memory additions carry over as ways to keep such systems honest. The general lesson, that a small alignment module can unlock the combined power of two large pretrained models, is one the wider field can reuse.

The limitations keep it grounded. The model may be learning correlations rather than deep structural meaning, it handles only single turn questions, it stumbles on high precision continuous quantities, and its frozen language model can still hallucinate against the structural evidence. The authors name all of this plainly and point toward deeper alignment, multiturn dialogue, and stronger reliability safeguards as the road ahead.

What lingers is how naturally the two halves fit. Materials science has spent years building models that see structure and, separately, models that command language, and the missing piece was simply a way to introduce them. Teach a small bridge to translate a crystal into words a language model can read, and the machine can suddenly both calculate a material’s properties and talk you through them. For a field trying to accelerate the discovery of new materials, a tool that computes and explains in the same breath is a genuinely useful thing to have.

Frequently asked questions