How RO LMM Plans Breast Cancer Radiotherapy End to End

The problem with chaining AI tasks in a clinical workflow

Foundation models trained the way large language models are trained have gotten good at a strange range of things at once, reasoning across images and text, summarizing dense documents, and even segmenting regions in an image when guided by a written description. Medicine is an obvious place to point that versatility, because a real clinical decision rarely rests on one type of data. A radiation oncologist planning breast cancer treatment reads an MRI report, an ultrasound report, and a pathology report, synthesizes them into a clinical note, decides on a radiotherapy strategy such as whole breast irradiation with a particular dose and fractionation schedule, and then has to translate that written plan into a precise three dimensional contour on a CT scan showing exactly which tissue the radiation beam should target.

The paper’s introduction points out a real structural gap here. Over 500 AI models already carry FDA approval for medical use, but the overwhelming majority are narrow tools built for one task with one type of input, an image classifier here, a report generator there. None of them cover the whole sequence a radiation oncologist actually walks through. The authors set out to build something that does, and they call it RO LMM, short for Radiation Oncology Large Multimodal Model.

Why stitching three models together is harder than it sounds

The obvious approach is to train three separate models, one per task, and feed the output of each into the next. Summarize the reports into a note, feed the note into a strategy suggester, feed the strategy into a segmentation network. The authors tried exactly this and ran into a problem that will sound familiar to anyone who has built a multi stage pipeline of any kind. Each stage introduces its own small errors and stylistic quirks, and those imperfections do not stay contained, they get passed forward and can compound by the time you reach the final output. A segmentation network trained only on clean, human written treatment plans has never seen the slightly off phrasing or occasional inaccuracy that a language model generated plan will actually contain, so at test time it receives an input distribution quite different from what it trained on, and its accuracy can suffer as a result.

This mismatch between training time and inference time inputs, sometimes called exposure bias in the language modeling literature, is really the central problem the paper is trying to solve. Their answer is not a bigger model or more data, it is a training technique that deliberately teaches each stage to treat noisy, imperfect input the same way it treats clean input.

Three experts, one workflow

RO LMM is really three specialized models trained for three sequential tasks, all built on a LLaMA2 7B chat backbone for the text stages. RO-LMM-S, the summary expert, reads raw MRI, ultrasound, and pathology reports and writes a concise clinical note in the style a radiation oncologist would produce. RO-LMM-P, the plan expert, reads that clinical note and proposes a radiotherapy strategy, specifying things like which side to treat, whether the aim is definitive treatment after breast conserving surgery or postoperative treatment after mastectomy, how far the radiation field should extend, and the dose and fractionation schedule. RO-LMM-SEG, the segmentation expert, takes the proposed radiotherapy strategy together with a 3D CT scan and outputs a segmentation mask marking the target volume for treatment.

Each of these building blocks already existed in some form in prior work, and the authors are upfront that RO LMM extends their own earlier LLMSeg model, which focused only on the multimodal segmentation piece. What is new here is stitching the full pipeline together end to end and dealing honestly with the error accumulation that stitching creates.

Why the training set for the plan expert does not quite match what it sees at test time

Here is a detail the paper is refreshingly candid about, since it is exactly the kind of mismatch that causes real production headaches. During training, the plan expert learns from human written clinical notes paired with human written radiotherapy strategies, both collected from real patient records. That makes sense, because generating plan expert training notes with the summary model would be expensive and would bake in whatever errors the summary model happens to make. But at inference time, in the actual end to end pipeline, the plan expert does not receive a human written note, it receives the note generated a moment earlier by the summary expert, which will inevitably differ in wording and occasionally in accuracy from what a human clinician would write. The plan expert has never seen this style of input during training, so its performance on the real deployed pipeline can degrade relative to its performance in isolated benchmark testing.

How CEFTune actually works

The starting point is a technique called Noisy Embedding Fine Tuning, or NEFTune, borrowed from prior language modeling research. NEFTune trains a model by adding a small amount of uniform random noise directly to the token embeddings during fine tuning, which in earlier work was shown to reduce overfitting to the exact wording and formatting of a training set and to improve how well a fine tuned model generalizes.

Here L is the token length, C is the embedding dimension, and alpha is a tunable scale controlling how much noise gets injected. The authors reasoned that NEFTune alone might be a reasonable first attempt at bridging the gap between clean training notes and generated inference notes, since injecting noise during training at least exposes the model to perturbed embeddings. But they found a subtlety. The generated notes the plan expert actually receives at inference are not uniformly noisy, some of them happen to land very close to what a human would have written, and others deviate more substantially. A model trained only to be robust to noise, without any explicit signal about what the correct answer should be regardless of how noisy the input got, does not necessarily preserve accuracy on the inputs that were already close to clean.

Their fix, Consistency Embedding Fine Tuning, adds a second loss term on top of the standard NEFTune objective. It takes the model’s output when given a clean embedding, and separately when given a noise perturbed version of that same embedding, and penalizes the difference between the two outputs after detokenizing them back into sentences.

The consistency term itself is measured in sentence embedding space rather than raw token embedding space, using a variant of SentenceBERT trained on PubMed text, called PubMedBERT. The authors explain this choice plainly, arguing that minimizing distance directly in the token embedding space would mostly enforce robustness without necessarily preserving semantic meaning, whereas comparing detokenized sentences in a pretrained sentence embedding space more directly captures whether the noisy and clean versions of the output actually say the same clinical thing. The distance metric used is one minus cosine similarity between the two sentence embeddings.

Extending consistency into the segmentation network

The same fix gets applied a second time, one stage later in the pipeline, where the segmentation model faces essentially the same clean versus noisy input problem. The segmentation network trains on ground truth radiotherapy strategies but at inference receives the plan expert’s generated strategy text as its conditioning input. The authors first try a direct analogue of NEFTune for this setting, injecting noise into the text embedding before it conditions the segmentation network, and they call this variant NESEG, corresponding to the model RO-LMM-SEG+.

Then they add the consistency piece, calling the result CESEG, corresponding to RO-LMM-SEG++. Instead of regularizing full generated sentences, CESEG regularizes the learnable text prompt embeddings used to condition the segmentation network, comparing the prompt embedding produced from a noisy input against the one produced from the clean ground truth input using cosine similarity.

Architecturally, the segmentation model pairs a 3D Residual U-Net image encoder and decoder with a frozen large language model that produces text embeddings from the radiotherapy strategy, and a multimodal alignment module borrowing the interactive image to text and text to image attention idea from the Segment Anything Model. Only a set of learnable text prompts and a handful of linear layers get trained on the text side, while the language model itself stays frozen, which keeps the segmentation training comparatively lightweight even though the whole system is large.

Does any of this actually help, and by how much

The evaluation is thorough enough to be worth walking through in some detail, since the headline claim, that consistency regularization solves the error accumulation problem, needs numbers to back it up rather than just a plausible mechanism.

Clinical report summarization

Trained on 5674 internal patients from Yonsei Cancer Center and tested on both a 121 patient internal validation set and an 81 patient external validation set from a second hospital, Yongin Severance Hospital, RO-LMM-S posted a Rouge 1 score of 0.783 internally and 0.788 externally. That comfortably beat every open medical language model tested, including MedAlpaca, ChatDoctor, Asclepius, and PMC-LLaMA, and it also edged out ChatGPT using few shot in context learning, which scored 0.657 internally and 0.651 externally. Two board certified clinical experts additionally scored a sample of summaries against a rubric covering relevance, formatting, length, and consistency, and RO-LMM-S scored roughly triple the total points of ChatGPT and vanilla LLaMA2 on both the internal and external samples, with inter clinician agreement measured above 0.85 internally and above 0.95 externally, which is a meaningfully strong correlation for a subjective clinical rubric.

Radiotherapy strategy suggestion, where the consistency idea earns its keep

This is the task where the paper’s central claim gets tested most directly, because it is the one caught in the middle of the error accumulation chain, receiving generated notes rather than ground truth ones. The plain fine tuned RO-LMM-P scored a Rouge 1 of 0.634 internally and 0.591 externally. Adding NEFTune alone, giving RO-LMM-P+, actually made the external score slightly worse, dropping to 0.588, a small but telling failure that supports the authors’ argument that naive noise injection is not a reliable fix on its own. Adding the full CEFTune approach, giving RO-LMM-P++, pushed the external score up to 0.615, the best of any variant, while also improving the internal score to 0.655. Compared against GPT-4.0 using few shot prompting, which scored only 0.356 internally and 0.316 externally on this task, the domain specific fine tuned models were not close competitors, they were in an entirely different tier of performance.

Clinical expert scoring against a five criterion rubric, covering laterality, surgical aim, treatment scope, dose scheme, and hallucination, told the same story. On the external validation set, RO-LMM-P++ reached a total expert score of 46.2 out of a possible 50 from one clinician, compared with 32.3 for GPT-4.0, and correlation between the two human clinicians came out above 0.7, with correlation between the clinicians and GPT-4 acting as an automated judge above 0.4, which the authors present as evidence their rubric based evaluation approach is measuring something clinically meaningful rather than noise.

3D target volume segmentation

For the final stage, RO-LMM-SEG++ with the full CESEG training reached a Dice score of 0.838 internally and 0.761 externally, both notably ahead of the plain unimodal 3D Residual U-Net baseline, which scored 0.802 internally and only 0.689 externally, a gap that widens once you leave the training hospital’s data distribution. Boundary accuracy told an even sharper story. The 95th percentile Hausdorff distance, a measure of the worst case boundary error in millimeters, dropped from 45.852 millimeters for the unimodal baseline down to 13.658 millimeters for RO-LMM-SEG++ internally, a more than threefold tightening of the worst case error.

Two transformer and state space model based segmentation architectures, 3D SegMamba and 3D UNETR, actually performed considerably worse than the simpler 3D Residual U-Net in this setting, with Dice scores around 0.56 to 0.58. The authors attribute this to memory constraints. Those architectures needed smaller input crops to fit in GPU memory, and for a task like radiotherapy target segmentation, where the network genuinely needs to see and align information across the entire breast volume at once, that forced reduction in input size turned out to hurt more than a fancier architecture helped.

Breaking results down by patient subgroup revealed one of the more genuinely striking findings in the paper. Patients were split into a typical group who had breast conserving surgery and a rarer atypical group who underwent total mastectomy. In the atypical, lower prevalence subgroup on the external validation set, the multimodal RO-LMM-SEG++ approach improved external Dice by roughly 22 percentage points over the unimodal baseline, jumping from 0.403 to 0.624, compared with a smaller though still solid single digit improvement in the more common typical subgroup. The authors read this as evidence that the textual conditioning is doing real work precisely where image data alone is scarcest, since a model that can lean on a written description of the treatment aim has an extra source of information to compensate for having seen fewer examples of an uncommon presentation.

A closer look at what CESEG is actually buying you

One of the ablation experiments isolates the consistency mechanism’s effect especially clearly. The authors compared segmentation accuracy when the model received the actual ground truth treatment plan text versus the AI generated plan text, both with and without CESEG active. Without CESEG, the gap between ground truth input and generated input was a Dice difference of 0.025 and an IoU difference of 0.033, a real but not catastrophic drop from using imperfect generated text. With CESEG active, that same gap shrank to a Dice difference of just 0.004 and an IoU difference of 0.004, essentially closing it. That is about as direct a demonstration as an ablation study can offer that the consistency loss is doing exactly what it was designed to do, making the segmentation network largely indifferent to whether its text conditioning came from a human or from the upstream language model.

Testing generalization on a synthetic public dataset

Because both validation cohorts in the main experiments came from hospitals in the same broader health system, the authors ran one more check using a synthetic dataset modeled on a third, unrelated hospital, Gachon University Gil Hospital. They used GPT-4 to generate synthetic clinical notes, ultrasound and MRI reports, and pathology findings spanning a range of breast cancer types, stages, and treatment scenarios in that hospital’s documentation style, then had a board certified radiation oncologist select the 20 most realistic cases from an initial pool of 50. On this fifteen case public style benchmark, RO-LMM-S reached a Rouge 1 of 0.795 against ChatGPT’s 0.765, and RO-LMM-P++ reached 0.669 against GPT-4.0’s 0.390, a gap the authors describe as demonstrating robustness to error accumulation carried over into a genuinely unseen institutional style. This synthetic dataset, stripped of any real patient information, has been released publicly, which is a useful gesture toward reproducibility in a field where real clinical data is almost always locked behind institutional review board restrictions.

What the ablations reveal about the design choices

A few of the smaller experiments in the paper are worth calling out because they explain design decisions that could otherwise look arbitrary.

Training one unified model to handle both summarization and strategy suggestion, rather than the separate expert approach RO LMM actually uses, produced a dramatic performance collapse, with internal Rouge 1 on summarization dropping from 0.783 down to 0.516. The authors interpret this as showing that summarization, which mostly involves extracting and reorganizing information already present in the input reports, and strategy suggestion, which requires proposing information not contained in the input at all, draw on different enough underlying skills that a single shared model does a mediocre job of both rather than a good job of either.

On noise type and intensity, uniform noise outperformed Gaussian noise, and a smaller noise scale, alpha equal to 5, outperformed larger values of 10 or 15, with excessive noise specifically hurting external generalization. Full parameter fine tuning substantially outperformed the more common parameter efficient LoRA approach, with internal Rouge 1 dropping all the way to 0.384 under LoRA, which the authors attribute to the specialized, narrow nature of the target clinical domain demanding more thorough adaptation than a lightweight adapter can provide. Scaling the backbone up from 7 billion to 13 billion parameters preserved the same ranking among methods without changing the overall conclusion, and the authors ultimately kept the smaller 7B model as their default specifically for cost effectiveness.

What it takes to actually run this

A detail that matters a great deal for anyone thinking about hospital deployment rather than benchmark performance alone is compute footprint. The authors report that the complete three stage pipeline runs on a single NVIDIA A6000 GPU within 24 gigabytes of memory, with RO-LMM-S using about 13.68 gigabytes and 6.89 seconds per patient, RO-LMM-P++ using about 13.57 gigabytes and 1.75 seconds per patient, and RO-LMM-SEG++ using about 17.89 gigabytes and 1.54 seconds per patient, adding up to roughly 10 seconds of total inference time per patient. That is a modest enough footprint that the authors frame local, on premises deployment as realistic even for a hospital without a large GPU cluster, which matters directly for data privacy, since patient reports and CT scans would not need to leave the institution to be processed by a closed source commercial model.

The clinical translation gap

It is worth being direct about how far this sits from routine clinical use, and the authors themselves are candid about several of these gaps. The training and validation data cover only patients with an initial breast cancer diagnosis who went on to receive radiotherapy after curative surgery, explicitly excluding recurrent or metastatic disease, so the model’s demonstrated competence does not extend to those harder, less standardized cases. Real world breast radiotherapy frequently uses techniques such as sequential or simultaneous integrated boost to deliver extra dose around the tumor bed, and accurately delineating a tumor bed for boost planning would require integrating additional imaging modalities beyond what this framework currently handles, so the authors deliberately left boost volume delineation out of scope for this study. The training data also reflects the specific, sometimes non standardized dose prescription conventions used at the source institutions, which the authors flag as a real limit on how directly the suggested strategies would generalize to a hospital following different regional or institutional prescribing norms, something that would need standardized, evidence based training data to address safely. Finally, due to context length limits on the language model, the system only ingests MRI, ultrasound, and pathology reports rather than a patient’s full electronic medical record, leaving out other information a human radiation oncologist would typically also weigh.

Where this fits in the wider push toward generalist medical AI

The paper situates itself against a backdrop of narrow, single task medical AI models, noting that most of the hundreds of FDA cleared AI tools in medicine handle one job with one type of input, an image classifier or a report generator working in isolation. The broader ambition behind generalist medical AI research, echoed in work the authors cite around foundation models for medicine, is that a system capable of synthesizing multiple modalities the way a clinician does could support more holistic decision making than any single narrow tool. RO LMM is a fairly concrete, workflow specific attempt at that ambition, deliberately scoped to one cancer type and one clinical pathway rather than trying to be everything at once, which is arguably why it was able to get far enough to run real multi center validation rather than staying at the demonstration stage.

The consistency embedding idea itself also has a life beyond this particular paper. It sits in a lineage of adversarial and noise based robustness training techniques for language models, including approaches like SMART, FreeLB, and R3F that the paper cites, all of which inject small perturbations during training to improve robustness. What distinguishes CEFTune and CESEG is the explicit pairing of noise injection with a consistency loss anchored in a task appropriate embedding space, sentence embeddings for text and learnable prompt embeddings for the segmentation conditioning, rather than just injecting noise and hoping robustness follows. That combination, noise plus explicit consistency rather than noise alone, is a pattern that plausibly generalizes to other multi stage AI pipelines well beyond radiation oncology, anywhere one model’s imperfect output becomes another model’s input.

Honest limitations, summarized

• The study population is limited to patients with an initial, non recurrent, non metastatic breast cancer diagnosis treated with standard postoperative radiotherapy, so results should not be assumed to extend to more complex or atypical disease presentations.
• Tumor bed boost delineation, a common real world component of breast radiotherapy planning, is explicitly outside the scope of the current segmentation framework.
• Training data reflects specific institutional dose prescribing conventions, which the authors note could limit how directly the suggested strategies generalize to hospitals following different regional or institutional norms.
• The clinical report summarization and strategy suggestion stages only ingest MRI, ultrasound, and pathology reports rather than a complete electronic medical record, due to context length constraints on the underlying language model.
• Internal and external validation cohorts, 121 and 81 patients for the text tasks and 79 and 81 for segmentation, are moderate in size for a two center study, and the authors themselves point to broader multi center and multi cancer type validation as necessary future work.

A simplified PyTorch implementation of CEFTune

The snippet below reimplements the core mechanics of NEFTune and CEFTune described in equations 1 and 2 of the paper, showing how uniform noise gets injected into token embeddings and how the consistency loss compares a clean and a noisy forward pass in sentence embedding space. It uses a small transformer stand in rather than a full LLaMA2 checkpoint so it runs end to end as a readable smoke test.

Two caveats if you plan to adapt this. First, the real system fine tunes a full 7 billion parameter LLaMA2 checkpoint rather than a toy transformer, and the paper reports using FSDP, bf16 precision, and gradient checkpointing across four GPUs to make that tractable, none of which this simplified script needs to demonstrate the loss mechanics. Second, CESEG for the segmentation stage follows the same pattern but compares learnable text prompt embeddings rather than full sentence embeddings, which would mean swapping the sentence encoder here for whatever prompt embedding layer sits between the frozen LLM and the 3D Residual U-Net in a full reimplementation.

The bigger picture

What makes this paper worth attention beyond the breast cancer specifics is the argument it makes about how to actually chain AI systems together for a real workflow rather than just benchmarking each piece in isolation. A great many medical AI papers report strong numbers for a single task, a classifier, a segmenter, a summarizer, tested against clean, carefully curated inputs. Far fewer grapple honestly with what happens when the output of one imperfect model becomes the input to the next, which is exactly the situation any real multi step clinical AI system will face the moment it leaves a benchmark and enters an actual workflow.

The conceptual shift here, treating consistency between clean and noisy predictions as something to explicitly train for rather than hope emerges from scale, is fairly portable. It does not obviously depend on radiation oncology, or even on medicine, and the authors’ own ablations, particularly the one showing CESEG shrinking the ground truth versus generated input gap from a Dice difference of 0.025 down to 0.004, make a compelling case that the mechanism does what it claims. Whether CEFTune and CESEG specifically become a standard tool for chained clinical AI pipelines, or get superseded by some other robustness technique, the underlying diagnosis, that error accumulation across sequential AI generated steps is a real and measurable problem deserving its own dedicated fix, seems likely to remain relevant regardless of which specific method wins out.

The honest remaining gaps are the ones the authors themselves name, a patient population narrowed to initial diagnoses, a segmentation scope that excludes tumor bed boost planning, training data tied to specific institutional prescribing conventions, and a text pipeline that cannot yet see a full electronic medical record. Closing those gaps is unglamorous, incremental engineering work rather than a single algorithmic breakthrough, which is often exactly what separates a promising research prototype from something a hospital could actually run. For a team building a similar chained clinical pipeline today, the more transferable lesson from this paper may be less about the specific numbers and more about the discipline of testing what happens when each stage feeds the next stage its actual, imperfect output rather than a clean benchmark input, since that is precisely where this kind of system tends to quietly fail.

Frequently asked questions

Related reading