Why DPO Is Beating RLHF at Aligning AI Images

The problem worth solving

Diffusion models got very good very fast at turning text into images. What they did not automatically get good at was matching what a person actually wants to see. A prompt asking for a cat sleeping in a sunbeam might come back technically coherent and still miss the mood entirely, or drift from the description in some small but noticeable way, or carry a stylistic bias nobody asked for. The paper’s authors point to three recurring failure patterns worth naming directly. Text and image content that does not quite line up. Output that diverges from ordinary human aesthetic judgment. And the occasional generation of content that is toxic or biased in ways that trace straight back to the training data.

The fix borrowed from an unlikely neighbor. Large language models had already shown that reinforcement learning with human feedback could nudge a model’s behavior toward what people actually rate highly, rather than just what scores well on a fixed loss function. Applying that same idea to image generation meant training a reward model on human comparisons, then using that reward signal to steer the diffusion process itself. The appeal is obvious. The execution turned out to be considerably messier, and mapping out exactly how messy is the job this survey set for itself.

Worth noting for context. Earlier surveys existed on diffusion models generally, and a few touched on reinforcement learning applied to diffusion, but the authors argue none of them treated preference alignment as its own subject with its own taxonomy, its own tradeoffs, and its own open problems. A prior survey the authors cite as an example focused specifically on image restoration and super resolution, which is a related but genuinely different question, since it is about pixel level fidelity rather than matching abstract human intent. You can read the full survey and its complete method list in the published paper in Computer Science Review.

How this survey was built

The team followed PRISMA reporting guidelines and searched four databases, Google Scholar, arXiv, IEEE Xplore, and the ACM Digital Library, for work published between January 2020 and February 2025. That window is deliberate. It starts right around when denoising diffusion probabilistic models were first formalized and runs through the most recent alignment techniques available as they were writing.

The search string combined diffusion model terminology with preference alignment terms including reinforcement learning, reinforcement learning with human feedback, direct preference optimization, and reward modeling, intersected with image generation or image editing. That combination returned more than 600 candidate articles. A screening pass based on three inclusion rules, the paper must use a diffusion model as its backbone, it must involve an explicit alignment mechanism, and the task must be visual content generation or editing, narrowed the pool. Papers focused purely on language models without a visual component, generic diffusion speed improvements with no preference component, and non English or clearly under developed submissions were excluded.

Title and abstract screening brought the corpus down to roughly 150 papers, which then went through full text review for technical depth and fit. One honest gap worth flagging. The paper never states an exact final count after that full text review stage, it moves from over 600, to about 150, to a final selection described only as representing the field’s most significant and peer reviewed contributions. Compare that with the taxonomy table itself, which lists 37 named methods across online and offline categories, and you get a rough sense of scale, but readers looking for a precise PRISMA style final N, the kind the other systematic reviews in this space typically report, will not find one stated directly in the text.

Three phases the field moved through

The authors organize the whole research trajectory into three periods, and the shape of that progression tells its own story about how a young technique matures.

The exploration phase ran roughly 2022 to 2023. This is when researchers directly ported reinforcement learning with human feedback pipelines over from language models, mostly just to see if the idea would work at all. DDPO and DPOK, both built on proximal policy optimization, were the pioneering efforts here, alongside ImageReward, which established a reward model trained on 137,000 expert comparisons.

The efficiency phase followed from about 2023 to 2024, driven by a very practical complaint. Online reinforcement learning was expensive and unstable. Diffusion-DPO reformulated the alignment objective to skip reward modeling entirely. D3PO and Diffusion-KTO pushed further, enabling fine tuning directly on pre collected datasets or even simple binary feedback signals rather than paired comparisons.

The application phase, from 2024 onward and still ongoing, is where these techniques get bent toward specific, often safety critical industries. TrafficRLHF and Gen-Drive for autonomous driving decision making. MedSegDiff and CXRL for medical image synthesis. This is also, not coincidentally, where a general purpose optimization technique starts having to answer to domain specific standards around safety and interpretability rather than just looking good.

The math underneath, briefly

It helps to have the basic mechanics in view before getting into the alignment methods themselves, since almost every technique in this survey is really just a different answer to the question of how to steer this process.

A diffusion model learns by first destroying an image, adding a little Gaussian noise at each of many time steps until nothing recognizable is left, then learning to reverse that process one step at a time. The forward noising step looks like this.

Training then teaches a model to reverse this, starting from pure noise and denoising step by step back toward a coherent image, guided by a learned mean and variance at each step. Once that base model can reliably turn prompts into plausible images, the preference fine tuning stage begins. This is where proximal policy optimization, borrowed straight from mainstream reinforcement learning, usually enters the picture for the online methods.

That clipping term is doing real work. Without it, a policy that stumbles onto a temporarily high reward can overcorrect hard, and diffusion models fine tuned this way are especially prone to that kind of instability because every single training step requires actually generating a full image first, through the entire multi step denoising chain, before the reward model can even score it.

Two philosophies of alignment

Nearly everything in this survey’s taxonomy traces back to one structural choice. Does the model generate fresh samples during training, or does it learn from a dataset collected in advance. The authors treat this as the fundamental dividing line, not a minor implementation detail, because it decides the computational bottleneck, the training stability, and how far the model can theoretically improve beyond its starting point.

How RLHF actually gets applied here

Reinforcement learning with human feedback for diffusion models works in three rough stages. The policy model interacts with its environment and gets updated through standard reinforcement learning mechanics. Pairs of outputs get shown to human annotators for comparison. Then the model updates again based on the reward signal that comparison produces.

DDPO frames this directly, maximizing expected reward with no regularization term at all.

DPOK adds a penalty term that pulls the model back toward its original, pretrained behavior, which the authors of that method found meaningfully improved output quality over the unregularized version.

That divergence penalty exists for a real reason, and it points at the core weakness of online reinforcement learning in this setting. Left unchecked, a model being optimized purely to maximize a reward score will happily find loopholes in that score rather than genuinely improving. This is Goodhart’s Law showing up in a very visual form, models generating chaotic, oversaturated patterns because that happens to spike an aesthetic classifier’s output, not because anyone finds it appealing. Researchers call this reward hacking, and it is one of the recurring themes across the whole survey.

A wave of follow up work tried to work around this instability from different angles. The Human Preference Score trains a classifier on large scale human choice data. Reward Feedback Learning trains directly on 137,000 expert comparisons. Parrot balances multiple competing objectives, aesthetics, prompt alignment, sentiment, all at once through a Pareto optimal selection process. Diffusion-MVP folds in visual market signals and CLIP relevance scores. Some of this work moved past pure human annotation entirely, using large vision language models as automatic judges, RAHF flags implausible regions and misaligned keywords for correction, LVLM-REFL applies an iterative correction loop guided by a vision language model’s own assessment.

A separate branch skips reward models altogether and backpropagates directly from a differentiable reward function into the policy. DRaFT and AlignProp both work this way, and they are genuinely effective at boosting aesthetic scores, but this approach runs into a memory and time tradeoff that is easy to underestimate. Backpropagating through an entire denoising chain, often twenty to fifty steps, means storing gradients at every one of those steps, which gets expensive fast.

Why direct preference optimization changed the calculus

Direct preference optimization, DPO for short, took a different approach entirely, and it is worth understanding why it caught on so quickly. Instead of training a separate reward model and then running reinforcement learning against it, DPO uses a mathematical shortcut. Given a set of paired comparisons, where one output is preferred over another, there turns out to be a closed form relationship between the optimal policy and the reward function under a standard statistical model of pairwise preference.

The practical upshot is that alignment turns into something close to an ordinary supervised classification problem, maximizing the margin between what the model prefers and what it does not, rather than the much messier actor critic setup reinforcement learning requires. That reformulation is why DPO trains more stably and uses less memory than its online counterparts.

It is not free of tradeoffs though. Because DPO learns entirely from a fixed dataset, it has no way to explore beyond what that dataset contains. If the preference data is narrow, biased, or simply not very diverse, DPO has no mechanism to self correct the way an online reinforcement learning agent theoretically could. Its alignment ceiling is set by data quality, full stop.

Diffusion-DPO adapted the framework specifically for diffusion by incorporating the model’s evidence lower bound into the objective. D3PO reframes the denoising process as a multi step decision problem to cut memory use even further, updating parameters at each individual denoising step instead of the whole trajectory at once. Diffusion-KTO goes a step further still, working from only per image binary feedback rather than requiring paired comparisons at all, which matters because paired comparison data is expensive to collect. TailorPO ranks intermediate noisy samples by their step wise reward to fix a gradient direction problem that earlier methods struggled with. A newer hybrid category, online DPO, blurs the line again by dynamically refreshing the training dataset using the model’s own outputs judged by reward models or language model judges, effectively borrowing DPO’s stable loss function while keeping some of online learning’s adaptability.

Beyond DPO and standard reinforcement learning, a smaller cluster of methods explores alternatives. Generative flow networks aim to sample outputs proportionally to their reward rather than purely maximizing it, which helps preserve output diversity. Self play methods like SPIN-Diffusion remove the need for human preference data entirely by having the model compete against earlier versions of itself. And the same alignment logic has started showing up in adjacent low level vision tasks too, Reti-Diff for lighting restoration, DiffBIR for blind image restoration, SeeSR for super resolution, all borrowing the same basic idea of aligning generated output with human perceptual expectations rather than pure pixel accuracy.

What the numbers actually say

The survey backs its comparisons with real benchmark data rather than leaving it at description, which is worth taking seriously. Three metrics dominate the field. Human Preference Score version 2 and PickScore act as learned reward models trained on large scale human choice datasets, scoring how well an image matches typical human aesthetic judgment. The underlying comparison logic uses the Bradley-Terry model, a standard statistical approach to pairwise preference.

CLIP Score checks semantic alignment between an image and its prompt without needing any human labels at all, computed as cosine similarity between image and text embeddings from a pretrained CLIP model. It is useful but the survey notes it correlates only loosely with fine grained aesthetic judgment. Win Rate is the most intuitive metric of the three, simply the share of head to head comparisons a method wins against a baseline, judged by either human annotators or a model like GPT-4V.

Here is where the survey’s meta analysis gets genuinely interesting. Pulled together from the original papers, the reported numbers show a consistent pattern.

Look closely at the pairing that matters most here. D3PO and DDPO both start from the identical Stable Diffusion 1.5 base model, yet D3PO scores meaningfully higher on both metrics, 30.5 against 29.1 on HPS v2, and a 64.1 percent win rate against DDPO’s 61.2 percent, while training on a static dataset instead of continuously sampling new images during training. The authors read this as evidence that data quality and objective stability are more distinctive drivers of performance than online exploration, at least for diffusion alignment as currently practiced.

Cost tells a matching story. The survey compares peak memory use and relative training time across methods.

Put plainly, DDPO costs roughly four and a half times as long to train as a plain supervised baseline and doubles the memory footprint, while producing worse alignment scores than D3PO, which trains at close to baseline cost. That is a striking result for a field that has generally assumed more environment interaction should translate into better outcomes.

Where preference aligned diffusion models are actually being used

The application landscape the survey maps out spans four broad areas, and the common thread across all of them is that these deployments care about controllability and safety as much as visual polish.

Medical imaging

Diffusion models are increasingly used for medical image segmentation, separating organs or lesions from surrounding tissue in a scan, which supports both diagnosis and image guided surgery. MedSegDiff enhances step wise regional attention inside the diffusion encoder specifically to make faint lesion boundaries more visible. Its successor, MedSegDiff-V2, adds transformer based dual conditioning to improve accuracy further. Because real patient imaging data is scarce and tightly regulated, a separate thread of work focuses on synthetic generation instead, one study trains latent diffusion models under differential privacy on the UK Biobank dataset specifically to protect patient identity while still producing useful synthetic training images. CXRL applies reinforcement learning with human feedback directly to chest X-ray generation to improve output quality.

Robotics

In robotics, preference fine tuning shows up across motion generation, trajectory planning, and policy learning. One line of work combines offline diffusion learning with online preference alignment to improve quadrupedal locomotion, targeting stability and the ability to generalize to real robots with zero additional training. Trajectory planning work embeds the entire optimization process into the denoising chain itself, enabling long horizon planning while keeping generated trajectories consistent with stated conditions. Preference fine tuned diffusion models also show up in soft robot design and in training world simulators used to test other autonomous systems.

Autonomous driving

This is probably the most developed application area in the survey, split across three distinct needs. Data generation, since real driving data is expensive to collect and has limited coverage of rare situations, DrivingDiffusion generates spatially and temporally consistent multi view video from synthetic layouts, and Panacea produces controllable panoramic driving scenes with adjustable weather and lighting specifically to simulate edge cases. Decision making, where Gen-Drive uses a generation then evaluation approach, producing multiple plausible future traffic scenarios and then using a learned evaluator to pick the best one, while other work focuses on shortening the denoising process so a car does not have to wait through fifty steps of diffusion before deciding what to do next. And world models, VISTA builds a highly generalizable world model with strong prediction accuracy that can even construct its own reward signal for evaluating real world driving actions without needing access to real action data at all.

Other domains

Beyond these three, the survey notes preference aligned diffusion models being applied to protein structure prediction in biology, realistic virtual world generation for gaming, and creative content production spanning digital art and music, anywhere that matching a nuanced human standard matters more than simply generating something plausible.

data quality and objective stability are more distinctive drivers of performance Wu, Si, Xing, Wang, Jin, Cheng, and Huang, Computer Science Review, 2026

Where this still breaks

The survey is refreshingly direct that this field has not solved its problems just because the benchmark numbers look good. Three issues stand out as structural rather than easily patched.

The synthetic data paradox

Offline methods like DPO and KTO increasingly rely on preference labels generated by AI judges rather than humans, since human annotation does not scale. That solves one problem and creates another. Training generative models recursively on their own outputs, or outputs judged by another AI system, risks what researchers call model collapse or model autophagy, a gradual narrowing of the output distribution and an amplification of whatever bias already exists in the labeling model. If a reward model happens to favor certain textures or saturation levels for reasons that have nothing to do with genuine human taste, the aligned model will overfit to that quirk rather than to anything real.

Reward hacking and the alignment tax

Goodhart’s Law again. Models optimized against a scalar reward function tend to find the shortest path to a high score, which is not always the same path as genuinely better output. The survey also flags something called the alignment tax, where aggressive tuning for one objective, safety for instance, can quietly erode a model’s general capability, its ability to produce a specific artistic style or an accurate medical diagram, say, simply because that capability was not part of what got optimized for.

The evaluation crisis

Static benchmarks like Pick-a-Pic and HPS v2 measure something real, an averaged snapshot of human preference, but that average is not the same as any one person’s actual taste, and it does not move with shifting trends the way real aesthetic judgment does. Optimizing hard against a fixed score risks producing images that are broadly inoffensive rather than genuinely good, technically pleasing to a statistical average of raters rather than compelling to any real person.

The authors lay out a three phase roadmap in response. Short term work should focus on resolving the memory bottleneck in reinforcement learning through single step DPO variants that can train on consumer grade graphics cards. Medium term work should push toward finer grained control, dense reward signals that operate at the pixel or object level rather than scoring a whole image at once, and extending these techniques from still images into video. Long term, the goal shifts toward genuinely personalized and value aligned systems, work the authors describe as weak to strong generalization research aimed at controlling increasingly capable generative models against abstract human values rather than a single averaged score.

What this means if you are actually building one of these systems

The survey’s own model selection guidance is genuinely practical, and worth restating directly rather than leaving buried in a methods section. If you already have, or can reasonably collect, a decent static preference dataset and efficiency matters to you, start with an offline direct preference optimization method rather than reaching straight for a full reinforcement learning pipeline. The evidence in this survey’s own benchmark tables says offline methods are both cheaper and currently scoring better, which undercuts the instinct that a more sophisticated reinforcement learning setup must be the stronger choice.

Online reinforcement learning still earns its keep in a narrower set of cases, specifically when the objective you care about is complex, non differentiable, or defined by something like a simulator or black box scorer rather than a static dataset you can collect in advance. That is precisely why autonomous driving and robotics, where success might mean something like a simulated crash rate rather than a simple aesthetic score, lean more heavily on online methods than pure image generation work does.

And if retraining model weights at all is not feasible, inference time reward guidance offers a plug and play alternative, applying alignment constraints directly during sampling. The tradeoff there is real though, that approach avoids training cost entirely but adds meaningful latency at inference time, since it typically requires gradient calculations at every denoising step, which makes it a poor fit for anything that needs to respond in real time.

Honest limits of this survey

A few things are worth knowing before treating this survey as the final word on the field. As mentioned earlier, the paper never states a precise final study count after full text screening, moving from over 600 candidates to approximately 150 to an undefined final selection, which is looser reporting than some comparable systematic reviews provide. The paper also does not include a dedicated section reflecting on its own methodological limitations, things like database coverage, language restrictions, or potential publication bias toward papers reporting positive results, the kind of self critique that some other reviews in this space build in explicitly. That is not necessarily a flaw in the underlying research, but it does mean a reader has to supply some of that skepticism themselves.

There is also the simple matter of how fast this field moves. The literature search cut off in February 2025, and the paper appeared online in February 2026, a year is a long time in diffusion model research, and it would be reasonable to expect at least a few newer alignment techniques have already emerged that this survey could not have captured.

Conclusion

Step back and the overall shape of this field looks like a fairly familiar engineering story. A promising but expensive idea, adapting reinforcement learning with human feedback from language models, gets tested for feasibility first, then gets systematically simplified and cheapened once the community understands what actually matters about it. What is less familiar, and genuinely worth sitting with, is that the cheaper approach did not just match the expensive one, it outperformed it on the field’s own preferred benchmarks. That is not the outcome reinforcement learning intuition usually predicts.

The conceptual shift underneath all of this is a move away from treating alignment as an exploration problem and toward treating it as a data quality problem. Diffusion-DPO and D3PO succeed not because they search more cleverly, but because they turn a messy actor critic optimization into something closer to ordinary supervised learning, and it turns out that stability and clean data matter more here than the theoretical ability to discover novel high reward trajectories.

There is real transferability in this lesson beyond image generation specifically. Anywhere reinforcement learning with human feedback gets applied, video generation, audio, robotic control, the same tension between exploration and stability is likely to resurface, and this survey’s finding that offline methods can outcompete online ones is a data point worth remembering the next time a team defaults to the more complicated pipeline simply because it sounds more principled.

That said, the honest gaps the authors themselves flag are real and unresolved. Synthetic preference data creates a genuine risk of narrowing what these models can produce over successive training rounds. Reward hacking is not a hypothetical, it shows up in the literature as chaotic, oversaturated images that happened to satisfy a classifier without satisfying anyone. And the benchmarks the whole field currently optimizes against measure an averaged taste that may be actively pulling generated art toward blandness rather than genuine quality.

Where this goes next probably depends less on bigger reward models and more on the unglamorous work the roadmap points toward, personalization mechanisms that respect individual taste rather than an aggregate mean, evaluation protocols that can track shifting aesthetic trends instead of freezing them, and constrained optimization approaches that treat safety as a non negotiable boundary rather than one more score to average into a single number. A diffusion model that knows what one specific person finds beautiful, rather than what a committee of raters found acceptable on average, is still mostly a research goal rather than a shipped product, and closing that gap is the work that remains.

Frequently asked questions