DreamFuse Teaches Diffusion Models Real Image Fusion

A compositing problem that outgrew copy and paste

Image fusion sounds like a simple ask. Take an object out of one photo and put it convincingly into another. Anyone who has actually tried it in Photoshop knows how quickly convincing becomes the hard part. A pasted object needs the right shadow falling in the right direction, a plausible reflection if the surface below it is glossy, matching color temperature, and often a change of pose or angle so the object looks like it belongs in the new scene rather than sitting on top of it.

Earlier diffusion based attempts split into two camps. One camp adjusted the pasted region afterward, tweaking lighting, shadows and contrast so the seams were less obvious. The other camp reconstructed the whole image through inversion, which improved how natural the foreground looked but often let the background drift away from the original. DreamFuse’s authors frame this as a genuine trade off rather than a solved problem, and their stated goal is to hit both targets at once, a background that stays put and a foreground that adapts.

The paper also pushes past a narrower definition of the task that dominates prior work. Most existing systems assume you are inserting a whole object cleanly into a background. Real editing requests are messier. Sometimes an object is only partially visible, held in a hand, worn on a body, or seen through a gap in another object. Sometimes the request is not insertion at all but replacement, swapping one object in a scene for another while keeping everything around it untouched. And sometimes the ask is narrower still, changing just the color or material of an object that is already sitting in a photo, turning a plain sedan into something styled like brushed gold without touching the rest of the frame. DreamFuse is built to handle all three of those request types with one framework, which is a meaningfully broader target than object placement alone.

Where 160,000 training pairs actually came from

The unglamorous truth about most fusion research is that good training data is the bottleneck, not model architecture. You need matched triples, a clean foreground object with its mask, a background with that object convincingly removed including its shadows and reflections, and a fused image showing the two combined correctly. Segmentation and inpainting pipelines can approximate this, but the paper points out the predictable failure modes, residual shadow ghosts left behind after an object is erased, and inpainting that cannot handle a foreground that is only partially visible in the source photo.

Starting from 86 samples and a lot of human review

Rather than trust automated segmentation and inpainting to produce the entire dataset, the team started deliberately small. They pulled a foreground object and a fused image from an existing subject driven dataset, then manually cleaned up the inpainted background by hand, removing not just the object but its reflections and shadows too. That gave them 86 genuinely clean triples, each paired with a caption written by GPT four oh. Those 86 samples trained a first generation data generation model built on Flux Dev, and three rounds of human inspection and correction grew the usable set from 86 to 1,000 and finally to 4,000 high quality pairs.

Only after that seed set proved reliable did the team scale up automatically, using the now competent data generation model to expand coverage across scenes and styles, including animals, products, portraits and logos, indoor and outdoor settings, and fusion types ranging from simple placement to logo printing, wearable items and full style transfer. They even folded in existing style oriented LoRA adapters covering depth of field, photographic realism and different ethnic representations, specifically to counter a stylistic bias baked into the Flux base model. The filtering that followed combined a multimodal model checking for foreground consistency, background integrity and artifacts, alongside rule based scoring on CLIP similarity, aesthetic quality and facial similarity. The category that survived filtering least often was text and symbols, which the authors note lines up with a known weakness of Flux style models in rendering legible text, a nice example of a dataset audit actually confirming something the field already suspected rather than papering over it.

Calling in existing tools for replacement and attribute editing

The original in context generation approach above only produced object placement data. To cover replacement and attribute referenced editing, the authors layered in a second pipeline that leans on existing specialist editing tools. For replacement tasks they used editing models such as SeedEdit and Kontext to swap an object in a fused image for a different category of object, and repurposed the edited result as new background data. For attribute editing, they first generated images with a specific attribute already applied, then used editing tools to transfer that attribute onto matched foreground and background pairs, effectively engineering paired examples that would be very expensive to shoot for real.

The two pipelines together, after a further round of quality inspection using edge similarity checks between backgrounds and fused images, produced the final 160,000 sample dataset spanning insertion, replacement and attribute referenced editing. That is a genuinely large and structurally diverse corpus for a task that most competing papers have trained on datasets an order of magnitude smaller.

How DreamFuse actually fuses an image

With the data problem addressed, the architectural choices in DreamFuse are where the paper gets specific about mechanism rather than just claiming better numbers.

Foreground and background as conditions, not paste targets

DreamFuse is built on the Diffusion Transformer, or DiT, architecture, using Flux Dev as the base model. Rather than pasting a foreground into a background and asking a model to clean up the seams, the framework treats both the foreground image and the background image as conditioning inputs alongside a fixed timestep of zero, while the fused image is the actual denoising target. During training the model learns to regress a velocity field under a flow matching objective, effectively learning how to move from noise toward the correct fused image given both conditioning images and a text prompt.

That equation describes how a noisy version of the fused image is built during training, interpolating between the clean fused image and pure noise. The network is then trained to minimize the difference between its predicted velocity and the true velocity that would move the noisy sample back toward the clean target.

The mechanism that actually lets the foreground and background talk to each other inside the network is a shared attention layer. The original DiT runs attention independently per sample. DreamFuse instead concatenates the query, key and value components across the text prompt, the foreground image, the background image and the fused image, so a single attention computation can pull relevant detail from the foreground while respecting the layout of the background.

To keep insertion and replacement tasks from being confused with each other, the model does not rely on a caption of the fused image alone. It is fed a combined prompt containing both an explicit instruction token and a caption token, and the ablation results later in the paper show why that distinction earns its keep.

The positional affine trick for placing an object precisely

Telling a diffusion model exactly where to put an object, and at what scale, turns out to be a surprisingly awkward problem. The most obvious approach, physically resizing and repositioning the foreground image before feeding it in, compresses the pixel information of small objects and throws away detail. A second approach encodes a placement mask through a tokenizer, but that representation needs a lot of training data to learn well.

DreamFuse’s answer is a positional affine transform applied not to the pixels, but to the two dimensional position index that Flux’s rotary position embedding, RoPE, already assigns to every patch of the image. Foreground and background each get their own position index. When the foreground needs to land in a target region of the background, the paper computes an affine matrix that maps the target region’s coordinates back onto the foreground’s own coordinate space.

The practical effect is that the model can place a small object into a specific region of a large background without ever compressing or distorting the object itself, because the transform only ever moves position indices, not pixels. An ablation later in the paper backs this up directly, the positional affine strategy scores highest on both CLIP similarity and a vision reward metric compared to the direct transform and tokenizer approaches, 35.22 and 5.28 respectively, and the authors attribute the gap to how much fine grained foreground detail the other two approaches simply lose.

For situations where a user does not want to specify an exact position at all, and would rather let the model choose something sensible from the prompt alone, the framework falls back to an identity style affine matrix and lets a text instruction carry the placement information instead. The ablation in the paper’s Table VI shows this fallback stays competitive with the explicit positional version, which matters for anyone who wants a simpler text only interface rather than a bounding box input.

Teaching the model taste with localized preference optimization

Getting the geometry right does not automatically get the aesthetics right. A model can place an object in the correct spot and still generate a background that has subtly drifted, or a foreground that looks pasted despite sitting in the right position. To close that gap the authors turn to Localized Direct Preference Optimization, LDPO, built on top of Diffusion DPO.

The idea is to train on pairs of fused outputs where one is judged better than the other by a human relevant signal, and push the model’s predictions toward the preferred sample while pulling away from the rejected one. What makes it localized is a mask that restricts part of that preference signal to the region around the foreground object, expanded slightly by a dilation factor so shadows and reflections just outside the object’s bounding box are still covered.

Two kinds of rejected samples feed this stage. Some are simply weaker outputs generated by the model itself after the first training stage. Others are copy paste composites, a foreground pasted flatly onto a background with no adaptation at all. The clever part is treating those two kinds of negative examples differently. For the copy paste negatives, the localized loss lets the model learn that the pasted region should have been handled differently while still crediting the background for being consistent, since copy paste at least preserves the background perfectly. Ablation results show this approach lifts an image reward score by about 0.11 over the first stage model, and the qualitative examples in the paper show it specifically helping the model learn perspective and affine adjustments rather than just learning to copy pixels.

What the numbers actually show

The paper evaluates against a long list of prior fusion and composition methods, including TF-ICON, AnyDoor, MADD, ControlCom, InsertAnything, MimicBrush, ACE++ and OmniGen2, across the authors’ own DreamFuse test set plus two external benchmarks, FOSCom and TF-ICON’s own dataset of photorealistic, sketch, oil painting and cartoon images. Two scoring axes carry most of the analysis, a semantic consistency score that checks whether the foreground stayed recognizable and the background avoided over editing, and a perceptual quality score that checks for artifacts and overall naturalness.

A 50 participant user study adds a human judgment layer on top of the automated metrics, ranking DreamFuse against InsertAnything, OmniGen2, AnyDoor and TF-ICON across harmony, consistency and overall quality on a one to five scale where lower is better. DreamFuse comes out ahead on all three dimensions, including an average harmony rank of 1.85, and the authors report the result held up under a statistical significance test with Kendall’s coefficient of concordance confirming reasonable agreement between raters.

What still breaks, in the authors’ own words

The most credible part of this paper is arguably its limitations section, because the authors quantify their own weak points rather than gesturing at them. On the AnyInsertion benchmark, which specifically probes garment try on and identity preservation, DreamFuse’s scores drop on metrics like PSNR, SSIM, LPIPS, FID and a dedicated face score built on AdaFace. The authors trace this to three specific causes rather than a vague admission of imperfection. The base model itself, Flux, carries a documented bias toward Western facial features that limits identity preservation for other populations. Excessive preservation of the background can sometimes produce duplicated body parts around an inserted person. And the training data itself is thinner for domain specific try on scenarios than for general object placement.

That third point becomes a genuinely interesting finding rather than an excuse, because the paper tests it directly. Swapping the base model from Flux or SD3 to the newer Qwen Image backbone lifts the Face Score to 50.25, a clear jump, which suggests the identity problem is substantially inherited from the base generative model rather than being a flaw specific to DreamFuse’s own fusion logic. The authors frame their own contribution as backbone agnostic for exactly this reason, arguing the framework will keep improving as the underlying foundation models do, rather than needing to be rebuilt each time a better base model appears.

Robustness testing under deliberately noisy conditions tells a more reassuring story for object level fusion. Random perturbations to bounding box position and scale of up to 50 percent produced only a marginal 0.2 drop in semantic consistency when a text prompt was present alongside the positional input, and performance only degraded sharply once perturbation intensity reached the 50 to 80 percent range, well past what a normal editing workflow would introduce by accident. Inference resolution also matters more than it might seem. The model was trained at a 512 pixel short side but performs better at moderately higher inference resolutions, before quality drops again once the gap from the training resolution becomes too large, at 1536 pixels the semantic consistency score falls to 7.74.

Why the broader idea travels beyond this one paper

Strip away the specific architecture and DreamFuse is really making an argument about how compositing should be framed for generative models. Instead of treating placement as a pixel level paste operation to be cleaned up afterward, it treats placement as a position embedding transform that a transformer’s own attention mechanism already knows how to use. Instead of treating aesthetic quality as something a loss function should get right on the first try, it treats aesthetic quality as a second stage preference tuning problem, closer to how large language models are aligned after pretraining than to how earlier image editing pipelines were built.

That second point is probably the more transferable idea. Localized preference optimization, tuning only the region around an edit rather than the whole image, is a pattern that could plausibly extend to other conditional generation tasks where a global preference signal is too blunt an instrument, style transfer, local inpainting, or even video editing where only a specific object track needs correction across frames. The data generation story is equally exportable. Bootstrapping from a tiny, carefully human verified seed set, then using each generation of a trained model to help generate the next, larger batch of training data, is a recipe that other groups facing a data scarcity problem in a narrow visual task could reasonably copy wholesale.

Limitations worth keeping in view

Beyond the identity preservation gap already discussed, a few other constraints are worth flagging plainly. The dataset, while large at 160,000 samples, is synthetically generated and inspected by a multimodal model plus rule based filters rather than fully human verified end to end, which leaves open the possibility of systematic biases the filtering did not catch. The paper’s own experiment mixing in 1,300 real world pairs from an external dataset found that real data improved consistency and text alignment but slightly hurt aesthetic scores, since real world photos generally lack the polish of the synthetic training distribution, a trade off the authors are transparent about rather than resolving definitively. And several of the strongest results depend on swapping in a more capable base model like Qwen Image, meaning some of the reported gains are partly attributable to base model capacity rather than to the fusion framework alone.

Complete PyTorch implementation

The following code is an original, simplified educational implementation inspired by the mechanisms described in the paper, the shared attention layer, the positional affine transform, the flow matching objective and the localized preference loss. It is not the authors’ released code, since no public repository was available to link at the time of writing, and it uses small dummy tensors so it can run end to end on a CPU as a sanity check of the logic.

# dreamfuse_lite.py
# Educational reimplementation of the core DreamFuse mechanisms.
# Not the authors' official code. Runs a smoke test on random dummy data.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F


class SharedAttention(nn.Module):
    """Concatenated query/key/value attention across text, foreground,
    background and fused image tokens, following the shared attention
    idea used inside DreamFuse's DiT blocks."""

    def __init__(self, dim, num_heads=8):
        super().__init__()
        self.num_heads = num_heads
        self.head_dim = dim // num_heads
        self.q_proj = nn.Linear(dim, dim)
        self.k_proj = nn.Linear(dim, dim)
        self.v_proj = nn.Linear(dim, dim)
        self.out_proj = nn.Linear(dim, dim)

    def forward(self, text_tok, fg_tok, bg_tok, fused_tok):
        # concatenate along the sequence dimension, matching Q_g = [Qc; Qi]
        query_src = torch.cat([text_tok, fused_tok], dim=1)
        key_src = torch.cat([text_tok, fg_tok, bg_tok, fused_tok], dim=1)
        value_src = key_src

        q = self.q_proj(query_src)
        k = self.k_proj(key_src)
        v = self.v_proj(value_src)

        b, nq, d = q.shape
        nk = k.shape[1]
        q = q.view(b, nq, self.num_heads, self.head_dim).transpose(1, 2)
        k = k.view(b, nk, self.num_heads, self.head_dim).transpose(1, 2)
        v = v.view(b, nk, self.num_heads, self.head_dim).transpose(1, 2)

        scale = 1.0 / math.sqrt(self.head_dim)
        attn = torch.softmax((q @ k.transpose(-2, -1)) * scale, dim=-1)
        out = attn @ v
        out = out.transpose(1, 2).reshape(b, nq, d)
        return self.out_proj(out)


def positional_affine_matrix(target_box, region_size):
    """Builds the affine matrix A that maps a target region (u, v) in the
    background back onto the foreground's own position index space,
    matching Equation 5 and 7 in the paper."""
    h_r, w_r = target_box[2], target_box[3]
    h_prime, w_prime = region_size

    A = torch.tensor([
        [w_r / w_prime, 0.0, w_r],
        [0.0, h_r / h_prime, h_r],
        [0.0, 0.0, 1.0],
    ], dtype=torch.float32)
    return A


def remap_position_index(pos_idx, affine_matrix):
    """Applies the inverse affine transform to a batch of 2D position
    indices, producing the new foreground position index P_idx'."""
    ones = torch.ones(pos_idx.shape[0], 1)
    homogeneous = torch.cat([pos_idx, ones], dim=1)
    inv_A = torch.inverse(affine_matrix)
    remapped = (inv_A @ homogeneous.T).T
    return remapped[:, :2]


class TinyFusionDiT(nn.Module):
    """A drastically simplified DiT block stack standing in for Flux Dev,
    just enough structure to demonstrate conditioning and the training
    objective end to end."""

    def __init__(self, dim=64, depth=2, num_heads=4):
        super().__init__()
        self.token_embed = nn.Linear(3, dim)
        self.time_embed = nn.Linear(1, dim)
        self.blocks = nn.ModuleList([SharedAttention(dim, num_heads) for _ in range(depth)])
        self.norm = nn.LayerNorm(dim)
        self.velocity_head = nn.Linear(dim, 3)

    def forward(self, text_tok, fg_pixels, bg_pixels, fused_noisy, t):
        fg_tok = self.token_embed(fg_pixels)
        bg_tok = self.token_embed(bg_pixels)
        fused_tok = self.token_embed(fused_noisy)
        time_tok = self.time_embed(t.view(-1, 1, 1)).expand_as(fused_tok)
        fused_tok = fused_tok + time_tok

        h = fused_tok
        for block in self.blocks:
            h = h + block(text_tok, fg_tok, bg_tok, h)
        h = self.norm(h)
        velocity = self.velocity_head(h)
        return velocity


def flow_matching_loss(model, text_tok, fg, bg, fused, t):
    """Implements Equation 4, regressing the velocity field between noise
    and the clean fused image under a flow matching schedule."""
    noise = torch.randn_like(fused)
    t_expand = t.view(-1, 1, 1)
    noisy_fused = (1 - t_expand) * fused + t_expand * noise
    target_velocity = noise - fused

    pred_velocity = model(text_tok, fg, bg, noisy_fused, t)
    return F.mse_loss(pred_velocity, target_velocity)


def localized_dpo_loss(model, ref_model, text_tok, fg, bg,
                       fused_win, fused_lose, mask, t, beta=5000.0):
    """Implements the LDPO objective from Equation 8 and 10, splitting the
    preference loss inside and outside a dilated foreground mask."""
    noise = torch.randn_like(fused_win)
    t_expand = t.view(-1, 1, 1)

    x_win_t = (1 - t_expand) * fused_win + t_expand * noise
    x_lose_t = (1 - t_expand) * fused_lose + t_expand * noise

    v_win_true = noise - fused_win
    v_lose_true = noise - fused_lose

    v_win_pred = model(text_tok, fg, bg, x_win_t, t)
    v_lose_pred = model(text_tok, fg, bg, x_lose_t, t)

    with torch.no_grad():
        v_win_ref = ref_model(text_tok, fg, bg, x_win_t, t)
        v_lose_ref = ref_model(text_tok, fg, bg, x_lose_t, t)

    err_win_policy = ((v_win_pred - v_win_true) ** 2).mean(dim=-1)
    err_lose_policy = ((v_lose_pred - v_lose_true) ** 2).mean(dim=-1)
    err_win_ref = ((v_win_ref - v_win_true) ** 2).mean(dim=-1)
    err_lose_ref = ((v_lose_ref - v_lose_true) ** 2).mean(dim=-1)

    diff_win = err_win_policy - err_win_ref
    diff_lose = err_lose_policy - err_lose_ref

    # mask selects the localized foreground region vs the rest of the image
    w_diff = mask * diff_win + (1 - mask) * diff_lose
    l_diff = mask * diff_lose + (1 - mask) * diff_win

    logits = -0.5 * beta * (w_diff - l_diff)
    loss = -F.logsigmoid(-logits).mean()
    return loss


def train_step(model, ref_model, optimizer, batch, stage="flow"):
    optimizer.zero_grad()
    t = torch.rand(batch["fused"].shape[0])

    if stage == "flow":
        loss = flow_matching_loss(
            model, batch["text"], batch["fg"], batch["bg"],
            batch["fused"], t,
        )
    else:
        loss = localized_dpo_loss(
            model, ref_model, batch["text"], batch["fg"], batch["bg"],
            batch["fused_win"], batch["fused_lose"], batch["mask"], t,
        )

    loss.backward()
    optimizer.step()
    return loss.item()


def evaluate(model, batch):
    model.eval()
    with torch.no_grad():
        t = torch.full((batch["fused"].shape[0],), 0.5)
        loss = flow_matching_loss(
            model, batch["text"], batch["fg"], batch["bg"],
            batch["fused"], t,
        )
    model.train()
    return loss.item()


def make_dummy_batch(batch_size=2, seq_len=16):
    return {
        "text": torch.randn(batch_size, 4, 64),
        "fg": torch.randn(batch_size, seq_len, 3),
        "bg": torch.randn(batch_size, seq_len, 3),
        "fused": torch.randn(batch_size, seq_len, 3),
        "fused_win": torch.randn(batch_size, seq_len, 3),
        "fused_lose": torch.randn(batch_size, seq_len, 3),
        "mask": torch.cat([torch.ones(batch_size, seq_len // 2),
                          torch.zeros(batch_size, seq_len - seq_len // 2)], dim=1),
    }


if __name__ == "__main__":
    torch.manual_seed(0)
    model = TinyFusionDiT(dim=64, depth=2, num_heads=4)
    ref_model = TinyFusionDiT(dim=64, depth=2, num_heads=4)
    ref_model.load_state_dict(model.state_dict())
    for p in ref_model.parameters():
        p.requires_grad_(False)

    optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4)

    # stage one, flow matching smoke test
    batch = make_dummy_batch()
    stage_one_losses = [train_step(model, ref_model, optimizer, batch, stage="flow") for _ in range(5)]
    print("stage one flow matching losses", stage_one_losses)

    # stage two, localized DPO smoke test
    dpo_batch = make_dummy_batch()
    stage_two_losses = [train_step(model, ref_model, optimizer, dpo_batch, stage="dpo") for _ in range(5)]
    print("stage two LDPO losses", stage_two_losses)

    eval_batch = make_dummy_batch(batch_size=1)
    eval_loss = evaluate(model, eval_batch)
    print("eval loss on a fresh dummy batch", eval_loss)

    # quick sanity check that the affine remapping runs without error
    box = (0, 0, 32, 48)
    A = positional_affine_matrix(box, region_size=(16, 16))
    sample_pos = torch.tensor([[8.0, 8.0]])
    remapped = remap_position_index(sample_pos, A)
    print("remapped position index", remapped)

Running that file prints five loss values for each training stage and a final evaluation loss, confirming the shapes and gradients flow correctly through both the flow matching objective and the localized preference loss, with the position remapping utility producing a sensible coordinate transform on a toy example.

Conclusion

The headline achievement in this paper is not any single benchmark win, it is that DreamFuse manages to fold three previously separate editing tasks, object insertion, object replacement and attribute referenced editing, into one diffusion transformer trained end to end, and it does so on a dataset the team had to invent a data generation pipeline to build in the first place. The 86 to 4,000 to 160,000 scaling story is arguably as significant a contribution as the model architecture, since data scarcity has quietly capped progress across the whole fusion subfield for years.

The conceptual shift worth remembering is the move away from thinking about compositing as a pixel operation. By encoding placement as a transform on position indices rather than on pixels, and by treating aesthetic harmony as a preference tuning problem handled after the main training stage rather than something a single loss function must nail immediately, DreamFuse borrows two ideas that have already proven their worth elsewhere in generative modeling and applies them cleanly to a task that had mostly been approached with segmentation and inpainting tricks.

Those same two ideas look genuinely transferable outside image fusion. Localized preference tuning, applying a human judged reward signal to only the region of an image that actually changed, seems like a natural fit for any conditional generation task where a single global reward is too blunt, style transfer and localized video editing both come to mind. The bootstrapped data generation loop, starting from a small human verified seed and using each model generation to help build the next, larger training set, is a pattern any team facing a narrow data scarce visual task could adapt directly.

None of that erases the honest gaps the authors report. Identity preservation for people, and virtual try on scenarios specifically, remain measurably weaker than object placement, and the paper is upfront that this traces back to biases already present in the Flux and SD3 base models rather than to a flaw unique to their framework. That distinction matters for anyone deciding whether to build on this work today. It also points toward the most likely path for the field’s next improvement, since the authors show that simply swapping in a stronger backbone like Qwen Image lifts identity scores substantially without changing the fusion logic at all.

Where this leaves practitioners is with a genuinely useful blueprint rather than a finished product. Object placement, product photography style compositing, logo insertion and style transfer look solidly production ready based on the reported numbers and the robustness tests under noisy bounding boxes. Anything involving human identity or garment fitting should still be treated as an active research problem, one that will likely improve on its own as the base diffusion models it depends on keep getting better at representing people fairly and consistently.

Frequently asked questions

Related reading