HybridAgent Routes Image Restoration More Intelligently

Analysis by the aitrendblend editorial team · Machine Learning · Agent Systems

HybridAgent addresses two different bottlenecks

Bingchen Li, Xin Li, Yiting Lu and Zhibo Chen of the University of Science and Technology of China describe the framework in Hybrid Agents for Image Restoration. The publisher identifies the paper as part of the CVPR 2026 proceedings.

The first bottleneck is deciding what to do. A user may describe a problem precisely, vaguely or incorrectly. A restoration system must translate that request into an operation without spending unnecessary computation on information the user has already supplied.

The second bottleneck is executing a useful operation. Images often contain several interacting distortions. Even a perfectly chosen sequence of individually capable restoration models can behave poorly when their assumptions about the input distribution no longer hold.

These are related problems, but they require different evidence. Better prompt routing addresses the first. Better restoration tools address the second. A convincing evaluation should show where each improvement comes from rather than attribute every gain to the presence of agents.

This places the work within the Agent Systems collection. The interesting contribution is the connection between a controller and a deliberately constructed tool library.

Why clear instructions deserve a different route

Consider a request that explicitly identifies noise as the unwanted distortion. The system can potentially select a denoising tool from the text alone. Image inspection may add little value when the instruction is both accurate and complete.

A vague request is different. Asking for a cleaner image leaves the system to determine whether the problem is noise, blur, compression, poor illumination or some combination. That decision requires evidence from the pixels.

HybridAgent assigns the first kind of request to FastAgent and sends ambiguous requests to SlowAgent. The architecture therefore makes a useful distinction between understanding an instruction and diagnosing an image.

The separation also creates a failure mode. A clear instruction can still be mistaken. Someone may call compression artifacts noise, or describe motion blur as a lack of sharpness. Textual confidence is not evidence that the proposed operation matches the image.

An implementation should therefore preserve the reason for a routing decision. Recording whether a tool was selected from explicit user intent or visual diagnosis makes later failures easier to interpret.

SlowAgent performs visual diagnosis and planning

The author manuscript uses a specialized multimodal model based on Co-Instruct for SlowAgent. Additional instruction tuning connects degradation recognition to restoration tool selection.

This matters because an ordinary visual language model may describe an image convincingly while failing to identify the distortion that determines which tool should be applied. Semantic description and restoration diagnosis are different capabilities.

The system also uses repeated candidate decisions and majority voting. This can reduce variability in an unstable prediction, but the votes should not be treated as independent clinical style measurements or as a calibrated probability.

If the same model repeatedly misunderstands a texture, its votes can agree on the wrong answer. Agreement measures consistency under the sampling procedure. It does not establish that the image has been diagnosed correctly.

A practical controller should validate any returned tool identifier against its supported library. A plausible sentence from a language model is not yet an executable action.

FeedbackAgent decides when further work is justified

Choosing an operation does not settle whether another operation is needed afterward. A denoiser may leave compression blocks visible. A blur removal model may sharpen edges while amplifying grain. The controller needs a stopping decision that considers the resulting image.

HybridAgent gives this responsibility to FeedbackAgent. The manuscript describes a model trained to judge whether a restored image is relatively clean, with restoration history included as context.

History is useful because the same visible artifact can mean different things after different operations. A checker that knows a denoiser has already been applied can interpret a remaining pattern differently from a checker examining the untouched input.

The stopping problem still has no perfect visual answer. A photograph can become smoother while losing fine texture. An enhancement can look more attractive while shifting colors. Cleanliness, fidelity and user preference need not improve together.

Mixed distortions change the restoration problem

A familiar pipeline applies one specialist after another. It may remove compression artifacts, then noise, then blur. The apparent logic is that each tool eliminates one component until the image is clean.

That logic assumes the tools can tolerate the inputs produced by their neighbors. A denoiser trained on noisy photographs may not have learned how to process ringing left by a deblurring model. A sharpening operation can alter the frequency pattern that another model uses to distinguish noise from texture.

The order of degradations also matters. Noise added before compression is not equivalent to noise added afterward. Each process changes the evidence available to later processes.

This equation is an explanatory abstraction. It does not imply that each restoration operator is an exact inverse. In general, information has been lost, and the learned operators make assumptions about what the missing information should look like.

A mixed distortion tool learns to address combinations jointly. It can potentially recognize interactions that are poorly represented by a sequence of isolated corrections.

The neighboring article on composite image restoration examines another approach to coupled distortions. Its results belong to a different method, but the shared question is whether the restoration problem can safely be broken into independent steps.

The tool library is trained in three stages

The first stage trains a shared restoration backbone across several degradation types. This gives later tools a common starting representation rather than building every specialist independently.

The second stage adapts that backbone to individual distortions. The manuscript uses LoRA together with learned prompt components. The prompts encode degradation related information, while the adapters alter selected transformations inside the network.

The equation describes the usual low rank update mechanism. Its value is that a task specific change can be represented without retraining every entry of the original weight matrix.

The third stage creates a mixed distortion tool. This is a separate learned capability rather than an assumption that adding individual adapters automatically yields a model that understands their combined distortions.

Shared weights can reduce duplicated storage, but actual savings depend on deployment. Keeping a single backbone with switchable adapters differs from exporting every tool as a complete model. The latter can duplicate the supposedly shared parameters.

The architecture also remains more than the small adapter matrices. Prompt components, language models, image encoders and the restoration backbone all contribute to the total memory and runtime budget.

The fast route trades some recognition capability for latency

The manuscript compares the complete system with a version that sends all requests through SlowAgent. That experiment uses direct prompts, making it a focused test of whether visual reasoning is necessary when the requested action is already specified.

The reported time is agent inference time on the stated hardware. It should not be relabeled as the runtime of the entire restoration application. Loading images, running the restoration network and performing repeated feedback checks can contribute additional cost.

These values come from the accessible manuscript’s routing ablation. They show why speed and correctness must be reported separately. The lightweight route is faster in these examples, while the visual route selects the intended tool more often.

A useful product policy might let an informed user request an operation directly while escalating uncertain cases to visual inspection. That is a design suggestion, not an additional policy evaluated by the paper.

The mixed tool comparison is strong but not uniform

The author manuscript compares a library containing only individual distortion tools with a library that also includes the mixed tool. Its average metrics favor the expanded library.

Higher PSNR and SSIM are preferable in this comparison, while lower LPIPS is preferable. These metrics measure different aspects of agreement with a reference. None establishes that all meaningful scene information has been preserved.

The averages also hide variation. For the motion blur and JPEG combination, the manuscript reports PSNR of 23.79 for individual tools and 23.77 for the expanded library. The corresponding structural and perceptual measures move in the other direction.

That small exception matters because it prevents an absolute claim that every metric improves for every corruption. The stronger conclusion is that the expanded library performs substantially better on average across the reported mixed distortion cases.

Restoration can also produce disagreement between metrics and human preferences. A model may improve pixel agreement while smoothing fine details, or create convincing texture that is not present in the reference. Evaluation should retain several views of quality.

Training data shape the agent’s idea of clean

A feedback model learns its stopping behavior from examples. If clean examples are primarily outputs from the expected restoration tools, the model can learn properties of those tools as well as properties of genuine image quality.

This creates a distinction between an image being relatively clean under a training convention and an image satisfying an independent fidelity standard. The former can be useful for controlling a workflow without guaranteeing the latter.

The accessible manuscript also contains an arithmetic discrepancy in its feedback dataset description. It lists 30000 clean and 33000 unclean examples, then states a total of 66000. Those components sum to 63000. The final proceedings version or released data should resolve that before an exact dataset size is repeated.

Prompt diversity has a similar limitation. Synthetic paraphrases help a router avoid depending on one wording, but they do not cover every way users describe images. Negation, conflicting requests and instructions to preserve particular content deserve separate evaluation.

For example, a request to remove noise while preserving film grain contains an aesthetic constraint. A tool selector that reduces it to denoising may technically follow a category label while missing the user’s intent.

A useful evaluation must follow the whole workflow

Agent accuracy should be measured against the intended action. Restoration quality should be measured against the resulting image. Stopping quality should assess whether continuing would help. These outcomes can fail independently.

A controller can invoke the expected tool and still obtain a poor image because the tool does not generalize. It can select an unexpected operation that nevertheless produces a good result. A feedback model can then accept or reject either result incorrectly.

Runtime comparisons should therefore include the route distribution. A workload dominated by explicit requests will favor a fast text path more than a workload dominated by ambiguous photographs. Reporting one mean without that distribution makes transfer to another application difficult.

The same applies to degradation composition. The frequency of noise, darkness, compression and mixed cases determines which tools matter most. A library optimized for one corruption mixture may require different adapters when the source camera or codec changes.

Our analysis of realistic degradation modeling discusses why the training distribution deserves attention. HybridAgent changes tool coordination, but coordination cannot fully compensate for a library trained on the wrong image formation process.

Practical implementation needs explicit boundaries

A restoration run should preserve the original input and record each selected operation. That allows a user to compare outputs and prevents an apparently successful final step from concealing damage introduced earlier.

An iteration budget is also useful. Feedback can remain uncertain, and successive tools can alternate between sharpening and smoothing. A maximum budget turns an open ended conversation into a bounded computation with an inspectable result.

Repeated tool selection should be visible rather than silently ignored. Sometimes another pass is justified, but repeated calls without measurable progress can reveal a controller that is failing to understand its own output.

The implementation should distinguish a tool that is unsupported from an image that needs no restoration. Returning the input unchanged after an invalid tool name should not be reported as a successful clean result.

Efficient deployment may ultimately depend more on the restoration tools than on the router. The article on compact super resolution tools provides another perspective on the importance of counting the component that actually processes the pixels.

Limitations of the available evidence

The accessible experiments concern a defined collection of distortions and test images. They do not certify correct operation on every camera pipeline, artistic effect or unfamiliar compression artifact.

Synthetic corruption combinations make controlled comparisons possible, but real acquisition pipelines may combine distortions differently. A mixed tool can still encounter distributions outside its training support.

The fast route depends on instruction interpretation. A precise but incorrect user description can bypass the visual evidence that would have exposed the mistake. Whether this behavior is acceptable depends on how much control the user intends to exercise.

The feedback labels are operational judgments rather than a proof of recovered ground truth. A clean decision should not be translated into a statement that all original details have been reconstructed faithfully.

Finally, the numerical analysis here comes from the accessible author manuscript. The uploaded proceedings PDF remains the outstanding source check. This draft should not be presented as a verified transcription of that final version.

“error propagation occurs when step-by-step single degradation removal tools are applied” Li, Li, Lu and Chen, explaining the motivation for mixed restoration tools.

Better orchestration starts with better choices

HybridAgent offers a useful division of work. Clear requests can take a lightweight route, ambiguous requests can trigger visual reasoning, and the result can be checked before another operation is applied.

Its more substantial restoration lesson concerns the tool library. When distortions interact, changing the order of isolated tools may be insufficient. A jointly trained tool can address a different problem from the one solved by its individual counterparts.

Shared pretraining and adapters provide a practical way to organize that library. Their benefit depends on how the tools are stored, trained and invoked, so the complete system budget still needs measurement.

The experiments support this design while retaining important tradeoffs. Faster routing can select the wrong tool more often, averages conceal exceptions, and automated cleanliness decisions remain imperfect.

The next useful advance would connect these pieces through transparent evaluation of intent, fidelity, stopping behavior and total cost. An image restoration agent becomes valuable when it makes better decisions about both the work to perform and the work to avoid.

Complete educational PyTorch reference

The code below implements a small restoration tool prototype with shared training, separate low rank adapters and a mixed distortion adapter. It includes the model, synthetic degradation generation, loss, training, evaluation and a runnable smoke test.

The orchestration interface accepts external planner and feedback functions. It does not disguise a keyword router as the paper’s language model or invent a substitute for Co-Instruct. The small convolutional backbone and L1 objective are editorial implementation choices.

This prototype does not reproduce the paper’s architecture, instruction tuning or benchmark results. Runtime and syntax checks could not be executed because the workspace was unavailable.

# Educational prototype. Requires Python and PyTorch.
# Not an implementation of the complete published HybridAgent system.
import copy
import math
import torch
from torch import nn
from torch.nn import functional as F


class LoRALinear(nn.Module):
    def __init__(self, channels, rank=4):
        super().__init__()
        self.base = nn.Linear(channels, channels)
        self.down = nn.Linear(channels, rank, bias=False)
        self.up = nn.Linear(rank, channels, bias=False)
        self.scale = 1.0
        self.reset_adapter()

    def reset_adapter(self):
        nn.init.normal_(self.down.weight, std=0.02)
        nn.init.zeros_(self.up.weight)

    def forward(self, x):
        return self.base(x) + self.scale * self.up(self.down(x))


class RestorationTool(nn.Module):
    def __init__(self, channels=24):
        super().__init__()
        self.entry = nn.Conv2d(3, channels, 3, padding=1)
        self.spatial = nn.Conv2d(channels, channels, 3, padding=1)
        self.channel = LoRALinear(channels)
        self.prompt = nn.Parameter(torch.zeros(channels))
        self.exit = nn.Conv2d(channels, 3, 3, padding=1)

    def forward(self, image):
        z = F.gelu(self.entry(image))
        z = F.gelu(self.spatial(z))
        z = z.permute(0, 2, 3, 1)
        z = F.gelu(self.channel(z) + self.prompt)
        correction = self.exit(z.permute(0, 3, 1, 2))
        return image + correction


def configure_shared(model):
    for parameter in model.parameters():
        parameter.requires_grad_(True)
    for parameter in model.channel.down.parameters():
        parameter.requires_grad_(False)
    for parameter in model.channel.up.parameters():
        parameter.requires_grad_(False)


def configure_adapter(model, reset_prompt):
    for parameter in model.parameters():
        parameter.requires_grad_(False)
    for parameter in model.channel.down.parameters():
        parameter.requires_grad_(True)
    for parameter in model.channel.up.parameters():
        parameter.requires_grad_(True)
    model.prompt.requires_grad_(True)
    model.channel.reset_adapter()
    if reset_prompt:
        with torch.no_grad():
            model.prompt.normal_(0, 0.02)


def degrade(clean, kind):
    if kind not in {"noise", "blur", "mixed"}:
        raise ValueError("Unsupported synthetic degradation")

    image = clean
    if kind in {"blur", "mixed"}:
        padded = F.pad(image, (1, 1, 1, 1), mode="reflect")
        image = F.avg_pool2d(padded, kernel_size=3, stride=1)

    if kind in {"noise", "mixed"}:
        image = image + 0.08 * torch.randn_like(image)

    return image.clamp(0, 1)


def train_tool(model, clean, kinds, steps, learning_rate=1e-3):
    parameters = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.AdamW(parameters, lr=learning_rate)
    model.train()

    for step in range(steps):
        kind = kinds[step % len(kinds)]
        corrupted = degrade(clean, kind)
        prediction = model(corrupted)

        # Editorial loss for the small prototype.
        loss = F.l1_loss(prediction, clean)
        if not torch.isfinite(loss):
            raise FloatingPointError("Nonfinite restoration loss")

        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()

    model.eval()
    return model


def build_library(clean):
    shared = RestorationTool()
    configure_shared(shared)
    train_tool(shared, clean, ["noise", "blur", "mixed"], steps=12)

    library = {}
    for kind in ["noise", "blur"]:
        tool = copy.deepcopy(shared)
        configure_adapter(tool, reset_prompt=True)
        library[kind] = train_tool(tool, clean, [kind], steps=8)

    mixed = copy.deepcopy(shared)
    with torch.no_grad():
        mixed.prompt.copy_(library["noise"].prompt)
    configure_adapter(mixed, reset_prompt=False)
    library["mixed"] = train_tool(mixed, clean, ["mixed"], steps=8)
    return library


@torch.no_grad()
def evaluate(tool, corrupted, clean):
    prediction = tool(corrupted).clamp(0, 1)
    mse = F.mse_loss(prediction, clean).item()
    psnr = math.inf if mse == 0 else -10 * math.log10(mse)
    return {"mse": mse, "psnr": psnr}


@torch.no_grad()
def restore(image, prompt, library, planner=None,
            feedback=None, maximum_steps=3):
    """
    planner(image, prompt, history) must return a library key.
    feedback(image, history) must return a Boolean stopping decision.

    These are external interfaces, not simulated language models.
    """
    if maximum_steps < 1:
        raise ValueError("maximum_steps must be positive")

    direct = {
        "remove noise": "noise",
        "remove blur": "blur",
        "remove mixed distortion": "mixed",
    }

    current = image.clone()
    history = []

    for step in range(maximum_steps):
        if step == 0 and prompt.strip().lower() in direct:
            selected = direct[prompt.strip().lower()]
            route = "explicit prototype command"
        else:
            if planner is None:
                return current, history, "planner required"
            selected = planner(current, prompt, tuple(history))
            route = "external planner"

        if selected not in library:
            raise ValueError("Planner selected an unsupported tool")

        current = library[selected](current).clamp(0, 1)
        history.append({"tool": selected, "route": route})

        if feedback is None:
            return current, history, "not assessed"

        decision = feedback(current, tuple(history))
        if not isinstance(decision, bool):
            raise TypeError("Feedback must return a Boolean")
        if decision:
            return current, history, "accepted by external feedback"

    return current, history, "iteration budget reached"


def smoke_test():
    torch.manual_seed(42)

    # Synthetic data only. No benchmark or natural image claim.
    clean = F.interpolate(
        torch.rand(4, 3, 6, 6),
        size=(24, 24),
        mode="bilinear",
        align_corners=False,
    )
    library = build_library(clean)
    held_out = F.interpolate(
        torch.rand(2, 3, 6, 6),
        size=(24, 24),
        mode="bilinear",
        align_corners=False,
    )
    noisy = degrade(held_out, "noise")

    output, history, status = restore(
        noisy, "remove noise", library
    )
    assert output.shape == noisy.shape
    assert torch.isfinite(output).all()
    assert len(history) == 1
    assert status == "not assessed"

    print({
        "educational_evaluation": evaluate(
            library["noise"], noisy, held_out
        ),
        "history": history,
        "status": status,
    })

    # An actual multimodal planner and trained feedback model are
    # required to evaluate the paper's agent behavior.


if __name__ == "__main__":
    smoke_test()

Frequently asked questions

What does HybridAgent contribute?

It combines lightweight instruction routing, visual degradation recognition, feedback and a restoration library containing individual and mixed distortion tools.

Does FastAgent inspect the image?

The fast route interprets clear user instructions to select a tool. Ambiguous requests are passed to a visual reasoning component.

Why use a mixed distortion tool?

Several distortions can interact in ways that isolated restoration models do not handle well. A mixed tool learns to address combinations jointly.

Does the fast route always select the correct tool?

No. The accessible manuscript reports differences in selection success between fast and slow routes, including cases where the slow route is more accurate.

Does a clean feedback decision prove faithful restoration?

No. It is a learned stopping judgment. It does not establish that every original detail has been recovered correctly.

Does the reference code reproduce HybridAgent?

No. It is an educational restoration tool prototype with external planner and feedback interfaces. It does not reproduce the language models, full backbone or published results.

Read the source and inspect the prototype

Related reading