DamageRAG Shows Why Inspection AI Needs Better Perception

Analysis by the aitrendblend editorial team · 11 November 2026 · Research analysis

An inspector returns from a bridge survey with a folder of photographs. Finding one image that contains rust is manageable. Finding the image with the most rust among all images that also contain cracks requires a different kind of attention. A useful assistant must see the damage, select the right photographs, and compare the right measurements. DamageRAG shows how easily a system can get the last two steps right while the first step quietly undermines the answer.

The most useful result is the uncomfortable one

Ziyu Hu, from the Department of Electrical Engineering and Automation at Aalto University, presents DamageRAG in the published Knowledge Based Systems paper. The article became available online on 1 November 2026 and belongs to volume 355, dated 2027. That distinction matters when citing it. This is a published research article, rather than an unpublished proposal for an inspection product.

The contribution is a diagnostic benchmark and a deliberately separable pipeline. Its strongest evidence concerns what prevents useful performance. With accurate structured descriptions supplied in advance, exact numerical tools and field matching work well. With descriptions generated from the actual images, much of that apparent competence disappears. An assistant can calculate correctly from observations that were wrong before calculation began.

That is a valuable angle for the Practical AI Tools collection. Anyone building a retrieval assistant faces a version of this problem. The retrieved evidence may look orderly and precise without being reliable. DamageRAG makes that distinction measurable rather than leaving it hidden inside a fluent final response.

“fix perception first, then the pipeline pays off”

Ziyu Hu, DamageRAG, discussion of deployment implications

Why inspection questions exceed ordinary image search

A similarity search engine can find images that resemble the words in a query. An inspection question often demands a logical condition instead. Consider asking for photographs containing both cracks and rust, then ranking only those photographs by rust instance count. A visually similar rust image without a crack is an invalid candidate even if its embedding is close to the query.

There are several distinct operations here. The system must establish which damage classes are present, intersect those conditions, identify a measurement, and apply a comparison. A strong language model does not eliminate those operations. It can only perform them explicitly, approximate them, or make mistakes while producing a plausible description.

Hu places this work alongside Lewis and colleagues’ retrieval augmented generation framework, Radford and colleagues’ CLIP, and the program aided reasoning approach of Gao and colleagues. These are references 17, 14, and 42 in the source paper’s reference list. Their roles explain the design choices. Retrieval supplies evidence, visual embeddings offer a generic search mechanism, and program execution supplies exact computation.

The architectural idea is therefore a combination of established components applied to a demanding inspection setting. It is not a newly trained universal vision architecture. The practical question is whether the interface between those components preserves the facts needed to answer the query. Readers interested in evaluation design can also explore our article on benchmarking prompt optimization.

What the benchmark measures

The study uses dacl10k v2 bridge inspection images with polygon annotations for 13 damage classes. The working corpus contains 7910 annotated images, divided into 6935 training images and 975 evaluation images. The full generated benchmark contains 5794 multiple choice questions across 11 task types and three levels of reasoning depth. Those totals describe the benchmark, not the sample behind every accuracy reported in the paper.

The main comparison uses 2466 questions derived from the 975 image split. There are 378 questions at the first hop level, 1372 at the second, and 716 at the third. Later appendices call this evaluation partition the validation split, while the main setup calls it the test split. Following the image and question counts is more reliable than treating those labels as separate evaluation datasets.

Annotations produce reference reports containing damage presence, instance counts, relative areas, and spatial regions. Questions and answers are then constructed from these records. A single hop question might ask for the image with the most instances of one damage type. A second hop adds a filter. A third can identify an extreme image and compare one of its attributes with a corpus statistic.

The benchmark excludes certain ties and ambiguous cases, requires sufficient distractors, and balances answer positions. These controls make automatic grading defensible. They also shape the task distribution. Real inspection queries can be ambiguous, evidence can be incomplete, and several images can be equally relevant. Performance on filtered questions should not be read as performance on every question an engineer might ask.

Four answer choices create a random baseline of 25.0%. That baseline makes 29.5% much less encouraging than the same percentage would be in a huge answer space. Yet statistical detectability and practical usefulness are different judgments. The reported gain over Direct QA is statistically significant in the paired comparison. That does not make the system dependable for maintenance decisions.

Inside the three stage pipeline

Build a structured record for every image

DamageRAG begins offline. A multimodal model answers closed questions about each image instead of generating an unrestricted inspection narrative. It first assesses the presence of the 13 damage classes. For each detected class, it supplies attributes for instance count, area coverage, and spatial region. The resulting Damage Report becomes the evidence used by later stages.

The count choices are one, two, three, four, or at least five. Area estimates fall into five bins from below 1% to above 20%. Spatial responses select among five regions. These choices make the outputs easier to parse and compare. They do not turn approximate visual judgments into accurate measurements. In particular, a count category that saturates at five cannot retain every distinction available in polygon annotations.

The paper reports an average of 4.8 perception calls per image under its described procedure. This moves substantial work ahead of interactive answering. A cached report can support many later queries, but the cache also preserves mistakes until it is regenerated or corrected. Report provenance and versioning would therefore be useful engineering additions. The study does not establish their effect experimentally.

Retrieve by conditions rather than resemblance

The retrieval stage operates on report fields. It can filter by damage presence, intersect two labels, sort counts, select an extreme area, or calculate an aggregate. The appeal is straightforward. Once the report vocabulary matches the query vocabulary, the system can execute a condition exactly instead of hoping that embedding similarity approximates the intended constraint.

This precision is conditional on report correctness. If rust is omitted from a report, an exact rust filter will faithfully exclude that image. Better execution does not recover evidence that the representation has already discarded. This is the central tradeoff of structured intermediates. They make operations inspectable while making upstream omissions consequential.

Choose the appropriate reasoning engine

A Qwen2.5 model with 0.5 billion parameters routes queries between quantitative and qualitative reasoning. Numerical tasks go to programmatic solvers. Other tasks go to a multimodal language model reasoning over the reports. The paper’s operator taxonomy determines the branch assignments, so readers should not infer every assignment from ordinary meanings of the words quantitative and qualitative.

The distinction also matters when reading retrieval ablations. The programmatic branch can inspect report fields directly rather than being limited to the same retrieved context as the language model. A retrieval change can therefore affect one branch much more strongly than the other. An overall score hides that asymmetry unless the task level results are examined.

The mathematics behind an apparently simple question

The following notation is an explanatory reconstruction of the report operations, rather than a claim that the paper introduces a new optimization objective. Let a report for image i contain a set of present damage classes P, counts c, area fractions a, and spatial regions Q. This is enough to express a filtered ranking without a language model doing the arithmetic.

For the question about rust among images containing cracks, the eligible set contains images whose reports include the filtering class. The winning image is the eligible image with the largest reported count for the ranking class. A real implementation must define how missing values and ties are handled. The benchmark removes some ambiguous cases during construction.

Polygon area is obtained using the shoelace calculation and can be divided by image area to form a relative measure. Below, x and y are polygon vertex coordinates, with the final vertex linked back to the first. W and H are image dimensions. Combining several instances requires a declared aggregation rule, especially if annotations overlap. This expression describes one polygon and does not silently equate a sum of overlapping polygons with union area.

Relative image area is a visual attribute. It is not automatically a calibrated physical measurement or a structural severity score. A close photograph and a distant photograph can show different image fractions of the same defect. Interpreting area rankings as repair priorities would require additional engineering evidence beyond the benchmark.

For adaptation, the general supervised language modeling objective rewards the correct answer tokens given the image and prompt. The main DamageRAG pipeline itself has no training loss. The separate LoRA experiment adapts the perception model. The expression below explains ordinary supervised answer token learning and should not be mistaken for a novel DamageRAG loss.

Read the results by evidence condition

The table below draws from Table 6 and the adaptation analysis in Appendix F.33. All accuracy values refer to the evaluation set of 2466 questions, except the theoretical random baseline. Ground truth reports, generated reports, and adapted generated reports are separate experimental conditions. Mixing them would turn a useful diagnostic study into an exaggerated performance claim.

With ground truth reports, routing suitable questions to code raises accuracy from 57.9% to 82.9%. This establishes that exact computation can remove errors introduced by model based numerical reasoning in this setting. Replacing those reports with generated reports then lowers accuracy to 29.5%. The downstream machinery remains useful, but its inputs no longer support the same answers.

Against Direct QA, the generated report pipeline gains 3.0 percentage points with a reported p value of 0.020. Against naive CLIP retrieval, its 29.5% trails 31.0%, with a reported p value of 0.249. The second comparison does not establish a statistically significant difference. It certainly does not support advertising the unadapted pipeline as superior to the strongest naive baseline.

The 100% programmatic result with ground truth reports deserves special care. The reference answers and the symbolic computations derive from the same annotated information. That is a consistency check on a controlled task, not proof that arbitrary inspection questions can be answered perfectly. The value is that a correct reference makes errors in other stages easier to localize.

Why improved perception helps, and why it is insufficient

The sensitivity experiment adapts Qwen2.5-VL-7B on the 6935 training images. Macro perception F1 increases from approximately 0.37 to 0.60 after three epochs. The realistic adaptive pipeline reaches 37.47%, with the reported interval from 35.56% to 39.33%. The headline rounds this to 37.5%. This is evidence that changing the perception model can improve the complete system without redesigning the report based pipeline.

The gain is conditional on how the rest of the system uses that adaptation. Giving the same perception adapter to naive CLIP retrieval leaves its aggregate accuracy at 31.0%. A text only Direct QA baseline adapted on its own task reaches 30.7%. Under these stronger controls, the adapted pipeline leads the best naive baseline by about 6.5 percentage points. It leads adapted Direct QA by about 6.8 points.

That is more informative than comparing adapted DamageRAG only with an unadapted weak baseline. It supports an interaction between structured reports and improved perception. It does not show that any adapter improves any retrieval architecture, or that every gain belongs exclusively to the pipeline design.

Rare classes also complicate the average. The appendix reports improvement for several difficult classes but regression for rock pocket and concrete corrosion. Class balancing does not clearly solve that problem. A practitioner should therefore inspect the classes relevant to the intended application instead of treating macro F1 as a guarantee that all important defects improved.

The language interface has its own failure modes

The main experiments do not require the system to recover all query structure from unrestricted user language. Appendix F.7 explains that the structured operators and damage arguments come from benchmark records. This is a substantial simplifying assumption. Reading a supplied operation is easier than recognizing the operation in a question phrased by an inspector.

The appendix removes that assumption and examines rewritten questions. In its matched quantitative evaluation, performance falls from 79.85% with supplied structure and template wording to 41.93% and 46.35% when structure must be recovered from two rewritten versions. These figures concern ground truth reports and a particular matched protocol. They must not be directly combined with the 29.5% generated report result.

The 79.85% reference also differs from the headline 82.9% because the experiment drops a benchmark field containing relevant image candidates. That change avoids giving one side privileged information unavailable to a parser. The nearby full benchmark result of 79.5% in another appendix is a different number under a different evaluation. Similar looking percentages do not establish interchangeable experiments.

Routing is another concentrated source of loss. The small router agrees with oracle routing on 90.4% of template questions, but the 236 misrouted questions suffer a much larger downstream penalty. Accuracy on that subset is 25.85% with the learned router versus 76.27% with oracle routing. An apparently strong routing average can conceal precisely the mistakes that most damage the final answer.

A practical evaluation plan for researchers

Begin with reference reports to verify the report schema and numerical operations. Then replace those reports with model generated observations while keeping the queries and evaluation protocol fixed. The resulting change isolates what the perception layer contributes. Only after that comparison should a researcher change the retrieval method, reasoning model, or parser.

Keep the router fixed when testing retrieval. Keep the report source fixed when testing reasoning. Keep answer candidates, question wording, and any privileged metadata consistent between compared systems. These controls prevent a change in information access from being mistaken for a better architecture. The paper’s revisions offer a useful example of why matched comparisons can alter the interpretation of an attractive headline.

Log evidence identifiers, report fields, selected operators, routing decisions, and final answers together. This is an editorial recommendation, not a reported deployment feature. Such a record allows a reviewer to distinguish an omitted defect from a failed filter or an incorrect numerical comparison. Without it, the final wrong answer leaves too many plausible explanations.

Human oversight should be evaluated as a workflow rather than asserted as a solution. A reviewer may catch a visible inconsistency, yet still miss an image excluded during retrieval. Our discussion of evaluating human and AI teams provides a related perspective on testing whether combined work actually improves outcomes.

Limitations that should shape the next experiment

The evidence comes from one bridge inspection dataset. Transfer to tunnels, facades, or pipelines remains a hypothesis. Even within bridge imagery, acquisition conditions, camera distance, annotation practice, and the frequency of individual damage types can change the difficulty. A successful result on one collection does not establish general inspection reliability.

Multiple choice evaluation measures selection among supplied alternatives. It does not measure the ability to write a complete inspection report, discover an unexpected defect category, or communicate uncertainty in an open response. The benchmark also generates many questions from the same image corpus. Question count and independent visual diversity should not be treated as the same quantity.

The paper finds that chain of thought prompting reduces accuracy for its arithmetic heavy report tasks. That is a scoped observation about the tested prompts and systems. It does not establish that explicit reasoning is universally harmful. Similarly, adding images to accurate reports slightly reduces performance in the tested reasoning condition. It does not establish that images are unnecessary for inspection.

The gap between benchmark answers and structural decisions remains substantial. The study does not validate autonomous repair prioritization or replace engineering assessment. Inspection deployment would require evaluation against operational outcomes and the relevant professional workflow. This article explains the research and does not provide a procedure for judging the safety of a real structure.

Where this leaves DamageRAG

DamageRAG’s core achievement is to make a failed answer more informative. It separates the observation, retrieval, and reasoning stages well enough to show that improving the latter two cannot compensate for unreliable damage descriptions. The benchmark supplies a controlled setting in which that claim can be tested rather than merely suspected.

The conceptual shift is toward evaluating the intermediate evidence. A report can be syntactically valid and internally consistent while describing the wrong things. Once retrieval operates on that report, the system can become precise about an inaccurate world. The difference between correct computation and correct observation deserves its own evaluation budget.

The design pattern may transfer to other applications that depend on structured visual evidence, including equipment maintenance and visual quality control. What transfers first is the method of diagnosis. Neither the bridge specific label set nor the reported accuracy should be carried into another domain without new annotations and new tests.

Several limitations remain unresolved. Generated reports still produce weak question answering accuracy after adaptation. Counting remains difficult, some rare classes regress, and free wording exposes fragility in both decomposition and routing. These are distinct problems, so solving one should not be presented as solving the others.

The next useful experiments would separately improve counting, assess report uncertainty, test query interpretation outside templates, and evaluate inspection workflows with expert review. These are research directions suggested by the findings, not capabilities demonstrated by the current system. A reliable assistant must preserve the right evidence before its reasoning can earn trust.

Executable implementation and reproduction boundary

The code below is a complete educational report engine with all 11 task intents, an evaluation function, a runnable synthetic smoke test, and an optional PyTorch supervised adaptation loop. It is not the authors’ implementation and does not reproduce their scores. The report mechanics smoke test was executed successfully. The optional PyTorch training path was syntax checked but not executed because PyTorch and the pretrained model were unavailable in the authoring environment.

The implementation intentionally receives structured queries and reports as inputs. It does not pretend to perform image perception or unrestricted query understanding. It solves qualitative intents symbolically for teaching, whereas DamageRAG sends those tasks to its language model. For comparisons whose precise multiple choice thresholds are not established here, it returns numerical evidence instead of inventing answer thresholds.

No custom neural layers or novel losses are introduced in the paper’s training free pipeline, so none are invented for this article. The optional training loop uses the model’s causal language modeling loss. It requires prepared multimodal batches and an appropriately configured model. It omits the exact adapter setup, cosine schedule, preprocessing, and official benchmark harness, and therefore cannot serve as an exact adaptation reproduction. Use the repository cited by the paper for that work.

"""Educational report-level DamageRAG mechanics, not the official reproduction.
Run: python demo.py
Optional train_perception requires PyTorch and an already configured causal
vision-language model (with LoRA attached if desired) plus prepared batches.
No image model, free-text parser, or learned router is simulated here.
"""
from dataclasses import dataclass
from collections import Counter
from math import isfinite

@dataclass(frozen=True)
class Report:
    image: str
    counts: dict
    areas: dict
    quadrants: dict

    def __post_init__(self):
        if not self.image or not self.counts:
            raise ValueError("Image identifier and explicit class fields required")
        if set(self.counts) != set(self.areas) or set(self.counts) != set(self.quadrants):
            raise ValueError("All attributes must cover the same classes")
        for label, count in self.counts.items():
            if type(count) is not int or count < 0:
                raise ValueError("Counts must be nonnegative integers")
            area = self.areas[label]
            if not isfinite(area) or not 0 <= area <= 1:
                raise ValueError("Areas must be finite image fractions")
            if count == 0 and (area != 0 or self.quadrants[label]):
                raise ValueError("Absent class has inconsistent attributes")

    @property
    def present(self):
        return {d for d, n in self.counts.items() if n > 0}

@dataclass(frozen=True)
class Query:
    # Structured input is supplied explicitly, not recovered from prose.
    op: str
    labels: tuple
    quadrant: str = "center"

class Abstain(ValueError):
    pass

def unique_max(items, score):
    if not items:
        raise Abstain("No eligible evidence")
    ranked = sorted(items, key=score, reverse=True)
    if len(ranked) > 1 and abs(score(ranked[0]) - score(ranked[1])) < 1e-12:
        raise Abstain("Ambiguous maximum")
    return ranked[0]

class ReportEngine:
    """Symbolic teaching engine for all 11 task intents.

    It also solves qualitative intents symbolically, unlike the paper's MLLM
    branch. compare_avg returns a numeric ratio, spatial_cooccurrence a share,
    and filter_compare_area mean areas. Official MCQ thresholds are not guessed.
    It uses the entire corpus; no benchmark answer candidates enter retrieval.
    """
    def __init__(self, reports):
        self.reports = list(reports)
        if not self.reports:
            raise ValueError("Empty report corpus")
        if len({r.image for r in self.reports}) != len(self.reports):
            raise ValueError("Duplicate image identifiers")
        self.classes = set(self.reports[0].counts)
        if any(set(r.counts) != self.classes for r in self.reports):
            raise ValueError("Explicit common vocabulary required")

    def retrieve(self, labels):
        return [r for r in self.reports if set(labels) <= r.present]

    def answer(self, q):
        two = {"filter_rank", "filter_chain", "filter_locate",
               "filter_compare_area", "spatial_cooccurrence"}
        one = {"max_instance", "max_area", "filter_diversity", "compare_avg",
               "conditional_aggregate", "extreme_attribute"}
        if q.op not in one | two:
            raise ValueError("Unknown operator")
        expected = 2 if q.op in two else 1
        if len(q.labels) != expected or not set(q.labels) <= self.classes:
            raise ValueError("Invalid query arguments")
        d = q.labels[0]
        e = q.labels[-1]
        rows = self.retrieve(q.labels if q.op in {
            "filter_chain", "filter_compare_area", "spatial_cooccurrence"
        } else (d,))
        if not rows:
            raise Abstain("No eligible evidence")
        if q.op in {"max_instance", "filter_rank"}:
            return unique_max(rows, lambda r: r.counts[e]).image
        if q.op == "max_area":
            return unique_max(rows, lambda r: r.areas[d]).image
        if q.op in {"filter_diversity", "filter_chain"}:
            return unique_max(rows, lambda r: len(r.present - set(q.labels))).image
        if q.op == "filter_locate":
            votes = Counter(z for r in rows for z in set(r.quadrants[e]))
            return unique_max(list(votes), votes.get)
        if q.op == "filter_compare_area":
            return {k: sum(r.areas[k] for r in rows) / len(rows) for k in (d, e)}
        if q.op == "compare_avg":
            winner = unique_max(rows, lambda r: r.counts[d])
            # Explicit teaching convention: average over ALL corpus images.
            mean = sum(r.areas[d] for r in self.reports) / len(self.reports)
            if mean == 0:
                raise Abstain("Zero denominator")
            return winner.areas[d] / mean
        if q.op == "conditional_aggregate":
            eligible = [r for r in rows if q.quadrant in r.quadrants[d]]
            votes = Counter(k for r in eligible for k in r.present - {d})
            return unique_max(list(votes), votes.get)
        if q.op == "extreme_attribute":
            winner = unique_max(rows, lambda r: r.counts[d])
            return unique_max(list(winner.present - {d}), winner.areas.get)
        return sum(bool(set(r.quadrants[d]) & set(r.quadrants[e]))
                   for r in rows) / len(rows)

def evaluate(engine, cases):
    correct = answered = 0
    for query, expected in cases:
        try:
            predicted = engine.answer(query)
            answered += 1
            correct += int(predicted == expected)
        except Abstain:
            pass  # Abstentions count as incorrect in overall accuracy.
    n = len(cases)
    return {"n": n, "accuracy": correct / n if n else None,
            "coverage": answered / n if n else None}

def train_perception(model, loader, epochs=3, lr=1e-4, accumulation=8):
    """Optional PyTorch SFT loop, NOT an exact Appendix F.6 reproduction.

    Model must expose a causal LM loss. Batches must already include images,
    input_ids, attention_mask and labels. Prompt and padding labels must be
    -100, leaving only assistant answer tokens supervised. The caller must
    freeze the vision encoder and attach the intended adapters beforehand.
    No new loss or architecture is attributed to DamageRAG.
    """
    import torch
    trainable = [p for p in model.parameters() if p.requires_grad]
    if not trainable or len(loader) == 0:
        raise ValueError("Need trainable adapter parameters and nonempty data")
    device = next(model.parameters()).device
    optimizer = torch.optim.AdamW(trainable, lr=lr)
    history = []
    model.train()
    for _ in range(epochs):
        optimizer.zero_grad(set_to_none=True)
        total = 0.0
        for i, batch in enumerate(loader):
            batch = {k: v.to(device) if hasattr(v, "to") else v
                     for k, v in batch.items()}
            loss = model(**batch).loss
            if not torch.isfinite(loss):
                raise ValueError("Nonfinite loss")
            total += loss.detach().item()
            start = (i // accumulation) * accumulation
            window = min(accumulation, len(loader) - start)
            (loss / window).backward()
            if (i + 1) % accumulation == 0 or i + 1 == len(loader):
                torch.nn.utils.clip_grad_norm_(trainable, 1.0)
                optimizer.step()
                optimizer.zero_grad(set_to_none=True)
        history.append(total / len(loader))
    return history

def smoke_test():
    def report(name, rust, crack, rust_area, crack_area):
        return Report(name, {"Rust": rust, "Crack": crack},
                      {"Rust": rust_area, "Crack": crack_area},
                      {"Rust": ("center",) if rust else (),
                       "Crack": ("center",) if crack else ()})
    rows = [report("a.jpg", 2, 1, .10, .03),
            report("b.jpg", 4, 1, .30, .02),
            report("c.jpg", 6, 0, .20, 0)]
    engine = ReportEngine(rows)
    cases = [(Query("max_instance", ("Rust",)), "c.jpg"),
             (Query("max_area", ("Rust",)), "b.jpg"),
             (Query("filter_rank", ("Crack", "Rust")), "b.jpg"),
             (Query("filter_locate", ("Crack", "Rust")), "center")]
    assert evaluate(engine, cases)["accuracy"] == 1.0
    assert engine.answer(Query("spatial_cooccurrence", ("Rust", "Crack"))) == 1
    try:
        engine.answer(Query("filter_diversity", ("Rust",)))
    except Abstain:
        pass
    else:
        raise AssertionError("Ties should trigger abstention")
    # A perception mistake propagates into a different numerical answer.
    corrupted = [rows[0], rows[1], report("c.jpg", 1, 0, .20, 0)]
    assert ReportEngine(corrupted).answer(Query("max_instance", ("Rust",))) == "b.jpg"
    print("Report mechanics smoke test passed. No paper result reproduced.")

if __name__ == "__main__":
    smoke_test()

Related reading