Multi-Agent System Explain Why a Defect Actually Happened

The core problem, a defect means different things at different points in its life

Most industrial defect detection systems answer two questions. Where is the damage, and what category does it fall into. That has real value for automated inspection, but it stops well short of what a quality manager actually needs, which is an explanation good enough to act on. The paper’s authors illustrate this with a simple, concrete example. A metal mesh product can show up broken at several different points in its life, storage, pressing or welding, or surface treatment, and each of those moments has a completely different explanation. In storage, the likely cause is mechanical compression from overstacking. During pressing or welding, it is more likely improper tension control during the operation itself. During surface treatment, uneven processing temperatures or a bad coating procedure are the more plausible culprits.

A defect detection system that cannot tell these three situations apart will generate the same generic explanation regardless of which one actually happened, and a generic explanation is not an actionable one. A prior system built by some of this paper’s own authors attempted to fix part of this problem by generating free text reports covering causes, impacts, and solutions, an improvement over plain bounding boxes and category labels. But that earlier system still had no explicit notion of supply chain stage, so it could not distinguish the storage explanation from the welding explanation from the surface treatment explanation for the same visual defect. The current paper’s whole premise is that stage awareness is not an optional nice to have, it is the difference between a plausible sounding answer and a genuinely correct one.

Why multiple choice, and why a new dataset

The authors also make a smaller but important methodological choice worth calling out. Free text generation, the format used in the prior system, is hard to score objectively, since two reasonable sounding explanations can differ in wording while both being defensible, which usually means a human has to sit down and judge each answer. A multiple choice format sidesteps that problem entirely. Either the model picks the option that matches the correct cause, impact, and solution for the correct stage, or it does not, which supports fast, consistent, statistical evaluation at scale. The tradeoff is that building a good multiple choice dataset, one where the wrong options are genuinely tempting rather than obviously wrong, takes real design work, which is most of what Section 3 of the paper is actually about.

How the dataset was actually built

The object categories all come from MVTec AD, a well known defect detection benchmark, giving the dataset a familiar and credible base. On top of that, the authors built a normalized eight stage supply chain taxonomy that is explicitly not meant as a universal industrial standard, just a consistent scaffold for this specific benchmark. The eight stages are manufacturing or forming, assembly or joining, surface treatment or finishing, quality inspection, packaging or storage, transportation or logistics, sales or distribution, and installation, usage, or maintenance. Only the stages that genuinely apply to a given product category are kept, and product specific stage names are allowed as long as they map back to one of these eight general categories.

Two separate volunteer groups worked in sequence. The first group researched and drafted stage definitions per object category using prior literature and open resources, then used GPT-4o to reconcile sources that emphasized different aspects of the same product’s lifecycle. The second group ran a three round verification pass, first checking whether each proposed stage actually applied to that product, then checking whether it was visually and semantically distinguishable from the other stages, and finally resolving any remaining disagreements, escalating to domain experts in industrial inspection when the volunteers could not agree themselves.

Images came from two sources, ordinary web search and Midjourney, an AI image generation tool, used specifically for stages that were hard to find real photographs of. Every candidate image had to be unanimously approved by three independent reviewers before it made it into the dataset. Comprehensive information, meaning the cause, impact, and solution text tied to each defect and stage combination, went through a similarly disciplined two stage annotation and review process, with an explicit factual correctness pass checking that each description was logically consistent with the defect type and stage, practically relevant, feasible, and free of unsupported or invented specifics. The validation rules the paper lays out for this pass are genuinely careful, requiring reviewers to check consistency, plausibility, and stage alignment for every single annotation rather than trusting the initial draft.

The result is measured two ways worth noting. A repetition analysis of question and correct answer pairs found an overall average repetition rate of about 16 percent across the harder two question types, meaning the large majority of questions are genuinely unique rather than recycled templates, though a handful of categories with naturally limited defect type diversity, such as bottles and toothbrushes, show somewhat higher repetition. Separately, the authors report Cohen’s kappa agreement between expert judgment and the final dataset labels, coming in at 0.9182 for defect type questions, 0.8920 for stage prediction questions, and 0.8812 for comprehensive information questions, all comfortably in the range conventionally read as strong agreement, even though the two harder question types involve real ambiguity between similar looking stages such as storage and sales.

How SA-VQA actually works

Instead of asking a single vision model to guess the answer from an image, this multi-agent system divides the reasoning into four distinct steps handled by specialized virtual workers.

Two design choices stand out as more than just plumbing. First, the Stage Concluder deliberately generates comprehensive information for every candidate stage rather than only the one it thinks is most likely, which means that information is ready and waiting no matter what the Stage Predictor later decides, rather than needing to be regenerated after the fact. Second, the Summarizer’s job is explicitly not to just read everything and average it together. Its prompt instructs it to identify the comprehensive information tied specifically to the predicted stage, filter out the information tied to every other candidate stage, and only then compare that narrowed, stage matched evidence against the answer options. That filtering step is what lets the same visual defect produce a different final answer depending on which stage the system believes it is looking at, which is the entire point of the task.

All four agents communicate through fixed, structured JSON formatted prompts rather than free form back and forth, which the authors argue keeps the information exchange consistent and avoids agents drifting off task. During evaluation, every agent runs in a deterministic mode, a fixed low temperature setting and instructions to output only a single option letter, with a simple fallback rule that extracts the first valid option letter if the model adds extra text anyway.

What the experiments actually show

SA-VQA is compared against seven baselines spanning three families, models that skip region proposals entirely such as ViLT, models built on Faster R-CNN region features such as VisualBERT, UNITER, LXMERT, and ModCR, and general purpose large vision language models used directly without fine tuning, GPT-4V and LLaVA-NeXT.

SA-VQA leads every single category against every baseline. Its margin over the best conventional fine tuned baseline, ModCR, is 20.03 percentage points on the macro average, and its margin over the best zero shot general purpose model, GPT-4V, is 17.66 points. Worth clarifying since the paper’s own text describes this as an improvement of over 20 percent. A jump from 39.98 to 60.01 is 20.03 percentage points, which is actually closer to a 50 percent relative improvement over that baseline’s own score, a noticeably bigger number than the phrasing in the paper might suggest at a glance.

A few individual results are worth sitting with. GPT-4V, used with no fine tuning at all, beats every conventional fine tuned baseline on the macro average, which says something about how far general purpose multimodal reasoning has come, but it still lands more than 17 points behind SA-VQA, suggesting that generic visual and language competence alone does not automatically confer the specific, stage aware reasoning this task demands. ModCR, the strongest fine tuned conventional baseline, does especially well on comprehensive information questions specifically, hinting that its multi level alignment approach genuinely helps with the harder reasoning category even though its Faster R-CNN based visual features are a full generation behind large vision language models.

Does using AI generated stage images bias the results

Since a meaningful share of the stage related images came from Midjourney rather than real photographs, the authors ran a direct comparison, sampling 100 test cases with real stage images and 100 with generated stage images and running SA-VQA on both subsets separately. The result was 58.43 percent accuracy on real images against 61.11 percent on generated images, a gap of 2.68 points, with generated images doing slightly better, which the authors attribute to generated images tending to show cleaner, more prototypical visual cues than real photographs with their messier backgrounds and occlusions.

That is a reasonable explanation, and the direction of the gap, generated images looking easier rather than harder, is a comforting one. Still, it is worth being a little cautious about how much weight to put on a 2.68 point difference measured across only 100 samples per group. That is a small enough sample that a gap this size sits well within the kind of noise you would expect from resampling, so treating this result as confirmation that synthetic images introduce no meaningful bias is a bit stronger a conclusion than 100 samples can really support on its own, even though the direction and the underlying explanation both make intuitive sense.

What the ablation study reveals about where the value comes from

The authors tested four configurations, the full four agent pipeline, and three reduced versions with specific agents removed.

The single most informative result here is which individual agent removal hurts the most. It is not the Defect Discerner, and it is not the Stage Predictor, it is the Supply Chain Stage Concluder, the agent whose entire job is generating a rich, candidate rich set of possible stages and their comprehensive information before anyone commits to a specific answer. Losing it drops accuracy further than losing either of the other two agents individually. That is a meaningful finding because it points to where the system’s real advantage lives, not in the final step of picking the best matching option, but earlier, in having a wide, well reasoned set of candidate explanations available to filter through in the first place.

The bottom row is its own kind of sanity check. Stripping away everything except the Summarizer and feeding it the raw image and question directly turns SA-VQA into something close to a single shot large vision language model baseline, and its score, 38.89 percent, lands right around where the weaker conventional baselines sit in Table 5. That is reassuring in the sense that it confirms the big jump to 60.01 percent really is coming from the multi agent decomposition itself, not from some hidden advantage baked into the Summarizer or the underlying model alone.

What the case studies show about how errors actually happen

The paper walks through three representative cases that make the failure modes concrete. In the first, every agent gets its part right, and the pipeline lands on the correct answer cleanly. In the second, the Stage Concluder mislabels one candidate stage using slightly wrong terminology, calling it post tanning rather than the true dyeing and finishing stage, yet the Stage Predictor still manages to pick the correct stage using the actual visual evidence in the image, and the Summarizer still lands on the right final answer. That is a genuinely useful result, since it shows the system does not depend on any single agent getting every detail exactly right, later agents can compensate for an earlier agent’s imprecise but not fatally wrong output.

The third case is where things actually break. The Stage Predictor misreads a quality inspection scene as a usage scene, an understandable mistake given how visually similar an assembly or inspection setting can look to a product already in use. Because the Summarizer trusts the Stage Predictor’s output and filters its evidence accordingly, it ends up reasoning from the wrong stage’s causes, impacts, and solutions entirely, even though the Defect Discerner correctly identified the object and defect from the start. This is a clean illustration of exactly the kind of error propagation you would expect from a sequential pipeline, a single wrong link partway through can undo work that every other agent got right.

Cost, latency, and where the authors themselves draw the line

To their credit, the authors do not oversell this as ready for the shop floor. Each of the four agents in SA-VQA is its own call to GPT-4o, so the pipeline costs roughly four times what a single direct model call would cost, and the authors report that a full run typically takes tens of seconds per sample rather than the millisecond level response a real time inspection line would need. They frame the current system as suited to decision support, quality analysis, and maintenance planning rather than live production line screening, and they suggest several concrete ways to bring the cost down in a real deployment, caching the Stage Concluder’s output for object and defect combinations that repeat often, swapping some agents for smaller or distilled local models, and only triggering the full four agent pipeline for cases a faster, lighter screening step flags as ambiguous or high risk.

Honest limitations of this study

Reference implementation, the SA-VQA agent pipeline in Python

SA-VQA is not a model trained end to end by backpropagation, it is an orchestration of prompts sent to a frozen, pretrained large vision language model. So rather than force an artificial loss function onto a system that does not use one, the snippet below implements the actual moving parts described in the paper, the shared information pool, the four agents, the sequential orchestration, an ablation harness matching the shape of the paper’s own Table 7, and the macro average accuracy metric the paper uses to score everything. Real GPT-4o calls are replaced with a small deterministic mock function so the whole thing runs anywhere without an API key, and it has been executed end to end to confirm it produces no errors.

# -----------------------------------------------------------------------
# Educational reference implementation inspired by SA-VQA, described in
# Xie et al., "Stage-aware industrial defect understanding via multi-
# agent collaboration" (Knowledge-Based Systems, 2026, doi 10.1016/
# j.knosys.2026.116321). Unlike a typical neural network paper, SA-VQA
# is a multi-agent orchestration pipeline built on frozen, pretrained
# large vision language models rather than a model trained end to end
# with backpropagation. This script mirrors that reality. It implements
# the shared information pool, the four cooperating agents, the
# sequential orchestration logic, the macro average accuracy metric the
# paper uses, and an ablation harness that reproduces the structure of
# the paper's own ablation table, all with small deterministic mock
# functions standing in for real large vision language model API calls.
# -----------------------------------------------------------------------

import random
from dataclasses import dataclass
from typing import Optional


# -------------------------------------------------------------------
# 1. The shared information pool, Section 4.6. A simple structured
#    store that every agent reads from and writes to, mirroring the
#    paper's blackboard style design rather than direct agent to
#    agent messaging.
# -------------------------------------------------------------------
@dataclass
class SharedInformationPool:
    defect_object_report: Optional[dict] = None
    comprehensive_information_report: Optional[list] = None
    supply_chain_report: Optional[dict] = None


# -------------------------------------------------------------------
# 2. Agent one, the Defect Discerner, Section 4.1. In the real system
#    this wraps an AnomalyGPT style model. Here it is a deterministic
#    mock lookup so the pipeline is runnable without any real model
#    weights or API key.
# -------------------------------------------------------------------
def defect_discerner(sample, pool, backend_fn):
    result = backend_fn("defect_discerner", sample)
    pool.defect_object_report = result
    return result


# -------------------------------------------------------------------
# 3. Agent two, the Supply Chain Stage Concluder, Section 4.2. It reads
#    the defect object report and proposes a full candidate set of
#    plausible supply chain stages, each with its own cause, impact,
#    and solution, rather than committing to a single stage early.
# -------------------------------------------------------------------
def stage_concluder(sample, pool, backend_fn):
    result = backend_fn("stage_concluder", sample, pool.defect_object_report)
    pool.comprehensive_information_report = result
    return result


# -------------------------------------------------------------------
# 4. Agent three, the Supply Chain Stage Predictor, Section 4.3. It
#    combines the stage related image with the candidate stage set to
#    decide which stage is actually depicted.
# -------------------------------------------------------------------
def stage_predictor(sample, pool, backend_fn):
    result = backend_fn("stage_predictor", sample, pool.comprehensive_information_report)
    pool.supply_chain_report = result
    return result


# -------------------------------------------------------------------
# 5. Agent four, the Summarizer, Section 4.4. It filters the candidate
#    stage information down to just the predicted stage, compares that
#    evidence against the multiple choice options, and returns a single
#    option label.
# -------------------------------------------------------------------
def summarizer(sample, pool, backend_fn):
    predicted_stage = pool.supply_chain_report["stage"] if pool.supply_chain_report else None
    matching_evidence = None
    if pool.comprehensive_information_report:
        for entry in pool.comprehensive_information_report:
            if entry["stage"] == predicted_stage:
                matching_evidence = entry
                break
    return backend_fn("summarizer", sample, pool.defect_object_report, matching_evidence)


# -------------------------------------------------------------------
# 6. The orchestrator. Agents run sequentially, exactly as in the
#    paper, and any agent can be switched off to reproduce the
#    structure of the paper's own ablation study in Table 7.
# -------------------------------------------------------------------
def run_sa_vqa(sample, backend_fn, use_discerner=True, use_concluder=True,
               use_predictor=True, use_summarizer=True):
    pool = SharedInformationPool()

    if use_discerner:
        defect_discerner(sample, pool, backend_fn)
    if use_concluder:
        stage_concluder(sample, pool, backend_fn)
    if use_predictor:
        stage_predictor(sample, pool, backend_fn)

    if not use_summarizer:
        # Matches the paper's most extreme ablation direction in spirit,
        # answering directly without the final evidence filtering step.
        return backend_fn("direct_answer", sample)

    return summarizer(sample, pool, backend_fn)


# -------------------------------------------------------------------
# 7. Evaluation, Section 5.2. Accuracy per question category, then a
#    macro average across the three categories, matching the paper's
#    own metric definition, where each category counts equally
#    regardless of how many questions it contains.
# -------------------------------------------------------------------
def evaluate(predictions, ground_truth, categories):
    per_category_correct = {}
    per_category_total = {}
    for pred, truth, cat in zip(predictions, ground_truth, categories):
        per_category_total[cat] = per_category_total.get(cat, 0) + 1
        if pred == truth:
            per_category_correct[cat] = per_category_correct.get(cat, 0) + 1

    per_category_accuracy = {
        cat: 100.0 * per_category_correct.get(cat, 0) / per_category_total[cat]
        for cat in per_category_total
    }
    macro_average = sum(per_category_accuracy.values()) / len(per_category_accuracy)
    return per_category_accuracy, macro_average


# -------------------------------------------------------------------
# 8. A small deterministic mock backend standing in for real GPT-4o
#    style agent calls, so the whole pipeline runs without any network
#    access, API key, or model weights.
# -------------------------------------------------------------------
def make_mock_backend(error_rate=0.15, seed=0):
    rng = random.Random(seed)

    def backend_fn(role, sample, *context):
        if role == "defect_discerner":
            return {"object_type": sample["object_type"], "defect_type": sample["defect_type"]}
        if role == "stage_concluder":
            return [
                {"stage": stage, "cause": f"cause for {stage}", "impact": f"impact for {stage}",
                 "solution": f"solution for {stage}"}
                for stage in sample["candidate_stages"]
            ]
        if role == "stage_predictor":
            correct = rng.random() > error_rate
            predicted = sample["true_stage"] if correct else rng.choice(sample["candidate_stages"])
            return {"stage": predicted}
        if role == "summarizer":
            defect_report, evidence = context
            if evidence is None:
                return rng.choice(sample["options"])
            correct = rng.random() > error_rate
            return sample["correct_option"] if correct else rng.choice(sample["options"])
        if role == "direct_answer":
            return rng.choice(sample["options"])
        raise ValueError(f"unknown role {role}")

    return backend_fn


# -------------------------------------------------------------------
# 9. Smoke test on dummy data, running both the full pipeline and the
#    ablation variants, matching the shape of the paper's Table 7.
# -------------------------------------------------------------------
def run_smoke_test():
    stages = ["storage", "sales", "usage", "quality_inspection"]
    samples = []
    for i in range(40):
        true_stage = random.Random(i).choice(stages)
        samples.append({
            "object_type": "metal_grid",
            "defect_type": "broken",
            "candidate_stages": stages,
            "true_stage": true_stage,
            "options": ["A", "B", "C", "D"],
            "correct_option": "A",
            "category": "comprehensive_information_understanding",
        })

    ablation_settings = {
        "full pipeline": dict(use_discerner=True, use_concluder=True, use_predictor=True, use_summarizer=True),
        "without stage concluder": dict(use_discerner=True, use_concluder=False, use_predictor=True, use_summarizer=True),
        "without stage predictor": dict(use_discerner=True, use_concluder=True, use_predictor=False, use_summarizer=True),
        "summarizer only": dict(use_discerner=False, use_concluder=False, use_predictor=False, use_summarizer=False),
    }

    for label, settings in ablation_settings.items():
        backend_fn = make_mock_backend(error_rate=0.2, seed=42)
        predictions, ground_truth, categories = [], [], []
        for sample in samples:
            pred = run_sa_vqa(sample, backend_fn, **settings)
            predictions.append(pred)
            ground_truth.append(sample["correct_option"])
            categories.append(sample["category"])
        _, macro_avg = evaluate(predictions, ground_truth, categories)
        print(f"{label}: {macro_avg:.2f} percent accuracy on this dummy sample")

    print("smoke test finished without errors")


if __name__ == "__main__":
    run_smoke_test()

Conclusion

The most useful idea in this paper travels well beyond industrial defects specifically. Context changes meaning, and a system that answers questions about the physical world without an explicit model of context is answering a simpler question than the one that actually matters. SA-IDU forces a model to reason about where something sits in a process, not just what it looks like, and that framing applies just as easily to medical imaging read alongside a patient’s treatment stage, to security footage read alongside a building’s occupancy schedule, or to any other domain where the same visual evidence means different things depending on when and where it was captured.

When we look closely at what makes this multi-agent system work so well we find that the magic does not happen during the final decision phase. It is generating a rich, deliberately unresolved set of candidate explanations before committing to any single interpretation. The Stage Concluder’s job of proposing every plausible stage with its own reasoning, rather than jumping straight to a best guess, is what the rest of the pipeline actually leans on. That is a genuinely transferable lesson for anyone building a multi agent system, resist collapsing to a single answer too early, and give later stages real material to filter through rather than a single premature conclusion to simply rubber stamp.

The case studies make an equally important point about failure. This is a sequential pipeline, and sequential pipelines are only as reliable as their weakest, most ambiguous link. Here, that weak link is distinguishing visually similar supply chain stages, quality inspection scenes that look like usage scenes, storage scenes that look like sales scenes. Any team adopting a similar multi agent design should expect the same pattern, correctness bottlenecked by whichever single reasoning step is hardest to get right, rather than by the overall architecture.

The honest limitations matter here as much as the headline number. A macro average accuracy just above 60 percent, a dataset the authors themselves cannot share, a handful of framing and reporting inconsistencies uncovered in a close read, and a cost and latency profile the authors openly admit rules out real time deployment all deserve equal billing with the result that SA-VQA convincingly beats every baseline tested. This is a benchmark oriented contribution, not a finished industrial product, and the paper is admirably clear about that distinction itself.

Even with those caveats, the core empirical claim holds up under scrutiny. Every single question category, across every baseline tested, favors the multi agent decomposition over a single model answering directly, and the ablation study backs that claim up with a specific, falsifiable explanation for where the advantage actually comes from. That is a stronger foundation than most new benchmark papers manage to build in one sitting.

Frequently asked questions

Related reading on aitrendblend