How to Test If a Human AI Team Beats Working Alone

Analysis by the aitrendblend editorial team  ·  Practical AI tools and agent systems  ·  Explains a preprint, not deployment or clinical advice  ·  Reading time about 17 minutes

• Human AI Teams
• Evaluation Design
• Agentic AI
• Selective Replay
• Decision Targeted Testing
• Experimental Design

A coding agent writes the pull request and an engineer reviews it. A model flags the nodule and a radiologist signs the report. In both cases the team ships something, and in both cases a quietly important question goes unanswered. Was the team actually the best option, or would the human alone or the AI alone have done better? A new preprint from Georgia Institute of Technology argues that most evaluations never answer this, because they measure the wrong thing under a fixed budget. The authors propose a rule called TEAM-DESIGN that spends a limited pool of expensive rechecks exactly where they settle the deployment decision fastest, and they prove it is the optimal way to do so.

The question that hides behind AI helps humans

There is a comfortable finding that shows up again and again in studies of assisted work. Give a professional an AI tool and they do better than they did without it. Anthropic reports that its own staff use its models constantly, and fintech teams are moving toward engineers reviewing an agent pull requests rather than writing everything themselves. In medicine, many studies show that a clinician reading a case with model support outperforms the same clinician reading it alone. The natural conclusion is that the team is the way to go.

The authors point out that this conclusion skips a step. Showing that the workflow beats the human alone says nothing about whether it beats the AI alone. If the model on its own is more accurate than the human plus the model, then the human is not adding value, they are subtracting it, and the honest deployment choice might be to let the model run unassisted. The workflow earns its place only when it clears both bars at once, when it beats the human alone and it beats the AI alone. That is a two comparison test, and most evaluations only run one side of it.

Why you cannot just measure everything

Here is the practical wrinkle that makes this hard. Once the workflow is deployed, the only outcome you actually observe is the workflow outcome. You see what the team produced. You do not see what the human would have produced alone on that same task, and you do not see what the model would have produced alone, because those alternatives were never run. To learn them you have to replay the task, and replays are expensive. A human only replay can mean pulling a senior engineer off other work to redo the task from scratch, or asking a clinician to read a case again with no assistance. An AI only replay burns model calls, tools, and compute.

Running both replays on every task would cost more than the evaluation is worth. So the real question is not which tasks to fully audit, but how to spread a limited replay budget across two different comparisons and many tasks so that the deployment decision becomes as clear as possible. The paper frames this as its central design problem.

The workflow runs, but the two alternatives it must beat are never seen. Every answer costs an expensive replay, and the budget will not cover them all.

Two probabilities per task, chosen on purpose

TEAM-DESIGN does not pick a fixed subset of tasks to audit in advance. Instead it hands every task two replay probabilities. One says how likely this task is to get a human only replay, the other how likely it is to get an AI only replay. The two draws are independent, so a task might get neither replay, one of them, or both. A higher probability simply means that this task will be replayed more often among tasks that carry similar information, which is how the budget flows toward the measurements that matter.

The intuition behind where the budget should go rests on two ideas. First, replay is most valuable on tasks where the missing outcome is hard to predict from what you already know. If you can already guess with confidence how the human alone would have done, replaying that task teaches you little. If the missing outcome is genuinely uncertain, a replay is informative. Second, the budget should favour the comparison that is closer to failing. If the workflow beats the human alone by a mile but only barely beats the AI alone, then the AI comparison is the one that needs more evidence, because it is the one that could still flip the decision.

That objective has a satisfying shape. The inner maximum picks out the harder of the two comparisons, the one that will decide how many tasks you need. The outer minimum then chooses the replay probabilities to shrink that hardest comparison as much as the budget allows. Because the workflow has to beat both alternatives, making an already easy comparison even easier does nothing, so all the effort flows to the bottleneck.

The square root rule

The elegant part is that this minimax program is convex, and the authors solve it exactly rather than approximately. The optimal replay probability for a task turns out to follow a clean square root rule. Replay rises with the square root of prediction difficulty, the harder the missing outcome is to guess, the more you replay. Replay falls with the square root of cost, the more expensive the replay, the less you do it. And two multipliers tune the balance between the two comparisons and hold the total spend to the budget.

The clipping matters. Every task keeps a floor probability of being replayed for each baseline, so no task is ever locked out entirely, which is what keeps the later estimates honest. The two multipliers are found by a pair of one dimensional searches rather than a heavy solver, so the whole rule is cheap to compute once you have the ingredients.

Those ingredients come from a pilot. Because prediction difficulty and how close each comparison sits to its threshold both depend on the unseen baseline outcomes, TEAM-DESIGN cannot know them in advance. It runs a small separate pilot set where all three configurations, the workflow, the human alone, and the AI alone, are executed together, learns predictors of the missing outcomes and estimates of the gaps, and then uses those to plan the replay probabilities for the much larger evaluation pool.

Correcting for the fact that you only replayed some tasks

Once replay is selective, a naive average is biased, because the tasks you chose to replay are not a random sample. TEAM-DESIGN corrects for this with a score that combines a prediction of the missing baseline with an observed correction whenever a replay actually happened. The correction is scaled by the reciprocal of the replay probability, so a rarely observed correction counts for more, exactly the reweighting that keeps the estimate unbiased.

A nice property falls out of this. Because the replay probability is known and fixed before any outcome is seen, the corrected estimate is unbiased whether or not the pilot predictor is any good. A poor predictor only makes the estimate noisier, it does not make it wrong. That separation between validity and efficiency is what lets the method lean on imperfect pilots without compromising the final decision.

A reference implementation of the rule

The whole method is small enough to write out. The routine below takes a pilot estimate of prediction difficulty, the gap above threshold, and the replay cost for each comparison, then produces the square root replay probabilities by tuning the budget multiplier with a one dimensional search. A second routine draws the selective replays and forms the unbiased gain estimate. There is no neural network here, the paper contribution is the design rule itself.

import numpy as np

def team_design_probs(difficulty, gap2, cost, budget,
                      lam=0.5, q_min=0.05):
    """Square root replay rule for one baseline comparison.

    difficulty : pilot estimate s_j(W) per task, higher means harder
    gap2       : stabilised squared margin above threshold, d_j^2
    cost       : replay cost c_j(W) per task
    budget     : average replay budget allowed per task for this baseline
    returns    : replay probability q_j(W) per task, clipped to [q_min, 1]
    """
    difficulty = np.asarray(difficulty, float)
    cost = np.asarray(cost, float)

    def spend(eta):
        raw = np.sqrt(lam * difficulty / (eta * gap2 * cost))
        q = np.clip(raw, q_min, 1.0)
        return q, float(np.mean(cost * q))

    # Bisect the budget multiplier eta so mean cost meets the budget.
    lo, hi = 1e-8, 1e8
    for _ in range(80):
        mid = np.sqrt(lo * hi)
        q, mean_cost = spend(mid)
        if mean_cost > budget:
            lo = mid          # too much spend, raise eta to shrink q
        else:
            hi = mid
    q, _ = spend(hi)
    return q


def estimate_gain(T, Y, replayed, q, m_pred):
    """Unbiased gain estimate under selective replay, one baseline.

    T        : deployed workflow outcomes on the evaluation pool
    Y        : baseline outcomes, valid only where replayed is True
    replayed : boolean mask, whether baseline j was replayed per task
    q        : the replay probabilities used to draw the mask
    m_pred   : pilot predictor of the baseline outcome per task
    """
    T = np.asarray(T, float)
    m_pred = np.asarray(m_pred, float)
    R = replayed.astype(float)
    correction = np.where(replayed, (Y - m_pred), 0.0)
    psi = T - m_pred + (R / q) * correction
    est = float(np.mean(psi))
    se = float(np.std(psi, ddof=1) / np.sqrt(len(psi)))
    return est, se


def demo():
    rng = np.random.default_rng(0)
    n = 4000

    # Synthetic pool. The AI comparison is deliberately the harder one.
    diff_H = rng.gamma(2.0, 1.0, n)      # human replay difficulty
    diff_A = rng.gamma(2.0, 1.0, n) * 3  # AI replay harder to predict
    cost_H = np.full(n, 4.0)             # human replays are pricey
    cost_A = np.full(n, 1.0)             # AI replays are cheap

    qH = team_design_probs(diff_H, gap2=0.02, cost=cost_H, budget=0.6)
    qA = team_design_probs(diff_A, gap2=0.004, cost=cost_A, budget=0.6)

    print("mean human replay prob:", round(qH.mean(), 3))
    print("mean AI replay prob   :", round(qA.mean(), 3))
    print("budget goes to the harder AI comparison:", qA.mean() > qH.mean())

if __name__ == "__main__":
    demo()

Run the demo and the budget tilts toward the AI comparison, because that is the harder one to settle in this synthetic pool. That tilt is the whole point of the method. When one comparison clearly needs more evidence, the square root rule finds it and pours the replay budget there.

What the clinical data actually showed

The first thing the authors do is check whether the two comparison question even matters in the real world, and the answer is a firm yes. They reanalysed two families of studies in which clinicians read the same cases with and without model support, covering six settings and more than eight thousand reads. In every single setting the assisted clinician beat the clinician alone. And in every single setting the assisted clinician lost to the model alone. None of the six human and AI workflows beat both alternatives.

Read that table slowly, because it is the paper strongest argument for why the second comparison is not optional. Any evaluation that stopped at the human comparison would have concluded that assistance works and the workflow should ship. The full two comparison test reaches the opposite verdict for these particular studies, that the model alone was the stronger option in the settings measured. The authors are careful about scope here, these are matched reader studies with their own limitations, not live deployments, and the point is about evaluation design rather than any clinical recommendation. Still, the direction of the result is exactly the trap the method is built to avoid.

A coding benchmark told the mirror image story. There the agent alone had a low pass rate, the human alone landed in the middle, and the workflow scored highest, so at least one workflow does beat both alternatives. But that benchmark reports one average per configuration rather than three outcomes on each individual task, which means it cannot support selective replay at all, and the improvement over the human alone by itself still does not settle the deployment question. The gap between what benchmarks report and what the deployment decision needs is a recurring theme.

When directing the budget helps, and when it does not

The second experiment tests whether steering replay toward the harder comparison actually pays off. It does, in the regime the method targets. When one comparison sits much closer to its threshold than the other, TEAM-DESIGN declared the workflow a winner over both baselines in about seventy seven percent of replications, against about sixty five percent for Neyman allocation, the standard variance based method, at the same cost. It gets there by raising the replay probabilities for the harder comparison and settling near an equal split when neither comparison is harder.

The honest row is the second one. When the two comparisons are equally difficult, there is no imbalance to exploit, and TEAM-DESIGN actually trails the variance based method. The authors do not hide this. Their objective measures large sample difficulty and steers toward the harder comparison, so when neither is harder that steering has nothing to grab and the simpler allocation does better. They frame the method plainly as a decision targeted design for unequal comparisons, not a rule that wins everywhere.

Where this sits in the evaluation landscape

It helps to place TEAM-DESIGN against its neighbours. Agent benchmarks and reader studies establish that a configuration works on average, but they rank configurations you can already measure, they do not choose which missing baseline to spend a replay on. Variance based sampling, the classical Neyman idea, spends measurement where outcomes are uncertain but ignores which of your two comparisons is closer to failing. Bayesian information methods focus on learning model parameters rather than making the deployment call. Ranking and best system selection methods allocate efficiently but choose among outcomes that are actually observed. What is new here is a design that treats both baseline outcomes as missing and costly, and targets the single downstream decision of whether the workflow beats both.

The flavour will feel familiar to anyone who has followed the harder problem of judging AI systems fairly. Work on whether language models can reliably judge subjective quality wrestles with the same tension between cheap noisy raters and expensive expert ones, and studies that put models through validated psychometric tests share the instinct that evaluation methods borrowed from the social sciences transfer surprisingly well to AI. TEAM-DESIGN pushes that instinct one step further, into the economics of which measurement to buy when every measurement has a price.

What to take away

The lasting idea in this preprint is a reframing. Stop asking whether AI helps, which almost always returns yes, and start asking whether the human and AI workflow beats both of the simpler options it is meant to replace. That question has a missing data problem baked in, because the alternatives are never run, and a budget problem baked in, because rechecking them costs real money and expert time. TEAM-DESIGN answers both at once with a rule that is provably the best way to spend a fixed replay budget toward settling the decision, spending most where the missing outcome is hard to predict and the comparison is close to flipping.

For anyone who evaluates assisted work, whether that is a team shipping code with agents or an organisation weighing clinical AI, the practical lesson is to design the evaluation around the decision, not around the average. Measure both comparisons, direct scarce rechecks toward the one that is genuinely in doubt, and be honest that when both comparisons are equally hard a plain balanced allocation is fine. The reframing costs nothing to adopt, and as the clinical reanalysis shows, it can flip a confident ship it verdict into a much more careful one.

Frequently asked questions

What deployment question does TEAM-DESIGN actually answer?

It answers whether a human and AI workflow is worth keeping in production. A workflow earns its place only if it beats both the human working alone and the AI working alone. Showing it beats the human alone is not enough, because the model on its own might be the stronger option, so the method targets that full two comparison decision rather than a single average gain.

Why can you not just measure both alternatives on every task?

Because once the workflow is deployed you only observe the workflow outcome, and learning what the human alone or the AI alone would have done requires replaying the task. A human only replay can mean a senior expert redoing the work, and an AI only replay burns compute and tools. Under a fixed budget you cannot replay everything, so the design decides which comparison to replay on which tasks.

How does TEAM-DESIGN decide where to spend the replay budget?

It gives every task two replay probabilities, one per baseline, following a square root rule proven optimal for the design problem. Replay rises with the square root of how hard the missing outcome is to predict and falls with the square root of the replay cost, and two tuning multipliers push the budget toward the comparison that is closer to failing while holding total spend to the budget.

What did the clinical reanalysis reveal?

Across six clinical reader study settings and more than eight thousand reads, the assisted clinician beat the clinician alone in every setting but lost to the model alone in every setting, so none of the six workflows beat both alternatives. An evaluation that only checked the human comparison would have concluded the workflow should ship, while the full two comparison test reached the opposite verdict for those measured settings.

When does TEAM-DESIGN not help?

It helps when one comparison is clearly harder to settle than the other, which is where directing the budget pays off. When both comparisons are equally difficult there is no imbalance to exploit, and the method can actually do worse than standard variance based allocation. The authors present it as a decision targeted design for unequal comparisons, not a universally better replay rule.

Should I be cautious that this is a preprint?

Yes, treat the results as promising rather than settled. The work is posted as an arXiv preprint and has not yet completed peer review, so its proofs and numbers have not been independently vetted by referees. The clinical evidence comes from reanalysing matched reader studies rather than live deployments, and independent replication is the right standard before relying on the method in practice.

Source paper. Hamed Khosravi and Xiaoming Huo. Beyond AI Helps Humans, Decision Targeted Evaluation Design for Human Agent Teams in the Agentic Era. arXiv preprint arXiv:2609.05527v1, 1 September 2026. Affiliation, H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology. This article explains a preprint that has not yet been peer reviewed and reanalyses published clinical studies as data. It is general technical education, not medical, engineering, or deployment advice.