Key points
- MITRE ATLAS extends the ATT&CK model with fourteen tactics built for AI specific risks, from reconnaissance through impact, and the review sorts sixty three papers under six broad attack families that sit inside it.
- Evasion and poisoning remain the deepest studied areas, with attacks such as the Carlini and Wagner method achieving a full success rate against defensively distilled models, and a single poisoned image flipping an SVM’s error rate by double digits.
- Large language models introduce attack surfaces the original ATT&CK framework never had to account for, including prompt injection, jailbreaks, and plugins that quietly exfiltrate a user’s conversation history.
- The paper pairs sixty three attacks with twenty four concrete mitigations, turning an academic taxonomy into something a security team can actually check against.
- Real world case studies, from a fraudulent tax scheme built on deepfake video to a chatbot poisoned into racism within a day, show that these are not hypothetical risks confined to benchmark datasets.
The problem traditional security frameworks never had to solve
For decades, security teams have leaned on MITRE ATT&CK to reason about attackers. It gives everyone the same vocabulary for phishing, privilege escalation, lateral movement, all the familiar beats of a network intrusion. What it was never built for is a system that learns from data, updates its own behavior, and can be fooled by a pattern of pixels a human eye would never register. Machine learning models introduce failure modes software never had. A model can be poisoned during training rather than breached after deployment. It can leak the private records it was trained on simply by answering questions honestly. It can be tricked by an input that looks completely ordinary to a person and completely different to the model doing the classifying.
MITRE responded to this gap with ATLAS, short for Adversarial Threat Landscape for AI Systems. Where ATT&CK organizes attacker behavior against conventional infrastructure, ATLAS organizes it against the AI lifecycle itself, from the reconnaissance an attacker performs on a target’s published research through to the eventual impact on the deployed model. The review this article is based on takes that taxonomy and, for what its authors say is the first time, systematically maps a large body of academic literature onto it rather than treating ATLAS as background color for a handful of case studies.
Why this matters beyond academia
The paper’s authors note that attacks documented in ATLAS have already undermined facial recognition systems, misled autonomous vehicles, and slipped past fraud detection. These are not laboratory curiosities. A California man used stolen identities paired with deepfake video to file over 180 fraudulent unemployment claims and pocket more than $3.4 million before he was caught, a case the review cites as an example of how evasion techniques translate directly into financial crime.
Fourteen tactics, six attack families
ATLAS organizes adversary behavior into fourteen tactics that roughly follow the life of an attack. Reconnaissance and resource development come first, where an adversary studies a target’s public research papers or scans for exposed model endpoints. Initial access and ML model access follow, describing how an attacker gets a foothold or starts querying a model. Later tactics cover execution, persistence, privilege escalation, defense evasion, credential access, discovery, collection, staging an attack, exfiltration, and finally impact, the stage where the damage actually lands.
To make sense of sixty three individual research papers against fourteen tactics, the review groups the relevant techniques into six broad attack families. Evasion covers inputs crafted at inference time to force a misclassification. Poisoning covers corruption of the training process itself. Model extraction covers attempts to steal a model’s functionality through queries alone. Inference attacks, most notably membership inference, try to determine whether a specific record was part of a training set. Model inversion goes a step further and tries to reconstruct the training data itself. And a newer category, attacks specific to large language models, covers prompt injection, jailbreaking, and the leakage that comes from a model’s own memorization.
| Attack family | Primary ATLAS home | Example technique | What it targets |
|---|---|---|---|
| Evasion | ML Model Access, Defense Evasion, Impact | Crafting inputs that cross a decision boundary, such as the Carlini and Wagner method | Model output at inference time |
| Poisoning | Resource Development, Persistence, ML Attack Staging | Embedding a hidden trigger during training, as in backdoor attacks against federated learning | The training pipeline and long term model integrity |
| Model extraction | ML Attack Staging, Exfiltration | Querying a service enough times to train a functionally equivalent copy | Model intellectual property |
| Inference attacks | Discovery | Using a shadow model to guess whether a record was in the training set | Training data confidentiality |
| Model inversion | Exfiltration via ML Inference API | Using confidence scores to reconstruct a plausible training image or record | Training data privacy |
| LLM attacks | Initial Access, Execution, Privilege Escalation | Prompt injection that hijacks an assistant’s retrieval augmented generation pipeline | Model guidance, plugins, and connected data |
How a black box learns your model without ever seeing its weights
Evasion is the oldest studied branch of adversarial machine learning, and the review traces it from its earliest roots. Szegedy and colleagues first showed in 2013 that a neural network could be pushed into misclassifying an input using a perturbation invisible to a human observer. The following year, Goodfellow, Shlens, and Szegedy proposed the fast gradient sign method, a computationally cheap way to generate such perturbations by following the sign of the model’s own loss gradient. From there the field moved quickly. Projected gradient descent added iterative refinement and random restarts. Auto attack combined several methods into a standard benchmark. The Carlini and Wagner method, still cited as one of the strongest baselines, achieved a full success probability against models specifically hardened through defensive distillation.
What makes this branch genuinely unsettling for anyone deploying a model as a service is how little access an attacker actually needs. The Zeroth Order Optimization attack estimates gradients purely from repeated queries and their resulting confidence scores, without ever seeing the model’s internals. The boundary attack goes further still, needing nothing but the model’s final hard label. Square Attack, a more recent method built around randomly perturbing small regions of an image, has in some experiments outperformed white box techniques that had full access to gradients. None of this requires the attacker to know anything about the model’s architecture. It only requires the ability to ask it questions and watch how it answers.
Poisoning turns the training process itself into the attack surface
If evasion happens at the moment a model makes a decision, poisoning happens long before that, while the model is still learning. Biggio, Nelson, and Laskov demonstrated one of the earliest versions of this against support vector machines back in 2012, showing that a single carefully placed poisoned point could push a classifier’s error rate up by thirteen to fifteen percentage points. More recent work has pushed the idea into clean label territory, where the attacker never even touches a label. Shafahi and colleagues showed that a single modified image, nudged toward the feature space of a target class while keeping its original label intact, was enough to achieve full success in a transfer learning setting.
Federated learning turns out to be an especially attractive target for this family of attacks, since the server aggregating updates from many participants generally has no way to inspect what any one participant actually trained on. Bagdasaryan and colleagues showed that replacing an entire local model with a poisoned one could plant a backdoor with perfect reliability while barely denting overall accuracy. A follow up study introduced distributed backdoor attacks, where the trigger itself is split into fragments across multiple clients, each learning only its own piece, so the full pattern only assembles once the updates are combined on the server. That kind of fragmentation is difficult for any single participant’s contribution to catch, which is exactly the point.
Stealing a model without ever seeing its weights
Model extraction attacks treat a deployed model as a black box worth copying. Tramer and colleagues showed early on that even API only access, paired with carefully chosen queries, was enough to reverse engineer simpler models with perfect fidelity. Orekondy and colleagues later scaled the idea up with Knockoff Nets, training a substitute model purely on a target’s output probabilities using unrelated public data, and hitting over seventy percent of the original model’s performance without any knowledge of what that model was trained on.
The review also highlights a more surprising extraction pathway, one aimed squarely at large language models. Carlini and colleagues showed that querying a production language model and analyzing the structure of its output logits could reveal the size of its hidden dimension, and from there, parts of its final embedding projection matrix. Their demonstration against OpenAI’s smaller commercial models reportedly cost less than twenty dollars in API calls. That number does more to explain why model providers restrict raw logit access than any amount of theoretical discussion could.
What a model remembers about you
Membership inference and model inversion sit closer to privacy than to sabotage, but their consequences can be just as serious. Shokri and colleagues formalized the shadow model approach to membership inference, training auxiliary models to mimic the target and using their behavior to teach an attack model the difference between a record the target has seen before and one it has not. Later work by Carlini and colleagues, introducing the Likelihood Ratio Attack, showed that focusing on the right statistical framing, rather than simply chasing raw accuracy, produced a tenfold improvement in detection power at low false positive rates, a distinction that matters enormously once an attack moves from a benchmark paper into a real audit.
Model inversion pushes further still, attempting to reconstruct plausible training examples rather than just confirm their membership. One of the earliest and most sobering demonstrations came from work on a warfarin dosing model, where researchers showed that combining published marginal statistics with an individual’s known clinical attributes could expose sensitive genetic information the model was never meant to reveal directly. More recent GAN based approaches, including Plug and Play attacks and reinforcement learning guided inversion, have pushed reconstruction accuracy on face recognition models well above eighty percent using nothing but confidence scores.
A pattern worth noticing
The review points out that these attacks rarely happen in isolation. A common chain starts with a black box evasion attack against a model offered as a service, which produces samples clustered near the true decision boundary. Those samples make it far cheaper to train a proxy model through extraction, and that proxy in turn lets an attacker craft precise white box evasion examples with very few additional API calls. Each stage compounds the one before it, and most published defenses still address only one stage at a time.
Large language models brought a new kind of attack surface
Everything covered so far predates the current wave of large language models, and much of it still applies to them. What ATLAS had to add on top is a set of tactics built around natural language as the attack vector itself. Prompt injection lets an attacker embed instructions inside content a model is asked to process, whether that is a webpage, an email, or a document fetched through a plugin. Jailbreaking uses carefully engineered inputs to override a model’s built in safety behavior. Meta prompt extraction tries to recover a system’s hidden instructions, and data leakage exploits a model’s tendency to memorize and later regurgitate fragments of its training data.
Zou and colleagues showed that adversarial suffixes discovered through a hybrid greedy and gradient search transferred surprisingly well across models, achieving over eighty six percent success against GPT-3.5 despite being optimized on smaller open models. The review also covers ActorAttack, a multi turn method that builds a network of semantically related conversational threads to steer a model toward a harmful output gradually rather than all at once, a strategy that reportedly worked even against GPT-4. On the automation side, PentestGPT and AutoAttacker both demonstrate that language models can be pointed at real penetration testing tasks with meaningful success, which cuts in both directions. The same capability that helps a defender triage vulnerabilities faster can just as easily automate an attacker’s reconnaissance.
Perhaps the most concrete illustration in the whole review is the smallest one. A Streamlit application built on GPT-3 to solve math problems turned out to have a prompt injection flaw that let an attacker extract the application’s own API key simply by asking the right question, an incident MITRE catalogues as AML.CS0016. No sophisticated tooling was involved. Just a well phrased prompt and a model that answered honestly.
From taxonomy to a checklist a team can actually use
The most practically useful part of the review is not the taxonomy itself but what it does with it afterward. Section 6 of the paper maps twenty four concrete mitigations directly onto the ATLAS techniques they counter, turning an academic classification into something closer to an audit checklist.
| Attack family | Representative defenses named in the paper |
|---|---|
| Poisoning | Verifying AI artifacts, maintaining dataset provenance, sanitizing training data, model hardening through adversarial training |
| Evasion | Ensemble methods, input restoration, adversarial input detection, capping the number of queries a user can make |
| LLM attacks | Generative AI guardrails, alignment techniques such as reinforcement learning from human feedback, telemetry logging, code signing on plugins |
| Inference and model inversion | Obscuring or rounding output confidence scores, restricting query volume, encrypting sensitive information at rest |
| Model extraction | Limiting public release of architecture details, telemetry logging, limiting how much of a model’s artifacts are ever released |
Two of these mitigations show up again and again across nearly every attack family, and they are worth calling out on their own. The first is simply restricting how many queries a user is allowed to make. Nearly every black box attack described in the review, from evasion to extraction to inversion, depends on repeated querying to work at all, which makes rate limiting one of the cheapest defenses with the broadest reach. The second is passive obfuscation of model output, meaning rounding or truncating confidence scores before they reach the user. Several of the strongest attacks in the review, including the Likelihood Ratio Attack and Plug and Play model inversion, lean heavily on fine grained probability values that a coarser output would simply deny them.
Where the framework still falls short
The review is honest about ATLAS having real gaps, and this section is arguably where it adds the most original thinking. Explainable AI tools, now built into platforms from Google Cloud and AWS, are meant to make models more transparent to users, but that same transparency hands an attacker useful information about a model’s internal reasoning, and the current framework does not account for that tradeoff at all. Reinforcement learning, newly relevant again through techniques used to align large language models, brings its own poisoning risks around manipulated rewards, and ATLAS has only thin coverage of it.
The gap that stood out most while reading through the paper is around multi agent systems. As AI deployments increasingly involve several models cooperating or competing, whether that is a fleet of coordinating drones or a set of automated trading agents, the attack surface stops being about one model’s vulnerabilities and starts being about how agents can be turned against each other. The review calls this largely unexplored territory, and given how quickly agentic workflows are being adopted in production, that gap will not stay theoretical for long. Multi modal models, continual learning systems that keep adapting after deployment, and neurosymbolic architectures that combine neural networks with rule based reasoning round out the list of areas the authors flag as underserved by the current taxonomy.
A useful caveat
The authors also note that ATLAS depends on real world incidents and voluntary reporting to populate its case studies, which means quieter failures such as silent model extraction or unnoticed data leakage are likely underrepresented compared to headline grabbing incidents like chatbot manipulation. That reporting bias is worth keeping in mind whenever ATLAS coverage is used as a proxy for how common an attack actually is.
Honest limitations
This is a review of research literature, not new experimental work, so its conclusions are only as strong as the sixty three papers it draws from and the judgment calls made in mapping each one to a tactic. The authors describe a two reviewer coding process with a third senior adjudicator for disagreements, which is a reasonable safeguard, but ATLAS mappings for genuinely multi stage attacks still involve some subjectivity about which technique counts as primary. The paper also draws heavily on benchmark datasets such as ImageNet and CIFAR-10, and results measured there do not automatically transfer to production systems running on proprietary data with different distributions. Several of the case studies, including the most eye catching ones like ShadowRay and the tax fraud scheme, come from security researchers and press reporting rather than peer reviewed venues, and financial figures such as the reported billion dollar impact should be read as reported estimates rather than audited numbers.
Conclusion
What this review ultimately demonstrates is that adversarial machine learning has grown well past the point where it can be treated as one exotic risk category. Evasion, poisoning, extraction, inference, inversion, and the newer wave of large language model attacks are not six unrelated topics that happen to share a bibliography. They are stages in a shared lifecycle that ATLAS was built specifically to describe, and the paper’s contribution is showing that description actually holds up once you try to fit sixty three real research contributions inside it.
The conceptual shift worth sitting with is the move away from thinking about model security as a single hardening step applied once before deployment. Poisoning happens during training. Evasion happens at inference. Extraction and inversion happen continuously, every time a model answers a query. Treating any one of these as solved while ignoring the others leaves the door open for exactly the kind of chained attack the paper describes, where a cheap black box evasion attempt quietly sets up a far more damaging extraction or inversion attack later.
That lifecycle framing also travels well beyond the six categories this paper focuses on. The same logic applies to robotics systems making real time decisions from sensor data, to healthcare models trained on protected patient records, and to any federated system where a server has to trust updates it cannot fully inspect. Wherever a model learns continuously or serves predictions to untrusted users, the same fourteen ATLAS tactics apply with only the specifics changing.
The honest remaining limitation is that a framework built on cataloguing real incidents will always trail the frontier of what is theoretically possible. Multi agent exploitation, attacks on the reasoning of neurosymbolic systems, and poisoning that unfolds gradually inside continual learning pipelines are all areas where research is already ahead of what ATLAS currently documents. That lag is not a flaw unique to this framework. It is the price of grounding a taxonomy in things that have actually happened rather than things that merely could.
Where this leaves practitioners is with a genuinely useful starting point rather than a finished playbook. The defense mapping in Section 6 gives a team a real checklist to work from today, while the gaps the authors identify around explainability, agentic systems, and continual learning are a fair warning about where the next wave of incidents is likely to come from. Given how fast agentic AI deployments are moving, that warning deserves more attention than it is currently getting.
Frequently asked questions
What is MITRE ATLAS and how is it different from MITRE ATT&CK
MITRE ATLAS is an offshoot of the widely used ATT&CK framework, built specifically to describe adversarial tactics against AI and machine learning systems. Where ATT&CK organizes attacker behavior against conventional software infrastructure, ATLAS organizes it around the AI lifecycle, covering everything from reconnaissance on a target’s published research through to the eventual impact on a deployed model.
How many attack families does the review cover
The paper groups its sixty three analyzed papers into six broad families. Evasion, poisoning, model extraction, membership inference and related inference attacks, model inversion, and a newer category covering attacks specific to large language models such as prompt injection and jailbreaking.
Can a model be attacked without any access to its internal weights
Yes, and the paper covers several such attacks in detail. Black box methods such as the Zeroth Order Optimization attack, the boundary attack, and Square Attack can all craft effective adversarial examples using only the model’s output labels or confidence scores, without ever seeing its architecture or parameters.
What is the single most broadly useful defense the paper identifies
Restricting how many queries a user or API caller is allowed to make shows up as a mitigation across nearly every attack family the paper covers, since most black box attacks depend on repeated querying to function at all. Obscuring the precision of output confidence scores is a close second.
Does the paper cover attacks on ChatGPT and similar commercial systems
Yes. Several case studies involve real production systems, including a prompt injection flaw in a GPT-3 powered math tutoring application that exposed its own API key, and a documented case where ChatGPT recommended a nonexistent software package that was later exploited through supply chain confusion.
What does the paper say ATLAS is currently missing
The authors identify several underserved areas including attacks that specifically target explainable AI outputs, poisoning of reinforcement learning systems used to align large language models, coordinated attacks across multi agent AI systems, cross modal attacks on models that process text, images, and audio together, and security for continual learning systems that keep adapting after deployment.
Read the original research
This analysis draws on the full peer reviewed paper published in Computer Science Review, which includes the complete methodology, all sixty three analyzed studies, and the full defense mapping table.
Related reading on aitrendblend
Academic citation. Sachpelidis Brozos N, Katsaros E, Radoglou Grammatikis P, Kalitsios G, Sarigiannidis A, Seritan G C, Politis I, Xenakis C, Goudos S, Sarigiannidis P. Hacking intelligence, mapping the anatomy of adversarial threats in artificial intelligence with MITRE ATLAS. Computer Science Review, volume 61, 2026, article 100923. Published under a CC BY license.
This analysis is based on the published paper and an independent evaluation of its claims.