A malware sample scores as almost certainly malicious on a machine-learning antivirus engine and is blocked. The attacker appends a large block of harmless text strings, copied from a popular online game, to the end of the file and tries again. This time the engine scores it as benign and the file runs, even though not one byte of the malicious code changed. In 2019, researchers did almost exactly this to a commercial AI-based antivirus product and got most of the samples they tried past it.
AI systems have the weaknesses of ordinary software plus attack points of their own: the data a model learns from, the file that holds what it learned, and the way it reacts to inputs nobody anticipated. On the CompTIA Security+ SY0-801 exam, AI risks sit under objective 2.6, which expects you to describe how AI systems can be attacked and the risks they bring when an organization uses them. This lesson covers the attacks aimed at the model itself: poisoning the data it learns from or looks up, backdooring and tampering with the model, fooling it with crafted inputs, and copying it through its own interface.
Each attack on an AI model targets a different stage of its life
A machine-learning model is built and used in stages, and naming the stage is the fastest way to name the attack.
| Stage | What the attacker touches | Attack |
|---|---|---|
| Data collection and training | Training sets, labels, user feedback | Poisoning |
| Retrieval at answer time | Knowledge bases and document indexes | Retrieval (RAG) poisoning |
| Model artifact | Weights files, fine-tuned versions, settings | Model manipulation and backdoors |
| Inference | A single input sent to the deployed model | Evasion |
| Inference at scale | Many queries and their answers | Model theft (extraction) |
The key distinction is timing. Poisoning and manipulation change the model, or what it relies on, before the attacker's real target ever arrives. Evasion leaves the model untouched and attacks it at the moment it makes a decision.
Two references are worth knowing by name: MITRE ATLAS, which catalogs attacks on AI systems in the style of MITRE ATT&CK, with real case studies, and NIST's adversarial machine learning taxonomy, which sorts attacks into classes such as evasion, poisoning and privacy attacks. Using AI to run security operations, such as agentic AI and predictive analysis, is covered in the automation and orchestration section.
Poisoning corrupts what a model learns from
Data poisoning means inserting crafted or mislabeled records into the data a model is trained or retrained on, so the model learns the attacker's lesson. It is an integrity attack on the model, delivered through its data.
Untargeted poisoning degrades the model overall: feed a spam filter enough mislabeled examples and it simply gets worse, a denial of service against the detection. Targeted poisoning is quieter. The model behaves normally on almost everything but makes the mistake the attacker wants on one class of input. The most dangerous form is a backdoor, where the poisoned examples teach the model to react to a trigger only the attacker knows, such as a rare word, a pixel pattern or a specific sender domain.
Any system that keeps learning from outside input is exposed. Microsoft's Tay chatbot learned from public conversations in 2016 and was pulled offline within a day after coordinated users taught it to post offensive messages, and a spam filter that retrains on "not spam" clicks can be steered the same way. Web-scale training data is exposed too: researchers showed in 2023 that for about 60 US dollars an attacker could have controlled 0.01 percent of two widely used public image datasets, by buying expired domains the datasets still pointed to. A 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that about 250 poisoned documents could plant a simple backdoor in language models from 600 million to 13 billion parameters, however much clean data surrounded them.
Defenses focus on data provenance and change control. Know where every training source comes from, restrict who can add or relabel data, keep versioned snapshots you can roll back to, and don't let anonymous feedback flow straight into retraining. Test each retrained model against a fixed evaluation set before it replaces the old one.