What is model extraction?
Model extraction, also called model stealing or model theft, is an attack in which someone recreates a machine learning model they have no right to. They do not need the model file. They send the model large numbers of chosen inputs through its normal interface, usually a paid or public API, record every output, and use those input and output pairs to train a substitute of their own. If the substitute behaves closely enough like the original, the attacker has taken the result of the owner's data collection and training spend.
Why attackers steal models
- Intellectual property. A trained model can represent years of data gathering and large compute bills. A replica lets a competitor or reseller skip both.
- Avoiding fees. Once copied, the model can be run without paying per query.
- Building better attacks. A local copy is a private test bench. The attacker can tune evasion inputs against the replica as often as needed, then use them on the original, which often falls for the same inputs.
- A step toward privacy attacks. A close copy makes model inversion and membership inference easier.
How it works in practice
A 2016 study showed that models offered through commercial machine-learning-as-a-service APIs could be duplicated with a modest number of queries, and that returning detailed confidence values made the job much easier. Large language models are exposed too: in 2024 researchers recovered part of the internal structure of production LLMs through ordinary API calls. A related practice, distillation, trains a smaller model on a larger model's answers. It is a normal technique when the owner does it, and usually breaks the provider's terms of service when a competitor does it to another provider's model.
What the query log shows
Extraction is noisy if anyone looks. A fraud-scoring API that normally serves a few thousand requests a day from checkout servers might show one new API key submitting 400,000 requests over a weekend, each a synthetic transaction that changes one field at a time, such as the amount rising in one-dollar steps. Real customers rarely sweep the input space that methodically.
Defending against model extraction
- Require authentication for every API caller, and tie each key to an account with a known owner.
- Enforce per-client rate limits and quotas, and alert on volumes or patterns that do not match normal use.
- Return only the detail the use case needs: a decision or label rather than raw probabilities.
- Watch for systematic queries that probe decision boundaries or cover the input space evenly.
- Prohibit training competing models on outputs in the terms of service, and consider watermarking outputs so a copy can be recognized.
Model extraction vs. model inversion and evasion
Extraction steals the model. Model inversion and membership inference use queries to learn about the training data instead. Evasion feeds one crafted input to an unchanged model to get a wrong decision, and extraction is often the preparation for it. None of these modifies the model, which separates them from poisoning and model manipulation.
Model extraction on the CompTIA Security+ exam
Copying a model through its own interface is taught with the attacks on AI models under SY0-801 objective 2.6, where it is often called model theft. Scenario wording such as thousands of queries were used to build a copy, a competitor released a model that mimics ours or one API key made an unusual volume of systematic requests points here. The expected controls are API controls: authentication, rate limiting, monitoring and returning less detail. Distractors include model inversion, when the attacker rebuilds training data, evasion, when one input is crafted to fool the model, and poisoning, when the training data was changed.