In 2023, lawyers in a New York personal injury case filed a brief citing six court decisions that did not exist. A chatbot had produced them, complete with case names, citations and quotations, and when one lawyer asked whether the cases were real, it assured him they were. The judge sanctioned the lawyers and their firm 5,000 dollars. No attacker was involved. The tool did what it was built to do, which is produce fluent, plausible text, and people treated that text as fact.
On the CompTIA Security+ SY0-801 exam, objective 2.6 includes bias, explainability, hallucinations and ethical considerations alongside attacks like poisoning and prompt injection. These are risks of using AI even when nobody attacks it, and attackers have learned to exploit some of them. This lesson covers where bias comes from and how it shows up in security tools, why explainability matters when a model makes a decision, why models invent facts and how attackers turn invented package names into malware delivery, and the ethical principles that govern responsible AI use.
Bias means a model's mistakes fall unevenly because its data was skewed
A model learns patterns from its training data, including patterns nobody intended it to learn. If the data underrepresents a group, reflects past unfairness or was labeled inconsistently, the model reproduces the skew at machine speed, with an appearance of objectivity that makes it harder to question.
Two well-known cases show the mechanism. Amazon scrapped an experimental resume-screening model, as reported in 2018, after finding that it penalized resumes containing the word "women's", because it had learned from a decade of resumes that came mostly from men. The 2018 Gender Shades study found that commercial facial analysis systems misclassified the gender of darker-skinned women up to 34.7 percent of the time, against under 1 percent for lighter-skinned men.
Bias usually enters in one of four ways:
- Sampling bias: the training data doesn't represent the population the model will serve.
- Historical bias: the data accurately records a past that was itself unfair.
- Proxy features: a protected attribute is removed, but a stand-in such as postal code or school carries the same signal.
- Feedback loops: the model's own decisions shape the next round of training data, reinforcing the original skew.
Security tools are not exempt. A user and entity behavior analytics (UEBA) baseline trained only on headquarters staff flags night-shift workers and overseas offices constantly; analysts learn to ignore those alerts, and a real intrusion in that group gets ignored with them. A phishing classifier trained mostly on English email misses lures written in other languages. An insider-risk or fraud model that flags one group of employees or customers far more often than others creates legal exposure for discrimination. In security, bias is an ethics problem and a detection gap at the same time.
The controls are representative training data, testing error rates separately for each relevant group before deployment, monitoring for drift as the population changes, and keeping people involved in decisions that affect people. If an attacker skews the data on purpose, the issue is no longer bias but poisoning, covered in the first lesson of this section.
Explainability is being able to say why a model made its decision
Explainability is the ability to describe why a model produced a particular output, in terms a person can check. Simple models such as decision trees are explainable by design; you can follow the branches. Deep neural networks and large language models, with billions of parameters, are largely black boxes. You see the input and the output, not the reasoning.
That matters every time a security tool makes a call. Compare two alerts for the same event:
ALERT 2026-10-21 09:14:07 host=FIN-WS-22 action=quarantined
verdict=malicious score=0.94 engine=behavior-ml-v7
details: none available
ALERT 2026-10-21 09:14:07 host=FIN-WS-22 action=quarantined
verdict=malicious score=0.94 engine=behavior-ml-v7
top factors: winword.exe spawned powershell.exe -enc ... (+0.41)
connection to domain registered 2 days ago (+0.27)
process read lsass.exe memory (+0.19)
The first alert asks the analyst to trust the model. The second lets them verify the verdict in minutes and, if it is wrong, see which factor to tune.
Lack of explainability causes concrete problems. Analysts can't separate true positives from false ones without redoing the investigation. Managers, auditors and courts expect a reason when a system blocks a user, quarantines an executive's laptop or denies a transaction. Poisoning and evasion are harder to spot when you can't see what the model relied on. Regulation adds pressure: the GDPR gives people rights concerning decisions made solely by automated processing that significantly affect them, including meaningful information about the logic involved, and the EU AI Act requires transparency and human oversight for systems it classifies as high-risk.
The remedies are to choose interpretable models where the stakes are high, to use feature attribution methods such as SHAP and LIME that estimate which inputs drove a result, and to buy tools that show the evidence behind each verdict. One caution: when an LLM explains its own answer, the explanation is more generated text, not a trace of the computation, and it may not reflect why the model actually answered as it did.