What is a regular expression?
A regular expression, usually shortened to regex, is a compact pattern that describes a set of strings. Instead of searching for one exact value, you describe the shape of what you want, such as five digits, or any line that begins with an IP address, and a regex engine finds every match. Security tools use them constantly: log parsers extract fields with them, SIEM and IDS rules match attack strings with them, data loss prevention tools spot card numbers with them, and applications validate input with them.
Core syntax
| Pattern | Meaning |
|---|---|
. | Any one character |
[abc], [0-9] | One character from a set or range; [^0-9] is any character outside it |
\d, \w, \s | A digit, a word character, a whitespace character (Perl-style engines) |
*, +, ? | Repeat the preceding item zero or more times, one or more times, or make it optional |
{3}, {2,5} | Exactly three, or two to five, of the preceding item |
^, $ | Anchor to the start or end of the line or string |
| | Match the pattern on either side |
( ) | Group items and capture what they matched |
\b | A word boundary |
Engines differ. grep's basic syntax needs backslashes before some operators, its extended mode (grep -E) doesn't, and the Perl-compatible engines in most programming languages add shortcuts such as \d and lookarounds. A pattern that works in one tool may need adjusting in another.
Examples in security work
- Log parsing:
grep -oE 'user=[a-z0-9._-]+' proxy.logpulls every user name field out of a proxy log. - Detection: a rule matching
(?i)union\s+selectflags a common SQL injection fragment in web requests regardless of case. - Data loss prevention:
\b4[0-9]{15}\bmatches 16-digit numbers beginning with 4, the shape of most Visa card numbers. DLP products pair patterns like this with a Luhn checksum test and nearby keywords to cut false positives, since many 16-digit numbers aren't card numbers. - Input validation:
^[0-9]{5}(-[0-9]{4})?$accepts a US ZIP code and nothing else. The anchors matter: without ^ and $, the pattern would accept any input that merely contains five digits somewhere.
ReDoS: when a pattern becomes the vulnerability
Most regex engines in programming languages use backtracking: when part of a pattern fails, the engine goes back and tries other ways to match. Patterns with nested or overlapping quantifiers, such as (a+)+$, can force an exponential number of attempts when the input almost matches. Thirty letter a's followed by an exclamation mark can tie up a processor core for seconds or far longer. Attackers exploit this as regular expression denial of service (ReDoS), sending crafted input to any endpoint that runs a vulnerable pattern. In July 2019, one badly behaved regex in a web application firewall rule pushed Cloudflare's servers to full CPU and caused a global outage of roughly half an hour.
The defenses are to avoid nested quantifiers and overlapping alternatives, limit input length before matching, set match timeouts where the language supports them, and use engines that guarantee linear-time matching, such as RE2, for patterns applied to untrusted input.
Regular expressions vs. wildcards
Shell wildcards, also called globs, look similar but mean something different. In a shell, *.log matches any file name ending in .log, because * stands for any run of characters. In a regular expression, * means zero or more of the item before it, so the equivalent pattern is .*\.log$. Mixing the two up is a common reason a search returns nothing, or everything.
Regular expression on the CompTIA Security+ exam
Regular expressions come up in two SY0-801 areas. Under objective 4.8 they are part of parsing logs during an investigation: expect to read a short pattern and say what it matches, such as one that extracts IP addresses or failed login lines. Under objective 4.1 they are a common way to implement input validation: a server-side allow-list pattern anchored with ^ and $ is the strong answer, while a client-side check or a block list of known-bad strings is the distractor. Remember that an unescaped dot matches any character, and that a pattern run against untrusted input can itself be a denial-of-service risk.