The web server's auth.log has 48,000 lines from the past week, the domain controller's Security log holds two million events, and the incident manager wants to know by noon which accounts were compromised. Nobody reads that many lines. Parsing is how you cut them down to the dozen that matter, and once you have an answer, the people who act on it (HR, accounts and legal) decide what happens next.
On the CompTIA Security+ SY0-801 exam this sits under objective 4.8, which expects you to work with investigation data in a scenario, and it names log-parsing techniques explicitly. Be ready to read a command and say what it outputs, or to read a few log lines and pick out the attacker's address. This lesson covers command-line parsing with grep, awk, sort and uniq, regular expressions, following an attacker through a Linux host's logins, cron jobs and connections, PowerShell's Get-WinEvent, time normalization and timelines, and the three stakeholders the objective names: HR, accounts and legal.
Parsing turns raw lines into fields you can filter, count and sort
A raw log line is text. Parsing means splitting it into named fields so you can ask questions of it:
Oct 14 02:13:07 web01 sshd[2231]: Failed password for invalid user admin from 203.0.113.50 port 51234 ssh2
time=Oct 14 02:13:07 host=web01 process=sshd pid=2231 result=failed user=admin (does not exist)
src_ip=203.0.113.50 src_port=51234
With fields, "how many failures came from each address" becomes a count, and "did that address ever succeed" becomes a filter. A SIEM does this automatically at ingest with parsers that normalize every source into common field names, which the monitoring section covers. Investigators still parse by hand constantly: exported logs from a system that never reached the SIEM, a log type the SIEM has no parser for, or a quick question that is faster at a shell than in a query builder.
grep, awk, sort and uniq answer most first questions about a Linux log
Each tool does one job. grep selects lines, awk picks fields, sort orders lines, uniq -c counts repeats, and head trims the list. Chained with pipes, they answer the first questions in minutes:
# how many failed SSH password attempts are in the log?
grep -c "Failed password" /var/log/auth.log
4380
# which source addresses failed most often?
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -5
4127 203.0.113.50
212 198.51.100.77
38 192.0.2.14
3 10.1.20.45
Two details make that pipeline work. First, awk splits on whitespace and numbers fields from $1, and NF is the number of fields, so $(NF-3) is the fourth field from the end. Counting from the end matters here because "invalid user" adds two words to some lines, which moves the address if you count from the front, and some newer distributions write ISO 8601 timestamps that change the front of the line too. Second, uniq -c collapses only adjacent identical lines, so you must sort before it. The final sort -rn sorts numerically in reverse, putting the biggest count first.
The next question is whether the noisy address ever got in:
grep "Accepted" /var/log/auth.log | grep -Fw "203.0.113.50"
Oct 14 02:31:44 web01 sshd[2410]: Accepted password for deploy from 203.0.113.50 port 52011 ssh2
The -F flag treats the search as fixed text instead of a pattern, and -w matches whole words only. Without -F, each dot in an address matches any character; without -w, searching for 10.1.1.5 also matches 10.1.1.50 and 10.1.1.55. The same tools work on web server logs. In the common Apache and Nginx format the client address is field 1 and the status code is field 9, so awk '{print $1}' access.log | sort | uniq -c | sort -rn | head lists the busiest clients and awk '$9 == 500' access.log shows every request the application failed to handle.
Regular expressions match patterns instead of exact text
A regular expression describes a pattern, so one search finds every address, every injection attempt or every user name, whatever the exact value. The basics cover most investigative work:
| Pattern | Matches |
|---|---|
. |
Any single character |
* |
Zero or more of the item before it |
+ |
One or more of the item before it |
[0-9] |
One digit; [a-z] is one lowercase letter |
{1,3} |
One to three of the item before it |
^ and $ |
The start and end of the line |
\. |
A literal dot |
( ) |
Grouping, and capturing in tools that support it |
A vertical bar between two patterns means either one. With grep, the -E flag turns on extended syntax, -o prints only the matching text, -i ignores case and -v inverts the match:
# pull every IPv4-looking address out of a web log and count them
grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort | uniq -c | sort -rn | head
# find common SQL injection markers in requests, ignoring case
grep -iE "union(%20|\+)+select|%27|or(%20|\+)1(%3d|=)1" access.log
# extract the targeted user name from both kinds of failure line
grep "Failed password" /var/log/auth.log | sed -E 's/.*Failed password for (invalid user )?([^ ]+) from .*/\2/' | sort | uniq -c | sort -rn | head
The address pattern is deliberately loose. It would also match 999.999.1.1, which is fine for pulling addresses out of a log but wrong for validating input. The sed line shows capturing: the optional group (invalid user )? absorbs the extra words when present, and the second group captures the user name, which \2 prints.