What is IPsec?
IPsec, short for Internet Protocol Security, is a suite of protocols that protects traffic at the network layer. Because it secures the IP packets themselves, it protects every protocol above it, including older applications that have no encryption of their own. IPsec provides confidentiality, integrity, authentication of the peers and protection against replayed packets. It is the standard technology for site-to-site VPNs and is common for remote access VPNs and host-to-host protection.
The parts of IPsec: IKE, AH and ESP
- IKE (Internet Key Exchange) sets up the connection before any data flows. The peers authenticate each other with certificates or a pre-shared key, run a Diffie-Hellman exchange and agree on security associations (SAs) that record the algorithms and keys. IKE uses UDP 500, and UDP 4500 when NAT sits in the path (NAT traversal). IKEv2 is the current version.
- AH (Authentication Header, IP protocol 51) provides integrity, origin authentication and replay protection, covering most of the IP header as well as the payload. It does not encrypt. Because it covers the IP addresses, AH breaks when NAT rewrites them.
- ESP (Encapsulating Security Payload, IP protocol 50) provides encryption as well as integrity, authentication and replay protection. Nearly every real deployment uses ESP.
Transport mode vs. tunnel mode
| Transport mode | Tunnel mode | |
|---|---|---|
| What is protected | The packet's payload | The entire original packet |
| IP header | Original header kept | New outer header added |
| Internal addresses | Visible | Hidden inside the protected packet |
| Typical use | Host to host, such as server to server | Gateway to gateway and remote access VPNs |
In tunnel mode, a branch firewall takes a packet from 10.20.5.14 to 10.10.1.30, encrypts the whole thing inside ESP and sends it in a new packet between the two gateways' public addresses. Routers on the internet see only the gateways.
Example: a site-to-site tunnel proposal
A typical modern configuration for a tunnel between two offices:
- IKEv2, with the gateways authenticated by certificates
- AES-256-GCM encryption
- Diffie-Hellman group 19 or 20 (elliptic curve), or group 14 (2048-bit) at minimum
- ESP in tunnel mode, with perfect forward secrecy enabled when keys are renewed
DH groups 1, 2 and 5 are too weak and should be removed. Certificates scale better than a pre-shared key, which has to be entered on both devices and is rarely changed.
IPsec vs. TLS VPNs
IPsec protects all IP traffic at the network layer and is the natural fit for connecting two networks. A TLS VPN, still often called an SSL VPN, runs inside TLS on TCP 443, so it connects from hotels and customer sites where IPsec's UDP ports may be blocked, and it can offer clientless browser access to specific web applications. Older protocols appear as distractors: PPTP is broken, and L2TP provides no encryption on its own, which is why it is paired with IPsec as L2TP/IPsec.
IPsec on the CompTIA Security+ exam
SY0-801 covers IPsec as a protocol for protecting data in transit under objective 1.3 and as a VPN and tunneling choice in the secure architecture scenarios of objective 3.2. Map the wording: encrypt the traffic means ESP, integrity without confidentiality means AH, connect two sites or hide internal addressing means tunnel mode, and protect traffic between two servers end to end means transport mode. When every protocol between two networks must be protected at once, the answer is IPsec with ESP in tunnel mode. If the scenario stresses getting through restrictive firewalls or browser-only access, a TLS VPN fits better. Know the values for firewall questions: UDP 500 for IKE, UDP 4500 for NAT traversal, and IP protocols 50 and 51 for ESP and AH.