IPSec (Internet Protocol Security) is a suite of protocols that secures communication across IP networks. Think of it as a security layer that can be added to the standard Internet Protocol (IP). It operates at the network layer (Layer 3) of the OSI model, meaning it secures communication between devices regardless of the applications running on them.
IPSec is widely used to create Virtual Private Networks (VPNs).
→ Encapsulating Security Payload (ESP) provides the same security services as AH (integrity, authentication, anti-replay) and adds encryption (confidentiality). ESP can encrypt the payload only or the payload and the IP header.
→ Authentication Header (AH) provides data integrity, data origin authentication, and anti-replay protection. It does not provide encryption (confidentiality). It authenticates the entire IP packet, including the IP header (except for mutable fields like TTL).
AH is rarely used alone in many modern implementations of IPSec. While AH was originally designed to provide authentication and integrity for IP packets, its standalone usage has diminished
Therefore, if confidentiality (encryption) is a requirement, ESP is the appropriate choice. AH alone would be insufficient.
B) When transmitting data across untrusted networks with minimal computational overhead: While AH has slightly less overhead than ESP due to the lack of encryption, the difference is usually negligible in modern hardware. This isn't the primary reason to choose one over the other.
C) When implementing security protocols for multicast network traffic with dynamic group membership: Both AH and ESP have limitations with multicast, particularly with dynamic group membership and NAT traversal. This isn't a distinguishing factor between them.
D) When ensuring integrity of IP packet headers without necessarily encrypting the payload content: This accurately describes a use case for AH. However, the question asks when AH is less effective compared to ESP. If confidentiality is required, AH is less effective.