The following excerpt is from Chapter 7 of the free e-book "The tips and tricks guide to securing Windows Server 2003" written by Robert a Brag g and available at Realtimepublishers.com. Click for the complete book excerpt series.
L2TP over IP Sec and NAT -- Traversal
One of the issues with IP sec and hence VPN s using L2TP over IP sec is the inability to use them in matted environments. In a typical scenario, a VPN tunnel is used to provide access from outside the fire wall to inside by opening the ports on the fire wall used by the VPN. Both PPTP and L2TP over IP sec VPN s can be configured this way -- unless the fire wall, router or other remote access device, which sits between the VPN client and the VPN server, uses NAT. The current IP sec standard does not address this issue, in fact, an implementation -- such as Win2K -- written to the standard, sees the NAT manipulation of the addressing as tampering and drops the packets.
The problem with NAT comes about because the NAT device must translates the source address, and might assign a new source port to maintain a table to be used in routing replies back to the originating host. Here's what's happening: The NAT device modifies an outgoing packet by changing the real source address, the address of the sending client, to that of the Internet routable address provided to the NAT device. When packets from the Internet return to the NAT device, it is able to modify the destination address (which arrives using the Internet routable address assigned as the source address of the outgoing packet). How does it know the new source address to use? It knows because it keeps a table of sources addresses and ports mapped to the assigned source address and ports it replaced in outgoing packets. It is able to match the incoming packets and modify the destination address and port. However, because of the built-in security mechanisms of IP sec such tampering with the address is not allowed, hence the packets are dropped. This is why a Win2K to Win2K VPN that must pass through a NAT device can only use PPTP.
Before any NAT Traversal can occur, the client must be capable of recognizing the use of NAT, and the server must be NAT Traversal enabled. To fully understand the process, you should know something of how IP sec works. The following text explains, in simplified form, how NAT Traversal works. The client detects the NAT Traversal capability of the server by an exchange of strings (in the draft an MD5 hash of the draft title) during the first messages of IP sec, Phase I negotiation. (Phase I negotiation of IP sec includes establishment of IKE communications and generation of master key that is used in Phase II to generate encryption keys.) If the server does not return a message that includes the hash, it is not considered to be NAT Traversal enabled.
Next, the presence and location of a NAT device is detected by using the NAT-D payload message. To discover if NAT is being used, each side looks to see whether the IP address of the message has been modified. This is done by including a hash of the original source address and port. When received, a new hash of the existing source address and port is made. If the new hash matches the original (included in the payload), there is no NAT device in-between and processing continues per the IP sec standard. If the hash does not match, NAT is being used.
Although port 500 is the standard port for IKE traffic, IP sec-aware NAT devices may respond in a different way than standard NAT when they detect IKE traffic. Because NAT Traversal does not include the ability to determine exactly what an IP sec-aware NAT device is doing, moving or floating, the IKE traffic to port 4500 avoids the problems that the non-standard IP sec-aware NAT device may pose. Additional modifications such as the inclusion of a non-ESP marker may also be necessary. After the initial NAT Traversal communication is established, subsequent negotiations must start on port 4500. An illustration of the IKE floating can be found in Figure 7.27.
Figure 7.27: IKE floating.
Finally, negotiation of NAT Traversal encapsulation occurs. Normal IP sec negotiations include the use of either Tunnel or Transport modes. NAT Traversal requires the use of either of two new modes: UDP-Encapsulated-Tunnel and UDP-Encapsulated-Transport. Either of the modes allows NAT manipulation of the source IP and port information as this information is now available in a header designed for this purpose. Figure 7.28 is an example rendering of the UDP-Encapsulated-Transport mode.
Figure 7.28: UDP-Encapsulated-Transport mode.