TCP/IP Troubleshooting: A Structured Approach – Part 2: Troubleshooting Routing Tables

If you missed the other parts in this article series please read:

• TCP/IP Troubleshooting: A Structured Approach – Part 1: An Introduction
• TCP/IP Troubleshooting: A Structured Approach – Part 3: Repairing Network Connections
• TCP/IP Troubleshooting: A Structured Approach – Part 4: Using Netdiag.exe

In the first article of this series, I outlined a structured approach for troubleshooting TCP/IP networking issues on Windows-based networks. Key to this structured approach were three things:

• Understanding the networking technologies and protocols that underlie the problem.
• Determining the different elements of the problem and their attributes.
• Determining which troubleshooting steps and tools should be applied to resolve the problem.

I displayed these items as a bullet list instead of a numbered list because network troubleshooting generally isn’t as easy as 1-2-3. In other words, it’s often more of an art (i.e. based on intuition) than a science (based on a methodology).

Get your copy of Windows Server Hacks!

At the foundation of TCP/IP networking is the routing table, a data construct on each host on a TCP/IP network. Routing tables serve the following three purposes:

• They are used to store the information about other subnets on the network and how you can reach hosts on these networks.
• They are used to determine which host (called the next-hop IP address) each packet should be forwarded to in order to reach the host this packet is ultimately destined for.
• They are used to determine which network interface (called the next-hop interface) should be used to forward this packet so it gets to its ultimate destination.

Understanding routing tables is therefore essential if you want to be able to effectively troubleshoot routing issues on a TCP/IP network. Let’s look at how routing tables work, what they look like in different scenarios, and what troubleshooting steps and tools might be indicated in different situations. We’ll begin by examining the routing table on a single-homed server (a server with a single network interface) that has a single IP address assigned to it. I’ve chosen this example because it’s the easiest one to understand, and in next month’s follow up article we’ll look at more complicated scenarios including servers with multiple IP addresses (such as web servers) and servers with multiple network interfaces (such as servers that are connected to both the LAN and to a separate network used for running backup jobs).

Routing Table for Single-Homed Server with a Single IP Address

The following routing table is for a server that has IP address 172.16.11.30 on the 172.16.11.0/24 network:

C:\>route print

IPv4 Route Table

===========================================================================

Interface List

0x1 ……………………… MS TCP Loopback interface

0x10003 …00 03 ff 25 88 8c …… Intel 21140-Based PCI Fast Ethernet Adapter

(Generic)

===========================================================================

===========================================================================

Active Routes:

Network Destination        Netmask          Gateway       Interface  Metric

          0.0.0.0          0.0.0.0      172.16.11.1     172.16.11.30     20

        127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1      1

      172.16.11.0    255.255.255.0     172.16.11.30     172.16.11.30     20

     172.16.11.30  255.255.255.255        127.0.0.1        127.0.0.1     20

   172.16.255.255  255.255.255.255     172.16.11.30     172.16.11.30     20

        224.0.0.0        240.0.0.0     172.16.11.30     172.16.11.30     20

  255.255.255.255  255.255.255.255     172.16.11.30     172.16.11.30      1

Default Gateway:       172.16.11.1

===========================================================================

Persistent Routes:

  None

To display this routing table, you open a command prompt window and type route print at the command line. Let’s take this table apart so we can understand how it works.

Each routing entry (or route) in the routing table is comprised of five fields:

• Network Destination. An IP address or subnet representing a possible target destination to which IP packets may be forwarded.
• Netmask. A bitmask used for matching the destination field in a packet’s IP address to one of the possible network destinations above.
• Gateway. The next-hop IP address to which the packet must be forwarded to reach the particular destination network.
• Interface. The next-hop interface which must be used to forward the packet to reach the particular destination network.
• Metric. The cost of the route.

Example 1: Destination Host on Local Subnet

For our first example, let’s say this particular server (172.16.11.30) has to send a packet to another host with IP address 172.16.11.80, which is on the same subnet. This packet will thus have source address 172.16.11.30 and destination address 172.16.11.80. Here is how Windows uses its routing table to decide which route to use:

1. Windows first takes each route from the table in turn and performs a bitwise AND between the destination address in the packet (172.16.11.80) and the bitmask (Netmask) of the selected route. Here are the results, where each route in the table is identified by its network destination:
   

   Route	Netmask	172.16.11.80 AND Netmask
   0.0.0.0	0.0.0.0	0.0.0.0
   127.0.0.0	255.0.0.0	172.0.0.0
   172.16.11.0	255.255.255.0	172.16.11.0
   172.16.11.30	255.255.255.255	172.16.11.80
   172.16.255.255	255.255.255.255	172.16.11.80
   224.0.0.0	224.0.0.0	160.0.0.0
   255.255.255.255	255.255.255.255	172.16.11.80

2. For each route, the result of this ANDing is then compared with the Network Destination field of the route, and a match means the route can be used to forward the packet to its destination address. If more than one match is found, Windows uses the route with the longest match (the route whose Netmask has the highest number of 1 bits). If this doesn’t result in a unique match, Windows uses the match that has the lowest cost (Metric). Finally, if more than one match has the same lowest cost, Windows arbitrarily chooses one of them as the route to use. From the table above, you can see that this ANDing process results in two matches (routes 1 and 3) so Windows chooses the one that has the longest match, which is row 3. The result of all this is that Windows now knows which route to use to get this packet to its destination. Here’s what this route looks like in the server’s routing table:

   Network Destination        Netmask          Gateway       Interface  Metric

    

         172.16.11.0    255.255.255.0     172.16.11.30     172.16.11.30     20

3. Windows now uses the following algorithm to decide what to do next:
   1. If the route’s Gateway field matches the address of one of the network interfaces on the server (or if the Gateway is empty) then Windows sends the packet directly to its destination address using the interface specified in the route.
   2. If the route’s Gateway field does not match the address of any of the network interfaces on the server, then Windows sends the packet to the address in the route’s Gateway field.

Clearly, condition A is the case here since the route’s Gateway field (172.16.11.30) is the address assigned to the server’s single network card. Windows therefore determines that the destination address is on the local subnet and that means Windows can send the packet directly to that address without needing to forward it to any routers. So in this case, Windows simply sends the packet to 172.16.11.80 using the server’s 172.16.11.30 network interface, and the receiving host gets it.

Example 2: Destination Host on Remote Subnet

Now let’s go through the same process, but this time let’s say the server is trying to send the packet to a host on a different subnet, say a host with the address 172.16.10.200. In other words, the packet has a source address of 172.16.11.30 and a destination address of 172.16.10.200. Here’s how Windows uses its routing table to decide which route to use this time:

1. Windows takes each route from the table and performs a bitwise AND between the destination address in the packet (172.16.10.200) and the bitmask (Netmask) of the route. The results this time are like this:

2. For each route, the result of ANDing is compared with the Network Destination field of the route, and a match means the route can be used to forward the packet to its destination address. From our second table above, you can see this time that there is only one match i.e. row one where the route’s Network Destination field 0.0.0.0 matches the result of the AND action. So the route that Windows will use to forward the packet to its destination will be the following route:

   Network Destination        Netmask          Gateway       Interface  Metric

    

             0.0.0.0          0.0.0.0      172.16.11.1     172.16.11.30     20

3. Windows then uses the previously-described algorithm to decide what to do next, and this time condition B is the case here since the route’s Gateway field (172.16.11.1) does not match the address assigned to the server’s single network card, which is 172.16.11.30. Windows therefore determines that the destination address is on a remote subnet and that means Windows cannot send the packet directly to its destination but must instead forward the packet a router, which will then take care of getting the packet to its destination by forwarding it along further. So in this case, Windows sends the packet to the address shown in the Gateway field of the selected route (172.16.11.1) using the server’s 172.16.11.30 network interface. Once the router at 172.16.11.1 receives the packet, it determines what actions need to be taken next to forward the packet on to its final destination of 172.16.10.200, and of course this depends on whether the 172.16.11.10/24 network is a neighboring subnet to 172.16.11.11/24 (i.e. connected by a single router to it) or a remote network (connected by a series of routers with intermediate networks in between them).

Troubleshooting Tips

What can fail in the above process? First, it’s possible that Windows might be unable to select a route whose Network Destination field matches the bitwise AND of the Netmask field of the route and the packet’s destination address. If this happens, you’ve got a routing error, and this will probably be indicated to you by some network application running on your server. What happens usually is that Windows uses TCP to notify the upper layer of the network stack that the packet can’t be sent, and an error message of some kind usually results.

In this situation, you probably have either a corrupt routing table or an invalid persistent route in your routing table. Persistent routes are routes you add manually to the table using the route -p add command and which persist across reboots since their values are stored in the registry. If you add routes that are invalid, they can produce strange results, though most often they simply result in traffic being dropped mysteriously.

On the other hand, if the destination host is on a remote subnet and Windows forwards the packet to a router (the default gateway address) and this router can’t select a route, then what usually happens in this case is that the route returns an ICMP message of “Destination Unreachable – Host Unreachable” to the host that sent the packet. In this case, TCP will notify upper layers and some sort of error message will be displayed.

In either situation, a useful way to proceed is to examine the routing tables on the sending host and any intermediate routers along the way to the destination host, and see if these routing tables are consistent or look corrupted. A corrupted routing table can be restored (at least on Windows machines) by resetting the TCP/IP stack using the netsh int ip reset command, see KB299357 for details. Note that this reset operation does not remove persistent routes you’ve added to your routing table.

Conclusion

Now that you know a bit about how routing tables work and how to troubleshoot them, in the next article we’ll look at more complex examples such as servers with multiple addresses and several network cards.

If you missed the other parts in this article series please read:

• TCP/IP Troubleshooting: A Structured Approach – Part 1: An Introduction
• TCP/IP Troubleshooting: A Structured Approach – Part 3: Repairing Network Connections
• TCP/IP Troubleshooting: A Structured Approach – Part 4: Using Netdiag.exe