This week was another travel week. I was out of town teaching the boys about the ISA 2004 Enterprise Edition firewall. Since these guys aim to be ISA firewall sharpies, I figured I’d bring my ISA 2006 beta firewall lab with me and show them all the new and cool things.
The ISA 2004 training went fine. They got jiggy on the Configuration Storage Server (CSS), Network Load Balancing (NLB), and the Cache Array Routing Protocol (CARP). While they we’re doing their labs, I figured I’d have time to set up the ISA 2006 firewall lab. The lab is pretty simple: two ISA firewall array members, a combined DC/Exchange Server, and a SharePoint Portal Server. No machine was designated as a dedicated CSS, as I planned to walk them through the rigors of placing CSSs on different machines on the network using both the domain (preferred) and the workgroup (reviled) scenarios.
I copied the VMs to a nice little laptop USB drive product called Wolverine, which I thought was pretty cool given that’s the code name for the ISA 2006 firewall product. When I got the production facility, I copied the VMs to a central server on their location where they had MS Virtual Server installed. I thought this was going to be interesting, because I’ve always used VMware and Virtual PC. Not that I have anything against MS Virtual Server, it’s just that I’ve never had to use it and since I’ve standardized on GSX server (which is gone, now VMware is giving it away under the name of VMware Virtual Server).
To my pleasant surprise, the VPCs started just fine on the MS Virtual Server box. But there was one thing I had to do — change the external IP addresses on the ISA firewall array machines. Before I did that, I had to figure out how to bind these external interfaces to the MS Virtual Server’s physical NIC. That wasn’t too hard. I then changed the IP addresses on the firewall array members external interfaces.
Everything was going well so I knew I was soon to be going in for a fall. I created an "all open" rule (for testing purposes only) and then pinged the proximal router and it worked. Maybe nothing bad was going to happen after all. Of course, I should have known better because when I pinged sites by their addresses, it was clear that DNS was not working.
I fired up Network Monitor on the machine I was testing from and tried some nslookup’s. The queries were going out but nothing was coming back. The next step was to check the ISA firewall’s real time log viewer. The pings were making it to the ISA firewall, but when I did DNS queries, they never appeared in the ISA firewall’s real time log viewer. I thought that was mighty interesting because I never saw anything like that before.
Of course, the ISA firewall’s real time log viewer is the poor man’s attempt at trying to figure out what’s happening on the wire. What I really needed to do was check out a NetMon trace. Unfortunately, I forgot to install NetMon on the ISA firewall array. This was a critical error on my part for two reasons:
- You should always install NetMon on the ISA firewall array servers for troubleshooting reasons. It is also a constant reminder that you should never treat the ISA firewall as a workstation, since if you do so, you risk that ISA firewall being owned and when it’s owned, not only did you provide them the opportunity to own it by being incompetent and violating basic network security principles, but you’ve also provided them a powerful tool to listen on your network traffic.
- I didn’t bring the Windows 2003 CD with me.
Of course, someone else had a Windows 2003 CD but we couldn’t figure out how to drag and drop files between desktops on MS Virtual Server. I still don’t know if MS VS has that feature. And since our networking was snafu’ed, we had to figure out what the problem was. All that said, there were other options available to us (such as attaching an iso to the VM, or changing the IP address and virtual network a server was located on), but we were more interested in solving the problem the hard way, without NetMon. If things didn’t pan out, we’d break down and find a way to get NetMon installed).
I pounded on the problem for about an hour. Checked the packet traces, checked the Event Viewers, messed with FWENGMON and more. I then recalled that I had NLB enabled on the array at one time and then disabled it in the ISA firewall console. I did this because I was going to show them how to use the MS script that enables you to whack the NLB settings on an array that has had NLB enabled in the past but then is disabled in the ISA firewall console. I was sure to disable NLB manually on each machine afterwards just to make sure it wasn’t still enabled and even restarted the machines after disabling NLB.
I was getting bored of doing the same thing over and over again and getting the same result, so some of the guys there said they had a mega-genius kind of guy who can solve any Windows networking problem. I always like to meet people smarter than me, so I said bring him on. Well, the smart guy starts checking things out. He did everything I did, but he did two things that I didn’t:
- He did an ipconfg /all on one of the array servers. This showed that the OLD external IP address was still bound as a secondary IP address on the interface. WTF? We removed the old IP addresses so that only the new valid IP address was bound to the NIC
- He checked NLB (this was after I restarted the computer after disabling myself) on the NIC and found that it was bound to the NIC again! He unbound himself as I did, and then he tested DNS name resolution without restarting
Guess what happened? It worked! The key was that he did the ipconfig /all and found the old IP address AND he tested name resolution immediately after disabling NLB and didn’t restart the firewall array servers. What must have happened is that NLB was rebound to the NICs after restarting the server.
So to test this hypothesis, I restarted the servers again and NLB was rebound to the interfaces. I didn’t know that would happen. I don’t remember reading about that happening. Nevertheless, now I know that’s what happens. Now that I had Internet connectivity, I downloaded the script that you can use on the NLB array members to unbind NLB from the interfaces and remove the Registry settings that the ISA NLB configuration adds to the firewall array. I ran the script and all was well. I also restarted the firewall array machines to confirm that the script actually worked.
MORAL OF THE STORY:
If for any reason you enable and then later disable NLB on the ISA firewall array’s interfaces, make sure that you run the script. You can find it at: http://www.microsoft.com/downloads/details.aspx?familyid=1B02DFA5-B982-48CD-AD4C-B72BBAC0002F&displaylang=en
HTH,
Tom
Thomas W Shinder, M.D.