Minimizing the effect of DOS attacks and overflows on your DNS servers

DNS is an essential part of the fundamental services that is required by all windows 2000 networks.   If DNS is DOS attacked then the organization may face major consequences.  It is vital that DNS is kept as secure as possible as a multitude of servers like Microsoft ISA, exchange 2000, and any other communication software have serious dependencies on the perfect running of the DNS.

It is highly recommended that the security administrator frequents websites where vulnerabilities are published and where many issues are discussed in an open forum.  This will give you an intruder’s view and will enable you to better protect your network against related intrusion attempts.  There is very little information that is not published on the internet in regards to vulnerabilities and news travels quicker than light.  As soon as new vulnerabilities are discovered hundreds of hackers are updated simultaneously and the race is on.  Hackers hurry over to well known websites attempting the newly discovered vulnerabilities and attempting every angle in order to exploit the new discovery.  This gives smaller organizations an opportunity to update their software.  IT is important to note that major organization have measures in place that alert them to new vulnerabilities.  Applications like GFI’s LANguard have host intrusion detection capabilities that inform the administrators of potential intrusions.  Always be aware that when the hackers have found that the large organizations have no vulnerabilities the next target tends to be smaller public companies.  Patch your machines and keep them current for the underground world is always on the prowl.

Security should be hard coded into the DNS design

When designing the DNS system it is imperative that security is hard coded into the design and implementation.  This technique ensures that the configuration is failsafe and enhances any pit falls that the organization may have within security strategies that are in place to protect the organizations best interest.

  • Ensure that the DNS server is placed behind a firewall .
  • Use the firewall to perform DNS queries; this greatly reduces organizational DNS risk.
  • Ensure that the DNS servers are clustered.
  • Do not chain DNS servers as this strategy has proven to have its downfalls.
  • Remove other services from your DNS servers as vulnerabilities can be exploited within badly written software.
  • Scan the DNS machine and ensure that there are no other ports that are listening except for the typical DNS ports.

Different design scenarios.

1.       Secure the DNS Design: usually DNS designs include a primary DNS server and multiple client DNS servers known as mater slave. Primary DNS servers should be hosted by the organization and further client DNS servers reference off the primary master DNS server. Your primary DNS server should have router and firewall protection as would be founding a DMZ environment.

2.       Split DNS Design: Split DNS design employs the separation of the internal DNS servers from the external DNS servers. Internal servers only contain internal DNS entries and the external server only contains external entries respectively. Intruders look for DNS servers that are not split and expose internal hosts to the Internet by reflecting internal IP addresses that the intruder can directly address. This information is then used to plot the networks coordinative points and is used like a tool to find the weak spot where the intruder can gain entry.

3.       DNS fail over design: duplication is vital when implementing DNS. A multitude of applications rely on DNS for name resolution in multiplatform environments. HTTP,SMTP and many other windows applications require DNS.  It is a good idea to have a well designed and implemented fail over strategy. Robust design should include at least two internal DNS servers for every set of 500 users. DNS servers should be distributed throughout the company as a load balancing strategy and using performance monitor you should identify the segments that need their own DNS servers. Furthermore if the segments are separated by WAN links it will be advisable to have a DNS server at each side to prevent clients from traveling over slow WAN links to resolve domain names. Secondary DNS servers should be setup on the local DHCP/WINS server at each site.

DOS attacking multiple DNS servers proves to be more of a challenge than attacking organizations that only have one DNS server running.  Do not rely on this strategy as a shield against hackers as it will only work temporarily and will fail once they rescan your network and find an alternate DNS server.  It is more effective to have a firewall do the DNS queries that to have the DNS server itself.  In most cases this is not a practical solution and that is why other strategies need to b explored. 

4.       LAN protection: As a further layer of protection it is advisable that a good HIDS be installed on the DNS server incase and intruder is lurking about. An excellent tried and tested software would be LANguard by GFI. It is a good idea to spread your DNS servers over different subnets as an interesting lesson can be learnt form a historical attack on Microsoft’s DNS servers in the last 5 years

Figure A:  The diagram above depicts the various layers of defense that may be implemented against DOS attacks. 

  1. The fist layer of defense typically being your internet router.  Your internet router can have access lists that specify that when communication takes place with a specific IP address on the DMZ packet transactions may only take place on specific ports to specific machines.  So the intruder would be limited to only the default ports on the DNS server this can prove to be very useful and will further harden the operating system flaws in a virtual way.  This is no excuse not to update your patches and IDS software keeping IDS software and operating systems fully patch further ads more armor to your defense mechanism.

  2. If the intruder gets past the router he will then get to your firewall.  Firewalls can be configured to drop sequential packets or SYN packets that have malicious intent.  Ensuring that your firewall supports state full packet inspection is advantageous and will ensure that no packet passes without interrogation.

  3. If these previous defense systems fail, the IDS should flag a DOS attack.  HIDS applications are installed locally on target machines and can be set to alert administrators that an intrusion attempt or malicious activity is taking place.  The cluster should remain up, but it’s a matter of time before the intruder realizes that other DNS servers are serving request.  Act quickly if you want your organization to stay up.

DNS hostname overflows

DNS hostname overflows occur when a DNS response intended for a host name exceeds a fixed length. Some applications do not check the length of the host names and may return overflow internal buffers when copying this host name.  This process may allow an intruder to execute unpredictable commands on pre-selected machine.

DNS length overflows

DNS responses for IP addresses include a length field, which typically comprise of four bytes. A DNS response can be formatted to reflect a larger value, various applications executing DNS lookups will reflect an overflow of internal buffers, allowing a remote attacker to execute unpredictable commands on any machine with this vulnerability exposed.

An article has been written on this vulnerability and on a simple solution (Configure ISA to enable a DNS intrusion detection filter.) This article can be found at

Proposed registry settings that assist in prevention of attacks

Apply the following registry settings to your Lab environment first and if you are happy that they work as intended then use them on your live environment.  Please note that the original registry should always be backed up before making changes to it. These registry settings have been used to reduce the probability of denial of service (DoS) attacks on any windows computers.

The diagram above depicts where chnges will be made to the registry after you have backed it up. The key will be HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.  All value names to be created will be value type Reg_dword.

  1. Create a value name called EnableDeadGWDetect set the value to 0 doing this enables the TCP/IP stack to change to an alternate gateway when a connection is experiencing an issue. Monitor both gateways to ensure you resolve the issue that may arise if DOS attacked.
  2. Create a value name called EnablePMTUDiscovery set the value to 0 doing this enables TCP/IP to decide the maximum transmissions unit (MTU) that can be transmitted over a route to the computer. By setting the value to 0 you force any transmission to 576 bytes.
  3. Create a value name called KeepAlive set the value to 300,000 this check to see if remote systems are still available and disconnects form then within the keepalive period if no response is received.
  4. Create a value name called SynAttackProtect set the value to 2 doing so protects the machine against precise type of denial of service attack known as a SYN Flood Attack. SYN Flood Attacks obstruct typical TCP acknowledgement handshake involving a client and server.

The diagram above depicts the SYN process as it should take place.

A SYN attack will be represented when a server receives an overflow of connection requests from a fake computer. The server waits for the final ACK message to be received from the client, this causes the server memory to fill up with invalid connections requests, and when real computers attempt connections the server can not service the requests and a denial of service is created.


If no intrusion detection software is installed on your DNS servers it proves to be very challenging when identify the problems whilst DOS attacks are taking place.  Intrusion detection is the answer when attempting to recognize malicious DOS attack attempts on you DNS server.  This whitepaper has shown you ways to minimize such attacks and recommendations have been surmised that soften the blow when DOS attacks are in fact degrading system performance or brining DNS systems to a stand still.  It is important to realize DOS attacks are ruthless and that there is very little defense against them if you are caught unprepared.  Arm your self with information and when the time comes apply the correct technology to remedy the problem.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top