Buffer Overflows, Data Execution Prevention, and You


I have always called it a variation on the bait and switch. Let’s say that all you want in the world is that shiny new 52″ big screen TV but you know your wife is going to be a little resistant to the idea. What do you do? Well, instead of pitching the 52″ idea, you go bigger and tell her you are eyeing the 65″ plasma. The chances of this going over well are pretty much zero, but, by pitching high you might have more room to negotiate. You can then start with a bombardment of information on the pro’s of getting such a TV, and then sneak in the idea of getting a smaller one as a compromise. This way she would be more inclined to go for the 52″ idea. If you succeed, you would have effectively filled her head with so much nonsense that you can sneak in the idea you want to move forward with!

A buffer overflow works in much the same way. Simply put, you end up sending too much data to a program to such an extent that the data is forced to be stored outside of the memory that the programmer set aside for the program. This overflow of data can cause all sorts of erratic behavior, but the one that is most concerning to us is that once a buffer is filled to a certain point, it becomes a playfield of program codes that “attackers” can execute on that system. It is a simple bait and switch.

In this article I am going to walk you through an example of a buffer overflow which can result in an attacker executing code on your system. Along with this, I am going to discuss Data Execution Preventions (DEP), a Windows feature designed to prevent buffer overflows from happening.

Recognizing a Buffer Overflow

In order to fully understand the intricacies of buffer overflows you need to have a deep understanding of a high level programming language such as C/C++, as well as a detailed knowledge of memory stack operations. Hundreds of books have been written on these subjects so there is no way to fully elaborate the process here in a few words, which is why I will only be giving a high level overview.

When writing a program, one of the concerns that must be dealt with is the size of buffer space allocated to specific functions. A buffer is a contiguous space of memory that a program can use to store data that can be passed to other functions. Let us consider the following code example:

Figure 1: A C function example that is vulnerable to buffer overflow

This function is fairly straightforward and begins by declaring two variables, bufferA and bufferB which have sizes of 50 and 16 respectively. The program outputs a question to the user requesting their name and uses the “gets” function in order to receive this input. Finally, the data supplied from the user is copied from bufferA to the buffer variable and the function is complete.

With such a simple program it may seem a bit odd that this is vulnerable to any sort of attack. The problem here lies in the use of the “gets” function. By itself, the “gets” function does not do any form of bounds checking. That is to say that it does not ensure that the data inserted into bufferA is actually 50 characters or less. If the user supplies more than fifty characters then the program will crash.

If you followed that, then you should see something else that looks odd. The strcopy function copies the contents of bufferA into bufferB. However, bufferB has a smaller size than bufferA. This means that even if a user enters something less than 50 but more than 16 characters into bufferA, it will cause an overflow situation when it is copied into bufferB, causing the program to crash. This small program has not one, but TWO buffer overflow vulnerabilities.

Exploiting the Overflow

Given these conditions that allow for a buffer overflow, how does this translate into a problem? In situations where buffer overflows occur, the data that overflows the assigned buffer space has to go somewhere. This data overflows into neighboring areas of memory which typically causes the program to crash because it does not know how to handle the extra data. On the other hand, when this is done by someone who understands how the memory stack and assembly language function, things can be a little different. In this scenario, an attacker can manipulate the buffer overflow in such a way that they can create their own system commands, convert them to low level byte code, and send them to the program in the proper format so that they are executed.

Figure 2: An example of C and Assembly shellcode designed to return a Windows C:\ prompt

The code that is executed at this point is done under the context of the user’s original vulnerable application. This means that if the program is run by a system administrator, the injected code also runs under the context of a system administrator. Depending on the size of the buffer, the attacker can inject many different types of codes. By far, the most common is what is called shellcode. This code will return a shell (a windows C:\ prompt for example) to the person who has injected the code. Given the proper user context this now means that this person now has complete control over the workstation. If you were not afraid of a buffer overflow at this point, you should be now. Buffer overflows come in all shapes and sizes and someone who is skilled in “stack dancing”, as it’s often called, has a very real ability to gain complete control over any vulnerable system.

Data Execution Prevention

The simplest solution to preventing the exploitation of buffer overflow vulnerabilities is for programmers to ensure their code is secure. This is not really an automated process, it requires a significant amount of man hours in code review to ensure the integrity of the program code is maintained, and as the number of lines of code grows, so does the time commitment. Realistically, we need other forms of protection against these vulnerabilities. In an effort to do this, Microsoft created a feature known as Data Execution Prevention (DEP).

DEP is a security feature originally released in Windows XP SP2 that is designed to prevent an application from executing code in a non-executable area of memory. DEP is available in both hardware-based and software-based configurations.

Hardware-Based DEP

The use of hardware-based DEP is considered the most secure implementation of DEP. In this implementation, the processor marks all memory locations as “non-executable” unless the location already explicitly contains executable code. The goal is that in doing this, DEP will intercept the attempted execution of code in non-executable areas.

The primary problem with the use of hardware-based DEP is that it is only supported by a minimal number of processes. The processor feature which allows for this is referred to as the NX feature for AMD processors and the XD feature by Intel processors.

Software-Based DEP

When hardware-based DEP is not available, software-based DEP must be used. This form of DEP is built into the Windows operation system. Software-based DEP works by detecting when exceptions are thrown by programs and ensures that those exceptions are actually a valid part of the program before allowing them to proceed.

Configuring Data Execution Prevention

The configuration of DEP can be handled the system control panel. You can access this screen by browsing the System control panel and selecting the Advanced System Settings area. Clicking on the Data Execution Prevention tab will get you to the proper screen in order to make configuration changes.

Figure 3: The default configuration of DEP as shown in Windows 7

DEP has two configuration options available. The default selection is shown in the image above and is what is called an OptIn configuration. As stated in the description, this option only applies DEP to important system programs and services. This level of security, called the OptOut configuration, is definitely better than a complete lack of DEP, but if you want achieve the highest level of security then you should select the second option which applies DEP to all programs and services on the system. You can also note that the lower portion of this screen indicates whether or not the CPU being used supports hardware-based DEP.

Although configuring DEP to be applied to all programs and services is the most secure, keep in mind that it does raise to possibility of potential compatibility issues. First of all, some programs will perform functions that are legitimate but may be blocked by DEP due to the way they work. In these cases you must create an exception for that program. This is done by clicking the Add button on the DEP configuration screen and selecting the appropriate executable. This form of DEP may also cause driver compatibility issues. Microsoft recommends only using signed drivers to prevent this from happening.

Figure 4: Configuring DEP for all programs and one exception

The result of this configuration will produce a warning message when DEP detects the execution of code from a non executable area.

Figure 5: DEP blocking an execution in Windows Explorer


The term buffer overflow is thrown around very loosely but it poses a more severe threat to system security than almost any other type of threat out there. Ultimately, attackers are going to continue to perform this bait and switch that allows the exploitation of systems. These attacks are continuing to evolve a few steps quicker than the good guys can keep up with. If you are a programmer you can start doing your part by ensuring that you are performing proper audits of your code and employing secure coding practices. If you are a system administrator you can follow the guidelines here to enforce a more strict form of DEP to help mitigate some of this risk.

If you are interested in learning more about buffer overflows and their defense I would highly recommend some of the following resources:

About The Author

1 thought on “Buffer Overflows, Data Execution Prevention, and You”

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top