Buffer overruns are probably the most common form of vulnerability found.
A program running in memory is broken down into various parts.
fig 1. A typical process in memory.
The code section contains the machine instructions to be executed; the data section contains data such as text. As the program executes, memory needed to store data for a short time is allocated on the heap. So the heap grows and shrinks as the program executes.
Program code is broken down into functions so that code used more than once can be reused. As control passes to the function, parameters and the address to return control to is added to the stack. At the end of the function, the address to continue the execution is removed from the stack and the program continues.
A buffer overrun is caused by a programming error where the input to a fixed-length storage area or buffer is not properly checked. For example, a function in a program expects an input parameter in the numerical range 1-5 but instead receives the number 11532. If this value is not correctly checked a buffer overrun may occur. The result is that memory areas are overwritten with the unexpected input and the program may crash or exhibit unexpected behaviour. In the case of a specific attack it is possible that malicious code could be executed.
fig 2. Data is correctly written to a buffer on the heap.
Sophos Endpoint Security and Control aims to protect users from buffer overruns by monitoring the individual memory areas of particular processes and suspending processes when attacks are detected.
The animated cursor exploit from earlier this year used this technique and was successfully caught using buffer overrun protection.