You can never test quality or security into an application. If the app is written in an insecure manner with poor coding practices or has a large attack surface, no amount of testing will make it secure.
Although known to only a few developers, non-security experts can conduct effective security testing, most notably by fuzz testing. Fuzz testing includes penetration testing, run-time verification, re-reviewing threat models and re-evaluating the attack surface. By fuzz testing all the file formats your app consumes and the “parser per” format and network protocols with man-in-the-middle techniques, you’ll provide a “sanity check” of code before release.
Originally developed to find reliability bugs, fuzz testing is an effective way to find certain classes of security bugs as well. Fuzzing means creating malformed data and having the app under test consume the data to see how it reacts. If the app fails unexpectedly, a bug will be found. The bug is a reliability bug and, possibly, a security bug. Fuzzing is aimed at exercising code that analyzes data structures — loosely referred to as parsers. There are three broad classes of parsers:
File format parsers
Network protocol parsers
APIs and miscellaneous parsers
Fuzzing file formats means building malformed files to be consumed by your app. For example, if your app parses and displays TIFF files, you could build a malformed TIFF and have the app read the file. Of course, you don’t create just one malformed file; SDL mandates that you create and test at least 100,000 malformed files for every file format and parser you support.
The process for fuzzing files is simple. It consists of the following steps:
The following paragraphs address each of these steps in detail.
Identify all valid file formats
The first step in the file-fuzzing process is to identify all file formats your app reads and handles.
Your app should fail gracefully if faced with a file format it does not render or understand. Similarly, any component you have developed should fail gracefully and, just as important, bubble errors up to the next level of code.
Collect a library of valid files
You should gather as many valid files from as many trusted sources as possible — and the files should represent a broad spectrum of content. Aim for at least 100 files of each supported file type to get reasonable coverage. For example, if you manufacture digital photography equipment, you’ll need representative files from every camera and scanner you build.
You should continue to build on this library over time as you define new formats or new format variants.
Malform a file
The work really starts when you begin malforming a file. You need to build or use a tool that chooses a file at random, malforms the file and then passes the file to the software under test (van Sprundel 2005, Sutton and Greene 2005, Oehlert 2005).
The two broad classes of file fuzzing are smart fuzzing and dumb fuzzing. Smart fuzzing is when you know the data structure of the file format and you change specific values within the file. Dumbfuzzing is when you change the data at random. For example, PNG files start with a well-known signature followed by a series of blocks, named chunks (Milano 1999). The signature is 8 bytes long and must have the following value: 0x89 0x50 0x4E 0x47 0x0D 0x0A 0x1A 0x0A.
Each chunk has the following format:
4-byte length
4-byte type
n-byte data
4-byte CRC (cyclical redundancy check)
The IHDR chunk type always follows the signature, specifies image dimensions and color information, and has the following format:
4-byte width
4-byte height
1-byte bit depth
1-byte color type
1-byte compression mode
1-byte filter mode
1-byte interlace mode
It’s important to know also that PNG files structure multibyte integers with the most significant byte first.
Knowing the basic PNG format and the IHDR chunk type, you can be very specific about how you corrupt a PNG file. We’ll give file corruption examples later.
Dumb fuzzing is a shotgun approach: you take a valid file and randomly corrupt it. It really is that simple. You can smart fuzz or dumb fuzz a file in many ways, including these:
Looking back at the PNG format, you could be very specific and smart fuzz a file by using the following techniques:
In the PNG example, you would also need to build a valid CRC for each malformed file; otherwise, a CRC failure would prevent most of the parsing code from being exercised.
Consume the file and observe the application
Finally, have the application consume the file — for example, via a command-line argument that includes the file name. As the app runs, monitor for failures such as access violations or core dumps, and watch for spiked CPU usage. In Microsoft Windows, you can set the fuzzing tool to be a mini-debugger by using debugging APIs (Robbins 2003), and then you can write failure information to a log file. The sample fuzzer, MiniFuzz, shows how to do this.