The US Defense Advanced Research Projects Agency, DARPA, recently initiated a two-year AI Cyber Challenge (AIxCC), welcoming elite AI and cybersecurity specialists to invent new AI systems to protect major open-source projects on which our critical infrastructure relies. As AI continues to evolve, it’s essential to invest in AI tools for Guardians, and this competition will push technology to achieve that goal. 

Google’s OSS-Fuzz and Security Engineering teams have been thrilled to support AIxCC organizers in shaping their challenges and competition structure. To evaluate the competition, we constructed a Cyber Reasoning System (CRS) to tackle DARPA’s exemplary challenge. 

This post will discuss our strategy for the exemplary challenge using open-source technology present in Google’s OSS-Fuzz, emphasizing areas where AI can enhance the platform’s capability to detect and correct vulnerabilities, which we anticipate will prompt creative solutions from participants.

Harnessing OSS-Fuzz

AIxCC challenges revolve around discovering and addressing vulnerabilities in open-source projects. OSS-Fuzz, the fuzz testing platform, has been detecting vulnerabilities in open source projects as a communal offering for several years, resulting in more than 11,000 vulnerabilities identified and resolved across 1200+ projects. OSS-Fuzz is complimentary, open source, and its projects and structure are analogous to AIxCC challenges. Rivals can conveniently reuse its existing toolchains, fuzzing engines, and sanitizers on AIxCC projects. Our fundamental Cyber Reasoning System (CRS) chiefly employs non-AI methodologies and comes with certain restrictions. We identify these as openings for rivals to investigate the potential of AI in advancing fuzz testing.

Fuzzing the AIxCC challenges

When it comes to userspace Java and C/C++ challenges, employing fuzzing engines like libFuzzer, AFL(++), and Jazzer is uncomplicated since they share the same framework as OSS-Fuzz.

For kernel fuzzing, the situation becomes more complex, so we deliberated on two alternatives:

• Syzkaller, an autonomous coverage guided kernel fuzzer

• A universal purpose coverage guided fuzzer, like AFL

Syzkaller has proven effective in identifying vulnerabilities in the Linux kernel, but is unsuitable for AIxCC due to its methodology of generating syscall sequences to fuzz the entire Linux kernel. Conversely, AIxCC kernel challenges (exemplar) include a userspace test environment to target specific sections of the kernel.

Instead, AFL was selected as it is commonly used for fuzzing userspace applications. To facilitate kernel fuzzing, a similar methodology to a past blog publication by Cloudflare was followed. The kernel was compiled with KCOV and KSAN instrumentation and operated virtually using QEMU. Subsequently, a userspace test environment mimics an AFL fork server, which processes the inputs by executing a series of syscalls for fuzzing purposes.

Following each input execution, the test environment gathered KCOV coverage data and transmitted it to AFL’s coverage counters via shared memory for coverage-guided fuzzing capabilities. Additionally, after each run, the test environment inspected the kernel dmesg logs to determine if a KASAN sanitizer trigger was provoked by the input.

Modifications were necessary to make Cloudflare’s harness adaptable with the given kernel challenges. It was essential to transform the harness into a library/wrapper that could be associated with various AIxCC kernel harnesses.

AIxCC challenges are equipped with their own main() function that accepts a file path. This function opens and reads the file, sending it to the harness() method, which receives a buffer and size representing the input. Our wrapper function was crafted by encapsulating the main() function during compilation using $CC -Wl,–wrap=main harness.c harness_wrapper.a 

The container commences by initializing KCOV, the AFL forkserver, and shared memory. In addition, the wrapper retrieves the input from stdin (which is the default expectation of AFL) and transfers it to the harness() function within the challenge harness. 

Considering that AIxCC’s harnesses are beyond our influence and could behave inadequately, we needed to be cautious regarding memory or FD leaks originating from the challenge harness. In reality, the provided harness exhibits multiple FD leaks, leading to a scenario where fuzzing it swiftly becomes ineffective once the FD limit is exceeded.

To resolve this issue, we have two potential options:

• Forcefully terminate FDs generated during the execution of harness by inspecting for newly created FDs through /proc/self/fd prior to and following the execution of the harness, or

• Simply duplicate the userspace harness by literally forking within the forkserver. 

The initial method proved effective for us. The latter option may be more dependable, although it could potentially degrade performance.

The Linux exemplar has been fuzzed successfully through the efforts applied. However, even after extensive fuzzing sessions, the vulnerability remains elusive unless seed inputs closely resembling the solution are provided.

Enhancing fuzzing utilizing Artificial Intelligence

This limitation in the fuzzing process presents an opportunity for competitors to leverage the capabilities of AI. The intricate input format and slow execution speeds make pinpointing the exact reproducer a challenging task. AI has the potential to expedite the discovery of vulnerabilities through techniques like having an LLM generate seed inputs closely resembling the anticipated input format based on the source code of the harness. Competitors can draw inspiration from intriguing experiments conducted by Brendan Dolan-Gavitt from NYU, which demonstrate potential for this concept.

An Alternative Method: Static Analysis

Instead of relying on fuzzing, another approach to detecting vulnerabilities is through static analysis. Traditionally, static analysis struggles with generating a high number of false positives and faces challenges in proving the exploitability and reachability of identified issues. By incorporating LLMs, bug detection capabilities can be significantly enhanced by improving the precision and analysis functionalities of conventional static analysis methods.

Establishing Proof of Understanding (PoU)

Once a reproducer is identified through fuzzing, essential evidence for the PoU can be obtained:

1. Determining the responsible commit, which can be traced back through the git history bisection.

2. Identifying the expected sanitizer by running the reproducer to provoke the crash and analyzing the resulting stack trace.

Next Course of Action: Refinement through Delta Debugging

Following the identification of the perpetrator, a straightforward method to “rectify” the loophole is to simply undo this action. Nevertheless, the action may encompass legitimate alterations that are indispensable for the proper execution of tests. To guarantee the continuity of operation without any disruptions, we can employ delta debugging: we gradually experiment with incorporating/excluding different portions of the aforementioned action until the loophole ceases to be triggered, while affirming that all functional tests continue to succeed.

This is quite a blunt method for “rectifying.” There lacks a grasp of the patched code which is unlikely to be successful for more intricate patches involving nuanced adjustments required to resolve the vulnerability without impairing the operation.

Enhancing the rectification process with AI

These limitations underscore another avenue for rivals to leverage the capabilities of AI. One potential strategy could involve utilizing a Language Model to propose patches. A 2024 paper authored by Google outlines a method to construct an automated patching pipeline based on a Language Model.

Rivals will have to confront the subsequent challenges:

• Validating the patches through executing crashes and tests to confirm the prevention of a crash and unaffected functionality

• Constricting prompts to encompass solely the operations available in the crashing stack trace, to adhere to prompt constraints

• Establishing a validation phase to sift out unauthorized patches

Utilizing an LLM agent is potentially another encouraging strategy, where rivals could merge an LLM’s generative capabilities with the capacity to compile and receive debug test failures or stacktraces in an iterative manner.

Enhancing security for all

Collaboration is vital to leverage the potential of AI as a widespread tool for defenders. As new developments surface, we will integrate them into OSS-Fuzz, signifying that the results from AIxCC will directly improve security for the open source environment. We anticipate the innovative solutions that stem from this competition!