Solving Capture The Flags (CTFs) with multi-agent systems

Solving Capture The Flags (CTFs) with multi-agent systems

Problem Statement:

Large Language Models (LLMs) have shown to be useful in a vast array of tasks, this is also true for the domain of cybersecurity. In the realm of penetration testing LLMs have shown to be able to solve Capture the Flags (CTFs), highlighting offensive cybersecurity capabilities within these models [1].  However, a framework for automated CTF breaking has not yet been developed. 

When using LLMs for problem solving multi-agent systems are a promising method for improving reliability and performance. MetaGPT is a multi-agent system that divides programming tasks, into sub-tasks solved by agents [2]. Such teams of LLM agents have already shown capabilities at exploiting vulnerabilities [3], however this remains to be explored for CTFs.

Task:

This project explores the feasibility of using multi-agent systems for solving CTFs. The goal is to see whether multi-agent approaches can increase the complexity of CTFs LLMs are able to solve. Advancing automated methods to be more competitive with current expert-level performance.

 Potential Research Questions (RQs):

- Can multi-agent systems improve the performance and reliability of LLMs solving CTFs?

- What sub-challenges are easy or difficult for the agents to solve?

- How does agents specialization affect the performance of the system?

 Related Literature:

 [1] (Shao et al. 2024. “An Empirical Evaluation of LLMs for Solving Offensive Security Challenges.”)

[2] (Hong et al. 2024. “MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.”)

[3] (Fang et al. 2024. “Teams of LLM Agents Can Exploit Zero-Day Vulnerabilities.”)