MASTERÂ Assignment
CyberBench: A Taxonomy, Dataset and Framework for Automated Benchmarking of Direct Security Risks in LLM-based Systems
Type : Master M-CS
Period: August 2025 - January, 2026
Student: Hartmans, L.A. (Lucas, Student M-CS)
Date Final project: January 29, 2026
Supervisors:
Abstract:
Large Language Model (LLM)-based systems increasingly operate with elevated autonomy through tools that grant access to sensitive data and infrastructure, making them vulnerable direct security risks such as prompt injection, sensitive information disclosure, and excessive agency. Yet, existing benchmarks largely fail to assess such risks, as they focus on safety rather than security. Furthermore, existing benchmarking approaches often rely on unstandardized evaluation methodologies and manual processes, limiting their suitability for scalable and industrial security assessments. To address these problems, we introduce CyberBench, an ecosystem for automated benchmarking of direct security risks in LLM-based systems. CyberBench comprises a taxonomy of 36 OWASP-aligned risk categories, a dataset of 10,000 adversarial prompts operationalizing this taxonomy, and an automated framework that unifies prompt generation, execution, and standardized evaluation. Experimental results across an existing benchmark, real-world LLM systems and CTF-based challenges show that CyberBench enables effective and efficient assessment of direct security risks that go beyond the scope and capabilities of prior benchmarks.