Master Assignment
Develop an LLM Benchmark for Dutch Healthcare
Type: Master
Period: TBD
Student: (Unassigned)
If you are interested, please contact :
Background
At Ziekenhuisgroep Twente (ZGT), data scientists design and deploy impactful apps that make use of open-source large language models such as Mistral 3.2 (24B) and Gemma4 (31B). Currently, model selection is largely based on qualitative judgment rather than structured evaluation. This project aims to address this gap by developing a quantitative benchmarking framework tailored to the hospital’s needs. In this project, you will collaborate closely with ZGT data scientists and engage with clinical departments to collect representative datasets. All models will be evaluated locally on the ZGT infrastructure, ensuring that all sensitive data remains within the hospital environment. Their hardware infrastructure consists of powerful GPUs running open-source LLMs through tools like, Ollama, vLLM, and OpenWebUI.
Goal
Design and implement a functional, model-agnostic Python benchmarking tool for ZGT data scientists to systematically evaluate and compare LLMs (such as Deepseek, Llama, Qwen, and Mistral). The benchmark must operate in Dutch and reflect real-world hospital use cases, such as structuring unstructured medical reports, summarizing clinical data, identifying patient inclusion/exclusion criteria for research, extracting complications from cardiology reports, and supporting internal chatbot applications.
The benchmark should be organized into multiple evaluation categories, each of which represents a distinct medical or operational task relevant to a Dutch hospital. You are free to define the scope of these categories. It is essential to design the tool as a modular blueprint that allows future extensions of the benchmark with minimal effort. The tool should produce clear, structured performance metrics per category. This allows the data scientists to easily compare newly released LLMs over time. The benchmark can include suitable existing benchmarks selected through a literature review.
Requirements:
- High proficiency in Dutch
- Affinity with Healthcare