PRE-MASTER'S RESEARCH ASSIGNMENT: Compressing Vision-Language Foundation Models for On-Board UAV Perception
Problem Statement
State-of-the-art vision-language foundation models (VLMs) can describe complex aerial scenes in natural language, making them interesting as a general-purpose perception layer for autonomous UAVs that cannot rely on cloud connectivity. Recent work in the Pervasive Systems group has shown that a fine-tuned 3-billion-parameter VLM can be deployed on an embedded platform (NVIDIA Jetson Orin NX), but with a per-frame latency of roughly 8 seconds. This is well above the 1 to 2 second range targeted for real-time UAV operation. That study examined the main factors affecting deployment latency and accuracy and found that reducing model size is one of the most promising ways to improve performance. However, it did not investigate methods that actually reduce the size or computational cost of the model. This assignment focuses on that problem.
The aim is to investigate whether model compression, primarily knowledge distillation and structured or unstructured pruning, optionally combined with quantization, can reduce the computational and memory requirements of a VLM enough for deployment on a UAV-class edge device while preserving useful scene-understanding accuracy. The main question is how much accuracy can be retained as inference cost is reduced, and whether a compressed VLM can approach the latency required for practical on-board use.
The work will build on the group's existing fine-tuning and deployment pipeline based on the Qwen2.5-VL family, VisDrone aerial data, and llama.cpp on Jetson. The student therefore starts from an existing baseline and can focus on the compression experiments. Each approach will be evaluated using the same hardware, dataset, and evaluation procedure.
Research Questions RQ1: How much can distillation and pruning reduce per-frame latency and memory usage on a UAV-class edge device, and what is the corresponding loss in accuracy? RQ2: Which compression approach provides the best trade-off between accuracy and inference cost, and does this depend on scene density? RQ3: Can a compressed model reach a target latency of ≤ 2 s/frame while retaining sufficient accuracy for aerial scene understanding? |
Tasks
The student will investigate one or more of the following approaches and evaluate the resulting models on the Jetson against the existing baseline.
- Knowledge Distillation (response or feature-based)
Description: A larger teacher VLM supervises a smaller student model. The student learns from the teacher's outputs and, where appropriate, intermediate representations rather than relying only on hard labels.
Application: Train a compact student using the teacher's aerial scene descriptions and evaluate how closely it can match the teacher while reducing model size and inference time. - Structured Pruning
Description: Structural components such as attention heads, channels, MLP neurons, or transformer layers are removed and the remaining model is fine-tuned to recover some of the lost accuracy. Because the resulting model has a smaller structure, this can provide actual speed-ups on standard hardware.
Application: Prune the deployed VLM at different ratios and measure the resulting changes in latency, memory usage, and scene-understanding accuracy. - Unstructured / Magnitude Pruning and Sparsity
Description: Individual low-importance weights are removed to create a sparse model. The practical benefit depends on whether the inference runtime and hardware can efficiently exploit that sparsity.
Application: Determine how much sparsity the model can tolerate before accuracy degrades significantly, and test whether the resulting sparse models provide measurable speed-ups on the Jetson. - Layer / Depth Reduction
Description: The number of transformer blocks is reduced by removing layers or, where appropriate, allowing the model to exit before processing the full network. This reduces the amount of sequential computation.
Application: Evaluate different model depths and measure the resulting latency and accuracy to identify where reducing depth becomes too costly in terms of scene understanding. - Combined Compression
Description: Multiple techniques can be combined, for example by distilling a large model into a smaller student, pruning the student, and then applying calibrated low-bit quantization.
Application: Evaluate whether combining methods provides a better trade-off than applying them individually and identify configurations that approach the target latency and memory budget.
Common Evaluation Harness All approaches will be evaluated using the same procedure. Measurements will include per-frame latency and peak memory on the Jetson, scene-understanding and counting accuracy on a held-out VisDrone test set, with results also broken down by scene density. Model size will be recorded for all configurations and, where practical, energy consumption per inference will also be measured. The results will be presented as latency, accuracy, and memory trade-offs relative to the uncompressed fine-tuned baseline. |
Recommended Background
● Solid Python skills and practical experience with PyTorch.
● Familiarity with training and fine-tuning deep learning models.
● Basic understanding of transformers and foundation models.
● Interest in efficient machine learning and embedded AI.
● Willingness to work with NVIDIA Jetson hardware and command-line inference tools.
Work
15% Theory, 65% Experiments & Implementation, 20% Writing
Note: The student will inherit a working fine-tuning and Jetson deployment pipeline from previous group work and build the compression experiments on top of it.
Contact
Adarsh Nanjaiya Latha (a.nanjaiyalatha@utwente.nl)
Pervasive Systems Group, EEMCS