LOLBin detection through unsupervised learning

MASTER Assignment

LOLBIN DETECTION THROUGH UNSUPERVISED LEARNING : AN APPROACH BASED ON EXPLICIT FEATURIZATION OF THE COMMAND LINE AND PARENT-CHILD RELATIONSHIPS

Type : Master M-CS

Period: Mar, 2022 - Oct, 2022

Student : Nisslmüller, U.J. (Utz, Student M-CS)

Date Final project: Oct 4, 2022

Thesis

Supervisors:

Abstract:

Over the last couple of years, LOLBins have become a staple in the arsenal of APTs and other organized threat actors. Compared to the usual modus operandi of performing one or more steps in the intrusion chain via custom binaries, the use of these onboard Windows programs is much harder to detect due to the deliberate closeness in syntax to legitimate program instances, with significant deviations in semantics. In an effort to improve the defenders’ toolkit in dealing with such adversarial behavior, we present a LOLBin detection algorithm that leverages unsupervised learning to distinguish benign system process executions from malicious ones. We extract our features from parent-child process pairs, with a particular focus on the command line of both. Using the IsolationForest anomaly detection algorithm, we were able to achieve an F1-score of 0.92 on proprietary log data from ReaQta, a Dutch EDR vendor. We were able to reproduce these findings on various open-source data sets, with F1-scores ranging from 0.85 − 0.93. We also found that omitting the parent portion of parent-child process pair from the model reduces performance only slightly, reaching F1-scores of up to 0.88 using this reduced, child-only feature set.