DISTRIBUTED ON-DEMAND INTEGRITY MONITORING OF LEGACY APPLICATIONS AND RELIABLE ANALYSIS OF MIXED-MODE USER-KERNEL LEVEL ROOTKITS

Abstract

The increasing number of malicious programs has become a serious threat. The growth of malware samples has led computer security researchers to design and develop automatic malware detection and analysis tools. At the same time, malware writers attempt to develop more sophisticated malware that makes detection and analysis hard or impossible. In my dissertation I explore the problems of current malware detection and analysis techniques by providing the proof-of-concept implementation of malware samples that cannot be detected or fully analyzed by current techniques. My dissertation identifies three problems in the current solutions.

First, regarding the limitations of monitoring the integrity of legacy programs such as expensive cost of migrating to modern and more secure platforms, code injection rootkit attacks on legacy applications are hard to detect. Second, the complex malware codes manipulate or intercept the malware analysis components which reside on their execution domain (user-mode and kernel-mode).Third, a mixed-mode malware, which contains interdependent user-mode and kernel-mode components, misleads or foils single-domain analysis techniques.

To address the first problem, I propose TDOIM (Tiny Distributed On-Demand Integrity Monitor). TDOIM is a client-server scheme that periodically monitors applications to detect the malicious behavior injected by an attack. Specifically, it periodically compares the runtime state of all instances of the legacy application. If some instances start to diverge from the rest, this is an indication that the diverging instances may have been manipulated by malware. In other words, the server periodically infers and updates a white-list directly from the monitored application instances and checks all clients against this dynamic white-list. TDOIM installs a tiny client-side agent on legacy platforms with minimum attack surface and it does not require recompilation or restart of the monitored legacy application.

In order to address the problems of the current malware analysis techniques, I present the first mixed-mode automatic malware analysis platform called SEMU (Secure Emulator). SEMU is a binary analysis framework that 1) it operates outside the operating system and thereby outside the domain of user-mode and kernel-mode malware. 2) it deploys a novel mixed-mode monitoring of malware operations that is effective against sophisticated user-kernel level rootkit samples and kernel-mode exploits.

The increasing number of malicious programs has become a serious threat. The growth of malware samples has led computer security researchers to design and develop automatic malware detection and analysis tools. At the same time, malware writers attempt to develop more sophisticated malware that makes detection and analysis hard or impossible. In my dissertation I explore the problems of current malware detection and analysis techniques by providing the proof-of-concept implementation of malware samples that cannot be detected or fully analyzed by current techniques. My dissertation identifies three problems in the current solutions.

First, regarding the limitations of monitoring the integrity of legacy programs such as expensive cost of migrating to modern and more secure platforms, code injection rootkit attacks on legacy applications are hard to detect. Second, the complex malware codes manipulate or intercept the malware analysis components which reside on their execution domain (user-mode and kernel-mode).Third, a mixed-mode malware, which contains interdependent user-mode and kernel-mode components, misleads or foils single-domain analysis techniques.

To address the first problem, I propose TDOIM (Tiny Distributed On-Demand Integrity Monitor). TDOIM is a client-server scheme that periodically monitors applications to detect the malicious behavior injected by an attack. Specifically, it periodically compares the runtime state of all instances of the legacy application. If some instances start to diverge from the rest, this is an indication that the diverging instances may have been manipulated by malware. In other words, the server periodically infers and updates a white-list directly from the monitored application instances and checks all clients against this dynamic white-list. TDOIM installs a tiny client-side agent on legacy platforms with minimum attack surface and it does not require recompilation or restart of the monitored legacy application.

In order to address the problems of the current malware analysis techniques, I present the first mixed-mode automatic malware analysis platform called SEMU (Secure Emulator). SEMU is a binary analysis framework that 1) it operates outside the operating system and thereby outside the domain of user-mode and kernel-mode malware. 2) it deploys a novel mixed-mode monitoring of malware operations that is effective against sophisticated user-kernel level rootkit samples and kernel-mode exploits.