PAM: process authentication mechanism for protecting system services against malicious code attacks

The kernel of the modern operating system fails to ensure the authenticity of a running process while servicing a system call. Verifying the origin and integrity of a system call is an important security issue in terms of ensuring the proper functioning of an end-system. The conventional process identification parameters such as process identifier, process names and the executable flow exercised by the operating system are not reliable. As a result, a stealthy malware may mimic other processes to carry out many computer crimes, thus compromising the end-system. In this paper, we present a novel idea in which system call invocations made by a malicious application are verified during runtime in Windows operating system. To ensure the authenticity of a process while servicing a system call, we propose a behavior-based mechanism, namely, the process authentication mechanism (PAM), for combating malicious code attacks that verifies the identity of each suspected process before being serviced by the kernel. The simulation and performance evaluation results confirm that our mechanism can effectively block all malicious samples that directly invoke system services in the kernel mode. PAM incurs no more than two percent overhead and helps to strengthen the overall system security.