Memory Access Validation Scheme Against Payload Injection Attacks

As more individuals and organizations become more dependent on computers and the Internet to create, manage, and share their resources efficiently, computer security issues are of increasing significance in every corner of our life. Due to insecure programming environments and structural limitations at the hardware level, many vulnerabilities are still being discovered. This thesis explores various issues concerning vulnerabilities and protection measures, including attack vectors and how protection measures are designed to address security threats. This thesis addresses threats from payload injection attacks at the architectural level by leveraging existing hardware techniques. The first work utilizes the address translation for virtual memory system. With a TLB (Translation Lookaside Buffer) that is usually split between data (DTLB) and instructions (ITLB) as found in virtual memory system of modern processors, a simple protection can be developed based on an observation that activating an injected code causes a DTLB hit under ITLB miss with dirty bit set in the hit TLB entry. To evaluate our idea, we have revised the address translation function in Bochs x86 simulator and conducted code injection attacks available over the Internet. The experimental results with two simulators show that the proposed protection can detect all the code injection attacks tested. The second work pursues more fine-grained protection against sophisticated attacks like return-oriented-programming attacks by leveraging existing hardware techniques. Two widely adopted hardware techniques -- the cache structure and the branch predictor -- increase performance in modern microprocessors by exploiting expected or predicted circumstances and events. As malicious payloads are prone to induce unprecedented or unexpected circumstances at control flow redirection, validating those circumstances or events at the associated handlers could be utilized in countering payload injection attacks. In order to utilize these components for protection, this thesis clarifies practical issues in distinguishing legitimate miss events from those caused by malicious attacks and integrating supporting mechanisms into multi-tasking environments. Based on the observation and discussion, we propose a memory-access validation scheme against payload injection attacks. This scheme consists of two parts -- the validation unit and taint-status data. The validation unit handles queries from other processor components, namely the cache structure and the branch predictor, and validates suspicious control flow redirections by referring to the active taint-status data set. Experimental results with two simulators show that the proposed validation scheme is able to detect simulated payload injection attacks under negligible to moderate performance degradation.