Intrusion Detection: Detecting Network Intruders Behind Anonymity Networks and Identifying Intruders on the Hosts by Modeling User Behaviors

Our society is facing a growing threat from data breaches, where confidential information is stolen from computer servers. To steal data, hackers must first gain entry into the targeted systems. However, commercial off-the-shelf cyber defense systems typically relax the restrictions for frequent services. This dissertation proposes a network-based and a host-based intrusion detection method in two separate scenarios. Most existing defense systems at the first line of cyber defense, such as firewalls, often allow useful services to pass through. However, recent security breaches reveal that malicious users hide their identities by taking advantage of the circuit-based anonymity networks, such as Tor and SOCKS proxy services, and launch attacks through SSH and HTTPS protocols. Related research only provides solutions in detecting stepping-stone attacks and cannot defend against such threats. The first part of this dissertation investigates strategies to detect SSH and HTTPS connections via the circuit-based anonymity networks, to help servers and websites defend against intruders. This research also addresses the issue of detecting intruders on host machines based on user behaviors. We hypothesize that intruders behave differently from normal users due to their malicious intent, which can be used to detect intruders on the hosts. To validate our hypothesis, we propose a graph model to model user behaviors and derive a set of features. We evaluate our detecting approach with a file access log dataset by adopting various classification and anomaly detection algorithms. We made two main contributions to the research area of intrusion detection. First, we proposed a network-based intrusion detection method to detect malicious users hiding behind Tor and SOCKS proxy service. This novel approach is based on the packet latency disparities derived from the protocol handshake process, which can effectively protect the data-sensitive server from exchanging data with anonymous intruders. Second, this dissertation formally proposed a graph model to model user behaviors on the host. Using the features derived from the graph model, we validated our host-based intrusion detection method by adopting an anomaly detection algorithm. Our method demonstrated high anomaly detection performance in the evaluation with a file access log dataset.