Intrusion Detection: Detecting Network Intruders Behind Anonymity Networks and Identifying Intruders on the Hosts by Modeling User Behaviors



Journal Title

Journal ISSN

Volume Title



Our society is facing a growing threat from data breaches, where confidential information is stolen from computer servers. To steal data, hackers must first gain entry into the targeted systems. However, commercial off-the-shelf cyber defense systems typically relax the restrictions for frequent services. This dissertation proposes a network-based and a host-based intrusion detection method in two separate scenarios. Most existing defense systems at the first line of cyber defense, such as firewalls, often allow useful services to pass through. However, recent security breaches reveal that malicious users hide their identities by taking advantage of the circuit-based anonymity networks, such as Tor and SOCKS proxy services, and launch attacks through SSH and HTTPS protocols. Related research only provides solutions in detecting stepping-stone attacks and cannot defend against such threats. The first part of this dissertation investigates strategies to detect SSH and HTTPS connections via the circuit-based anonymity networks, to help servers and websites defend against intruders. This research also addresses the issue of detecting intruders on host machines based on user behaviors. We hypothesize that intruders behave differently from normal users due to their malicious intent, which can be used to detect intruders on the hosts. To validate our hypothesis, we propose a graph model to model user behaviors and derive a set of features. We evaluate our detecting approach with a file access log dataset by adopting various classification and anomaly detection algorithms. We made two main contributions to the research area of intrusion detection. First, we proposed a network-based intrusion detection method to detect malicious users hiding behind Tor and SOCKS proxy service. This novel approach is based on the packet latency disparities derived from the protocol handshake process, which can effectively protect the data-sensitive server from exchanging data with anonymous intruders. Second, this dissertation formally proposed a graph model to model user behaviors on the host. Using the features derived from the graph model, we validated our host-based intrusion detection method by adopting an anomaly detection algorithm. Our method demonstrated high anomaly detection performance in the evaluation with a file access log dataset.



Intrusion detection, machine learning


Portions of this document appear in: Cao, Zechun, and Shou-Hsuan Stephen Huang. "Detecting intruders and preventing hackers from evasion by tor circuit selection." In 2018 17th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/12th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE), pp. 475-480. IEEE, 2018. And in: Huang, Shou-Hsuan Stephen, and Zechun Cao. "Detecting Malicious Users Behind Circuit-Based Anonymity Networks." IEEE Access 8 (2020): 208610-208622. And in: Huang, Shou-Hsuan S., Zechun Cao, Calvin E. Raines, Mai N. Yang, and Camille Simon. "Detecting Intruders by User File Access Patterns." In International Conference on Network and System Security, pp. 320-335. Springer, Cham, 2019.