Detecting Anomalous Behavior in Cloud Servers by Nested-Arc Hidden SEMI-Markov Model with State Summarization

Anomaly detection for cloud servers is important for detecting zero-day attacks. However, it is very challenging due to the large amount of accumulated data. In this paper, a new mathematical model for modeling dynamic usage behavior and detecting anomalies is proposed. It is constructed using state summarization and a novel nested-arc hidden semi-Markov model (NAHSMM). State summarization is designed to extract usage behavior reflective states from a raw sequence. The NAHSMM is comprised of exterior and interior hidden Markov chains. The exterior controls the propagation of raw sequences of system calls and, conditional on it, the interior one controls the summarized observation process from the transition less usage behavior reflective states. An anomaly detection algorithm is derived by integrating state summarization and NAHSMM. During training the algorithm is assisted by a forensic module to tune the behavioral threshold. Experimental data is collected using IXIA Perfect Storm in conjunction with the commercial security-test hardware platform cyber range. To evaluate the reliability of the proposed model, first, its accuracy and training costs are compared with those of existing machine-learning models and then its scalability and resistance capabilities are tested. The results indicate that this model could be used as a method for detecting anomalies in cloud servers.