RMIT University
Browse

Sub-curve HMM: A malware detection approach based on partial analysis of API call sequences

journal contribution
posted on 2024-11-01, 22:58 authored by Jakapan Suaboot, Zahir TariZahir Tari, Abdun Mahmood, Albert Zomaya, Wei Li
Malicious software (Malware) plays an important role in penetrating and extracting sensitive information. Based on dynamic program's behavior monitoring, existing solutions have shown that the Hidden Markov Model (HMM) is efficient in detecting malware using sequences of API calls. However, an obfuscation technique could insert minimal data stealing code into a large set of legitimate instructions, which makes the detector ineffective. Additionally, existing solutions require a whole picture of a program's actions, and hence a small chunk of activities is much harder to detect. Substantial performance degradation can occur during the detection when a long sequence of APIs is used. This paper proposes the Sub-Curve HMM feature extraction approach that focuses on matching subsets of activities from the running processes that potentially lead to data exfiltration incidents. Sequences of API calls are used to train HMMs and test the likelihood of matching to the model. Malicious and benign activities gain different matching scores over an adjoining set of API calls. By projecting a sequence of matching score into a curve, our approach discriminates malicious actions using discontinuities in the slope of the curve. The experimental results show that the proposed approach outperforms existing solutions in detecting six (6) families of malware: the detection accuracy of Sub-Curve HMM is over 94% compared to 83% for the baseline HMM approach and 73% for Information Gain.

Funding

Cloud-data centres resource allocation under bursty conditions

Australian Research Council

Find out more...

History

Journal

Computers and Security

Volume

92

Number

101773

Start page

1

End page

15

Total pages

15

Publisher

Elsevier Advanced Technology

Place published

United Kingdom

Language

English

Copyright

© 2020 Elsevier Ltd. All rights reserved

Former Identifier

2006097894

Esploro creation date

2020-09-08

Usage metrics

    Scholarly Works

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC