Deakin University
Browse

File(s) under permanent embargo

Control flow-based malware variant detection

Version 2 2024-06-03, 11:46
Version 1 2015-04-02, 16:33
journal contribution
posted on 2024-06-03, 11:46 authored by S Cesare, Y Xiang, W Zhou
Static detection of malware variants plays an important role in system security and control flow has been shown as an effective characteristic that represents polymorphic malware. In our research, we propose a similarity search of malware to detect these variants using novel distance metrics. We describe a malware signature by the set of control flowgraphs the malware contains. We use a distance metric based on the distance between feature vectors of string-based signatures. The feature vector is a decomposition of the set of graphs into either fixed size k-subgraphs, or q-gram strings of the high-level source after decompilation. We use this distance metric to perform pre-filtering. We also propose a more effective but less computationally efficient distance metric based on the minimum matching distance. The minimum matching distance uses the string edit distances between programs' decompiled flowgraphs, and the linear sum assignment problem to construct a minimum sum weight matching between two sets of graphs. We implement the distance metrics in a complete malware variant detection system. The evaluation shows that our approach is highly effective in terms of a limited false positive rate and our system detects more malware variants when compared to the detection rates of other algorithms. © 2013 IEEE.

History

Journal

IEEE transactions on dependable and secure computing

Volume

11

Pagination

304-317

Location

Piscataway, NJ

ISSN

1545-5971

Language

eng

Publication classification

C Journal article, C1 Refereed article in a scholarly journal

Copyright notice

2014, IEEE

Issue

4

Publisher

IEEE