RobOT: Robustness-Oriented Testing for Deep Learning Systems

Wang, J, Chen, J, Sun, Y, Ma, Xingjun, Wang, D, Sun, J and Cheng, P 2021, RobOT: Robustness-Oriented Testing for Deep Learning Systems, in ICSE 2021 : Proceedings of the IEEE/ACM International Conference on Software Engineering, IEEE, Piscataway, N.J., pp. 300-311, doi: 10.1109/ICSE43902.2021.00038.

Attached Files
Name Description MIMEType Size Downloads

Title RobOT: Robustness-Oriented Testing for Deep Learning Systems
Author(s) Wang, J
Chen, J
Sun, Y
Ma, XingjunORCID iD for Ma, Xingjun
Wang, D
Sun, J
Cheng, P
Conference name Software Engineering. Conference (2021 : 43rd : Madrid, Spain)
Conference location Madrid, Spain
Conference dates 2021/05/22 - 2021/05/30
Title of proceedings ICSE 2021 : Proceedings of the IEEE/ACM International Conference on Software Engineering
Publication date 2021
Series International Conference on Software Engineering
Start page 300
End page 311
Total pages 12
Publisher IEEE
Place of publication Piscataway, N.J.
Keyword(s) Computer Science
Computer Science, Software Engineering
Computer Science, Theory & Methods
Science & Technology
CORE2020 A*
Summary Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called Robustness-Oriented Testing (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metric to automatically generate test cases valuable for improving model robustness. The proposed metric is also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini.
ISBN 9781665402965
ISSN 0270-5257
Language eng
DOI 10.1109/ICSE43902.2021.00038
HERDC Research category E1 Full written paper - refereed
Persistent URL

Connect to link resolver
Unless expressly stated otherwise, the copyright for items in DRO is owned by the author, with all rights reserved.

Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 0 times in TR Web of Science
Scopus Citation Count Cited 0 times in Scopus
Google Scholar Search Google Scholar
Access Statistics: 8 Abstract Views, 1 File Downloads  -  Detailed Statistics
Created: Thu, 23 Sep 2021, 11:02:58 EST

Every reasonable effort has been made to ensure that permission has been obtained for items included in DRO. If you believe that your rights have been infringed by this repository, please contact