Deakin University
Browse

File(s) under permanent embargo

Stable bayesian optimization

Version 2 2024-06-05, 11:50
Version 1 2017-05-01, 13:30
conference contribution
posted on 2024-06-05, 11:50 authored by TD Nguyen, Sunil GuptaSunil Gupta, Santu RanaSantu Rana, Svetha VenkateshSvetha Venkatesh
Tuning hyperparameters of machine learning models is important for their performance. Bayesian optimization has recently emerged as a de-facto method for this task. The hyperparameter tuning is usually performed by looking at model performance on a validation set. Bayesian optimization is used to find the hyperparameter set corresponding to the best model performance. However, in many cases, where training or validation set has limited set of datapoints, the function representing the model performance on the validation set contains several spurious sharp peaks. The Bayesian optimization, in such cases, has a tendency to converge to sharp peaks instead of other more stable peaks. When a model trained using these hyperparameters is deployed in real world, its performance suffers dramatically. We address this problem through a novel stable Bayesian optimization framework. We construct a new acquisition function that helps Bayesian optimization to avoid the convergence to the sharp peaks. We conduct a theoretical analysis and guarantee that Bayesian optimization using the proposed acquisition function prefers stable peaks over unstable ones. Experiments with synthetic function optimization and hyperparameter tuning for Support Vector Machines show the effectiveness of our proposed framework.

History

Volume

54

Pagination

578-591

Location

Jeju, South Korea

Start date

2017-05-23

End date

2017-05-26

ISBN-13

9783319575292

Language

eng

Publication classification

E Conference publication, E1 Full written paper - refereed

Copyright notice

2017, Springer International Publishing

Editor/Contributor(s)

Kim J, Shim K, Cao L, Lee JG, Lin X, Moon YS

Title of proceedings

PAKDD 2017 : Advances in Knowledge Discovery and Data Mining : Proceedings of the 21st Pacific-Asia Conference

Event

Knowledge Discovery and Data Mining. Pacific-Asia Conference (21st : 2017 : Jeju, South Korea)

Issue

Part II

Publisher

Springer International Publishing

Place of publication

Cham, Switzerland

Series

Lecture notes in artificial intelligence