Deakin University
Browse

File(s) under permanent embargo

Stable bayesian optimization

conference contribution
posted on 2017-05-23, 00:00 authored by Thanh Dai Nguyen, Sunil GuptaSunil Gupta, Santu RanaSantu Rana, Svetha VenkateshSvetha Venkatesh
Tuning hyperparameters of machine learning models is important for their performance. Bayesian optimization has recently emerged as a
de-facto method for this task. The hyperparameter tuning is usually performed by looking at model performance on a validation set. Bayesian optimization is used to find the hyperparameter set corresponding to the best model performance. However, in many cases, where training or validation set has limited set of datapoints, the function representing the model performance on the validation set contains several spurious sharp peaks. The Bayesian optimization, in such cases, has a tendency to converge to sharp peaks instead of other more stable peaks. When a model trained using these hyperparameters is deployed in real world, its performance suffers dramatically. We address this problem through a novel stable Bayesian optimization framework. We construct a new acquisition function that helps Bayesian optimization to avoid the convergence to the sharp peaks. We conduct a theoretical analysis and guarantee that Bayesian optimization using the proposed acquisition
function prefers stable peaks over unstable ones. Experiments with
synthetic function optimization and hyperparameter tuning for Support
Vector Machines show the effectiveness of our proposed framework.

History

Event

Knowledge Discovery and Data Mining. Pacific-Asia Conference (21st : 2017 : Jeju, South Korea)

Volume

54

Issue

Part II

Series

Lecture notes in artificial intelligence

Pagination

578 - 591

Publisher

Springer International Publishing

Location

Jeju, South Korea

Place of publication

Cham, Switzerland

Start date

2017-05-23

End date

2017-05-26

ISBN-13

9783319575292

Language

eng

Publication classification

E Conference publication; E1 Full written paper - refereed

Copyright notice

2017, Springer International Publishing

Editor/Contributor(s)

J Kim, K Shim, L Cao, J Lee, X Lin, Y Moon

Title of proceedings

PAKDD 2017 : Advances in Knowledge Discovery and Data Mining : Proceedings of the 21st Pacific-Asia Conference