File(s) under permanent embargo
Stable bayesian optimization
conference contribution
posted on 2017-05-23, 00:00 authored by Thanh Dai Nguyen, Sunil GuptaSunil Gupta, Santu RanaSantu Rana, Svetha VenkateshSvetha VenkateshTuning hyperparameters of machine learning models is important for their performance. Bayesian optimization has recently emerged as a
de-facto method for this task. The hyperparameter tuning is usually performed by looking at model performance on a validation set. Bayesian optimization is used to find the hyperparameter set corresponding to the best model performance. However, in many cases, where training or validation set has limited set of datapoints, the function representing the model performance on the validation set contains several spurious sharp peaks. The Bayesian optimization, in such cases, has a tendency to converge to sharp peaks instead of other more stable peaks. When a model trained using these hyperparameters is deployed in real world, its performance suffers dramatically. We address this problem through a novel stable Bayesian optimization framework. We construct a new acquisition function that helps Bayesian optimization to avoid the convergence to the sharp peaks. We conduct a theoretical analysis and guarantee that Bayesian optimization using the proposed acquisition
function prefers stable peaks over unstable ones. Experiments with
synthetic function optimization and hyperparameter tuning for Support
Vector Machines show the effectiveness of our proposed framework.
de-facto method for this task. The hyperparameter tuning is usually performed by looking at model performance on a validation set. Bayesian optimization is used to find the hyperparameter set corresponding to the best model performance. However, in many cases, where training or validation set has limited set of datapoints, the function representing the model performance on the validation set contains several spurious sharp peaks. The Bayesian optimization, in such cases, has a tendency to converge to sharp peaks instead of other more stable peaks. When a model trained using these hyperparameters is deployed in real world, its performance suffers dramatically. We address this problem through a novel stable Bayesian optimization framework. We construct a new acquisition function that helps Bayesian optimization to avoid the convergence to the sharp peaks. We conduct a theoretical analysis and guarantee that Bayesian optimization using the proposed acquisition
function prefers stable peaks over unstable ones. Experiments with
synthetic function optimization and hyperparameter tuning for Support
Vector Machines show the effectiveness of our proposed framework.
History
Event
Knowledge Discovery and Data Mining. Pacific-Asia Conference (21st : 2017 : Jeju, South Korea)Volume
54Issue
Part IISeries
Lecture notes in artificial intelligencePagination
578 - 591Publisher
Springer International PublishingLocation
Jeju, South KoreaPlace of publication
Cham, SwitzerlandPublisher DOI
Start date
2017-05-23End date
2017-05-26ISBN-13
9783319575292Language
engPublication classification
E Conference publication; E1 Full written paper - refereedCopyright notice
2017, Springer International PublishingEditor/Contributor(s)
J Kim, K Shim, L Cao, J Lee, X Lin, Y MoonTitle of proceedings
PAKDD 2017 : Advances in Knowledge Discovery and Data Mining : Proceedings of the 21st Pacific-Asia ConferenceUsage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC