Bayesian Optimization with Particle Swarm

For most machine learning models, the mapping from the hyper-parameter set to the model's generalization error can be regarded as a complex black box function. Particle swarm optimization (PSO) methods cannot be directly used in the problem of hyper-parameters estimation since the mathematical formulation of the mapping from hyper-parameters to loss function or generalization accuracy is unclear. Functions with high evaluation costs can be solved by Bayesian optimization (BO) which converting the optimization of hyper-parameters into the optimization of an acquisition function. The proposed method in this paper uses the particle swarm method to optimize the acquisition function in the BO to get better hyper-parameters. The performances of proposed method in both of the classification and regression models are evaluated and demonstrated.