A two-stage framework with search space pruning for combined algorithm selection and hyperparameter optimization

With the increasing complexity of building and optimizing machine learning models, automated machine learning (AutoML) has attracted much attention over the last decade. Combined algorithm selection and hyperparameter optimization (CASH), which automatically selects an ML algorithm and tunes its hyperparameters in a unified manner, plays a crucial role in the AutoML process. However, due to the vast search space, identifying optimal algorithms and hyperparameters remains a significant challenge. We observe that the relative performance rankings of ML algorithms and hyperparameter configurations remain generally consistent when trained on the full dataset and on a reduced dataset obtained by subsampling and dimensionality reduction. Accordingly, we propose a Two-Stage framework for Combined Algorithm Selection and Hyperparameter optimization (TS-CASH) in this paper. The first stage aims to efficiently evaluate algorithm performance on a reduced dataset to identify promising algorithms while minimizing computational cost, and the second stage focuses on hyperparameter pruning followed by hyperparameter optimization on the pruned search space. This strategic pruning leads to a more focused search for optimal algorithms and hyperparameters, shrinking the search space by up to 76% in practice. With systematic experimental evaluations on 30 ML tasks, we demonstrate that TS-CASH outperforms state-of-the-art CASH methods on roughly 67%–73% of the tasks with only a slight increase in time overhead. The source code and experimental data are publicly available at https://github.com/ACMISLab/TS-CASH.