LMTree: Leveraging LLMs with Monte Carlo Tree Search for Automated Feature Engineering

Automated feature engineering (AutoFE) plays a pivotal role in improving the performance of machine learning (ML) models on tabular data. Traditional AutoFE methods are often limited by pre-specified search spaces and cannot effectively integrate domain knowledge into the feature generation process. Recently, large language models (LLMs) have shown great potential for AutoFE by enhancing feature generation through their semantic understanding capabilities. However, existing LLM-based AutoFE approaches still face two challenges: instability in feature synthesis and repetition across multiple rounds of generation. To address these issues, we propose LMTree, a new AutoFE method that leverages the global optimization capabilities of Monte Carlo tree search (MCTS) to encourage LLMs to generate more diverse and higher-quality features. Specifically, it uses MCTS to evaluate feature configurations with the upper confidence bound, prioritizing the generation of higher-value features. Then, a validation mechanism is designed to match historical features based on operator and attribute similarities, filtering duplicates and diversifying features. Extensive experimental results demonstrate that LMTree significantly outperforms state-of-the-art AutoFE methods, remarkably improving the accuracy of ML models across 20 tabular datasets.