From 74041bd5cb3d8a0b9e0e2f8c45b421510c5558fa Mon Sep 17 00:00:00 2001 From: Cyzus Chi Date: Wed, 30 Oct 2024 15:56:06 +0800 Subject: [PATCH 1/6] update readme intro --- metagpt/ext/sela/README.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/metagpt/ext/sela/README.md b/metagpt/ext/sela/README.md index a942fdb7d..d0fbcf4b8 100644 --- a/metagpt/ext/sela/README.md +++ b/metagpt/ext/sela/README.md @@ -1,5 +1,7 @@ # SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning +SELA is an innovative framework that enhances Automated Machine Learning (AutoML) by integrating Monte Carlo Tree Search (MCTS) with LLM-based agents. Traditional AutoML methods often generate low-diversity and suboptimal code, limiting their effectiveness in model selection and ensembling. SELA addresses these challenges by representing pipeline configurations as trees, enabling agents to intelligently explore the solution space and iteratively refine their strategies based on experimental feedback. + ## 1. Data Preparation You can either download the datasets from the link or prepare the datasets from scratch. @@ -82,4 +84,18 @@ ### Ablation Study - **Use a set of insights:** ```bash python run_experiment.py --exp_mode rs --task titanic --rs_mode set - ``` \ No newline at end of file + ``` + +## 4. Citation + +```bibtex +@misc{chi2024selatreesearchenhancedllm, + title={SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning}, + author={Yizhou Chi and Yizhang Lin and Sirui Hong and Duyi Pan and Yaying Fei and Guanghao Mei and Bangbang Liu and Tianqi Pang and Jacky Kwok and Ceyao Zhang and Bang Liu and Chenglin Wu}, + year={2024}, + eprint={2410.17238}, + archivePrefix={arXiv}, + primaryClass={cs.AI}, + url={https://arxiv.org/abs/2410.17238}, +} +``` From 73d2358d108f6ebdcf25afadc4f86a43d075a51d Mon Sep 17 00:00:00 2001 From: Cyzus Chi Date: Wed, 30 Oct 2024 16:11:42 +0800 Subject: [PATCH 2/6] update intro and citation --- metagpt/ext/sela/README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/metagpt/ext/sela/README.md b/metagpt/ext/sela/README.md index d0fbcf4b8..ceb832a98 100644 --- a/metagpt/ext/sela/README.md +++ b/metagpt/ext/sela/README.md @@ -1,6 +1,10 @@ # SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning -SELA is an innovative framework that enhances Automated Machine Learning (AutoML) by integrating Monte Carlo Tree Search (MCTS) with LLM-based agents. Traditional AutoML methods often generate low-diversity and suboptimal code, limiting their effectiveness in model selection and ensembling. SELA addresses these challenges by representing pipeline configurations as trees, enabling agents to intelligently explore the solution space and iteratively refine their strategies based on experimental feedback. + +Official implementation for paper [SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning](https://arxiv.org/abs/2410.17238). + + +SELA is an innovative system that enhances Automated Machine Learning (AutoML) by integrating Monte Carlo Tree Search (MCTS) with LLM-based agents. Traditional AutoML methods often generate low-diversity and suboptimal code, limiting their effectiveness in model selection and ensembling. SELA addresses these challenges by representing pipeline configurations as trees, enabling agents to intelligently explore the solution space and iteratively refine their strategies based on experimental feedback. ## 1. Data Preparation @@ -87,6 +91,7 @@ ### Ablation Study ``` ## 4. Citation +Please cite our paper if you use SELA or find it useful! ```bibtex @misc{chi2024selatreesearchenhancedllm, From 0a215a960bd436da478f4ac93d68242390b63b6d Mon Sep 17 00:00:00 2001 From: Cyzus Chi Date: Wed, 30 Oct 2024 16:31:13 +0800 Subject: [PATCH 3/6] update readme --- metagpt/ext/sela/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metagpt/ext/sela/README.md b/metagpt/ext/sela/README.md index ceb832a98..2a4054c64 100644 --- a/metagpt/ext/sela/README.md +++ b/metagpt/ext/sela/README.md @@ -91,7 +91,7 @@ ### Ablation Study ``` ## 4. Citation -Please cite our paper if you use SELA or find it useful! +Please cite our paper if you use SELA or find it cool or useful! ```bibtex @misc{chi2024selatreesearchenhancedllm, From 261a39d5479e117e77e034055dd8bcf5b2d237f8 Mon Sep 17 00:00:00 2001 From: Cyzus Chi Date: Wed, 30 Oct 2024 23:01:38 +0800 Subject: [PATCH 4/6] remove autogluon models in gitignore --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index 46c9b0dd4..0d6be14ad 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,6 @@ share/python-wheels/ MANIFEST metagpt/tools/schemas/ examples/data/search_kb/*.json -metagpt/ext/sela/AutogluonModels # PyInstaller # Usually these files are written by a python scripts from a template From 776ee4455cc59e49b46d0cda1c965c4b9a789328 Mon Sep 17 00:00:00 2001 From: Cyzus Chi Date: Thu, 31 Oct 2024 00:05:44 +0800 Subject: [PATCH 5/6] remove mlebench --- metagpt/ext/sela/runner/README.md | 32 +------------------------------ 1 file changed, 1 insertion(+), 31 deletions(-) diff --git a/metagpt/ext/sela/runner/README.md b/metagpt/ext/sela/runner/README.md index 7c031f1ee..4867aa4f0 100644 --- a/metagpt/ext/sela/runner/README.md +++ b/metagpt/ext/sela/runner/README.md @@ -165,34 +165,4 @@ ### 5. Custom Baselines To run additional baselines: - Each baseline must produce `dev_predictions.csv` and `test_predictions.csv` with a `target` column. -- Use the `evaluate_score` function for evaluation. - ---- - -## MLE-Bench - -**Note:** MLE-Bench requires Python 3.11 or higher. - -#### Setup - -Clone the repository and install: - -```bash -git clone https://github.com/openai/mle-bench.git -cd mle-bench -pip install -e . -``` - -Prepare the data: - -```bash -mlebench prepare -c --data-dir -``` - -#### Run the MLE-Bench Experiment - -Run the following command to execute the experiment: - -```bash -python run_experiment.py --exp_mode mcts --custom_dataset_dir --rollouts 10 --from_scratch --role_timeout 3600 -``` \ No newline at end of file +- Use the `evaluate_score` function for evaluation. \ No newline at end of file From b710365fabc4a150b2a6cb90ca0ec44d01f1e38b Mon Sep 17 00:00:00 2001 From: Cyzus Chi Date: Thu, 31 Oct 2024 00:06:37 +0800 Subject: [PATCH 6/6] use google drive link --- metagpt/ext/sela/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metagpt/ext/sela/README.md b/metagpt/ext/sela/README.md index 2a4054c64..6fb47b42c 100644 --- a/metagpt/ext/sela/README.md +++ b/metagpt/ext/sela/README.md @@ -9,7 +9,7 @@ # SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning ## 1. Data Preparation You can either download the datasets from the link or prepare the datasets from scratch. -- **Download Datasets:** [Dataset Link](https://deepwisdom.feishu.cn/drive/folder/RVyofv9cvlvtxKdddt2cyn3BnTc?from=from_copylink) +- **Download Datasets:** [Dataset Link](https://drive.google.com/drive/folders/151FIZoLygkRfeJgSI9fNMiLsixh1mK0r?usp=sharing) - **Download and prepare datasets from scratch:** ```bash cd data