apunkt/MetaGPT
1
0
Fork 0
mirror of https://github.com/FoundationAgents/MetaGPT.git synced 2026-06-11 15:15:18 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
MetaGPT/expo
History
Yizhou Chi c0262bcd8f 1. add support to hf dataset 
2. add support to datasets that have both train and test
3. create data folder
4. fix new instruction bug
2024-09-06 19:05:10 +08:00
..
data 1. add support to hf dataset 2024-09-06 19:05:10 +08:00
evaluation format code 2024-09-04 17:52:02 +08:00
experimenter 1. add support to hf dataset 2024-09-06 19:05:10 +08:00
insights 1. add support to hf dataset 2024-09-06 19:05:10 +08:00
results add expo 2024-08-30 16:26:05 +08:00
data.yaml update readme 2024-09-03 13:40:23 +08:00
datasets.yaml update readme 2024-09-03 13:40:23 +08:00
Greedy.py 1. avoid circular reference 2024-09-06 13:09:18 +08:00
MCTS.py 1. add support to hf dataset 2024-09-06 19:05:10 +08:00
README.md 更新Prompt相关readme 2024-09-05 14:30:36 +08:00
requirements.txt 1. 暂时在expo文件夹里单独放一个requirements.txt 2024-09-02 20:23:45 +08:00
research_assistant.py format code 2024-09-04 17:52:02 +08:00
run_experiment.py fix import 2024-09-04 18:08:59 +08:00
utils.py format code 2024-09-04 17:52:02 +08:00

README.md

Expo

1. Data Preparation

2. Configs

Data Config

datasets.yaml 提供数据集对应的指标和基础提示词

data.yaml 继承了datasets.yaml以及一些路径信息，需要将datasets_dir指到数据集合集的根目录下

LLM Config

llm:
  api_type: 'openai'
  model: deepseek-coder
  base_url: "https://oneapi.deepwisdom.ai/v1"
  api_key: sk-xxx
  temperature: 0.5

Budget

实验轮次 k = 10, 20

Prompt Usage

  • 通过执行dataset.py中的generate_task_requirement函数获取提示词
    • 非DI-based方法设置is_di=False
    • data_config用utils.DATA_CONFIG
  • 每一个数据集里有dataset_info.json，里面的内容需要提供给baselines以保证公平（generate_task_requirement已经默认提供）

3. Evaluation

运行各个框架，运行后框架需要提供Dev和Test的dev_predictions.csv和test_predictions.csv，每个csv文件只需要单个名为target的列

  • 使用CustomExperimenter
experimenter = CustomExperimenter(task="titanic")
score_dict = experimenter.evaluate_pred_files(dev_pred_path, test_pred_path)

4. Baselines

DS Agent

提供github链接，并说明使用的命令以及参数设置

AIDE

提供github链接，并说明使用的命令以及参数设置

Autogluon

Setup

pip install -U pip
pip install -U setuptools wheel
pip install autogluon

提供github链接，并说明使用的命令以及参数设置

Base DI

For setup, check 5.

  • python run_experiment.py --exp_mode base --task titanic --num_experiments 10

DI RandomSearch

For setup, check 5.

  • Single insight python run_experiment.py --exp_mode aug --task titanic --aug_mode single

  • Set insight python run_experiment.py --exp_mode aug --task titanic --aug_mode set

5. DI MCTS

Run DI MCTS

Setup

In the root directory,

pip install -e .

cd expo

pip install -r requirements.txt

Run

  • python run_experiment.py --exp_mode mcts --task titanic --rollout 10

If the dataset has reg metric, remember to use --low_is_better:

  • python run_experiment.py --exp_mode mcts --task househouse_prices --rollout 10 --low_is_better