MetaGPT/examples/di

Data Interpreter (DI)

What is Data Interpreter

Data Interpreter is an agent who solves data-related problems through codes. It understands user requirements, makes plans, writes codes for execution, and uses tools if necessary. These capabilities enable it to tackle a wide range of scenarios, please check out the examples below. For overall design and technical details, please see our paper.

Example List

• Data visualization
• Machine learning modeling
• Image background removal
• Solve math problems
• Receipt OCR
• Tool usage: web page imitation
• Tool usage: web crawling
• Tool usage: text2image
• Tool usage: email summarization and response
• More on the way!

Please see the docs for more explanation.

Paper Experiments

Before running the experiments, download the di_dataset and place it in the specified path (default DATA_PATH or any other path you prefer). Note that the DATA_PATH = METAGPT_ROOT / "data".

To reproduce the results in the paper, run the following commands:

python run_ml_benchmark.py --task_name 05_titanic
python run_open_ended_tasks.py --task_name 14_image_background_removal --data_dir directory_to_di_dataset --use_reflection True

The very simple run_ml_benchmark.py and run_open_ended_tasks.py scripts implement the pipeline of the Data Interpreter.

Some key arguments:

• --task_name: required, the name of the task that combines the ID and the dataset name or the scenario name, e.g., 05_titanic or 14_image_background_removal.
• --data_dir: optional, the directory that stores the di_dataset (default is DATA_PATH).
• --use_reflection: optional, the flag to use reflection or not (default is True).

Data Interpreter Benchmark

Data Interpreter Dataset Structure

di_dataset

• ml_benchmark
  • 05_titanic
  • 06_house-prices-advanced-regression-techniques
  • 07_santander-customer-transaction-prediction
  • 08_icr-identify-age-related-conditions
  • 09_santander-value-prediction-challenge
• open_ended_tasks
  • 01_invoice_ocr
  • 02_invoice_ocr
  • 03_invoice_ocr
  • 14_image_background_removal
  • 16_image_2_code_generation
  • 17_image_2_code_generation

ML-Benchmark Dataset and Requirements

ML-Benchmark contains 9 typical machine learning datasets.

ID	Dataset Name	User Requirement
01	Iris	Run data analysis on sklearn Iris dataset, include a plot
02	Diabetes	Run data analysis on sklearn Diabetes dataset, include a plot
03	Wine recognition	Run data analysis on sklearn Wine recognition dataset, include a plot, and train a model to predict wine class with 20% as test set, and show prediction accuracy
04	Breast Cancer Wisconsin (Diagnostic)	Run data analysis on sklearn Wisconsin Breast Cancer dataset, include a plot, train a model to predict targets (20% as validation), and show validation accuracy
05	Titanic	This is a titanic passenger survival dataset, your goal is to predict passenger survival outcome. The target column is Survived. Perform data analysis, data preprocessing, feature engineering, and modeling to predict the target. Report accuracy on the eval data. Train data path: '{DATA_PATH}/ml_benchmark/05_titanic/split_train.csv', eval data path: '{DATA_PATH}/ml_benchmark/05_titanic/split_eval.csv'.
06	House Prices - Advanced Regression Techniques	This is a house price dataset, your goal is to predict the sale price of a property based on its features. The target column is SalePrice. Perform data analysis, data preprocessing, feature engineering, and modeling to predict the target. Report RMSE between the logarithm of the predicted value and the logarithm of the observed sales price on the eval data. Train data path: '{DATA_PATH}/ml_benchmark/06_house-prices-advanced-regression-techniques/split_train.csv', eval data path: '{DATA_PATH}/ml_benchmark/06_house-prices-advanced-regression-techniques/split_eval.csv'.
07	Santander Customer Transaction Prediction	This is a customers financial dataset. Your goal is to predict which customers will make a specific transaction in the future. The target column is target. Perform data analysis, data preprocessing, feature engineering, and modeling to predict the target. Report AUC on the eval data. Train data path: '{DATA_PATH}/ml_benchmark/07_santander-customer-transaction-prediction/split_train.csv', eval data path: '{DATA_PATH}/ml_benchmark/07_santander-customer-transaction-prediction/split_eval.csv' .
08	ICR - Identifying Age-Related Conditions	This is a medical dataset with over fifty anonymized health characteristics linked to three age-related conditions. Your goal is to predict whether a subject has or has not been diagnosed with one of these conditions. The target column is Class. Perform data analysis, data preprocessing, feature engineering, and modeling to predict the target. Report F1 Score on the eval data. Train data path: '{DATA_PATH}/ml_benchmark/08_icr-identify-age-related-conditions/split_train.csv', eval data path: '{DATA_PATH}/ml_benchmark/08_icr-identify-age-related-conditions/split_eval.csv' .
09	Santander Value Prediction Challenge	This is a customers financial dataset. Your goal is to predict the value of transactions for each potential customer. The target column is target. Perform data analysis, data preprocessing, feature engineering, and modeling to predict the target. Report RMSLE on the eval data. Train data path: '{DATA_PATH}/ml_benchmark/09_santander-value-prediction-challenge/split_train.csv', eval data path: '{DATA_PATH}/ml_benchmark/09_santander-value-prediction-challenge/split_eval.csv' .

Open-Ended Tasks Dataset and Requirements

Open-Ended Tasks have collected and designed 20 moderately challenging open-ended tasks, requiring Data Interpreters to understand user requirements, plan and decompose tasks, and generate and execute code.

ID	Scenario	Scenario Description	User Requirement
1	Invoice OCR	Scan all the necessary fields and amounts from the given file and then create an Excel sheet with the extracted data.	This is an English invoice image. Your goal is to perform OCR on the image, extract the total amount from ocr result and save as table, using PaddleOCR. The PaddleOCR environment has been fully installed, try to use Paddleocr as much as possible. Image path: '{DATA_PATH}/open_ended_tasks/01_invoice_ocr.png
2	Invoice OCR	Scan all the necessary fields and amounts from the given file and then create an Excel sheet with the extracted data.	This is a Chinese invoice image. Your goal is to perform OCR on the image and only output the recognized text word results, nothing else is needed, then extract the total amount and receipt ID starting with 'No' from ocr text words results and save as table, using PaddleOCR. The PaddleOCR environment has been fully installed, try to use Paddleocr as much as possible. Image path: '{DATA_PATH}/open_ended_tasks/02_invoice_ocr.jpg'
3	Invoice OCR	Scan all the necessary fields and amounts from the given file and then create an Excel sheet with the extracted data.	This is an invoice image for OCR. Your goal is to perform OCR on the image, extract the total amount and save it into an Excel table format, using PaddleOCR with lang='en' The PaddleOCR environment has been fully installed, try to use Paddleocr as much as possible. Image path: '{DATA_PATH}/open_ended_tasks/03_invoice_ocr.jpg'
4	Web search and crawling	Crawling and organizing web form information	Get data from paperlist table in https://papercopic.com/statistics/iclr-statistics/iclr-2024-statistics/, and save it to a csv file. paper title must include multiagent or large language model. notice: print key variables
5	Web search and crawling	Crawling and organizing web form information	获取https://www.stats.gov.cn/sj/sjjd/202307/t20230718_1941322.html的cpi数据, 请按照这个计划一步一步执行: 1. 检测目标网页的编码类型和html结构. 2.爬取网页, 将网页正文内容去重，并转换为段落清晰适合阅读的纯文本, 并保存到target.txt. 3.设计多个正则匹配表达式来匹配target.txt中关键语句, 使用try-except语句组合各个正则匹配, 注意网页文本是中文. 4.最后使用中文总结概括关键语句回答用户的请求. 注意: 如果是代码块, 请将代码块的关键变量结果打印出来; 如果是网页文本就打印前200个字符.
6	Web search and crawling	Crawling and organizing web form information	爬取电子商务网站https://scrapeme.live/shop/中的商品数据并保存为csv文件。注意: 第一步要先解析网页编码和html结构; csv中保存商品名称、价格、url、图片网址;
7	Web search and crawling	Crawling and organizing web form information	从36kr创投平台https://pitchhub.36kr.com/financing-flash所有初创企业融资的信息, 注意: 这是⼀个中⽂⽹站; 下⾯是⼀个⼤致流程, 你会根据每⼀步的运⾏结果对当前计划中的任务做出适当调整: 1. 爬取并本地保存html结构; 2. 直接打印第7个快讯关键词后2000个字符的html内容, 作为快讯的html内容示例; 3. 反思快讯的html内容示例中的规律, 设计正则匹配表达式来获取快讯的标题、链接、时间; 4. 筛选最近3天的初创企业融资快讯, 以list[dict]形式打印前5个。5. 将全部结果存在本地csv中
8	Email reply	Filter through my emails and respond to them as necessary	You are an agent that automatically reads and replies to emails. I will give you your Outlook email account and password. You need to check the content of the latest email and return it to me. If the email address suffix of this email is @communication.microsoft.com, please automatically reply with "I've received your email and will reply as soon as possible. Thank you!" Email account: englishgpt@outlook.com Email Password: xxxx
9	Web page imitation	Using Selenium and WebDriver to access a webpage and convert it to an image, with the assistance of GPT-4V to mimic the creation of a one-page website.	This is a URL of webpage: https://medium.com/. Firstly, utilize Selenium and WebDriver for rendering. Secondly, convert image to a webpage including HTML, CSS and JS in one go. Finally, save webpage in a text file. All required dependencies and environments have been fully installed and configured.
10	Web page imitation	Using Selenium and WebDriver to access a webpage and convert it to an image, with the assistance of GPT-4V to mimic the creation of a one-page website.	This is a URL of webpage: https://pytorch.org/. Firstly, utilize Selenium and WebDriver for rendering. Secondly, convert image to a webpage including HTML, CSS and JS in one go. Finally, save webpage in a file. NOTE: All required dependencies and environments have been fully installed and configured.
11	Web page imitation	Using Selenium and WebDriver to access a webpage and convert it to an image, with the assistance of GPT-4V to mimic the creation of a one-page website.	This is a URL of webpage: https://www.kaggle.com/. Firstly, utilize Selenium and WebDriver to render the webpage, ensuring the browser window is maximized for an optimal viewing experience. Secondly, convert image to a webpage including HTML, CSS and JS in one go. Finally, save webpage in a file. NOTE: All required dependencies and environments have been fully installed and configured.
12	Web page imitation	Using Selenium and WebDriver to access a webpage and convert it to an image, with the assistance of GPT-4V to mimic the creation of a one-page website.	This is a URL of webpage: https://chat.openai.com/auth/login.. Firstly, utilize Selenium and WebDriver to render the webpage, ensuring the browser window is maximized for an optimal viewing experience. Secondly, convert image to a webpage including HTML, CSS and JS in one go. Finally, save webpage in a file. NOTE: All required dependencies and environments have been fully installed and configured.
13	Web page imitation	Using Selenium and WebDriver to access a webpage and convert it to an image, with the assistance of GPT-4V to mimic the creation of a one-page website.	This is a URL of webpage: https://deepmind.google/technologies/gemini/#introduction. Firstly, utilize Selenium and WebDriver to render the webpage, ensuring the browser window is maximized for an optimal viewing experience. Secondly, convert image to a webpage including HTML, CSS and JS in one go. Finally, save webpage in a file. NOTE: All required dependencies and environments have been fully installed and configured.
14	Image Background Removal	Remove the background of a given image	This is an image, you need to use python toolkit rembg remove the background of the image. image path:'{DATA_PATH}/open_ended_tasks/14_image_background_removal.jpg'; save path:'{DATA_PATH}/open_ended_tasks/14_image_background_removal.jpg'
15	Text2Img	Use SD tools to generate images	I want to generate an image of a beautiful girl using the stable diffusion text2image tool, sd_url = "http://your.sd.service.ip:port"
16	Image2Code Generation	Web code generation	This is a image. First, check if the path exists, then convert the image to webpage code including HTML, CSS and JS in one go, and finally save webpage code in a file.The image path: '{DATA_PATH}/open_ended_tasks/16_image_2_code_generation.png'. NOTE: All required dependencies and environments have been fully installed and configured.
17	Image2Code Generation	Web code generation	This is a image. First, check if the path exists, then convert the image to webpage code including HTML, CSS and JS in one go, and finally save webpage code in a file.The image path: '{DATA_PATH}/open_ended_tasks/16_image_2_code_generation.png'. NOTE: All required dependencies and environments have been fully installed and configured.
18	Generate games using existing repo	Game tool usage (pyxel)	Create a Snake game. Players need to control the movement of the snake to eat food and grow its body, while avoiding the snake's head touching their own body or game boundaries. Games need to have basic game logic, user interface. During the production process, please consider factors such as playability, beautiful interface, and convenient operation of the game. Note: pyxel environment already satisfied
19	Generate games using existing repo	Game tool usage (pyxel)	You are a professional game developer, please use pyxel software to create a simple jumping game. The game needs to include a character that can move left and right on the screen. When the player presses the spacebar, the character should jump. Please ensure that the game is easy to operate, with clear graphics, and complies with the functional limitations of pyxel software. Note: pyxel environment already satisfied
20	Generate games using existing repo	Game tool usage (pyxel)	Make a mouse click game that click button as many times as possible in 30 seconds using pyxel. Note: pyxel environment already satisfied