diff --git a/README.md b/README.md index e80082a3a..b3473a12c 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ + # MetaGPT: The Multi-Agent Framework

@@ -50,9 +51,9 @@ # Step 2: Clone the repository to your local machine for latest version, and ins cd MetaGPT pip3 install -e. # or pip3 install metagpt # for stable version -# Step 3: run the startup.py +# Step 3: run metagpt cli # setup your OPENAI_API_KEY in key.yaml copy from config.yaml -python3 startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # Step 4 [Optional]: If you want to save the artifacts like diagrams such as quadrant chart, system designs, sequence flow in the workspace, you can execute the step before Step 3. By default, the framework is compatible, and the entire process can be run completely without executing this step. # If executing, ensure that NPM is installed on your system. Then install mermaid-js. (If you don't have npm in your computer, please go to the Node.js official website to install Node.js https://nodejs.org/ and then you will have npm tool in your computer.) @@ -78,7 +79,7 @@ # Step 2: Run metagpt demo with container -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" ``` detail installation please refer to [docker_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-with-docker) @@ -117,7 +118,7 @@ ### Contact Information If you have any questions or feedback about this project, please feel free to contact us. We highly appreciate your suggestions! -- **Email:** alexanderwu@fuzhi.ai +- **Email:** alexanderwu@deepwisdom.ai - **GitHub Issues:** For more technical inquiries, you can also create a new issue in our [GitHub repository](https://github.com/geekan/metagpt/issues). We will respond to all questions within 2-3 business days. diff --git a/docs/FAQ-EN.md b/docs/FAQ-EN.md index f9df50caf..af6868509 100644 --- a/docs/FAQ-EN.md +++ b/docs/FAQ-EN.md @@ -98,7 +98,7 @@ 1. How to change the investment amount? - 1. You can view all commands by typing `python startup.py --help` + 1. You can view all commands by typing `metagpt --help` 1. Which version of Python is more stable? @@ -134,7 +134,7 @@ 1. Configuration instructions for SD Skills: The SD interface is currently deployed based on *https://github.com/AUTOMATIC1111/stable-diffusion-webui* **For environmental configurations and model downloads, please refer to the aforementioned GitHub repository. To initiate the SD service that supports API calls, run the command specified in cmd with the parameter nowebui, i.e., - 1. > python webui.py --enable-insecure-extension-access --port xxx --no-gradio-queue --nowebui + 1. > python3 webui.py --enable-insecure-extension-access --port xxx --no-gradio-queue --nowebui 1.     Once it runs without errors, the interface will be accessible after approximately 1 minute when the model finishes loading. 1. Configure SD_URL and SD_T2I_API in the config.yaml/key.yaml files. 1. ![](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/065295a67b0b4feea665d1372722d49d~tplv-k3u1fbpfcp-zoom-1.image) diff --git a/docs/README_CN.md b/docs/README_CN.md index 038925184..dd65c2a25 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -47,9 +47,9 @@ # 第 2 步:克隆最新仓库到您的本地机器,并进行安装。 cd MetaGPT pip3 install -e. # 或者 pip3 install metagpt # 安装稳定版本 -# 第 3 步:执行startup.py +# 第 3 步:执行metagpt # 拷贝config.yaml为key.yaml,并设置你自己的OPENAI_API_KEY -python3 startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # 第 4 步【可选的】:如果你想在执行过程中保存像象限图、系统设计、序列流程等图表这些产物,可以在第3步前执行该步骤。默认的,框架做了兼容,在不执行该步的情况下,也可以完整跑完整个流程。 # 如果执行,确保您的系统上安装了 NPM。并使用npm安装mermaid-js @@ -75,7 +75,7 @@ # 步骤2: 使用容器运行metagpt演示 -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" ``` 详细的安装请安装 [docker_install](https://docs.deepwisdom.ai/zhcn/guide/get_started/installation.html#%E4%BD%BF%E7%94%A8docker%E5%AE%89%E8%A3%85) @@ -114,7 +114,7 @@ ### 联系信息 如果您对这个项目有任何问题或反馈,欢迎联系我们。我们非常欢迎您的建议! -- **邮箱:** alexanderwu@fuzhi.ai +- **邮箱:** alexanderwu@deepwisdom.ai - **GitHub 问题:** 对于更技术性的问题,您也可以在我们的 [GitHub 仓库](https://github.com/geekan/metagpt/issues) 中创建一个新的问题。 我们会在2-3个工作日内回复所有问题。 diff --git a/docs/README_JA.md b/docs/README_JA.md index 411d190b4..05f718635 100644 --- a/docs/README_JA.md +++ b/docs/README_JA.md @@ -41,7 +41,7 @@ ## MetaGPT の能力 ## 例(GPT-4 で完全生成) -例えば、`python startup.py "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます +例えば、`metagpt "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます ![Jinri Toutiao Recsys データと API デザイン](resources/workspace/content_rec_sys/resources/data_api_design.png) @@ -60,16 +60,16 @@ ### 伝統的なインストール ```bash # ステップ 1: Python 3.9+ がシステムにインストールされていることを確認してください。これを確認するには: -python --version +python3 --version # ステップ 2: リポジトリをローカルマシンにクローンし、インストールする。 git clone https://github.com/geekan/MetaGPT.git cd MetaGPT pip install -e. -# ステップ 3: startup.py を実行する +# ステップ 3: metagpt を実行する # config.yaml を key.yaml にコピーし、独自の OPENAI_API_KEY を設定します -python3 startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # ステップ 4 [オプション]: 実行中に PRD ファイルなどのアーティファクトを保存する場合は、ステップ 3 の前にこのステップを実行できます。デフォルトでは、フレームワークには互換性があり、この手順を実行しなくてもプロセス全体を完了できます。 # NPM がシステムにインストールされていることを確認してください。次に mermaid-js をインストールします。(お使いのコンピューターに npm がない場合は、Node.js 公式サイトで Node.js https://nodejs.org/ をインストールしてください。) @@ -178,7 +178,7 @@ # ステップ 2: コンテナで metagpt デモを実行する -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" # コンテナを起動し、その中でコマンドを実行することもできます docker run --name metagpt -d \ @@ -188,7 +188,7 @@ # コンテナを起動し、その中でコマンドを実行することもで metagpt/metagpt:latest docker exec -it metagpt /bin/bash -$ python startup.py "Write a cli snake game" +$ metagpt "Write a cli snake game" ``` コマンド `docker run ...` は以下のことを行います: @@ -196,7 +196,7 @@ # コンテナを起動し、その中でコマンドを実行することもで - 特権モードで実行し、ブラウザの実行権限を得る - ホスト設定ファイル `/opt/metagpt/config/key.yaml` をコンテナ `/app/metagpt/config/key.yaml` にマップします - ホストディレクトリ `/opt/metagpt/workspace` をコンテナディレクトリ `/app/metagpt/workspace` にマップするs -- デモコマンド `python startup.py "Write a cli snake game"` を実行する +- デモコマンド `metagpt "Write a cli snake game"` を実行する ### 自分でイメージをビルドする @@ -225,11 +225,11 @@ ## チュートリアル: スタートアップの開始 ```shell # スクリプトの実行 -python startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # プロジェクトの実施にエンジニアを雇わないこと -python startup.py "Write a cli snake game" --implement False +metagpt "Write a cli snake game" --no-implement # エンジニアを雇い、コードレビューを行う -python startup.py "Write a cli snake game" --code_review True +metagpt "Write a cli snake game" --code_review ``` スクリプトを実行すると、`workspace/` ディレクトリに新しいプロジェクトが見つかります。 @@ -239,17 +239,17 @@ ### プラットフォームまたはツールの設定 要件を述べるときに、どのプラットフォームまたはツールを使用するかを指定できます。 ```shell -python startup.py "pygame をベースとした cli ヘビゲームを書く" +metagpt "pygame をベースとした cli ヘビゲームを書く" ``` ### 使用方法 ``` 会社名 - startup.py - 私たちは AI で構成されたソフトウェア・スタートアップです。私たちに投資することは、無限の可能性に満ちた未来に力を与えることです。 + metagpt - 私たちは AI で構成されたソフトウェア・スタートアップです。私たちに投資することは、無限の可能性に満ちた未来に力を与えることです。 シノプシス - startup.py IDEA + metagpt IDEA 説明 私たちは AI で構成されたソフトウェア・スタートアップです。私たちに投資することは、無限の可能性に満ちた未来に力を与えることです。 @@ -317,7 +317,7 @@ ## お問い合わせ先 このプロジェクトに関するご質問やご意見がございましたら、お気軽にお問い合わせください。皆様のご意見をお待ちしております! -- **Email:** alexanderwu@fuzhi.ai +- **Email:** alexanderwu@deepwisdom.ai - **GitHub Issues:** 技術的なお問い合わせについては、[GitHub リポジトリ](https://github.com/geekan/metagpt/issues) に新しい issue を作成することもできます。 ご質問には 2-3 営業日以内に回答いたします。 diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 005a59ab2..afc9ff445 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -16,12 +16,12 @@ ### Tasks To reach version v0.5, approximately 70% of the following tasks need to be completed. 1. Usability - 1. Release v0.01 pip package to try to solve issues like npm installation (though not necessarily successfully) + 1. ~~Release v0.01 pip package to try to solve issues like npm installation (though not necessarily successfully)~~ (v0.3.0) 2. Support for overall save and recovery of software companies - 3. Support human confirmation and modification during the process + 3. ~~Support human confirmation and modification during the process~~ (v0.3.0) New: Support human confirmation and modification with fewer constrainsts and a more user-friendly interface 4. Support process caching: Consider carefully whether to add server caching mechanism - 5. Resolve occasional failure to follow instruction under current prompts, causing code parsing errors, through stricter system prompts - 6. Write documentation, describing the current features and usage at all levels + 5. ~~Resolve occasional failure to follow instruction under current prompts, causing code parsing errors, through stricter system prompts~~ (v0.4.0, with function call) + 6. Write documentation, describing the current features and usage at all levels (ongoing, continuously adding contents to [documentation site](https://docs.deepwisdom.ai/guide/get_started/introduction.html)) 7. ~~Support Docker~~ 2. Features 1. Support a more standard and stable parser (need to analyze the format that the current LLM is better at) @@ -30,31 +30,33 @@ ### Tasks 4. Complete the design and implementation of module breakdown 5. Support various modes of memory: clearly distinguish between long-term and short-term memory 6. Perfect the test role, and carry out necessary interactions with humans - 7. Provide full mode instead of the current fast mode, allowing natural communication between roles - 8. Implement SkillManager and the process of incremental Skill learning + 7. Allowing natural communication between roles (expected v0.5.0) + 8. Implement SkillManager and the process of incremental Skill learning (experimentation done with game agents) 9. Automatically get RPM and configure it by calling the corresponding openai page, so that each key does not need to be manually configured + 10. IMPORTANT: Support incremental development (expected v0.5.0) 3. Strategies - 1. Support ReAct strategy - 2. Support CoT strategy + 1. Support ReAct strategy (experimentation done with game agents) + 2. Support CoT strategy (experimentation done with game agents) 3. Support ToT strategy - 4. Support Reflection strategy + 4. Support Reflection strategy (experimentation done with game agents) + 5. Support planning 4. Actions - 1. Implementation: Search + 1. ~~Implementation: Search~~ (v0.2.1) 2. Implementation: Knowledge search, supporting 10+ data formats - 3. Implementation: Data EDA + 3. Implementation: Data EDA (expected v0.6.0) 4. Implementation: Review - 5. Implementation: Add Document - 6. Implementation: Delete Document + 5. Implementation: Add Document (expected v0.5.0) + 6. Implementation: Delete Document (expected v0.5.0) 7. Implementation: Self-training - 8. Implementation: DebugError + 8. ~~Implementation: DebugError~~ (v0.2.1) 9. Implementation: Generate reliable unit tests based on YAPI 10. Implementation: Self-evaluation 11. Implementation: AI Invocation 12. Implementation: Learning and using third-party standard libraries 13. Implementation: Data collection 14. Implementation: AI training - 15. Implementation: Run code - 16. Implementation: Web access + 15. ~~Implementation: Run code~~ (v0.2.1) + 16. ~~Implementation: Web access~~ (v0.2.1) 5. Plugins: Compatibility with plugin system 6. Tools 1. ~~Support SERPER api~~ @@ -64,13 +66,13 @@ ### Tasks 1. Perfect the action pool/skill pool for each role 2. Red Book blogger 3. E-commerce seller - 4. Data analyst + 4. Data analyst (expected v0.6.0) 5. News observer - 6. Institutional researcher + 6. ~~Institutional researcher~~ (v0.2.1) 8. Evaluation - 1. Support an evaluation on a game dataset - 2. Reproduce papers, implement full skill acquisition for a single game role, achieving SOTA results - 3. Support an evaluation on a math dataset + 1. Support an evaluation on a game dataset (experimentation done with game agents) + 2. Reproduce papers, implement full skill acquisition for a single game role, achieving SOTA results (experimentation done with game agents) + 3. Support an evaluation on a math dataset (expected v0.6.0) 4. Reproduce papers, achieving SOTA results for current mathematical problem solving process 9. LLM 1. Support Claude underlying API diff --git a/docs/install/cli_install_cn.md b/docs/install/cli_install_cn.md index f351090ed..b1da1b813 100644 --- a/docs/install/cli_install_cn.md +++ b/docs/install/cli_install_cn.md @@ -15,7 +15,7 @@ # 第 1 步:确保您的系统上安装了 NPM。并使用npm安装mermaid-js sudo npm install -g @mermaid-js/mermaid-cli # 第 2 步:确保您的系统上安装了 Python 3.9+。您可以使用以下命令进行检查: -python --version +python3 --version # 第 3 步:克隆仓库到您的本地机器,并进行安装。 git clone https://github.com/geekan/MetaGPT.git diff --git a/docs/install/docker_install.md b/docs/install/docker_install.md index b803a5dae..37125bdbe 100644 --- a/docs/install/docker_install.md +++ b/docs/install/docker_install.md @@ -15,7 +15,7 @@ # Step 2: Run metagpt demo with container -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python3 startup.py "Write a cli snake game" + metagpt "Write a cli snake game" # You can also start a container and execute commands in it docker run --name metagpt -d \ @@ -25,7 +25,7 @@ # You can also start a container and execute commands in it metagpt/metagpt:latest docker exec -it metagpt /bin/bash -$ python3 startup.py "Write a cli snake game" +$ metagpt "Write a cli snake game" ``` The command `docker run ...` do the following things: @@ -33,7 +33,7 @@ # You can also start a container and execute commands in it - Run in privileged mode to have permission to run the browser - Map host configure file `/opt/metagpt/config/key.yaml` to container `/app/metagpt/config/key.yaml` - Map host directory `/opt/metagpt/workspace` to container `/app/metagpt/workspace` -- Execute the demo command `python3 startup.py "Write a cli snake game"` +- Execute the demo command `metagpt "Write a cli snake game"` ### Build image by yourself diff --git a/docs/install/docker_install_cn.md b/docs/install/docker_install_cn.md index 347fae10c..f360b49ed 100644 --- a/docs/install/docker_install_cn.md +++ b/docs/install/docker_install_cn.md @@ -15,7 +15,7 @@ # 步骤2: 使用容器运行metagpt演示 -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" # 您也可以启动一个容器并在其中执行命令 docker run --name metagpt -d \ @@ -25,7 +25,7 @@ # 您也可以启动一个容器并在其中执行命令 metagpt/metagpt:latest docker exec -it metagpt /bin/bash -$ python startup.py "Write a cli snake game" +$ metagpt "Write a cli snake game" ``` `docker run ...`做了以下事情: @@ -33,7 +33,7 @@ # 您也可以启动一个容器并在其中执行命令 - 以特权模式运行,有权限运行浏览器 - 将主机文件 `/opt/metagpt/config/key.yaml` 映射到容器文件 `/app/metagpt/config/key.yaml` - 将主机目录 `/opt/metagpt/workspace` 映射到容器目录 `/app/metagpt/workspace` -- 执行示例命令 `python startup.py "Write a cli snake game"` +- 执行示例命令 `metagpt "Write a cli snake game"` ### 自己构建镜像 diff --git a/docs/tutorial/usage.md b/docs/tutorial/usage.md index ee87b65c9..fbe4a8311 100644 --- a/docs/tutorial/usage.md +++ b/docs/tutorial/usage.md @@ -19,11 +19,11 @@ ### Initiating a startup ```shell # Run the script -python startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # Do not hire an engineer to implement the project -python startup.py "Write a cli snake game" --implement False +metagpt "Write a cli snake game" --no-implement # Hire an engineer and perform code reviews -python startup.py "Write a cli snake game" --code_review True +metagpt "Write a cli snake game" --code_review ``` After running the script, you can find your new project in the `workspace/` directory. @@ -33,17 +33,17 @@ ### Preference of Platform or Tool You can tell which platform or tool you want to use when stating your requirements. ```shell -python startup.py "Write a cli snake game based on pygame" +metagpt "Write a cli snake game based on pygame" ``` ### Usage ``` NAME - startup.py - We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities. + metagpt - We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities. SYNOPSIS - startup.py IDEA + metagpt IDEA DESCRIPTION We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities. diff --git a/docs/tutorial/usage_cn.md b/docs/tutorial/usage_cn.md index 4b3bdd2c3..1ef50d633 100644 --- a/docs/tutorial/usage_cn.md +++ b/docs/tutorial/usage_cn.md @@ -18,9 +18,9 @@ # 复制配置文件并进行必要的修改 ### 示例:启动一个创业公司 ```shell -python startup.py "写一个命令行贪吃蛇" +metagpt "写一个命令行贪吃蛇" # 开启code review模式会花费更多的金钱, 但是会提升代码质量和成功率 -python startup.py "写一个命令行贪吃蛇" --code_review True +metagpt "写一个命令行贪吃蛇" --code_review ``` 运行脚本后,您可以在 `workspace/` 目录中找到您的新项目。 @@ -29,17 +29,17 @@ ### 平台或工具的倾向性 可以在阐述需求时说明想要使用的平台或工具。 例如: ```shell -python startup.py "写一个基于pygame的命令行贪吃蛇" +metagpt "写一个基于pygame的命令行贪吃蛇" ``` ### 使用 ``` 名称 - startup.py - 我们是一家AI软件创业公司。通过投资我们,您将赋能一个充满无限可能的未来。 + metagpt - 我们是一家AI软件创业公司。通过投资我们,您将赋能一个充满无限可能的未来。 概要 - startup.py IDEA + metagpt IDEA 描述 我们是一家AI软件创业公司。通过投资我们,您将赋能一个充满无限可能的未来。 diff --git a/metagpt/actions/fix_bug.py b/metagpt/actions/fix_bug.py new file mode 100644 index 000000000..6bd550d3d --- /dev/null +++ b/metagpt/actions/fix_bug.py @@ -0,0 +1,14 @@ +# -*- coding: utf-8 -*- +""" +@Time : 2023-12-12 +@Author : mashenquan +@File : fix_bug.py +""" +from metagpt.actions import Action + + +class FixBug(Action): + """Fix bug action without any implementation details""" + + async def run(self, *args, **kwargs): + raise NotImplementedError diff --git a/metagpt/actions/write_code.py b/metagpt/actions/write_code.py index 4c138a124..1dda6466f 100644 --- a/metagpt/actions/write_code.py +++ b/metagpt/actions/write_code.py @@ -14,12 +14,14 @@ 3. Encapsulate the input of RunCode into RunCodeContext and encapsulate the output of RunCode into RunCodeResult to standardize and unify parameter passing between WriteCode, RunCode, and DebugError. """ +import json from tenacity import retry, stop_after_attempt, wait_random_exponential from metagpt.actions.action import Action from metagpt.config import CONFIG -from metagpt.const import CODE_SUMMARIES_FILE_REPO, TEST_OUTPUTS_FILE_REPO +from metagpt.const import CODE_SUMMARIES_FILE_REPO, TEST_OUTPUTS_FILE_REPO, TASK_FILE_REPO, BUGFIX_FILENAME, \ + DOCS_FILE_REPO from metagpt.logs import logger from metagpt.schema import CodingContext, Document, RunCodeResult from metagpt.utils.common import CodeParser @@ -54,6 +56,12 @@ ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenc {summary_log} ``` ----- +# Bug Feedback logs +```text +{feedback} +``` +----- + ## Code: {filename} Write code with triple quoto, based on the following list and context. 1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT. @@ -88,6 +96,7 @@ class WriteCode(Action): return code async def run(self, *args, **kwargs) -> CodingContext: + bug_feedback = await FileRepository.get_file(filename=BUGFIX_FILENAME, relative_path=DOCS_FILE_REPO) coding_context = CodingContext.loads(self.context.content) test_doc = await FileRepository.get_file( filename="test_" + coding_context.filename + ".json", relative_path=TEST_OUTPUTS_FILE_REPO @@ -101,11 +110,13 @@ class WriteCode(Action): if test_doc: test_detail = RunCodeResult.loads(test_doc.content) logs = test_detail.stderr + code_context = await self._get_codes(coding_context.task_doc) prompt = PROMPT_TEMPLATE.format( design=coding_context.design_doc.content, tasks=coding_context.task_doc.content if coding_context.task_doc else "", - code=coding_context.code_doc.content if coding_context.code_doc else "", + code=code_context, logs=logs, + feedback=bug_feedback.content if bug_feedback else "", filename=self.context.filename, summary_log=summary_doc.content if summary_doc else "", ) @@ -115,3 +126,21 @@ class WriteCode(Action): coding_context.code_doc = Document(filename=coding_context.filename, root_path=CONFIG.src_workspace) coding_context.code_doc.content = code return coding_context + + @staticmethod + async def _get_codes(task_doc) -> str: + if not task_doc: + return "" + if not task_doc.content: + task_doc.content = FileRepository.get_file(filename=task_doc.filename, relative_path=TASK_FILE_REPO) + m = json.loads(task_doc.content) + code_filenames = m.get("Task list", []) + codes = [] + src_file_repo = CONFIG.git_repo.new_file_repository(relative_path=CONFIG.src_workspace) + for filename in code_filenames: + doc = await src_file_repo.get(filename=filename) + if not doc: + continue + codes.append(doc.content) + return "\n----------\n".join(codes) + diff --git a/metagpt/actions/write_docstring.py b/metagpt/actions/write_docstring.py index dd3312bd5..0ad134157 100644 --- a/metagpt/actions/write_docstring.py +++ b/metagpt/actions/write_docstring.py @@ -16,7 +16,7 @@ Options: Default: 'google' Example: - python3 -m metagpt.actions.write_docstring startup.py --overwrite False --style=numpy + python3 -m metagpt.actions.write_docstring ./metagpt/startup.py --overwrite False --style=numpy This script uses the 'fire' library to create a command-line interface. It generates docstrings for the given Python code using the specified docstring style and adds them to the code. diff --git a/metagpt/actions/write_prd.py b/metagpt/actions/write_prd.py index 530a22def..aad2422ef 100644 --- a/metagpt/actions/write_prd.py +++ b/metagpt/actions/write_prd.py @@ -17,6 +17,7 @@ from pathlib import Path from typing import List from metagpt.actions import Action, ActionOutput +from metagpt.actions.fix_bug import FixBug from metagpt.actions.search_and_summarize import SearchAndSummarize from metagpt.config import CONFIG from metagpt.const import ( @@ -24,10 +25,10 @@ from metagpt.const import ( DOCS_FILE_REPO, PRD_PDF_FILE_REPO, PRDS_FILE_REPO, - REQUIREMENT_FILENAME, + REQUIREMENT_FILENAME, BUGFIX_FILENAME, ) from metagpt.logs import logger -from metagpt.schema import Document, Documents +from metagpt.schema import Document, Documents, Message, BugFixContext from metagpt.utils.common import CodeParser from metagpt.utils.file_repository import FileRepository from metagpt.utils.get_template import get_template @@ -227,7 +228,6 @@ There are no unclear points. }, } - OUTPUT_MAPPING = { "Language": (str, ...), "Original Requirements": (str, ...), @@ -305,15 +305,44 @@ output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like "Old P and only output the json inside this tag, nothing else """ +IS_BUGFIX_PROMPT = """ +{content} + +___ +You are a professional product manager; You need to determine whether the above content describes a requirement or provides feedback about a bug. +Respond with `YES` if it is a feedback about a bug, `NO` if it is not, and provide the reasons. Return the response in JSON format like below: + +```json +{{ + "is_bugfix": ..., # `YES` or `NO` + "reason": ..., # reason string +}} +``` +""" + class WritePRD(Action): def __init__(self, name="", context=None, llm=None): super().__init__(name, context, llm) - async def run(self, with_messages, format=CONFIG.prompt_format, *args, **kwargs) -> ActionOutput: + async def run(self, with_messages, format=CONFIG.prompt_format, *args, **kwargs) -> ActionOutput | Message: # Determine which requirement documents need to be rewritten: Use LLM to assess whether new requirements are # related to the PRD. If they are related, rewrite the PRD. - requirement_doc = await FileRepository.get_file(filename=REQUIREMENT_FILENAME, relative_path=DOCS_FILE_REPO) + docs_file_repo = CONFIG.git_repo.new_file_repository(relative_path=DOCS_FILE_REPO) + requirement_doc = await docs_file_repo.get(filename=REQUIREMENT_FILENAME) + if await self._is_bugfix(requirement_doc.content): + await docs_file_repo.save(filename=BUGFIX_FILENAME, content=requirement_doc.content) + await docs_file_repo.save(filename=REQUIREMENT_FILENAME, content="") + bug_fix = BugFixContext(filename=BUGFIX_FILENAME) + return Message(content=bug_fix.json(), instruct_content=bug_fix, + role=self.profile, + cause_by=FixBug, + sent_from=self, + send_to="Alex", # the name of Engineer + ) + else: + await docs_file_repo.delete(filename=BUGFIX_FILENAME) + prds_file_repo = CONFIG.git_repo.new_file_repository(PRDS_FILE_REPO) prd_docs = await prds_file_repo.get_all() change_files = Documents() @@ -405,7 +434,7 @@ class WritePRD(Action): if not quadrant_chart: return pathname = ( - CONFIG.git_repo.workdir / Path(COMPETITIVE_ANALYSIS_FILE_REPO) / Path(prd_doc.filename).with_suffix("") + CONFIG.git_repo.workdir / Path(COMPETITIVE_ANALYSIS_FILE_REPO) / Path(prd_doc.filename).with_suffix("") ) if not pathname.parent.exists(): pathname.parent.mkdir(parents=True, exist_ok=True) @@ -430,3 +459,11 @@ class WritePRD(Action): ws_name = CodeParser.parse_str(block="Project Name", text=prd) CONFIG.project_name = ws_name CONFIG.git_repo.rename_root(CONFIG.project_name) + + async def _is_bugfix(self, content): + prompt = IS_BUGFIX_PROMPT.format(content=content) + res = await self._aask(prompt=prompt) + logger.info(f"IS_BUGFIX:{res}") + if "YES" in res: + return True + return False diff --git a/metagpt/const.py b/metagpt/const.py index bd735a5e1..f6f64a27d 100644 --- a/metagpt/const.py +++ b/metagpt/const.py @@ -74,6 +74,7 @@ MESSAGE_ROUTE_TO_ALL = "" MESSAGE_ROUTE_TO_NONE = "" REQUIREMENT_FILENAME = "requirement.txt" +BUGFIX_FILENAME = "bugfix.txt" PACKAGE_REQUIREMENTS_FILENAME = "requirements.txt" DOCS_FILE_REPO = "docs" diff --git a/metagpt/roles/engineer.py b/metagpt/roles/engineer.py index 9f8eb6482..cedd2101f 100644 --- a/metagpt/roles/engineer.py +++ b/metagpt/roles/engineer.py @@ -24,6 +24,7 @@ from pathlib import Path from typing import Set from metagpt.actions import Action, WriteCode, WriteCodeReview, WriteTasks +from metagpt.actions.fix_bug import FixBug from metagpt.actions.summarize_code import SummarizeCode from metagpt.config import CONFIG from metagpt.const import ( @@ -78,7 +79,7 @@ class Engineer(Role): """Initializes the Engineer role with given attributes.""" super().__init__(name, profile, goal, constraints) self.use_code_review = use_code_review - self._watch([WriteTasks, SummarizeCode, WriteCode, WriteCodeReview]) + self._watch([WriteTasks, SummarizeCode, WriteCode, WriteCodeReview, FixBug]) self.code_todos = [] self.summarize_todos = [] self.n_borg = n_borg @@ -191,14 +192,14 @@ class Engineer(Role): async def _think(self) -> Action | None: if not CONFIG.src_workspace: CONFIG.src_workspace = CONFIG.git_repo.workdir / CONFIG.git_repo.workdir.name - write_code_filters = any_to_str_set([WriteTasks, SummarizeCode]) + write_code_filters = any_to_str_set([WriteTasks, SummarizeCode, FixBug]) summarize_code_filters = any_to_str_set([WriteCode, WriteCodeReview]) if not self._rc.news: return None msg = self._rc.news[0] if msg.cause_by in write_code_filters: logger.info(f"TODO WriteCode:{msg.json()}") - await self._new_code_actions() + await self._new_code_actions(bug_fix=msg.cause_by == any_to_str(FixBug)) return self._rc.todo if msg.cause_by in summarize_code_filters and msg.sent_from == any_to_str(self): logger.info(f"TODO SummarizeCode:{msg.json()}") @@ -232,10 +233,10 @@ class Engineer(Role): coding_doc = Document(root_path=str(src_file_repo.root_path), filename=filename, content=context.json()) return coding_doc - async def _new_code_actions(self): + async def _new_code_actions(self, bug_fix=False): # Prepare file repos src_file_repo = CONFIG.git_repo.new_file_repository(CONFIG.src_workspace) - changed_src_files = src_file_repo.changed_files + changed_src_files = src_file_repo.all_files if bug_fix else src_file_repo.changed_files task_file_repo = CONFIG.git_repo.new_file_repository(TASK_FILE_REPO) changed_task_files = task_file_repo.changed_files design_file_repo = CONFIG.git_repo.new_file_repository(SYSTEM_DESIGN_FILE_REPO) diff --git a/metagpt/roles/role.py b/metagpt/roles/role.py index 2651be7eb..52ac3cf28 100644 --- a/metagpt/roles/role.py +++ b/metagpt/roles/role.py @@ -286,6 +286,8 @@ class Role: cause_by=self._rc.todo, sent_from=self, ) + elif isinstance(response, Message): + msg = response else: msg = Message(content=response, role=self.profile, cause_by=self._rc.todo, sent_from=self) self._rc.memory.add(msg) diff --git a/metagpt/schema.py b/metagpt/schema.py index a8c1b7726..25281e399 100644 --- a/metagpt/schema.py +++ b/metagpt/schema.py @@ -97,14 +97,14 @@ class Message(BaseModel): send_to: Set = Field(default_factory={MESSAGE_ROUTE_TO_ALL}) def __init__( - self, - content, - instruct_content=None, - role="user", - cause_by="", - sent_from="", - send_to=MESSAGE_ROUTE_TO_ALL, - **kwargs, + self, + content, + instruct_content=None, + role="user", + cause_by="", + sent_from="", + send_to=MESSAGE_ROUTE_TO_ALL, + **kwargs, ): """ Parameters not listed below will be stored as meta info, including custom parameters. @@ -341,3 +341,7 @@ class CodeSummarizeContext(BaseModel): def __hash__(self): return hash((self.design_filename, self.task_filename)) + + +class BugFixContext(BaseModel): + filename: str = "" diff --git a/metagpt/tools/sd_engine.py b/metagpt/tools/sd_engine.py index c6676a247..a84812f7c 100644 --- a/metagpt/tools/sd_engine.py +++ b/metagpt/tools/sd_engine.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # @Date : 2023/7/19 16:28 -# @Author : stellahong (stellahong@fuzhi.ai) +# @Author : stellahong (stellahong@deepwisdom.ai) # @Desc : import asyncio import base64 diff --git a/metagpt/utils/git_repository.py b/metagpt/utils/git_repository.py index 5aec4509c..d372fd22e 100644 --- a/metagpt/utils/git_repository.py +++ b/metagpt/utils/git_repository.py @@ -8,15 +8,13 @@ """ from __future__ import annotations -import os +from gitignore_parser import parse_gitignore, rule_from_pattern, handle_negation import shutil from enum import Enum from pathlib import Path from typing import Dict, List - from git.repo import Repo from git.repo.fun import is_git_dir - from metagpt.const import DEFAULT_WORKSPACE_ROOT from metagpt.logs import logger from metagpt.utils.dependency_file import DependencyFile @@ -51,6 +49,7 @@ class GitRepository: """ self._repository = None self._dependency = None + self._gitignore_rules = None if local_path: self.open(local_path=local_path, auto_init=auto_init) @@ -63,6 +62,7 @@ class GitRepository: local_path = Path(local_path) if self.is_git_dir(local_path): self._repository = Repo(local_path) + self._gitignore_rules = parse_gitignore(full_path=str(local_path / ".gitignore")) return if not auto_init: return @@ -82,6 +82,7 @@ class GitRepository: writer.write("\n".join(ignores)) self._repository.index.add([".gitignore"]) self._repository.index.commit("Add .gitignore") + self._gitignore_rules = parse_gitignore(full_path=gitignore_filename) def add_change(self, files: Dict): """Add or remove files from the staging area based on the provided changes. @@ -204,8 +205,9 @@ class GitRepository: logger.info(f"Rename directory {str(self.workdir)} to {str(new_path)}") self._repository = Repo(new_path) - def get_files(self, relative_path: Path | str, root_relative_path: Path | str = None) -> List: - """Retrieve a list of files in the specified relative path. + def get_files(self, relative_path: Path | str, root_relative_path: Path | str = None, filter_ignored=True) -> List: + """ + Retrieve a list of files in the specified relative path. The method returns a list of file paths relative to the current FileRepository. @@ -213,6 +215,8 @@ class GitRepository: :type relative_path: Path or str :param root_relative_path: The root relative path within the repository. :type root_relative_path: Path or str + :param filter_ignored: Flag to indicate whether to filter files based on .gitignore rules. + :type filter_ignored: bool :return: A list of file paths in the specified directory. :rtype: List[str] """ @@ -231,10 +235,35 @@ class GitRepository: rpath = file_path.relative_to(root_relative_path) files.append(str(rpath)) else: - subfolder_files = self.get_files(relative_path=file_path, root_relative_path=root_relative_path) + subfolder_files = self.get_files(relative_path=file_path, root_relative_path=root_relative_path, + filter_ignored=False) files.extend(subfolder_files) except Exception as e: logger.error(f"Error: {e}") + if not filter_ignored: + return files + filtered_files = self.filter_gitignore(filenames=files, root_relative_path=root_relative_path) + return filtered_files + + def filter_gitignore(self, filenames: List[str], root_relative_path: Path | str = None) -> List[str]: + """ + Filter a list of filenames based on .gitignore rules. + + :param filenames: A list of filenames to be filtered. + :type filenames: List[str] + :param root_relative_path: The root relative path within the repository. + :type root_relative_path: Path or str + :return: A list of filenames that pass the .gitignore filtering. + :rtype: List[str] + """ + if root_relative_path is None: + root_relative_path = self.workdir + files = [] + for filename in filenames: + pathname = root_relative_path / filename + if self._gitignore_rules(str(pathname)): + continue + files.append(filename) return files @@ -244,6 +273,7 @@ if __name__ == "__main__": repo = GitRepository() repo.open(path, auto_init=True) + repo.filter_gitignore(filenames=["snake_game/snake_game/__pycache__", "snake_game/snake_game/game.py"]) changes = repo.changed_files print(changes) diff --git a/metagpt/utils/token_counter.py b/metagpt/utils/token_counter.py index 53d9de345..266a53268 100644 --- a/metagpt/utils/token_counter.py +++ b/metagpt/utils/token_counter.py @@ -16,6 +16,7 @@ TOKEN_COSTS = { "gpt-3.5-turbo-0613": {"prompt": 0.0015, "completion": 0.002}, "gpt-3.5-turbo-16k": {"prompt": 0.003, "completion": 0.004}, "gpt-3.5-turbo-16k-0613": {"prompt": 0.003, "completion": 0.004}, + "gpt-3.5-turbo-1106": {"prompt": 0.001, "completion": 0.002}, "gpt-4-0314": {"prompt": 0.03, "completion": 0.06}, "gpt-4": {"prompt": 0.03, "completion": 0.06}, "gpt-4-32k": {"prompt": 0.06, "completion": 0.12}, @@ -33,6 +34,7 @@ TOKEN_MAX = { "gpt-3.5-turbo-0613": 4096, "gpt-3.5-turbo-16k": 16384, "gpt-3.5-turbo-16k-0613": 16384, + "gpt-3.5-turbo-1106": 16384, "gpt-4-0314": 8192, "gpt-4": 8192, "gpt-4-32k": 32768, @@ -54,6 +56,7 @@ def count_message_tokens(messages, model="gpt-3.5-turbo-0613"): if model in { "gpt-3.5-turbo-0613", "gpt-3.5-turbo-16k-0613", + "gpt-3.5-turbo-1106", "gpt-4-0314", "gpt-4-32k-0314", "gpt-4-0613", diff --git a/requirements.txt b/requirements.txt index e622a4b9e..c3a8cb44f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -53,3 +53,4 @@ aiofiles==23.2.1 gitpython==3.1.40 zhipuai==1.0.7 socksio~=1.0.0 +gitignore-parser==0.1.9 diff --git a/setup.py b/setup.py index 6d3708c32..4dd453b3d 100644 --- a/setup.py +++ b/setup.py @@ -30,13 +30,13 @@ with open(path.join(here, "requirements.txt"), encoding="utf-8") as f: setup( name="metagpt", - version="0.3.0", - description="The Multi-Agent Framework", + version="0.4.0", + description="The Multi-Role Meta Programming Framework", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/geekan/MetaGPT", author="Alexander Wu", - author_email="alexanderwu@fuzhi.ai", + author_email="alexanderwu@deepwisdom.ai", license="MIT", keywords="metagpt multi-role multi-agent programming gpt llm metaprogramming", packages=find_packages(exclude=["contrib", "docs", "examples", "tests*"]), diff --git a/tests/metagpt/actions/test_ui_design.py b/tests/metagpt/actions/test_ui_design.py index b8be914ae..83590ec7d 100644 --- a/tests/metagpt/actions/test_ui_design.py +++ b/tests/metagpt/actions/test_ui_design.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # @Date : 2023/7/22 02:40 -# @Author : stellahong (stellahong@fuzhi.ai) +# @Author : stellahong (stellahong@deepwisdom.ai) # from tests.metagpt.roles.ui_role import UIDesign diff --git a/tests/metagpt/roles/test_ui.py b/tests/metagpt/roles/test_ui.py index 5904bee8f..2038a1aee 100644 --- a/tests/metagpt/roles/test_ui.py +++ b/tests/metagpt/roles/test_ui.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # @Date : 2023/7/22 02:40 -# @Author : stellahong (stellahong@fuzhi.ai) +# @Author : stellahong (stellahong@deepwisdom.ai) # from metagpt.roles import ProductManager from metagpt.team import Team diff --git a/tests/metagpt/roles/ui_role.py b/tests/metagpt/roles/ui_role.py index ee36befbd..8ac799bf3 100644 --- a/tests/metagpt/roles/ui_role.py +++ b/tests/metagpt/roles/ui_role.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # @Date : 2023/7/15 16:40 -# @Author : stellahong (stellahong@fuzhi.ai) +# @Author : stellahong (stellahong@deepwisdom.ai) # @Desc : import os import re diff --git a/tests/metagpt/test_startup.py b/tests/metagpt/test_startup.py index 53d3509ed..c34fd2c31 100644 --- a/tests/metagpt/test_startup.py +++ b/tests/metagpt/test_startup.py @@ -16,6 +16,7 @@ runner = CliRunner() @pytest.mark.asyncio async def test_team(): + # FIXME: we're now using "metagpt" cli, so the entrance should be replaced instead. company = Team() company.run_project("做一个基础搜索引擎,可以支持知识库") history = await company.run(n_round=5) diff --git a/tests/metagpt/tools/test_sd_tool.py b/tests/metagpt/tools/test_sd_tool.py index edb23df42..e457101a9 100644 --- a/tests/metagpt/tools/test_sd_tool.py +++ b/tests/metagpt/tools/test_sd_tool.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # @Date : 2023/7/22 02:40 -# @Author : stellahong (stellahong@fuzhi.ai) +# @Author : stellahong (stellahong@deepwisdom.ai) # import os diff --git a/tests/metagpt/tools/test_web_browser_engine.py b/tests/metagpt/tools/test_web_browser_engine.py index b08d0ca10..28dd0e15c 100644 --- a/tests/metagpt/tools/test_web_browser_engine.py +++ b/tests/metagpt/tools/test_web_browser_engine.py @@ -7,8 +7,8 @@ from metagpt.tools import WebBrowserEngineType, web_browser_engine @pytest.mark.parametrize( "browser_type, url, urls", [ - (WebBrowserEngineType.PLAYWRIGHT, "https://fuzhi.ai", ("https://fuzhi.ai",)), - (WebBrowserEngineType.SELENIUM, "https://fuzhi.ai", ("https://fuzhi.ai",)), + (WebBrowserEngineType.PLAYWRIGHT, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), + (WebBrowserEngineType.SELENIUM, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), ], ids=["playwright", "selenium"], ) diff --git a/tests/metagpt/tools/test_web_browser_engine_playwright.py b/tests/metagpt/tools/test_web_browser_engine_playwright.py index 69e1339e7..e9ea80b10 100644 --- a/tests/metagpt/tools/test_web_browser_engine_playwright.py +++ b/tests/metagpt/tools/test_web_browser_engine_playwright.py @@ -8,9 +8,9 @@ from metagpt.tools import web_browser_engine_playwright @pytest.mark.parametrize( "browser_type, use_proxy, kwagrs, url, urls", [ - ("chromium", {"proxy": True}, {}, "https://fuzhi.ai", ("https://fuzhi.ai",)), - ("firefox", {}, {"ignore_https_errors": True}, "https://fuzhi.ai", ("https://fuzhi.ai",)), - ("webkit", {}, {"ignore_https_errors": True}, "https://fuzhi.ai", ("https://fuzhi.ai",)), + ("chromium", {"proxy": True}, {}, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), + ("firefox", {}, {"ignore_https_errors": True}, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), + ("webkit", {}, {"ignore_https_errors": True}, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), ], ids=["chromium-normal", "firefox-normal", "webkit-normal"], ) diff --git a/tests/metagpt/tools/test_web_browser_engine_selenium.py b/tests/metagpt/tools/test_web_browser_engine_selenium.py index ce322f7bd..ac6eafee7 100644 --- a/tests/metagpt/tools/test_web_browser_engine_selenium.py +++ b/tests/metagpt/tools/test_web_browser_engine_selenium.py @@ -8,9 +8,9 @@ from metagpt.tools import web_browser_engine_selenium @pytest.mark.parametrize( "browser_type, use_proxy, url, urls", [ - ("chrome", True, "https://fuzhi.ai", ("https://fuzhi.ai",)), - ("firefox", False, "https://fuzhi.ai", ("https://fuzhi.ai",)), - ("edge", False, "https://fuzhi.ai", ("https://fuzhi.ai",)), + ("chrome", True, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), + ("firefox", False, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), + ("edge", False, "https://deepwisdom.ai", ("https://deepwisdom.ai",)), ], ids=["chrome-normal", "firefox-normal", "edge-normal"], ) diff --git a/tests/metagpt/utils/test_git_repository.py b/tests/metagpt/utils/test_git_repository.py index 23bebba7f..d800e9594 100644 --- a/tests/metagpt/utils/test_git_repository.py +++ b/tests/metagpt/utils/test_git_repository.py @@ -73,6 +73,13 @@ async def test_git1(): repo1 = GitRepository(local_path=local_path, auto_init=False) assert repo1.changed_files + file_repo = repo1.new_file_repository("__pycache__") + await file_repo.save("a.pyc", content="") + all_files = repo1.get_files(relative_path=".", filter_ignored=False) + assert "__pycache__/a.pyc" in all_files + all_files = repo1.get_files(relative_path=".", filter_ignored=True) + assert "__pycache__/a.pyc" not in all_files + repo1.delete_repository() assert not local_path.exists()