-
-描述
- 我们是一家AI软件创业公司。通过投资我们，您将赋能一个充满无限可能的未来。
-
-位置参数
- IDEA
- 类型: str
- 您的创新想法，例如"写一个命令行贪吃蛇。"
-
-标志
- --investment=INVESTMENT
- 类型: float
- 默认值: 3.0
- 作为投资者，您有机会向这家AI公司投入一定的美元金额。
- --n_round=N_ROUND
- 类型: int
- 默认值: 5
-
-备注
- 您也可以用`标志`的语法，来处理`位置参数`
-```
-
-### 代码实现
-
-```python
-from metagpt.software_company import SoftwareCompany
-from metagpt.roles import ProjectManager, ProductManager, Architect, Engineer
-
-async def startup(idea: str, investment: float = 3.0, n_round: int = 5):
- """运行一个创业公司。做一个老板"""
- company = SoftwareCompany()
- company.hire([ProductManager(), Architect(), ProjectManager(), Engineer()])
- company.invest(investment)
- company.start_project(idea)
- await company.run(n_round=n_round)
-```
-
-你可以查看`examples`，其中有单角色（带知识库）的使用例子与仅LLM的使用例子。
-
-## 快速体验
-对一些用户来说，安装配置本地环境是有困难的，下面这些教程能够让你快速体验到MetaGPT的魅力。
-
-- [MetaGPT快速体验](https://deepwisdom.feishu.cn/wiki/Q8ycw6J9tiNXdHk66MRcIN8Pnlg)
-
-可直接在Huggingface Space体验
-
-- https://huggingface.co/spaces/deepwisdom/MetaGPT
-
-## 联系信息
+### 联系信息
如果您对这个项目有任何问题或反馈，欢迎联系我们。我们非常欢迎您的建议！
@@ -222,13 +119,17 @@ ## 联系信息
我们会在2-3个工作日内回复所有问题。
-## 演示
+## 引用
-https://github.com/geekan/MetaGPT/assets/2707039/5e8c1062-8c35-440f-bb20-2b0320f8d27d
+引用 [arXiv paper](https://arxiv.org/abs/2308.00352):
-## 加入我们
-
-📢 加入我们的Discord频道！
-https://discord.gg/ZRHeExS6xv
-
-期待在那里与您相见！🎉
+```bibtex
+@misc{hong2023metagpt,
+ title={MetaGPT: Meta Programming for Multi-Agent Collaborative Framework},
+ author={Sirui Hong and Xiawu Zheng and Jonathan Chen and Yuheng Cheng and Jinlin Wang and Ceyao Zhang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu},
+ year={2023},
+ eprint={2308.00352},
+ archivePrefix={arXiv},
+ primaryClass={cs.AI}
+}
+```
diff --git a/docs/README_JA.md b/docs/README_JA.md
index 2b2c35a62..411d190b4 100644
--- a/docs/README_JA.md
+++ b/docs/README_JA.md
@@ -19,7 +19,6 @@ # MetaGPT: マルチエージェントフレームワーク
- 
@@ -60,17 +59,22 @@ ### インストールビデオガイド
### 伝統的なインストール
```bash
-# ステップ 1: NPM がシステムにインストールされていることを確認してください。次に mermaid-js をインストールします。(お使いのコンピューターに npm がない場合は、Node.js 公式サイトで Node.js https://nodejs.org/ をインストールしてください。）
-npm --version
-sudo npm install -g @mermaid-js/mermaid-cli
-
-# ステップ 2: Python 3.9+ がシステムにインストールされていることを確認してください。これを確認するには:
+# ステップ 1: Python 3.9+ がシステムにインストールされていることを確認してください。これを確認するには:
python --version
-# ステップ 3: リポジトリをローカルマシンにクローンし、インストールする。
-git clone https://github.com/geekan/metagpt
-cd metagpt
+# ステップ 2: リポジトリをローカルマシンにクローンし、インストールする。
+git clone https://github.com/geekan/MetaGPT.git
+cd MetaGPT
pip install -e.
+
+# ステップ 3: startup.py を実行する
+# config.yaml を key.yaml にコピーし、独自の OPENAI_API_KEY を設定します
+python3 startup.py "Write a cli snake game"
+
+# ステップ 4 [オプション]: 実行中に PRD ファイルなどのアーティファクトを保存する場合は、ステップ 3 の前にこのステップを実行できます。デフォルトでは、フレームワークには互換性があり、この手順を実行しなくてもプロセス全体を完了できます。
+# NPM がシステムにインストールされていることを確認してください。次に mermaid-js をインストールします。(お使いのコンピューターに npm がない場合は、Node.js 公式サイトで Node.js https://nodejs.org/ をインストールしてください。）
+npm --version
+sudo npm install -g @mermaid-js/mermaid-cli
```
**注:**
@@ -159,6 +163,7 @@ # ステップ 3: リポジトリをローカルマシンにクローンし、
注: この方法は pdf エクスポートに対応していません。
### Docker によるインストール
+> Windowsでは、"/opt/metagpt"をDockerが作成する権限を持つディレクトリに置き換える必要があります。例えば、"D:\Users\x\metagpt"などです。
```bash
# ステップ 1: metagpt 公式イメージをダウンロードし、config.yaml を準備する
@@ -270,12 +275,12 @@ ### 使用方法
### コードウォークスルー
```python
-from metagpt.software_company import SoftwareCompany
+from metagpt.team import Team
from metagpt.roles import ProjectManager, ProductManager, Architect, Engineer
async def startup(idea: str, investment: float = 3.0, n_round: int = 5):
"""スタートアップを実行する。ボスになる。"""
- company = SoftwareCompany()
+ company = Team()
company.hire([ProductManager(), Architect(), ProjectManager(), Engineer()])
company.invest(investment)
company.start_project(idea)
@@ -295,12 +300,12 @@ ## クイックスタート
## 引用
-現時点では、[Arxiv 論文](https://arxiv.org/abs/2308.00352)を引用してください:
+現時点では、[arXiv 論文](https://arxiv.org/abs/2308.00352)を引用してください:
```bibtex
@misc{hong2023metagpt,
- title={MetaGPT: Meta Programming for Multi-Agent Collaborative Framework},
- author={Sirui Hong and Xiawu Zheng and Jonathan Chen and Yuheng Cheng and Jinlin Wang and Ceyao Zhang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu},
+ title={MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework},
+ author={Sirui Hong and Mingchen Zhuge and Jonathan Chen and Xiawu Zheng and Yuheng Cheng and Ceyao Zhang and Jinlin Wang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu and Jürgen Schmidhuber},
year={2023},
eprint={2308.00352},
archivePrefix={arXiv},
diff --git a/docs/install/cli_install.md b/docs/install/cli_install.md
new file mode 100644
index 000000000..80deda771
--- /dev/null
+++ b/docs/install/cli_install.md
@@ -0,0 +1,109 @@
+## Traditional Command Line Installation
+
+### Support System and version
+| System Version | Python Version | Supported |
+| ---- | ---- | ----- |
+| macOS 13.x | python 3.9 | Yes |
+| Windows 11 | python 3.9 | Yes |
+| Ubuntu 22.04 | python 3.9 | Yes |
+
+### Detail Installation
+```bash
+# Step 1: Ensure that NPM is installed on your system. Then install mermaid-js. (If you don't have npm in your computer, please go to the Node.js official website to install Node.js https://nodejs.org/ and then you will have npm tool in your computer.)
+npm --version
+sudo npm install -g @mermaid-js/mermaid-cli
+
+# Step 2: Ensure that Python 3.9+ is installed on your system. You can check this by using:
+python3 --version
+
+# Step 3: Clone the repository to your local machine, and install it.
+git clone https://github.com/geekan/MetaGPT.git
+cd MetaGPT
+pip install -e.
+```
+
+**Note:**
+
+- If already have Chrome, Chromium, or MS Edge installed, you can skip downloading Chromium by setting the environment variable
+ `PUPPETEER_SKIP_CHROMIUM_DOWNLOAD` to `true`.
+
+- Some people are [having issues](https://github.com/mermaidjs/mermaid.cli/issues/15) installing this tool globally. Installing it locally is an alternative solution,
+
+ ```bash
+ npm install @mermaid-js/mermaid-cli
+ ```
+
+- don't forget to the configuration for mmdc in config.yml
+
+ ```yml
+ PUPPETEER_CONFIG: "./config/puppeteer-config.json"
+ MMDC: "./node_modules/.bin/mmdc"
+ ```
+
+- if `pip install -e.` fails with error `[Errno 13] Permission denied: '/usr/local/lib/python3.11/dist-packages/test-easy-install-13129.write-test'`, try instead running `pip install -e. --user`
+
+- To convert Mermaid charts to SVG, PNG, and PDF formats. In addition to the Node.js version of Mermaid-CLI, you now have the option to use Python version Playwright, pyppeteer or mermaid.ink for this task.
+
+ - Playwright
+ - **Install Playwright**
+
+ ```bash
+ pip install playwright
+ ```
+
+ - **Install the Required Browsers**
+
+ to support PDF conversion, please install Chrominum.
+
+ ```bash
+ playwright install --with-deps chromium
+ ```
+
+ - **modify `config.yaml`**
+
+ uncomment MERMAID_ENGINE from config.yaml and change it to `playwright`
+
+ ```yaml
+ MERMAID_ENGINE: playwright
+ ```
+
+ - pyppeteer
+ - **Install pyppeteer**
+
+ ```bash
+ pip install pyppeteer
+ ```
+
+ - **Use your own Browsers**
+
+ pyppeteer allows you use installed browsers, please set the following envirment
+
+ ```bash
+ export PUPPETEER_EXECUTABLE_PATH = /path/to/your/chromium or edge or chrome
+ ```
+
+ please do not use this command to install browser, it is too old
+
+ ```bash
+ pyppeteer-install
+ ```
+
+ - **modify `config.yaml`**
+
+ uncomment MERMAID_ENGINE from config.yaml and change it to `pyppeteer`
+
+ ```yaml
+ MERMAID_ENGINE: pyppeteer
+ ```
+
+ - mermaid.ink
+ - **modify `config.yaml`**
+
+ uncomment MERMAID_ENGINE from config.yaml and change it to `ink`
+
+ ```yaml
+ MERMAID_ENGINE: ink
+ ```
+
+ Note: this method does not support pdf export.
+
\ No newline at end of file
diff --git a/docs/install/cli_install_cn.md b/docs/install/cli_install_cn.md
new file mode 100644
index 000000000..f351090ed
--- /dev/null
+++ b/docs/install/cli_install_cn.md
@@ -0,0 +1,43 @@
+## 命令行安装
+
+### 支持的系统和版本
+| 系统版本 | Python 版本 | 是否支持 |
+| ---- | ---- | ----- |
+| macOS 13.x | python 3.9 | 是 |
+| Windows 11 | python 3.9 | 是 |
+| Ubuntu 22.04 | python 3.9 | 是 |
+
+### 详细安装
+
+```bash
+# 第 1 步：确保您的系统上安装了 NPM。并使用npm安装mermaid-js
+npm --version
+sudo npm install -g @mermaid-js/mermaid-cli
+
+# 第 2 步：确保您的系统上安装了 Python 3.9+。您可以使用以下命令进行检查：
+python --version
+
+# 第 3 步：克隆仓库到您的本地机器，并进行安装。
+git clone https://github.com/geekan/MetaGPT.git
+cd MetaGPT
+pip install -e.
+```
+
+**注意：**
+
+- 如果已经安装了Chrome、Chromium或MS Edge，可以通过将环境变量`PUPPETEER_SKIP_CHROMIUM_DOWNLOAD`设置为`true`来跳过下载Chromium。
+
+- 一些人在全局安装此工具时遇到问题。在本地安装是替代解决方案，
+
+ ```bash
+ npm install @mermaid-js/mermaid-cli
+ ```
+
+- 不要忘记在config.yml中为mmdc配置配置，
+
+ ```yml
+ PUPPETEER_CONFIG: "./config/puppeteer-config.json"
+ MMDC: "./node_modules/.bin/mmdc"
+ ```
+
+- 如果`pip install -e.`失败并显示错误`[Errno 13] Permission denied: '/usr/local/lib/python3.11/dist-packages/test-easy-install-13129.write-test'`，请尝试使用`pip install -e. --user`运行。
diff --git a/docs/install/docker_install.md b/docs/install/docker_install.md
new file mode 100644
index 000000000..b803a5dae
--- /dev/null
+++ b/docs/install/docker_install.md
@@ -0,0 +1,44 @@
+## Docker Installation
+
+### Use default MetaGPT image
+
+```bash
+# Step 1: Download metagpt official image and prepare config.yaml
+docker pull metagpt/metagpt:latest
+mkdir -p /opt/metagpt/{config,workspace}
+docker run --rm metagpt/metagpt:latest cat /app/metagpt/config/config.yaml > /opt/metagpt/config/key.yaml
+vim /opt/metagpt/config/key.yaml # Change the config
+
+# Step 2: Run metagpt demo with container
+docker run --rm \
+ --privileged \
+ -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
+ -v /opt/metagpt/workspace:/app/metagpt/workspace \
+ metagpt/metagpt:latest \
+ python3 startup.py "Write a cli snake game"
+
+# You can also start a container and execute commands in it
+docker run --name metagpt -d \
+ --privileged \
+ -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
+ -v /opt/metagpt/workspace:/app/metagpt/workspace \
+ metagpt/metagpt:latest
+
+docker exec -it metagpt /bin/bash
+$ python3 startup.py "Write a cli snake game"
+```
+
+The command `docker run ...` do the following things:
+
+- Run in privileged mode to have permission to run the browser
+- Map host configure file `/opt/metagpt/config/key.yaml` to container `/app/metagpt/config/key.yaml`
+- Map host directory `/opt/metagpt/workspace` to container `/app/metagpt/workspace`
+- Execute the demo command `python3 startup.py "Write a cli snake game"`
+
+### Build image by yourself
+
+```bash
+# You can also build metagpt image by yourself.
+git clone https://github.com/geekan/MetaGPT.git
+cd MetaGPT && docker build -t metagpt:custom .
+```
diff --git a/docs/install/docker_install_cn.md b/docs/install/docker_install_cn.md
new file mode 100644
index 000000000..347fae10c
--- /dev/null
+++ b/docs/install/docker_install_cn.md
@@ -0,0 +1,44 @@
+## Docker安装
+
+### 使用MetaGPT镜像
+
+```bash
+# 步骤1: 下载metagpt官方镜像并准备好config.yaml
+docker pull metagpt/metagpt:latest
+mkdir -p /opt/metagpt/{config,workspace}
+docker run --rm metagpt/metagpt:latest cat /app/metagpt/config/config.yaml > /opt/metagpt/config/key.yaml
+vim /opt/metagpt/config/key.yaml # 修改配置文件
+
+# 步骤2: 使用容器运行metagpt演示
+docker run --rm \
+ --privileged \
+ -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
+ -v /opt/metagpt/workspace:/app/metagpt/workspace \
+ metagpt/metagpt:latest \
+ python startup.py "Write a cli snake game"
+
+# 您也可以启动一个容器并在其中执行命令
+docker run --name metagpt -d \
+ --privileged \
+ -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
+ -v /opt/metagpt/workspace:/app/metagpt/workspace \
+ metagpt/metagpt:latest
+
+docker exec -it metagpt /bin/bash
+$ python startup.py "Write a cli snake game"
+```
+
+`docker run ...`做了以下事情:
+
+- 以特权模式运行，有权限运行浏览器
+- 将主机文件 `/opt/metagpt/config/key.yaml` 映射到容器文件 `/app/metagpt/config/key.yaml`
+- 将主机目录 `/opt/metagpt/workspace` 映射到容器目录 `/app/metagpt/workspace`
+- 执行示例命令 `python startup.py "Write a cli snake game"`
+
+### 自己构建镜像
+
+```bash
+# 您也可以自己构建metagpt镜像
+git clone https://github.com/geekan/MetaGPT.git
+cd MetaGPT && docker build -t metagpt:custom .
+```
diff --git a/docs/tutorial/usage.md b/docs/tutorial/usage.md
new file mode 100644
index 000000000..ee87b65c9
--- /dev/null
+++ b/docs/tutorial/usage.md
@@ -0,0 +1,67 @@
+## MetaGPT Usage
+
+### Configuration
+
+- Configure your `OPENAI_API_KEY` in any of `config/key.yaml / config/config.yaml / env`
+- Priority order: `config/key.yaml > config/config.yaml > env`
+
+```bash
+# Copy the configuration file and make the necessary modifications.
+cp config/config.yaml config/key.yaml
+```
+
+| Variable Name | config/key.yaml | env |
+| ------------------------------------------ | ----------------------------------------- | ----------------------------------------------- |
+| OPENAI_API_KEY # Replace with your own key | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." |
+| OPENAI_API_BASE # Optional | OPENAI_API_BASE: "https:///v1" | export OPENAI_API_BASE="https:///v1" |
+
+### Initiating a startup
+
+```shell
+# Run the script
+python startup.py "Write a cli snake game"
+# Do not hire an engineer to implement the project
+python startup.py "Write a cli snake game" --implement False
+# Hire an engineer and perform code reviews
+python startup.py "Write a cli snake game" --code_review True
+```
+
+After running the script, you can find your new project in the `workspace/` directory.
+
+### Preference of Platform or Tool
+
+You can tell which platform or tool you want to use when stating your requirements.
+
+```shell
+python startup.py "Write a cli snake game based on pygame"
+```
+
+### Usage
+
+```
+NAME
+ startup.py - We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities.
+
+SYNOPSIS
+ startup.py IDEA
+
+DESCRIPTION
+ We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities.
+
+POSITIONAL ARGUMENTS
+ IDEA
+ Type: str
+ Your innovative idea, such as "Creating a snake game."
+
+FLAGS
+ --investment=INVESTMENT
+ Type: float
+ Default: 3.0
+ As an investor, you have the opportunity to contribute a certain dollar amount to this AI company.
+ --n_round=N_ROUND
+ Type: int
+ Default: 5
+
+NOTES
+ You can also use flags syntax for POSITIONAL ARGUMENTS
+```
\ No newline at end of file
diff --git a/docs/tutorial/usage_cn.md b/docs/tutorial/usage_cn.md
new file mode 100644
index 000000000..4b3bdd2c3
--- /dev/null
+++ b/docs/tutorial/usage_cn.md
@@ -0,0 +1,63 @@
+## MetaGPT 使用
+
+### 配置
+
+- 在 `config/key.yaml / config/config.yaml / env` 中配置您的 `OPENAI_API_KEY`
+- 优先级顺序：`config/key.yaml > config/config.yaml > env`
+
+```bash
+# 复制配置文件并进行必要的修改
+cp config/config.yaml config/key.yaml
+```
+
+| 变量名 | config/key.yaml | env |
+| ----------------------------------- | ----------------------------------------- | ----------------------------------------------- |
+| OPENAI_API_KEY # 用您自己的密钥替换 | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." |
+| OPENAI_API_BASE # 可选 | OPENAI_API_BASE: "https:///v1" | export OPENAI_API_BASE="https:///v1" |
+
+### 示例：启动一个创业公司
+
+```shell
+python startup.py "写一个命令行贪吃蛇"
+# 开启code review模式会花费更多的金钱, 但是会提升代码质量和成功率
+python startup.py "写一个命令行贪吃蛇" --code_review True
+```
+
+运行脚本后，您可以在 `workspace/` 目录中找到您的新项目。
+
+### 平台或工具的倾向性
+可以在阐述需求时说明想要使用的平台或工具。
+例如：
+```shell
+python startup.py "写一个基于pygame的命令行贪吃蛇"
+```
+
+### 使用
+
+```
+名称
+ startup.py - 我们是一家AI软件创业公司。通过投资我们，您将赋能一个充满无限可能的未来。
+
+概要
+ startup.py IDEA
+
+描述
+ 我们是一家AI软件创业公司。通过投资我们，您将赋能一个充满无限可能的未来。
+
+位置参数
+ IDEA
+ 类型: str
+ 您的创新想法，例如"写一个命令行贪吃蛇。"
+
+标志
+ --investment=INVESTMENT
+ 类型: float
+ 默认值: 3.0
+ 作为投资者，您有机会向这家AI公司投入一定的美元金额。
+ --n_round=N_ROUND
+ 类型: int
+ 默认值: 5
+
+备注
+ 您也可以用`标志`的语法，来处理`位置参数`
+```
diff --git a/examples/agent_creator.py b/examples/agent_creator.py
index 3618c0608..e724105a3 100644
--- a/examples/agent_creator.py
+++ b/examples/agent_creator.py
@@ -6,12 +6,13 @@ Author: garylin2099
import re
from metagpt.actions import Action
-from metagpt.const import PROJECT_ROOT, WORKSPACE_ROOT
+from metagpt.config import CONFIG
+from metagpt.const import METAGPT_ROOT
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
-with open(PROJECT_ROOT / "examples/build_customized_agent.py", "r") as f:
+with open(METAGPT_ROOT / "examples/build_customized_agent.py", "r") as f:
# use official example script to guide AgentCreator
MULTI_ACTION_AGENT_CODE_EXAMPLE = f.read()
@@ -48,7 +49,7 @@ class CreateAgent(Action):
pattern = r"```python(.*)```"
match = re.search(pattern, rsp, re.DOTALL)
code_text = match.group(1) if match else ""
- with open(WORKSPACE_ROOT / "agent_created_agent.py", "w") as f:
+ with open(CONFIG.workspace_path / "agent_created_agent.py", "w") as f:
f.write(code_text)
return code_text
diff --git a/examples/build_customized_agent.py b/examples/build_customized_agent.py
index ef274be8b..6805fd460 100644
--- a/examples/build_customized_agent.py
+++ b/examples/build_customized_agent.py
@@ -10,6 +10,7 @@ import subprocess
import fire
from metagpt.actions import Action
+from metagpt.llm import LLM
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
@@ -19,19 +20,10 @@ class SimpleWriteCode(Action):
PROMPT_TEMPLATE = """
Write a python function that can {instruction} and provide two runnnable test cases.
Return ```python your_code_here ``` with NO other texts,
- example:
- ```python
- # function
- def add(a, b):
- return a + b
- # test cases
- print(add(1, 2))
- print(add(3, 4))
- ```
your code:
"""
- def __init__(self, name="SimpleWriteCode", context=None, llm=None):
+ def __init__(self, name: str = "SimpleWriteCode", context=None, llm: LLM = None):
super().__init__(name, context, llm)
async def run(self, instruction: str):
@@ -52,7 +44,7 @@ class SimpleWriteCode(Action):
class SimpleRunCode(Action):
- def __init__(self, name="SimpleRunCode", context=None, llm=None):
+ def __init__(self, name: str = "SimpleRunCode", context=None, llm: LLM = None):
super().__init__(name, context, llm)
async def run(self, code_text: str):
@@ -74,13 +66,11 @@ class SimpleCoder(Role):
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
- todo = self._rc.todo
+ todo = self._rc.todo # todo will be SimpleWriteCode()
- msg = self._rc.memory.get()[-1] # retrieve the latest memory
- instruction = msg.content
-
- code_text = await SimpleWriteCode().run(instruction)
- msg = Message(content=code_text, role=self.profile, cause_by=todo)
+ msg = self.get_memories(k=1)[0] # find the most recent messages
+ code_text = await todo.run(msg.content)
+ msg = Message(content=code_text, role=self.profile, cause_by=type(todo))
return msg
@@ -94,44 +84,23 @@ class RunnableCoder(Role):
):
super().__init__(name, profile, **kwargs)
self._init_actions([SimpleWriteCode, SimpleRunCode])
-
- async def _think(self) -> None:
- if self._rc.todo is None:
- self._set_state(0)
- return
-
- if self._rc.state + 1 < len(self._states):
- self._set_state(self._rc.state + 1)
- else:
- self._rc.todo = None
+ self._set_react_mode(react_mode="by_order")
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
+ # By choosing the Action by order under the hood
+ # todo will be first SimpleWriteCode() then SimpleRunCode()
todo = self._rc.todo
- msg = self._rc.memory.get()[-1]
- if isinstance(todo, SimpleWriteCode):
- instruction = msg.content
- result = await SimpleWriteCode().run(instruction)
+ msg = self.get_memories(k=1)[0] # find the most k recent messages
+ result = await todo.run(msg.content)
- elif isinstance(todo, SimpleRunCode):
- code_text = msg.content
- result = await SimpleRunCode().run(code_text)
-
- msg = Message(content=result, role=self.profile, cause_by=todo)
+ msg = Message(content=result, role=self.profile, cause_by=type(todo))
self._rc.memory.add(msg)
return msg
- async def _react(self) -> Message:
- while True:
- await self._think()
- if self._rc.todo is None:
- break
- await self._act()
- return Message(content="All job done", role=self.profile)
-
-def main(msg="write a function that calculates the sum of a list"):
+def main(msg="write a function that calculates the product of a list and run it"):
# role = SimpleCoder()
role = RunnableCoder()
logger.info(msg)
diff --git a/examples/build_customized_multi_agents.py b/examples/build_customized_multi_agents.py
new file mode 100644
index 000000000..030a4b339
--- /dev/null
+++ b/examples/build_customized_multi_agents.py
@@ -0,0 +1,155 @@
+"""
+Filename: MetaGPT/examples/build_customized_multi_agents.py
+Created Date: Wednesday, November 15th 2023, 7:12:39 pm
+Author: garylin2099
+"""
+import re
+
+import fire
+
+from metagpt.actions import Action, UserRequirement
+from metagpt.llm import LLM
+from metagpt.logs import logger
+from metagpt.roles import Role
+from metagpt.schema import Message
+from metagpt.team import Team
+
+
+def parse_code(rsp):
+ pattern = r"```python(.*)```"
+ match = re.search(pattern, rsp, re.DOTALL)
+ code_text = match.group(1) if match else rsp
+ return code_text
+
+
+class SimpleWriteCode(Action):
+ PROMPT_TEMPLATE = """
+ Write a python function that can {instruction}.
+ Return ```python your_code_here ``` with NO other texts,
+ your code:
+ """
+
+ def __init__(self, name: str = "SimpleWriteCode", context=None, llm: LLM = None):
+ super().__init__(name, context, llm)
+
+ async def run(self, instruction: str):
+ prompt = self.PROMPT_TEMPLATE.format(instruction=instruction)
+
+ rsp = await self._aask(prompt)
+
+ code_text = parse_code(rsp)
+
+ return code_text
+
+
+class SimpleCoder(Role):
+ def __init__(
+ self,
+ name: str = "Alice",
+ profile: str = "SimpleCoder",
+ **kwargs,
+ ):
+ super().__init__(name, profile, **kwargs)
+ self._watch([UserRequirement])
+ self._init_actions([SimpleWriteCode])
+
+
+class SimpleWriteTest(Action):
+ PROMPT_TEMPLATE = """
+ Context: {context}
+ Write {k} unit tests using pytest for the given function, assuming you have imported it.
+ Return ```python your_code_here ``` with NO other texts,
+ your code:
+ """
+
+ def __init__(self, name: str = "SimpleWriteTest", context=None, llm: LLM = None):
+ super().__init__(name, context, llm)
+
+ async def run(self, context: str, k: int = 3):
+ prompt = self.PROMPT_TEMPLATE.format(context=context, k=k)
+
+ rsp = await self._aask(prompt)
+
+ code_text = parse_code(rsp)
+
+ return code_text
+
+
+class SimpleTester(Role):
+ def __init__(
+ self,
+ name: str = "Bob",
+ profile: str = "SimpleTester",
+ **kwargs,
+ ):
+ super().__init__(name, profile, **kwargs)
+ self._init_actions([SimpleWriteTest])
+ # self._watch([SimpleWriteCode])
+ self._watch([SimpleWriteCode, SimpleWriteReview]) # feel free to try this too
+
+ async def _act(self) -> Message:
+ logger.info(f"{self._setting}: ready to {self._rc.todo}")
+ todo = self._rc.todo
+
+ # context = self.get_memories(k=1)[0].content # use the most recent memory as context
+ context = self.get_memories() # use all memories as context
+
+ code_text = await todo.run(context, k=5) # specify arguments
+ msg = Message(content=code_text, role=self.profile, cause_by=type(todo))
+
+ return msg
+
+
+class SimpleWriteReview(Action):
+ PROMPT_TEMPLATE = """
+ Context: {context}
+ Review the test cases and provide one critical comments:
+ """
+
+ def __init__(self, name: str = "SimpleWriteReview", context=None, llm: LLM = None):
+ super().__init__(name, context, llm)
+
+ async def run(self, context: str):
+ prompt = self.PROMPT_TEMPLATE.format(context=context)
+
+ rsp = await self._aask(prompt)
+
+ return rsp
+
+
+class SimpleReviewer(Role):
+ def __init__(
+ self,
+ name: str = "Charlie",
+ profile: str = "SimpleReviewer",
+ **kwargs,
+ ):
+ super().__init__(name, profile, **kwargs)
+ self._init_actions([SimpleWriteReview])
+ self._watch([SimpleWriteTest])
+
+
+async def main(
+ idea: str = "write a function that calculates the product of a list",
+ investment: float = 3.0,
+ n_round: int = 5,
+ add_human: bool = False,
+):
+ logger.info(idea)
+
+ team = Team()
+ team.hire(
+ [
+ SimpleCoder(),
+ SimpleTester(),
+ SimpleReviewer(is_human=add_human),
+ ]
+ )
+
+ team.invest(investment=investment)
+ team.start_project(idea)
+ await team.run(n_round=n_round)
+
+
+if __name__ == "__main__":
+ fire.Fire(main)
diff --git a/examples/debate.py b/examples/debate.py
index 44602a11a..52f49e00e 100644
--- a/examples/debate.py
+++ b/examples/debate.py
@@ -10,17 +10,15 @@ import platform
import fire
-from metagpt.actions import Action, BossRequirement
+from metagpt.actions import Action, UserRequirement
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
-from metagpt.software_company import SoftwareCompany
-
-from metagpt.utils.common import any_to_str
+from metagpt.team import Team
-class ShoutOut(Action):
- """Action: Shout out loudly in a debate (quarrel)"""
+class SpeakAloud(Action):
+ """Action: Speak out aloud in a debate (quarrel)"""
PROMPT_TEMPLATE = """
## BACKGROUND
@@ -33,7 +31,7 @@ class ShoutOut(Action):
craft a strong and emotional response in 80 words, in {name}'s rhetoric and viewpoints, your will argue:
"""
- def __init__(self, name="ShoutOut", context=None, llm=None):
+ def __init__(self, name="SpeakAloud", context=None, llm=None):
super().__init__(name, context, llm)
async def run(self, context: str, name: str, opponent_name: str):
@@ -45,102 +43,56 @@ class ShoutOut(Action):
return rsp
-class Trump(Role):
+class Debator(Role):
def __init__(
self,
- name: str = "Trump",
- profile: str = "Republican",
+ name: str,
+ profile: str,
+ opponent_name: str,
**kwargs,
):
super().__init__(name, profile, **kwargs)
- self._init_actions([ShoutOut])
- self._watch([ShoutOut])
- self.name = "Trump"
- self.opponent_name = "Biden"
+ self._init_actions([SpeakAloud])
+ self._watch([UserRequirement, SpeakAloud])
+ self.opponent_name = opponent_name
async def _observe(self) -> int:
await super()._observe()
# accept messages sent (from opponent) to self, disregard own messages from the last round
-
self._rc.news = [msg for msg in self._rc.news if msg.send_to == {self.name}]
return len(self._rc.news)
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
+ todo = self._rc.todo # An instance of SpeakAloud
- msg_history = self._rc.memory.get_by_actions([ShoutOut])
- context = []
- for m in msg_history:
- context.append(str(m))
- context = "\n".join(context)
+ memories = self.get_memories()
+ context = "\n".join(f"{msg.sent_from}: {msg.content}" for msg in memories)
+ # print(context)
- rsp = await ShoutOut().run(context=context, name=self.name, opponent_name=self.opponent_name)
+ rsp = await todo.run(context=context, name=self.name, opponent_name=self.opponent_name)
msg = Message(
content=rsp,
role=self.profile,
- cause_by=ShoutOut,
+ cause_by=type(todo),
sent_from=self.name,
send_to=self.opponent_name,
)
+ self._rc.memory.add(msg)
return msg
-class Biden(Role):
- def __init__(
- self,
- name: str = "Biden",
- profile: str = "Democrat",
- **kwargs,
- ):
- super().__init__(name, profile, **kwargs)
- self._init_actions([ShoutOut])
- self._watch([BossRequirement, ShoutOut])
- self.name = "Biden"
- self.opponent_name = "Trump"
-
- async def _observe(self) -> int:
- await super()._observe()
- # accept the very first human instruction (the debate topic) or messages sent (from opponent) to self,
- # disregard own messages from the last round
- self._rc.news = [
- msg for msg in self._rc.news if msg.cause_by == any_to_str(BossRequirement) or msg.send_to == {self.name}
- ]
- return len(self._rc.news)
-
- async def _act(self) -> Message:
- logger.info(f"{self._setting}: ready to {self._rc.todo}")
-
- msg_history = self._rc.memory.get_by_actions([BossRequirement, ShoutOut])
- context = []
- for m in msg_history:
- context.append(str(m))
- context = "\n".join(context)
-
- rsp = await ShoutOut().run(context=context, name=self.name, opponent_name=self.opponent_name)
-
- msg = Message(
- content=rsp,
- role=self.profile,
- cause_by=ShoutOut,
- sent_from=self.name,
- send_to=self.opponent_name,
- )
-
- return msg
-
-
-async def startup(
- idea: str, investment: float = 3.0, n_round: int = 5, code_review: bool = False, run_tests: bool = False
-):
- """We reuse the startup paradigm for roles to interact with each other.
- Now we run a startup of presidents and watch they quarrel. :)"""
- company = SoftwareCompany()
- company.hire([Biden(), Trump()])
- company.invest(investment)
- company.start_project(idea)
- await company.run(n_round=n_round)
+async def debate(idea: str, investment: float = 3.0, n_round: int = 5):
+ """Run a team of presidents and watch they quarrel. :)"""
+ Biden = Debator(name="Biden", profile="Democrat", opponent_name="Trump")
+ Trump = Debator(name="Trump", profile="Republican", opponent_name="Biden")
+ team = Team()
+ team.hire([Biden, Trump])
+ team.invest(investment)
+ team.run_project(idea, send_to="Biden") # send debate topic to Biden and let him speak first
+ await team.run(n_round=n_round)
def main(idea: str, investment: float = 3.0, n_round: int = 10):
@@ -153,7 +105,7 @@ def main(idea: str, investment: float = 3.0, n_round: int = 10):
"""
if platform.system() == "Windows":
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
- asyncio.run(startup(idea, investment, n_round))
+ asyncio.run(debate(idea, investment, n_round))
if __name__ == "__main__":
diff --git a/examples/sk_agent.py b/examples/sk_agent.py
index a7513e838..647ea4380 100644
--- a/examples/sk_agent.py
+++ b/examples/sk_agent.py
@@ -13,7 +13,7 @@ from semantic_kernel.planning import SequentialPlanner
# from semantic_kernel.planning import SequentialPlanner
from semantic_kernel.planning.action_planner.action_planner import ActionPlanner
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.const import SKILL_DIRECTORY
from metagpt.roles.sk_agent import SkAgent
from metagpt.schema import Message
@@ -39,7 +39,7 @@ async def basic_planner_example():
role.import_semantic_skill_from_directory(SKILL_DIRECTORY, "WriterSkill")
role.import_skill(TextSkill(), "TextSkill")
# using BasicPlanner
- await role.run(Message(content=task, cause_by=BossRequirement))
+ await role.run(Message(content=task, cause_by=UserRequirement))
async def sequential_planner_example():
@@ -53,7 +53,7 @@ async def sequential_planner_example():
role.import_semantic_skill_from_directory(SKILL_DIRECTORY, "WriterSkill")
role.import_skill(TextSkill(), "TextSkill")
# using BasicPlanner
- await role.run(Message(content=task, cause_by=BossRequirement))
+ await role.run(Message(content=task, cause_by=UserRequirement))
async def basic_planner_web_search_example():
@@ -64,7 +64,7 @@ async def basic_planner_web_search_example():
role.import_skill(SkSearchEngine(), "WebSearchSkill")
# role.import_semantic_skill_from_directory(skills_directory, "QASkill")
- await role.run(Message(content=task, cause_by=BossRequirement))
+ await role.run(Message(content=task, cause_by=UserRequirement))
async def action_planner_example():
@@ -75,7 +75,7 @@ async def action_planner_example():
role.import_skill(TimeSkill(), "time")
role.import_skill(TextSkill(), "text")
task = "What is the sum of 110 and 990?"
- await role.run(Message(content=task, cause_by=BossRequirement)) # it will choose mathskill.Add
+ await role.run(Message(content=task, cause_by=UserRequirement)) # it will choose mathskill.Add
if __name__ == "__main__":
diff --git a/metagpt/actions/__init__.py b/metagpt/actions/__init__.py
index b004bd58e..79ff94b3e 100644
--- a/metagpt/actions/__init__.py
+++ b/metagpt/actions/__init__.py
@@ -9,7 +9,7 @@ from enum import Enum
from metagpt.actions.action import Action
from metagpt.actions.action_output import ActionOutput
-from metagpt.actions.add_requirement import BossRequirement
+from metagpt.actions.add_requirement import UserRequirement
from metagpt.actions.debug_error import DebugError
from metagpt.actions.design_api import WriteDesign
from metagpt.actions.design_api_review import DesignReview
@@ -28,7 +28,7 @@ from metagpt.actions.write_test import WriteTest
class ActionType(Enum):
"""All types of Actions, used for indexing."""
- ADD_REQUIREMENT = BossRequirement
+ ADD_REQUIREMENT = UserRequirement
WRITE_PRD = WritePRD
WRITE_PRD_REVIEW = WritePRDReview
WRITE_DESIGN = WriteDesign
diff --git a/metagpt/actions/action.py b/metagpt/actions/action.py
index 790295d55..f8016b8a2 100644
--- a/metagpt/actions/action.py
+++ b/metagpt/actions/action.py
@@ -30,6 +30,10 @@ class Action(ABC):
self.desc = ""
self.content = ""
self.instruct_content = None
+ self.env = None
+
+ def set_env(self, env):
+ self.env = env
def set_prefix(self, prefix, profile):
"""Set prefix for later usage"""
diff --git a/metagpt/actions/add_requirement.py b/metagpt/actions/add_requirement.py
index 16e14b3a4..d77d423ba 100644
--- a/metagpt/actions/add_requirement.py
+++ b/metagpt/actions/add_requirement.py
@@ -8,8 +8,8 @@
from metagpt.actions import Action
-class BossRequirement(Action):
- """Boss Requirement without any implementation details"""
+class UserRequirement(Action):
+ """User Requirement without any implementation details"""
async def run(self, *args, **kwargs):
raise NotImplementedError
diff --git a/metagpt/actions/debug_error.py b/metagpt/actions/debug_error.py
index 304b1bc3e..e66eadeaa 100644
--- a/metagpt/actions/debug_error.py
+++ b/metagpt/actions/debug_error.py
@@ -4,12 +4,19 @@
@Time : 2023/5/11 17:46
@Author : alexanderwu
@File : debug_error.py
+@Modified By: mashenquan, 2023/11/27.
+ 1. Divide the context into three components: legacy code, unit test code, and console log.
+ 2. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
"""
import re
from metagpt.actions.action import Action
+from metagpt.config import CONFIG
+from metagpt.const import TEST_CODES_FILE_REPO, TEST_OUTPUTS_FILE_REPO
from metagpt.logs import logger
+from metagpt.schema import RunCodeResult
from metagpt.utils.common import CodeParser
+from metagpt.utils.file_repository import FileRepository
PROMPT_TEMPLATE = """
NOTICE
@@ -19,7 +26,20 @@ Based on the message, first, figure out your own role, i.e. Engineer or QaEngine
then rewrite the development code or the test code based on your role, the error, and the summary, such that all bugs are fixed and the code performs well.
Attention: Use '##' to split sections, not '#', and '## ' SHOULD WRITE BEFORE the test case or script and triple quotes.
The message is as follows:
-{context}
+# Legacy Code
+```python
+{code}
+```
+---
+# Unit Test Code
+```python
+{test_code}
+```
+---
+# Console logs
+```text
+{logs}
+```
---
Now you should start rewriting the code:
## file name of the code to rewrite: Write code with triple quoto. Do your best to implement THIS IN ONLY ONE FILE.
@@ -30,24 +50,32 @@ class DebugError(Action):
def __init__(self, name="DebugError", context=None, llm=None):
super().__init__(name, context, llm)
- # async def run(self, code, error):
- # prompt = f"Here is a piece of Python code:\n\n{code}\n\nThe following error occurred during execution:" \
- # f"\n\n{error}\n\nPlease try to fix the error in this code."
- # fixed_code = await self._aask(prompt)
- # return fixed_code
+ async def run(self, *args, **kwargs) -> str:
+ output_doc = await FileRepository.get_file(
+ filename=self.context.output_filename, relative_path=TEST_OUTPUTS_FILE_REPO
+ )
+ if not output_doc:
+ return ""
+ output_detail = RunCodeResult.loads(output_doc.content)
+ pattern = r"Ran (\d+) tests in ([\d.]+)s\n\nOK"
+ matches = re.search(pattern, output_detail.stderr)
+ if matches:
+ return ""
- async def run(self, context):
- if "PASS" in context:
- return "", "the original code works fine, no need to debug"
-
- file_name = re.search("## File To Rewrite:\s*(.+\\.py)", context).group(1)
-
- logger.info(f"Debug and rewrite {file_name}")
-
- prompt = PROMPT_TEMPLATE.format(context=context)
+ logger.info(f"Debug and rewrite {self.context.code_filename}")
+ code_doc = await FileRepository.get_file(
+ filename=self.context.code_filename, relative_path=CONFIG.src_workspace
+ )
+ if not code_doc:
+ return ""
+ test_doc = await FileRepository.get_file(
+ filename=self.context.test_filename, relative_path=TEST_CODES_FILE_REPO
+ )
+ if not test_doc:
+ return ""
+ prompt = PROMPT_TEMPLATE.format(code=code_doc.content, test_code=test_doc.content, logs=output_detail.stderr)
rsp = await self._aask(prompt)
-
code = CodeParser.parse_code(block="", text=rsp)
- return file_name, code
+ return code
diff --git a/metagpt/actions/design_api.py b/metagpt/actions/design_api.py
index 75df8b909..c5787ba20 100644
--- a/metagpt/actions/design_api.py
+++ b/metagpt/actions/design_api.py
@@ -4,18 +4,28 @@
@Time : 2023/5/11 19:26
@Author : alexanderwu
@File : design_api.py
+@Modified By: mashenquan, 2023/11/27.
+ 1. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
+ 2. According to the design in Section 2.2.3.5.3 of RFC 135, add incremental iteration functionality.
"""
-import shutil
+import json
from pathlib import Path
from typing import List
from metagpt.actions import Action, ActionOutput
from metagpt.config import CONFIG
-from metagpt.const import WORKSPACE_ROOT
+from metagpt.const import (
+ DATA_API_DESIGN_FILE_REPO,
+ PRDS_FILE_REPO,
+ SEQ_FLOW_FILE_REPO,
+ SYSTEM_DESIGN_FILE_REPO,
+ SYSTEM_DESIGN_PDF_FILE_REPO,
+)
from metagpt.logs import logger
+from metagpt.schema import Document, Documents
from metagpt.utils.common import CodeParser
+from metagpt.utils.file_repository import FileRepository
from metagpt.utils.get_template import get_template
-from metagpt.utils.json_to_markdown import json_to_markdown
from metagpt.utils.mermaid import mermaid_to_file
templates = {
@@ -27,21 +37,21 @@ templates = {
## Format example
{format_example}
-----
-Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools
+Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
Requirement: Fill in the following missing information based on the context, each section name is a key in json
-Max Output: 8192 chars or 2048 tokens. Try to use them up.
-## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
+## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select appropriate open-source frameworks.
-## Python package name: Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
+## project_name: Provide as Plain text, concise and clear, characters only use a combination of all lowercase and underscores
-## File list: Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
+## File list: Provided as Python list[str], the list of files needed (including HTML & CSS IF NEEDED) to write the program. Only need relative paths. ALWAYS write a main.py or app.py here
-## Data structures and interface definitions: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
+## Data structures and interfaces: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
-## Anything UNCLEAR: Provide as Plain text. Make clear here.
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it.
output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like format example,
and only output the json inside this tag, nothing else
@@ -50,9 +60,9 @@ and only output the json inside this tag, nothing else
[CONTENT]
{
"Implementation approach": "We will ...",
- "Python package name": "snake_game",
+ "project_name": "snake_game",
"File list": ["main.py"],
- "Data structures and interface definitions": '
+ "Data structures and interfaces": '
classDiagram
class Game{
+int score
@@ -80,21 +90,21 @@ and only output the json inside this tag, nothing else
{format_example}
-----
Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
Requirement: Fill in the following missing information based on the context, note that all sections are response with code form separately
-Max Output: 8192 chars or 2048 tokens. Try to use them up.
-Attention: Use '##' to split sections, not '#', and '## ' SHOULD WRITE BEFORE the code and triple quote.
+ATTENTION: Output carefully referenced "Format example" in format.
## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
-## Python package name: Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
+## project_name: Provide as Plain text, concise and clear, characters only use a combination of all lowercase and underscores
-## File list: Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
+## File list: Provided as Python list[str], the list of code files (including HTML & CSS IF NEEDED) to write the program. Only need relative paths. ALWAYS write a main.py or app.py here
-## Data structures and interface definitions: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
+## Data structures and interfaces: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
-## Anything UNCLEAR: Provide as Plain text. Make clear here.
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it.
""",
"FORMAT_EXAMPLE": """
@@ -102,7 +112,7 @@ Attention: Use '##' to split sections, not '#', and '## ' SHOULD W
## Implementation approach
We will ...
-## Python package name
+## project_name
```python
"snake_game"
```
@@ -114,7 +124,7 @@ We will ...
]
```
-## Data structures and interface definitions
+## Data structures and interfaces
```mermaid
classDiagram
class Game{
@@ -141,13 +151,42 @@ The requirement is clear to me.
OUTPUT_MAPPING = {
"Implementation approach": (str, ...),
- "Python package name": (str, ...),
+ "project_name": (str, ...),
"File list": (List[str], ...),
- "Data structures and interface definitions": (str, ...),
+ "Data structures and interfaces": (str, ...),
"Program call flow": (str, ...),
"Anything UNCLEAR": (str, ...),
}
+MERGE_PROMPT = """
+## Old Design
+{old_design}
+
+## Context
+{context}
+
+-----
+Role: You are an architect; The goal is to incrementally update the "Old Design" based on the information provided by the "Context," aiming to design a SOTA PEP8-compliant python system; make the best use of good open source tools
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
+Requirement: Fill in the following missing information based on the context, note that all sections are response with code form separately
+ATTENTION: Output carefully referenced "Old Design" in format.
+
+## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
+
+## project_name: Provide as Plain text, concise and clear, characters only use a combination of all lowercase and underscores
+
+## File list: Provided as Python list[str], the list of code files (including HTML & CSS IF NEEDED) to write the program. Only need relative paths. ALWAYS write a main.py or app.py here
+
+## Data structures and interfaces: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
+
+## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
+
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it.
+
+output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like "Old Design" format,
+and only output the json inside this tag, nothing else
+"""
+
class WriteDesign(Action):
def __init__(self, name, context=None, llm=None):
@@ -158,60 +197,127 @@ class WriteDesign(Action):
"clearly and in detail."
)
- def recreate_workspace(self, workspace: Path):
- try:
- shutil.rmtree(workspace)
- except FileNotFoundError:
- pass # Folder does not exist, but we don't care
- workspace.mkdir(parents=True, exist_ok=True)
+ async def run(self, with_messages, format=CONFIG.prompt_format):
+ # Use `git diff` to identify which PRD documents have been modified in the `docs/prds` directory.
+ prds_file_repo = CONFIG.git_repo.new_file_repository(PRDS_FILE_REPO)
+ changed_prds = prds_file_repo.changed_files
+ # Use `git diff` to identify which design documents in the `docs/system_designs` directory have undergone
+ # changes.
+ system_design_file_repo = CONFIG.git_repo.new_file_repository(SYSTEM_DESIGN_FILE_REPO)
+ changed_system_designs = system_design_file_repo.changed_files
- async def _save_prd(self, docs_path, resources_path, context):
- prd_file = docs_path / "prd.md"
- if context[-1].instruct_content and context[-1].instruct_content.dict()["Competitive Quadrant Chart"]:
- quadrant_chart = context[-1].instruct_content.dict()["Competitive Quadrant Chart"]
- await mermaid_to_file(quadrant_chart, resources_path / "competitive_analysis")
+ # For those PRDs and design documents that have undergone changes, regenerate the design content.
+ changed_files = Documents()
+ for filename in changed_prds.keys():
+ doc = await self._update_system_design(
+ filename=filename, prds_file_repo=prds_file_repo, system_design_file_repo=system_design_file_repo
+ )
+ changed_files.docs[filename] = doc
- if context[-1].instruct_content:
- logger.info(f"Saving PRD to {prd_file}")
- prd_file.write_text(json_to_markdown(context[-1].instruct_content.dict()))
+ for filename in changed_system_designs.keys():
+ if filename in changed_files.docs:
+ continue
+ doc = await self._update_system_design(
+ filename=filename, prds_file_repo=prds_file_repo, system_design_file_repo=system_design_file_repo
+ )
+ changed_files.docs[filename] = doc
+ if not changed_files.docs:
+ logger.info("Nothing has changed.")
+ # Wait until all files under `docs/system_designs/` are processed before sending the publish message,
+ # leaving room for global optimization in subsequent steps.
+ return ActionOutput(content=changed_files.json(), instruct_content=changed_files)
- async def _save_system_design(self, docs_path, resources_path, system_design):
- data_api_design = system_design.instruct_content.dict()[
- "Data structures and interface definitions"
- ] # CodeParser.parse_code(block="Data structures and interface definitions", text=content)
- seq_flow = system_design.instruct_content.dict()[
- "Program call flow"
- ] # CodeParser.parse_code(block="Program call flow", text=content)
- await mermaid_to_file(data_api_design, resources_path / "data_api_design")
- await mermaid_to_file(seq_flow, resources_path / "seq_flow")
- system_design_file = docs_path / "system_design.md"
- logger.info(f"Saving System Designs to {system_design_file}")
- system_design_file.write_text((json_to_markdown(system_design.instruct_content.dict())))
-
- async def _save(self, context, system_design):
- if isinstance(system_design, ActionOutput):
- ws_name = system_design.instruct_content.dict()["Python package name"]
- else:
- ws_name = CodeParser.parse_str(block="Python package name", text=system_design)
- workspace = WORKSPACE_ROOT / ws_name
- self.recreate_workspace(workspace)
- docs_path = workspace / "docs"
- resources_path = workspace / "resources"
- docs_path.mkdir(parents=True, exist_ok=True)
- resources_path.mkdir(parents=True, exist_ok=True)
- await self._save_prd(docs_path, resources_path, context)
- await self._save_system_design(docs_path, resources_path, system_design)
-
- async def run(self, context, format=CONFIG.prompt_format):
+ async def _new_system_design(self, context, format=CONFIG.prompt_format):
prompt_template, format_example = get_template(templates, format)
prompt = prompt_template.format(context=context, format_example=format_example)
- # system_design = await self._aask(prompt)
system_design = await self._aask_v1(prompt, "system_design", OUTPUT_MAPPING, format=format)
- # fix Python package name, we can't system_design.instruct_content.python_package_name = "xxx" since "Python package name" contain space, have to use setattr
+ self._rename_project_name(system_design=system_design)
+ await self._rename_workspace(system_design)
+ return system_design
+
+ async def _merge(self, prd_doc, system_design_doc, format=CONFIG.prompt_format):
+ prompt = MERGE_PROMPT.format(old_design=system_design_doc.content, context=prd_doc.content)
+ system_design = await self._aask_v1(prompt, "system_design", OUTPUT_MAPPING, format=format)
+ # fix Python package name, we can't system_design.instruct_content.python_package_name = "xxx" since "Python
+ # package name" contain space, have to use setattr
+ self._rename_project_name(system_design=system_design)
+ system_design_doc.content = system_design.instruct_content.json(ensure_ascii=False)
+ return system_design_doc
+
+ @staticmethod
+ def _rename_project_name(system_design):
+ # fix project_name, we can't system_design.instruct_content.python_package_name = "xxx" since "project_name"
+ # contain space, have to use setattr
+ if CONFIG.project_name:
+ setattr(
+ system_design.instruct_content,
+ "project_name",
+ CONFIG.project_name,
+ )
+ return
setattr(
system_design.instruct_content,
- "Python package name",
- system_design.instruct_content.dict()["Python package name"].strip().strip("'").strip('"'),
+ "project_name",
+ system_design.instruct_content.dict()["project_name"].strip().strip("'").strip('"'),
)
- await self._save(context, system_design)
- return system_design
+
+ @staticmethod
+ async def _rename_workspace(system_design):
+ if CONFIG.project_path: # Updating on the old version has already been specified if it's valid. According to
+ # Section 2.2.3.10 of RFC 135
+ return
+
+ if isinstance(system_design, ActionOutput):
+ ws_name = system_design.instruct_content.dict()["project_name"]
+ else:
+ ws_name = CodeParser.parse_str(block="project_name", text=system_design)
+ CONFIG.git_repo.rename_root(ws_name)
+
+ async def _update_system_design(self, filename, prds_file_repo, system_design_file_repo) -> Document:
+ prd = await prds_file_repo.get(filename)
+ old_system_design_doc = await system_design_file_repo.get(filename)
+ if not old_system_design_doc:
+ system_design = await self._new_system_design(context=prd.content)
+ doc = Document(
+ root_path=SYSTEM_DESIGN_FILE_REPO,
+ filename=filename,
+ content=system_design.instruct_content.json(ensure_ascii=False),
+ )
+ else:
+ doc = await self._merge(prd_doc=prd, system_design_doc=old_system_design_doc)
+ await system_design_file_repo.save(
+ filename=filename, content=doc.content, dependencies={prd.root_relative_path}
+ )
+ await self._save_data_api_design(doc)
+ await self._save_seq_flow(doc)
+ await self._save_pdf(doc)
+ return doc
+
+ @staticmethod
+ async def _save_data_api_design(design_doc):
+ m = json.loads(design_doc.content)
+ data_api_design = m.get("Data structures and interface definitions")
+ if not data_api_design:
+ return
+ pathname = CONFIG.git_repo.workdir / Path(DATA_API_DESIGN_FILE_REPO) / Path(design_doc.filename).with_suffix("")
+ await WriteDesign._save_mermaid_file(data_api_design, pathname)
+ logger.info(f"Save class view to {str(pathname)}")
+
+ @staticmethod
+ async def _save_seq_flow(design_doc):
+ m = json.loads(design_doc.content)
+ seq_flow = m.get("Program call flow")
+ if not seq_flow:
+ return
+ pathname = CONFIG.git_repo.workdir / Path(SEQ_FLOW_FILE_REPO) / Path(design_doc.filename).with_suffix("")
+ await WriteDesign._save_mermaid_file(seq_flow, pathname)
+ logger.info(f"Saving sequence flow to {str(pathname)}")
+
+ @staticmethod
+ async def _save_pdf(design_doc):
+ await FileRepository.save_as(doc=design_doc, with_suffix=".md", relative_path=SYSTEM_DESIGN_PDF_FILE_REPO)
+
+ @staticmethod
+ async def _save_mermaid_file(data: str, pathname: Path):
+ pathname.parent.mkdir(parents=True, exist_ok=True)
+ await mermaid_to_file(data, pathname)
diff --git a/metagpt/actions/prepare_documents.py b/metagpt/actions/prepare_documents.py
new file mode 100644
index 000000000..b751dc970
--- /dev/null
+++ b/metagpt/actions/prepare_documents.py
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : git_repository.py
+@Desc: PrepareDocuments Action: initialize project folder and add new requirements to docs/requirements.txt.
+ RFC 135 2.2.3.5.1.
+"""
+import shutil
+from pathlib import Path
+
+from metagpt.actions import Action, ActionOutput
+from metagpt.config import CONFIG
+from metagpt.const import DEFAULT_WORKSPACE_ROOT, DOCS_FILE_REPO, REQUIREMENT_FILENAME
+from metagpt.schema import Document
+from metagpt.utils.file_repository import FileRepository
+from metagpt.utils.git_repository import GitRepository
+
+
+class PrepareDocuments(Action):
+ def __init__(self, name="", context=None, llm=None):
+ super().__init__(name, context, llm)
+
+ async def run(self, with_messages, **kwargs):
+ if not CONFIG.git_repo:
+ # Create and initialize the workspace folder, initialize the Git environment.
+ project_name = CONFIG.project_name or FileRepository.new_filename()
+ workdir = Path(CONFIG.project_path or DEFAULT_WORKSPACE_ROOT / project_name)
+ if not CONFIG.inc and workdir.exists():
+ shutil.rmtree(workdir)
+ CONFIG.git_repo = GitRepository()
+ CONFIG.git_repo.open(local_path=workdir, auto_init=True)
+
+ # Write the newly added requirements from the main parameter idea to `docs/requirement.txt`.
+ doc = Document(root_path=DOCS_FILE_REPO, filename=REQUIREMENT_FILENAME, content=with_messages[0].content)
+ await FileRepository.save_file(filename=REQUIREMENT_FILENAME, content=doc.content, relative_path=DOCS_FILE_REPO)
+
+ # Send a Message notification to the WritePRD action, instructing it to process requirements using
+ # `docs/requirement.txt` and `docs/prds/`.
+ return ActionOutput(content=doc.content, instruct_content=doc)
diff --git a/metagpt/actions/project_management.py b/metagpt/actions/project_management.py
index b395fa64e..3d59daeed 100644
--- a/metagpt/actions/project_management.py
+++ b/metagpt/actions/project_management.py
@@ -4,15 +4,27 @@
@Time : 2023/5/11 19:12
@Author : alexanderwu
@File : project_management.py
+@Modified By: mashenquan, 2023/11/27.
+ 1. Divide the context into three components: legacy code, unit test code, and console log.
+ 2. Move the document storage operations related to WritePRD from the save operation of WriteDesign.
+ 3. According to the design in Section 2.2.3.5.4 of RFC 135, add incremental iteration functionality.
"""
+import json
from typing import List
+from metagpt.actions import ActionOutput
from metagpt.actions.action import Action
from metagpt.config import CONFIG
-from metagpt.const import WORKSPACE_ROOT
-from metagpt.utils.common import CodeParser
+from metagpt.const import (
+ PACKAGE_REQUIREMENTS_FILENAME,
+ SYSTEM_DESIGN_FILE_REPO,
+ TASK_FILE_REPO,
+ TASK_PDF_FILE_REPO,
+)
+from metagpt.logs import logger
+from metagpt.schema import Document, Documents
+from metagpt.utils.file_repository import FileRepository
from metagpt.utils.get_template import get_template
-from metagpt.utils.json_to_markdown import json_to_markdown
templates = {
"json": {
@@ -24,22 +36,23 @@ templates = {
{format_example}
-----
Role: You are a project manager; the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
Requirements: Based on the context, fill in the following missing information, each section name is a key in json. Here the granularity of the task is a file, if there are any missing files, you can supplement them
-Attention: Use '##' to split sections, not '#', and '## ' SHOULD WRITE BEFORE the code and triple quote.
+ATTENTION: Output carefully referenced "Format example" in format.
-## Required Python third-party packages: Provided in requirements.txt format
+## Required Python third-party packages: Provide Python list[str] in requirements.txt format
-## Required Other language third-party packages: Provided in requirements.txt format
-
-## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
+## Required Other language third-party packages: Provide Python list[str] in requirements.txt format
## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
+## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
+
## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first.
-## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it. For example, don't forget a main entry. don't forget to init 3rd party libs.
output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like format example,
and only output the json inside this tag, nothing else
@@ -53,17 +66,17 @@ and only output the json inside this tag, nothing else
"Required Other language third-party packages": [
"No third-party ..."
],
+ "Logic Analysis": [
+ ["game.py", "Contains..."]
+ ],
+ "Task list": [
+ "game.py"
+ ],
"Full API spec": """
openapi: 3.0.0
...
description: A JSON object ...
""",
- "Logic Analysis": [
- ["game.py","Contains..."]
- ],
- "Task list": [
- "game.py"
- ],
"Shared Knowledge": """
'game.py' contains ...
""",
@@ -87,15 +100,15 @@ Attention: Use '##' to split sections, not '#', and '## ' SHOULD W
## Required Other language third-party packages: Provided in requirements.txt format
-## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
-
## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
+## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
+
## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first.
-## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it. For example, don't forget a main entry. don't forget to init 3rd party libs.
""",
"FORMAT_EXAMPLE": '''
@@ -127,14 +140,16 @@ description: A JSON object ...
## Logic Analysis
```python
[
- ["game.py", "Contains ..."],
+ ["index.js", "Contains ..."],
+ ["main.py", "Contains ..."],
]
```
## Task list
```python
[
- "game.py",
+ "index.js",
+ "main.py",
]
```
@@ -161,30 +176,118 @@ OUTPUT_MAPPING = {
"Anything UNCLEAR": (str, ...),
}
+MERGE_PROMPT = """
+# Context
+{context}
+
+## Old Tasks
+{old_tasks}
+-----
+Role: You are a project manager; The goal is to merge the new PRD/technical design content from 'Context' into 'Old Tasks.' Based on this merged result, break down tasks, give a task list, and analyze task dependencies to start with the prerequisite modules.
+Requirements: Based on the context, fill in the following missing information, each section name is a key in json. Here the granularity of the task is a file, if there are any missing files, you can supplement them
+Attention: Use '##' to split sections, not '#', and '## ' SHOULD WRITE BEFORE the code and triple quote.
+
+## Required Python third-party packages: Provided in requirements.txt format
+
+## Required Other language third-party packages: Provided in requirements.txt format
+
+## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
+
+## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
+
+## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
+
+## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first.
+
+## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
+
+output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like "Old Tasks" format,
+and only output the json inside this tag, nothing else
+"""
+
class WriteTasks(Action):
def __init__(self, name="CreateTasks", context=None, llm=None):
super().__init__(name, context, llm)
- def _save(self, context, rsp):
- if context[-1].instruct_content:
- ws_name = context[-1].instruct_content.dict()["Python package name"]
+ async def run(self, with_messages, format=CONFIG.prompt_format):
+ system_design_file_repo = CONFIG.git_repo.new_file_repository(SYSTEM_DESIGN_FILE_REPO)
+ changed_system_designs = system_design_file_repo.changed_files
+
+ tasks_file_repo = CONFIG.git_repo.new_file_repository(TASK_FILE_REPO)
+ changed_tasks = tasks_file_repo.changed_files
+ change_files = Documents()
+ # Rewrite the system designs that have undergone changes based on the git head diff under
+ # `docs/system_designs/`.
+ for filename in changed_system_designs:
+ task_doc = await self._update_tasks(
+ filename=filename, system_design_file_repo=system_design_file_repo, tasks_file_repo=tasks_file_repo
+ )
+ change_files.docs[filename] = task_doc
+
+ # Rewrite the task files that have undergone changes based on the git head diff under `docs/tasks/`.
+ for filename in changed_tasks:
+ if filename in change_files.docs:
+ continue
+ task_doc = await self._update_tasks(
+ filename=filename, system_design_file_repo=system_design_file_repo, tasks_file_repo=tasks_file_repo
+ )
+ change_files.docs[filename] = task_doc
+
+ if not change_files.docs:
+ logger.info("Nothing has changed.")
+ # Wait until all files under `docs/tasks/` are processed before sending the publish_message, leaving room for
+ # global optimization in subsequent steps.
+ return ActionOutput(content=change_files.json(), instruct_content=change_files)
+
+ async def _update_tasks(self, filename, system_design_file_repo, tasks_file_repo):
+ system_design_doc = await system_design_file_repo.get(filename)
+ task_doc = await tasks_file_repo.get(filename)
+ if task_doc:
+ task_doc = await self._merge(system_design_doc=system_design_doc, task_doc=task_doc)
else:
- ws_name = CodeParser.parse_str(block="Python package name", text=context[-1].content)
- file_path = WORKSPACE_ROOT / ws_name / "docs/api_spec_and_tasks.md"
- file_path.write_text(json_to_markdown(rsp.instruct_content.dict()))
+ rsp = await self._run_new_tasks(context=system_design_doc.content)
+ task_doc = Document(
+ root_path=TASK_FILE_REPO, filename=filename, content=rsp.instruct_content.json(ensure_ascii=False)
+ )
+ await tasks_file_repo.save(
+ filename=filename, content=task_doc.content, dependencies={system_design_doc.root_relative_path}
+ )
+ await self._update_requirements(task_doc)
+ await self._save_pdf(task_doc=task_doc)
+ return task_doc
- # Write requirements.txt
- requirements_path = WORKSPACE_ROOT / ws_name / "requirements.txt"
- requirements_path.write_text("\n".join(rsp.instruct_content.dict().get("Required Python third-party packages")))
-
- async def run(self, context, format=CONFIG.prompt_format):
+ async def _run_new_tasks(self, context, format=CONFIG.prompt_format):
prompt_template, format_example = get_template(templates, format)
prompt = prompt_template.format(context=context, format_example=format_example)
rsp = await self._aask_v1(prompt, "task", OUTPUT_MAPPING, format=format)
- self._save(context, rsp)
return rsp
+ async def _merge(self, system_design_doc, task_doc, format=CONFIG.prompt_format) -> Document:
+ prompt = MERGE_PROMPT.format(context=system_design_doc.content, old_tasks=task_doc.content)
+ rsp = await self._aask_v1(prompt, "task", OUTPUT_MAPPING, format=format)
+ task_doc.content = rsp.instruct_content.json(ensure_ascii=False)
+ return task_doc
+
+ @staticmethod
+ async def _update_requirements(doc):
+ m = json.loads(doc.content)
+ packages = set(m.get("Required Python third-party packages", set()))
+ file_repo = CONFIG.git_repo.new_file_repository()
+ requirement_doc = await file_repo.get(filename=PACKAGE_REQUIREMENTS_FILENAME)
+ if not requirement_doc:
+ requirement_doc = Document(filename=PACKAGE_REQUIREMENTS_FILENAME, root_path=".", content="")
+ lines = requirement_doc.content.splitlines()
+ for pkg in lines:
+ if pkg == "":
+ continue
+ packages.add(pkg)
+ await file_repo.save(PACKAGE_REQUIREMENTS_FILENAME, content="\n".join(packages))
+
+ @staticmethod
+ async def _save_pdf(task_doc):
+ await FileRepository.save_as(doc=task_doc, with_suffix=".md", relative_path=TASK_PDF_FILE_REPO)
+
class AssignTasks(Action):
async def run(self, *args, **kwargs):
diff --git a/metagpt/actions/run_code.py b/metagpt/actions/run_code.py
index f69d2cd1a..fa13a0980 100644
--- a/metagpt/actions/run_code.py
+++ b/metagpt/actions/run_code.py
@@ -4,14 +4,25 @@
@Time : 2023/5/11 17:46
@Author : alexanderwu
@File : run_code.py
+@Modified By: mashenquan, 2023/11/27.
+ 1. Mark the location of Console logs in the PROMPT_TEMPLATE with markdown code-block formatting to enhance
+ the understanding for the LLM.
+ 2. Fix bug: Add the "install dependency" operation.
+ 3. Encapsulate the input of RunCode into RunCodeContext and encapsulate the output of RunCode into
+ RunCodeResult to standardize and unify parameter passing between WriteCode, RunCode, and DebugError.
+ 4. According to section 2.2.3.5.7 of RFC 135, change the method of transferring file content
+ (code files, unit test files, log files) from using the message to using the file name.
+ 5. Merged the `Config` class of send18:dev branch to take over the set/get operations of the Environment
+ class.
"""
-import os
import subprocess
import traceback
from typing import Tuple
from metagpt.actions.action import Action
+from metagpt.config import CONFIG
from metagpt.logs import logger
+from metagpt.schema import RunCodeResult
PROMPT_TEMPLATE = """
Role: You are a senior development and qa engineer, your role is summarize the code running result.
@@ -51,8 +62,14 @@ CONTEXT = """
## Running Command
{command}
## Running Output
-standard output: {outs};
-standard errors: {errs};
+standard output:
+```text
+{outs}
+```
+standard errors:
+```text
+{errs}
+```
"""
@@ -77,17 +94,19 @@ class RunCode(Action):
additional_python_paths = [str(path) for path in additional_python_paths]
# Copy the current environment variables
- env = os.environ.copy()
+ env = CONFIG.new_environ()
# Modify the PYTHONPATH environment variable
additional_python_paths = [working_directory] + additional_python_paths
additional_python_paths = ":".join(additional_python_paths)
env["PYTHONPATH"] = additional_python_paths + ":" + env.get("PYTHONPATH", "")
+ RunCode._install_dependencies(working_directory=working_directory, env=env)
# Start the subprocess
process = subprocess.Popen(
command, cwd=working_directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env
)
+ logger.info(" ".join(command))
try:
# Wait for the process to complete, with a timeout
@@ -98,31 +117,46 @@ class RunCode(Action):
stdout, stderr = process.communicate()
return stdout.decode("utf-8"), stderr.decode("utf-8")
- async def run(
- self, code, mode="script", code_file_name="", test_code="", test_file_name="", command=[], **kwargs
- ) -> str:
- logger.info(f"Running {' '.join(command)}")
- if mode == "script":
- outs, errs = await self.run_script(command=command, **kwargs)
- elif mode == "text":
- outs, errs = await self.run_text(code=code)
+ async def run(self, *args, **kwargs) -> RunCodeResult:
+ logger.info(f"Running {' '.join(self.context.command)}")
+ if self.context.mode == "script":
+ outs, errs = await self.run_script(
+ command=self.context.command,
+ working_directory=self.context.working_directory,
+ additional_python_paths=self.context.additional_python_paths,
+ )
+ elif self.context.mode == "text":
+ outs, errs = await self.run_text(code=self.context.code)
logger.info(f"{outs=}")
logger.info(f"{errs=}")
context = CONTEXT.format(
- code=code,
- code_file_name=code_file_name,
- test_code=test_code,
- test_file_name=test_file_name,
- command=" ".join(command),
+ code=self.context.code,
+ code_file_name=self.context.code_filename,
+ test_code=self.context.test_code,
+ test_file_name=self.context.test_filename,
+ command=" ".join(self.context.command),
outs=outs[:500], # outs might be long but they are not important, truncate them to avoid token overflow
errs=errs[:10000], # truncate errors to avoid token overflow
)
prompt = PROMPT_TEMPLATE.format(context=context)
rsp = await self._aask(prompt)
+ return RunCodeResult(summary=rsp, stdout=outs, stderr=errs)
- result = context + rsp
+ @staticmethod
+ def _install_dependencies(working_directory, env):
+ install_command = ["python", "-m", "pip", "install", "-r", "requirements.txt"]
+ logger.info(" ".join(install_command))
+ try:
+ subprocess.run(install_command, check=True, cwd=working_directory, env=env)
+ except subprocess.CalledProcessError as e:
+ logger.warning(f"{e}")
- return result
+ install_pytest_command = ["python", "-m", "pip", "install", "pytest"]
+ logger.info(" ".join(install_pytest_command))
+ try:
+ subprocess.run(install_pytest_command, check=True, cwd=working_directory, env=env)
+ except subprocess.CalledProcessError as e:
+ logger.warning(f"{e}")
diff --git a/metagpt/actions/summarize_code.py b/metagpt/actions/summarize_code.py
new file mode 100644
index 000000000..88a37536b
--- /dev/null
+++ b/metagpt/actions/summarize_code.py
@@ -0,0 +1,115 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Author : alexanderwu
+@File : summarize_code.py
+"""
+
+from tenacity import retry, stop_after_attempt, wait_fixed
+
+from metagpt.actions.action import Action
+from metagpt.config import CONFIG
+from metagpt.logs import logger
+from metagpt.utils.file_repository import FileRepository
+
+PROMPT_TEMPLATE = """
+NOTICE
+Role: You are a professional software engineer, and your main task is to review the code.
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
+ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
+
+-----
+# System Design
+```text
+{system_design}
+```
+-----
+# Tasks
+```text
+{tasks}
+```
+-----
+{code_blocks}
+
+## Code Review All: 请你对历史所有文件进行阅读，在文件中找到可能的bug，如函数未实现、调用错误、未引用等
+
+## Call flow: mermaid代码，根据实现的函数，使用mermaid绘制完整的调用链
+
+## Summary: 根据历史文件的实现情况进行总结
+
+## TODOs: Python dict[str, str]，这里写出需要修改的文件列表与理由，我们会在之后进行修改
+
+"""
+
+FORMAT_EXAMPLE = """
+
+## Code Review All
+
+### a.py
+- 它少实现了xxx需求...
+- 字段yyy没有给出...
+- ...
+
+### b.py
+...
+
+### c.py
+...
+
+## Call flow
+```mermaid
+flowchart TB
+ c1-->a2
+ subgraph one
+ a1-->a2
+ end
+ subgraph two
+ b1-->b2
+ end
+ subgraph three
+ c1-->c2
+ end
+```
+
+## Summary
+- a.py:...
+- b.py:...
+- c.py:...
+- ...
+
+## TODOs
+{
+ "a.py": "implement requirement xxx...",
+}
+
+"""
+
+
+class SummarizeCode(Action):
+ def __init__(self, name="SummarizeCode", context=None, llm=None):
+ super().__init__(name, context, llm)
+
+ @retry(stop=stop_after_attempt(2), wait=wait_fixed(1))
+ async def summarize_code(self, prompt):
+ code_rsp = await self._aask(prompt)
+ return code_rsp
+
+ async def run(self):
+ design_doc = await FileRepository.get_file(self.context.design_filename)
+ task_doc = await FileRepository.get_file(self.context.task_filename)
+ src_file_repo = CONFIG.git_repo.new_file_repository(relative_path=CONFIG.src_workspace)
+ code_blocks = []
+ for filename in self.context.codes_filenames:
+ code_doc = await src_file_repo.get(filename)
+ code_block = f"```python\n{code_doc.content}\n```\n-----"
+ code_blocks.append(code_block)
+ format_example = FORMAT_EXAMPLE
+ prompt = PROMPT_TEMPLATE.format(
+ system_design=design_doc.content,
+ tasks=task_doc.content,
+ code_blocks="\n".join(code_blocks),
+ format_example=format_example,
+ )
+ logger.info("Summarize code..")
+ rsp = await self.summarize_code(prompt)
+ return rsp
diff --git a/metagpt/actions/write_code.py b/metagpt/actions/write_code.py
index 58499df62..59ccb49a5 100644
--- a/metagpt/actions/write_code.py
+++ b/metagpt/actions/write_code.py
@@ -6,35 +6,64 @@
@File : write_code.py
@Modified By: mashenquan, 2023-11-1. In accordance with Chapter 2.1.3 of RFC 116, modify the data type of the `cause_by`
value of the `Message` object.
+@Modified By: mashenquan, 2023-11-27.
+ 1. Mark the location of Design, Tasks, Legacy Code and Debug logs in the PROMPT_TEMPLATE with markdown
+ code-block formatting to enhance the understanding for the LLM.
+ 2. Following the think-act principle, solidify the task parameters when creating the WriteCode object, rather
+ than passing them in when calling the run function.
+ 3. Encapsulate the input of RunCode into RunCodeContext and encapsulate the output of RunCode into
+ RunCodeResult to standardize and unify parameter passing between WriteCode, RunCode, and DebugError.
"""
+
+
from tenacity import retry, stop_after_attempt, wait_fixed
-from metagpt.actions import WriteDesign
from metagpt.actions.action import Action
-from metagpt.const import WORKSPACE_ROOT
+from metagpt.const import TEST_OUTPUTS_FILE_REPO
from metagpt.logs import logger
-from metagpt.schema import Message
-
-from metagpt.utils.common import CodeParser, any_to_str
+from metagpt.schema import CodingContext, RunCodeResult
+from metagpt.utils.common import CodeParser
+from metagpt.utils.file_repository import FileRepository
PROMPT_TEMPLATE = """
NOTICE
Role: You are a professional engineer; the main goal is to write PEP8 compliant, elegant, modular, easy to read and maintain Python 3.9 code (but you can also use other programming language)
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
+-----
+# Design
+```json
+{design}
+```
+-----
+# Tasks
+```json
+{tasks}
+```
+-----
+# Legacy Code
+```python
+{code}
+```
+-----
+# Debug logs
+```text
+{logs}
+```
+-----
+
## Code: {filename} Write code with triple quoto, based on the following list and context.
1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT.
2. Requirement: Based on the context, implement one following code file, note to return only in code form, your code will be part of the entire project, so please implement complete, reliable, reusable code snippets
-3. Attention1: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
-4. Attention2: YOU MUST FOLLOW "Data structures and interface definitions". DONT CHANGE ANY DESIGN.
+3. Set default value: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
+4. Follow design: YOU MUST FOLLOW "Data structures and interfaces". DONT CHANGE ANY DESIGN.
5. Think before writing: What should be implemented and provided in this document?
6. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
7. Do not use public member functions that do not exist in your design.
+8. Before using a variable, make sure you reference it first
+9. Write out EVERY DETAIL, DON'T LEAVE TODO.
------
-# Context
-{context}
------
## Format example
-----
## Code: {filename}
@@ -47,39 +76,32 @@ ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenc
class WriteCode(Action):
- def __init__(self, name="WriteCode", context: list[Message] = None, llm=None):
+ def __init__(self, name="WriteCode", context=None, llm=None):
super().__init__(name, context, llm)
- def _is_invalid(self, filename):
- return any(i in filename for i in ["mp3", "wav"])
-
- def _save(self, context, filename, code):
- # logger.info(filename)
- # logger.info(code_rsp)
- if self._is_invalid(filename):
- return
-
- design = [i for i in context if i.cause_by == any_to_str(WriteDesign)][0]
-
- ws_name = CodeParser.parse_str(block="Python package name", text=design.content)
- ws_path = WORKSPACE_ROOT / ws_name
- if f"{ws_name}/" not in filename and all(i not in filename for i in ["requirements.txt", ".md"]):
- ws_path = ws_path / ws_name
- code_path = ws_path / filename
- code_path.parent.mkdir(parents=True, exist_ok=True)
- code_path.write_text(code)
- logger.info(f"Saving Code to {code_path}")
-
@retry(stop=stop_after_attempt(2), wait=wait_fixed(1))
- async def write_code(self, prompt):
+ async def write_code(self, prompt) -> str:
code_rsp = await self._aask(prompt)
code = CodeParser.parse_code(block="", text=code_rsp)
return code
- async def run(self, context, filename):
- prompt = PROMPT_TEMPLATE.format(context=context, filename=filename)
- logger.info(f"Writing {filename}..")
+ async def run(self, *args, **kwargs) -> CodingContext:
+ coding_context = CodingContext.loads(self.context.content)
+ test_doc = await FileRepository.get_file(
+ filename="test_" + coding_context.filename + ".json", relative_path=TEST_OUTPUTS_FILE_REPO
+ )
+ logs = ""
+ if test_doc:
+ test_detail = RunCodeResult.loads(test_doc.content)
+ logs = test_detail.stderr
+ prompt = PROMPT_TEMPLATE.format(
+ design=coding_context.design_doc.content,
+ tasks=coding_context.task_doc.content,
+ code=coding_context.code_doc.content,
+ logs=logs,
+ filename=self.context.filename,
+ )
+ logger.info(f"Writing {coding_context.filename}..")
code = await self.write_code(prompt)
- # code_rsp = await self._aask_v1(prompt, "code_rsp", OUTPUT_MAPPING)
- # self._save(context, filename, code)
- return code
+ coding_context.code_doc.content = code
+ return coding_context
diff --git a/metagpt/actions/write_code_review.py b/metagpt/actions/write_code_review.py
index 76adca255..364f6af57 100644
--- a/metagpt/actions/write_code_review.py
+++ b/metagpt/actions/write_code_review.py
@@ -4,79 +4,134 @@
@Time : 2023/5/11 17:45
@Author : alexanderwu
@File : write_code_review.py
+@Modified By: mashenquan, 2023/11/27. Following the think-act principle, solidify the task parameters when creating the
+ WriteCode object, rather than passing them in when calling the run function.
"""
from tenacity import retry, stop_after_attempt, wait_fixed
from metagpt.actions.action import Action
+from metagpt.config import CONFIG
from metagpt.logs import logger
-from metagpt.schema import Message
+from metagpt.schema import CodingContext
from metagpt.utils.common import CodeParser
PROMPT_TEMPLATE = """
NOTICE
Role: You are a professional software engineer, and your main task is to review the code. You need to ensure that the code conforms to the PEP8 standards, is elegantly designed and modularized, easy to read and maintain, and is written in Python 3.9 (or in another programming language).
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
-## Code Review: Based on the following context and code, and following the check list, Provide key, clear, concise, and specific code modification suggestions, up to 5.
-```
-1. Check 0: Is the code implemented as per the requirements?
-2. Check 1: Are there any issues with the code logic?
-3. Check 2: Does the existing code follow the "Data structures and interface definitions"?
-4. Check 3: Is there a function in the code that is omitted or not fully implemented that needs to be implemented?
-5. Check 4: Does the code have unnecessary or lack dependencies?
-```
-
-## Rewrite Code: {filename} Base on "Code Review" and the source code, rewrite code with triple quotes. Do your utmost to optimize THIS SINGLE FILE.
------
# Context
{context}
-## Code: {filename}
+## Code to be Reviewed: {filename}
```
{code}
```
+
-----
+## Code Review: Based on the "Code to be Reviewed", provide key, clear, concise, and specific code modification suggestions, up to 5.
+1. Is the code implemented as per the requirements? If not, how to achieve it? Analyse it step by step.
+2. Is the code logic completely correct? If there are errors, please indicate how to correct them.
+3. Does the existing code follow the "Data structures and interfaces"?
+4. Are all functions implemented? If there is no implementation, please indicate how to achieve it step by step.
+5. Have all necessary pre-dependencies been imported? If not, indicate which ones need to be imported
+6. Is the code implemented concisely enough? Are methods from other files being reused correctly?
+
+## Code Review Result: If the code doesn't have bugs, we don't need to rewrite it, so answer LGTM and stop. ONLY ANSWER LGTM/LBTM.
+LGTM/LBTM
+
+## Rewrite Code: if it still has some bugs, rewrite {filename} based on "Code Review" with triple quotes, try to get LGTM. Do your utmost to optimize THIS SINGLE FILE. Implement ALL TODO. RETURN ALL CODE, NEVER OMIT ANYTHING. 以任何方式省略代码都是不允许的。
+```
+```
+
## Format example
------
{format_example}
------
"""
FORMAT_EXAMPLE = """
-
-## Code Review
-1. The code ...
+-----
+# EXAMPLE 1
+## Code Review: {filename}
+1. No, we should add the logic of ...
2. ...
3. ...
4. ...
5. ...
+6. ...
+
+## Code Review Result: {filename}
+LBTM
## Rewrite Code: {filename}
```python
## {filename}
...
```
+-----
+# EXAMPLE 2
+## Code Review: {filename}
+1. Yes.
+2. Yes.
+3. Yes.
+4. Yes.
+5. Yes.
+6. Yes.
+
+## Code Review Result: {filename}
+LGTM
+
+## Rewrite Code: {filename}
+pass
+-----
"""
class WriteCodeReview(Action):
- def __init__(self, name="WriteCodeReview", context: list[Message] = None, llm=None):
+ def __init__(self, name="WriteCodeReview", context=None, llm=None):
super().__init__(name, context, llm)
@retry(stop=stop_after_attempt(2), wait=wait_fixed(1))
- async def write_code(self, prompt):
+ async def write_code_review_and_rewrite(self, prompt):
code_rsp = await self._aask(prompt)
+ result = CodeParser.parse_block("Code Review Result", code_rsp)
+ if "LGTM" in result:
+ return result, None
code = CodeParser.parse_code(block="", text=code_rsp)
- return code
+ return result, code
- async def run(self, context, code, filename):
- format_example = FORMAT_EXAMPLE.format(filename=filename)
- prompt = PROMPT_TEMPLATE.format(context=context, code=code, filename=filename, format_example=format_example)
- logger.info(f"Code review {filename}..")
- code = await self.write_code(prompt)
+ async def run(self, *args, **kwargs) -> CodingContext:
+ iterative_code = self.context.code_doc.content
+ k = CONFIG.code_review_k_times or 1
+ for i in range(k):
+ format_example = FORMAT_EXAMPLE.format(filename=self.context.code_doc.filename)
+ context = "\n----------\n".join(
+ [
+ "```text\n" + self.context.design_doc.content + "```\n",
+ "```text\n" + self.context.task_doc.content + "```\n",
+ "```python\n" + self.context.code_doc.content + "```\n",
+ ]
+ )
+ prompt = PROMPT_TEMPLATE.format(
+ context=context,
+ code=iterative_code,
+ filename=self.context.code_doc.filename,
+ format_example=format_example,
+ )
+ logger.info(
+ f"Code review and rewrite {self.context.code_doc.filename,}: {i+1}/{k} | {len(iterative_code)=}, {len(self.context.code_doc.content)=}"
+ )
+ result, rewrited_code = await self.write_code_review_and_rewrite(prompt)
+ if "LBTM" in result:
+ iterative_code = rewrited_code
+ elif "LGTM" in result:
+ self.context.code_doc.content = iterative_code
+ return self.context
# code_rsp = await self._aask_v1(prompt, "code_rsp", OUTPUT_MAPPING)
# self._save(context, filename, code)
- return code
+ # 如果rewrited_code是None（原code perfect），那么直接返回code
+ self.context.code_doc.content = iterative_code
+ return self.context
diff --git a/metagpt/actions/write_prd.py b/metagpt/actions/write_prd.py
index bd04ca79e..3967a0578 100644
--- a/metagpt/actions/write_prd.py
+++ b/metagpt/actions/write_prd.py
@@ -4,66 +4,81 @@
@Time : 2023/5/11 17:45
@Author : alexanderwu
@File : write_prd.py
+@Modified By: mashenquan, 2023/11/27.
+ 1. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
+ 2. According to the design in Section 2.2.3.5.2 of RFC 135, add incremental iteration functionality.
+ 3. Move the document storage operations related to WritePRD from the save operation of WriteDesign.
"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
from typing import List
from metagpt.actions import Action, ActionOutput
from metagpt.actions.search_and_summarize import SearchAndSummarize
from metagpt.config import CONFIG
+from metagpt.const import (
+ COMPETITIVE_ANALYSIS_FILE_REPO,
+ DOCS_FILE_REPO,
+ PRD_PDF_FILE_REPO,
+ PRDS_FILE_REPO,
+ REQUIREMENT_FILENAME,
+)
from metagpt.logs import logger
+from metagpt.schema import Document, Documents
+from metagpt.utils.file_repository import FileRepository
from metagpt.utils.get_template import get_template
+from metagpt.utils.mermaid import mermaid_to_file
templates = {
"json": {
"PROMPT_TEMPLATE": """
# Context
-## Original Requirements
-{requirements}
-
-## Search Information
-{search_information}
-
-## mermaid quadrantChart code syntax example. DONT USE QUOTO IN CODE DUE TO INVALID SYNTAX. Replace the with REAL COMPETITOR NAME
-```mermaid
-quadrantChart
- title Reach and engagement of campaigns
- x-axis Low Reach --> High Reach
- y-axis Low Engagement --> High Engagement
- quadrant-1 We should expand
- quadrant-2 Need to promote
- quadrant-3 Re-evaluate
- quadrant-4 May be improved
- "Campaign: A": [0.3, 0.6]
- "Campaign B": [0.45, 0.23]
- "Campaign C": [0.57, 0.69]
- "Campaign D": [0.78, 0.34]
- "Campaign E": [0.40, 0.34]
- "Campaign F": [0.35, 0.78]
- "Our Target Product": [0.5, 0.6]
-```
+{{
+ "Original Requirements": "{requirements}",
+ "Search Information": ""
+}}
## Format example
{format_example}
-----
Role: You are a professional product manager; the goal is to design a concise, usable, efficient product
-Requirements: According to the context, fill in the following missing information, each section name is a key in json ,If the requirements are unclear, ensure minimum viability and avoid excessive design
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
+Requirements: According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly.
+ATTENTION: Output carefully referenced "Format example" in format.
-## Original Requirements: Provide as Plain text, place the polished complete original requirements here
+## YOU NEED TO FULFILL THE BELOW JSON DOC
-## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals. If the requirement itself is simple, the goal should also be simple
-
-## User Stories: Provided as Python list[str], up to 5 scenario-based user stories, If the requirement itself is simple, the user stories should also be less
-
-## Competitive Analysis: Provided as Python list[str], up to 7 competitive product analyses, consider as similar competitors as possible
-
-## Competitive Quadrant Chart: Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
-
-## Requirement Analysis: Provide as Plain text. Be simple. LESS IS MORE. Make your requirements less dumb. Delete the parts unnessasery.
-
-## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards; no more than 5 requirements and consider to make its difficulty lower
-
-## UI Design draft: Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
-## Anything UNCLEAR: Provide as Plain text. Make clear here.
+{{
+ "Language": "", # str, use the same language as the user requirement. en_us / zh_cn etc.
+ "Original Requirements": "", # str, place the polished complete original requirements here
+ "project_name": "", # str, name it like game_2048 / web_2048 / simple_crm etc.
+ "Search Information": "",
+ "Requirements": "",
+ "Product Goals": [], # Provided as Python list[str], up to 3 clear, orthogonal product goals.
+ "User Stories": [], # Provided as Python list[str], up to 5 scenario-based user stories
+ "Competitive Analysis": [], # Provided as Python list[str], up to 8 competitive product analyses
+ # Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
+ "Competitive Quadrant Chart": "quadrantChart
+ title Reach and engagement of campaigns
+ x-axis Low Reach --> High Reach
+ y-axis Low Engagement --> High Engagement
+ quadrant-1 We should expand
+ quadrant-2 Need to promote
+ quadrant-3 Re-evaluate
+ quadrant-4 May be improved
+ Campaign A: [0.3, 0.6]
+ Campaign B: [0.45, 0.23]
+ Campaign C: [0.57, 0.69]
+ Campaign D: [0.78, 0.34]
+ Campaign E: [0.40, 0.34]
+ Campaign F: [0.35, 0.78]",
+ "Requirement Analysis": "", # Provide as Plain text.
+ "Requirement Pool": [["P0","P0 requirement"],["P1","P1 requirement"]], # Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards
+ "UI Design draft": "", # Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
+ "Anything UNCLEAR": "", # Provide as Plain text. Try to clarify it.
+}}
output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like format example,
and only output the json inside this tag, nothing else
@@ -71,6 +86,7 @@ and only output the json inside this tag, nothing else
"FORMAT_EXAMPLE": """
[CONTENT]
{
+ "Language": "",
"Original Requirements": "",
"Search Information": "",
"Requirements": "",
@@ -131,30 +147,33 @@ quadrantChart
{format_example}
-----
Role: You are a professional product manager; the goal is to design a concise, usable, efficient product
-Requirements: According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly. If the requirements are unclear, ensure minimum viability and avoid excessive design
+Language: Please use the same language as the user requirement to answer, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
+Requirements: According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly.
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. AND '## ' SHOULD WRITE BEFORE the code and triple quote. Output carefully referenced "Format example" in format.
+## Language: Provide as Plain text, use the same language as the user requirement.
+
## Original Requirements: Provide as Plain text, place the polished complete original requirements here
-## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals. If the requirement itself is simple, the goal should also be simple
+## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals.
-## User Stories: Provided as Python list[str], up to 5 scenario-based user stories, If the requirement itself is simple, the user stories should also be less
+## User Stories: Provided as Python list[str], up to 5 scenario-based user stories
## Competitive Analysis: Provided as Python list[str], up to 7 competitive product analyses, consider as similar competitors as possible
## Competitive Quadrant Chart: Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
-## Requirement Analysis: Provide as Plain text. Be simple. LESS IS MORE. Make your requirements less dumb. Delete the parts unnessasery.
+## Requirement Analysis: Provide as Plain text.
-## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards; no more than 5 requirements and consider to make its difficulty lower
+## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards
## UI Design draft: Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
-## Anything UNCLEAR: Provide as Plain text. Make clear here.
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it.
""",
"FORMAT_EXAMPLE": """
---
## Original Requirements
-The boss ...
+The user ...
## Product Goals
```python
@@ -205,7 +224,9 @@ There are no unclear points.
},
}
+
OUTPUT_MAPPING = {
+ "Language": (str, ...),
"Original Requirements": (str, ...),
"Product Goals": (List[str], ...),
"User Stories": (List[str], ...),
@@ -217,12 +238,101 @@ OUTPUT_MAPPING = {
"Anything UNCLEAR": (str, ...),
}
+IS_RELATIVE_PROMPT = """
+## PRD:
+{old_prd}
+
+## New Requirement:
+{requirements}
+
+___
+You are a professional product manager; You need to assess whether the new requirements are relevant to the existing PRD to determine whether to merge the new requirements into this PRD.
+Is the newly added requirement in "New Requirement" related to the PRD?
+Respond with `YES` if it is related, `NO` if it is not, and provide the reasons. Return the response in JSON format.
+"""
+
+MERGE_PROMPT = """
+# Context
+## Original Requirements
+{requirements}
+
+
+## Old PRD
+{old_prd}
+-----
+Role: You are a professional product manager; The goal is to incorporate the newly added requirements from the "Original Requirements" into the existing Product Requirements Document (PRD) in the "Old PRD" in order to design a concise, usable, and efficient product.
+Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
+Requirements: According to the context, fill in the following missing information, each section name is a key in json ,If the requirements are unclear, ensure minimum viability and avoid excessive design
+ATTENTION: Output carefully referenced "Old PRD" in format.
+
+## YOU NEED TO FULFILL THE BELOW JSON DOC
+
+{{
+ "Language": "", # str, use the same language as the user requirement. en_us / zh_cn etc.
+ "Original Requirements": "", # str, place the polished complete original requirements here
+ "project_name": "", # str, name it like game_2048 / web_2048 / simple_crm etc.
+ "Search Information": "",
+ "Requirements": "",
+ "Product Goals": [], # Provided as Python list[str], up to 3 clear, orthogonal product goals.
+ "User Stories": [], # Provided as Python list[str], up to 5 scenario-based user stories
+ "Competitive Analysis": [], # Provided as Python list[str], up to 8 competitive product analyses
+ # Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
+ "Competitive Quadrant Chart": "quadrantChart
+ title Reach and engagement of campaigns
+ x-axis Low Reach --> High Reach
+ y-axis Low Engagement --> High Engagement
+ quadrant-1 We should expand
+ quadrant-2 Need to promote
+ quadrant-3 Re-evaluate
+ quadrant-4 May be improved
+ Campaign A: [0.3, 0.6]
+ Campaign B: [0.45, 0.23]
+ Campaign C: [0.57, 0.69]
+ Campaign D: [0.78, 0.34]
+ Campaign E: [0.40, 0.34]
+ Campaign F: [0.35, 0.78]",
+ "Requirement Analysis": "", # Provide as Plain text.
+ "Requirement Pool": [["P0","P0 requirement"],["P1","P1 requirement"]], # Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards
+ "UI Design draft": "", # Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
+ "Anything UNCLEAR": "", # Provide as Plain text. Try to clarify it.
+}}
+
+output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like "Old PRD" format,
+and only output the json inside this tag, nothing else
+"""
+
class WritePRD(Action):
def __init__(self, name="", context=None, llm=None):
super().__init__(name, context, llm)
- async def run(self, requirements, format=CONFIG.prompt_format, *args, **kwargs) -> ActionOutput:
+ async def run(self, with_messages, format=CONFIG.prompt_format, *args, **kwargs) -> ActionOutput:
+ # Determine which requirement documents need to be rewritten: Use LLM to assess whether new requirements are
+ # related to the PRD. If they are related, rewrite the PRD.
+ requirement_doc = await FileRepository.get_file(filename=REQUIREMENT_FILENAME, relative_path=DOCS_FILE_REPO)
+ prds_file_repo = CONFIG.git_repo.new_file_repository(PRDS_FILE_REPO)
+ prd_docs = await prds_file_repo.get_all()
+ change_files = Documents()
+ for prd_doc in prd_docs:
+ prd_doc = await self._update_prd(
+ requirement_doc=requirement_doc, prd_doc=prd_doc, prds_file_repo=prds_file_repo, *args, **kwargs
+ )
+ if not prd_doc:
+ continue
+ change_files.docs[prd_doc.filename] = prd_doc
+ # If there is no existing PRD, generate one using 'docs/requirement.txt'.
+ if not change_files.docs:
+ prd_doc = await self._update_prd(
+ requirement_doc=requirement_doc, prd_doc=None, prds_file_repo=prds_file_repo, *args, **kwargs
+ )
+ if prd_doc:
+ change_files.docs[prd_doc.filename] = prd_doc
+ # Once all files under 'docs/prds/' have been compared with the newly added requirements, trigger the
+ # 'publish' message to transition the workflow to the next stage. This design allows room for global
+ # optimization in subsequent steps.
+ return ActionOutput(content=change_files.json(), instruct_content=change_files)
+
+ async def _run_new_requirement(self, requirements, format=CONFIG.prompt_format, *args, **kwargs) -> ActionOutput:
sas = SearchAndSummarize()
# rsp = await sas.run(context=requirements, system_text=SEARCH_AND_SUMMARIZE_SYSTEM_EN_US)
rsp = ""
@@ -231,11 +341,66 @@ class WritePRD(Action):
logger.info(sas.result)
logger.info(rsp)
+ # logger.info(format)
prompt_template, format_example = get_template(templates, format)
+ # logger.info(prompt_template)
+ # logger.info(format_example)
prompt = prompt_template.format(
requirements=requirements, search_information=info, format_example=format_example
)
- logger.debug(prompt)
+ # logger.info(prompt)
# prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING)
prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING, format=format)
return prd
+
+ async def _is_relative_to(self, new_requirement_doc, old_prd_doc) -> bool:
+ m = json.loads(old_prd_doc.content)
+ if m.get("Original Requirements") == new_requirement_doc.content:
+ # There have been no changes in the requirements, so they are considered unrelated.
+ return False
+ prompt = IS_RELATIVE_PROMPT.format(old_prd=old_prd_doc.content, requirements=new_requirement_doc.content)
+ res = await self._aask(prompt=prompt)
+ logger.info(f"[{new_requirement_doc.root_relative_path}, {old_prd_doc.root_relative_path}]: {res}")
+ if "YES" in res:
+ return True
+ return False
+
+ async def _merge(self, new_requirement_doc, prd_doc, format=CONFIG.prompt_format) -> Document:
+ prompt = MERGE_PROMPT.format(requirements=new_requirement_doc.content, old_prd=prd_doc.content)
+ prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING, format=format)
+ prd_doc.content = prd.instruct_content.json(ensure_ascii=False)
+ return prd_doc
+
+ async def _update_prd(self, requirement_doc, prd_doc, prds_file_repo, *args, **kwargs) -> Document | None:
+ if not prd_doc:
+ prd = await self._run_new_requirement(requirements=[requirement_doc.content], *args, **kwargs)
+ new_prd_doc = Document(
+ root_path=PRDS_FILE_REPO,
+ filename=FileRepository.new_filename() + ".json",
+ content=prd.instruct_content.json(ensure_ascii=False),
+ )
+ elif await self._is_relative_to(requirement_doc, prd_doc):
+ new_prd_doc = await self._merge(requirement_doc, prd_doc)
+ else:
+ return None
+ await prds_file_repo.save(filename=new_prd_doc.filename, content=new_prd_doc.content)
+ await self._save_competitive_analysis(new_prd_doc)
+ await self._save_pdf(new_prd_doc)
+ return new_prd_doc
+
+ @staticmethod
+ async def _save_competitive_analysis(prd_doc):
+ m = json.loads(prd_doc.content)
+ quadrant_chart = m.get("Competitive Quadrant Chart")
+ if not quadrant_chart:
+ return
+ pathname = (
+ CONFIG.git_repo.workdir / Path(COMPETITIVE_ANALYSIS_FILE_REPO) / Path(prd_doc.filename).with_suffix("")
+ )
+ if not pathname.parent.exists():
+ pathname.parent.mkdir(parents=True, exist_ok=True)
+ await mermaid_to_file(quadrant_chart, pathname)
+
+ @staticmethod
+ async def _save_pdf(prd_doc):
+ await FileRepository.save_as(doc=prd_doc, with_suffix=".md", relative_path=PRD_PDF_FILE_REPO)
diff --git a/metagpt/actions/write_test.py b/metagpt/actions/write_test.py
index 35ff36dc2..7cbb42e1d 100644
--- a/metagpt/actions/write_test.py
+++ b/metagpt/actions/write_test.py
@@ -3,10 +3,14 @@
"""
@Time : 2023/5/11 22:12
@Author : alexanderwu
-@File : environment.py
+@File : write_test.py
+@Modified By: mashenquan, 2023-11-27. Following the think-act principle, solidify the task parameters when creating the
+ WriteTest object, rather than passing them in when calling the run function.
"""
from metagpt.actions.action import Action
+from metagpt.config import CONFIG
from metagpt.logs import logger
+from metagpt.schema import TestingContext
from metagpt.utils.common import CodeParser
PROMPT_TEMPLATE = """
@@ -15,7 +19,7 @@ NOTICE
2. Requirement: Based on the context, develop a comprehensive test suite that adequately covers all relevant aspects of the code file under review. Your test suite will be part of the overall project QA, so please develop complete, robust, and reusable test cases.
3. Attention1: Use '##' to split sections, not '#', and '## ' SHOULD WRITE BEFORE the test case or script.
4. Attention2: If there are any settings in your tests, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
-5. Attention3: YOU MUST FOLLOW "Data structures and interface definitions". DO NOT CHANGE ANY DESIGN. Make sure your tests respect the existing design and ensure its validity.
+5. Attention3: YOU MUST FOLLOW "Data structures and interfaces". DO NOT CHANGE ANY DESIGN. Make sure your tests respect the existing design and ensure its validity.
6. Think before writing: What should be tested and validated in this document? What edge cases could exist? What might fail?
7. CAREFULLY CHECK THAT YOU DON'T MISS ANY NECESSARY TEST CASES/SCRIPTS IN THIS FILE.
Attention: Use '##' to split sections, not '#', and '## ' SHOULD WRITE BEFORE the test case or script and triple quotes.
@@ -47,12 +51,12 @@ class WriteTest(Action):
code = code_rsp
return code
- async def run(self, code_to_test, test_file_name, source_file_path, workspace):
+ async def run(self, *args, **kwargs) -> TestingContext:
prompt = PROMPT_TEMPLATE.format(
- code_to_test=code_to_test,
- test_file_name=test_file_name,
- source_file_path=source_file_path,
- workspace=workspace,
+ code_to_test=self.context.code_doc.content,
+ test_file_name=self.context.test_doc.filename,
+ source_file_path=self.context.code_doc.root_relative_path,
+ workspace=CONFIG.git_repo.workdir,
)
- code = await self.write_code(prompt)
- return code
+ self.context.test_doc.content = await self.write_code(prompt)
+ return self.context
diff --git a/metagpt/config.py b/metagpt/config.py
index d93640c1b..d04ae7291 100644
--- a/metagpt/config.py
+++ b/metagpt/config.py
@@ -2,13 +2,18 @@
# -*- coding: utf-8 -*-
"""
Provide configuration, singleton
+@Modified By: mashenquan, 2023/11/27.
+ 1. According to Section 2.2.3.11 of RFC 135, add git repository support.
+ 2. Add the parameter `src_workspace` for the old version project path.
"""
import os
+from copy import deepcopy
+from pathlib import Path
+from typing import Any
-import openai
import yaml
-from metagpt.const import PROJECT_ROOT
+from metagpt.const import DEFAULT_WORKSPACE_ROOT, METAGPT_ROOT, OPTIONS
from metagpt.logs import logger
from metagpt.tools import SearchEngineType, WebBrowserEngineType
from metagpt.utils.singleton import Singleton
@@ -35,25 +40,29 @@ class Config(metaclass=Singleton):
"""
_instance = None
- key_yaml_file = PROJECT_ROOT / "config/key.yaml"
- default_yaml_file = PROJECT_ROOT / "config/config.yaml"
+ home_yaml_file = Path.home() / ".metagpt/config.yaml"
+ key_yaml_file = METAGPT_ROOT / "config/key.yaml"
+ default_yaml_file = METAGPT_ROOT / "config/config.yaml"
def __init__(self, yaml_file=default_yaml_file):
- self._configs = {}
- self._init_with_config_files_and_env(self._configs, yaml_file)
+ self._init_with_config_files_and_env(yaml_file)
logger.info("Config loading done.")
+ self._update()
+
+ def _update(self):
+ # logger.info("Config loading done.")
self.global_proxy = self._get("GLOBAL_PROXY")
self.openai_api_key = self._get("OPENAI_API_KEY")
self.anthropic_api_key = self._get("Anthropic_API_KEY")
- if (not self.openai_api_key or "YOUR_API_KEY" == self.openai_api_key) and (
- not self.anthropic_api_key or "YOUR_API_KEY" == self.anthropic_api_key
+ self.zhipuai_api_key = self._get("ZHIPUAI_API_KEY")
+ if (
+ (not self.openai_api_key or "YOUR_API_KEY" == self.openai_api_key)
+ and (not self.anthropic_api_key or "YOUR_API_KEY" == self.anthropic_api_key)
+ and (not self.zhipuai_api_key or "YOUR_API_KEY" == self.zhipuai_api_key)
):
- raise NotConfiguredException("Set OPENAI_API_KEY or Anthropic_API_KEY first")
+ raise NotConfiguredException("Set OPENAI_API_KEY or Anthropic_API_KEY or ZHIPUAI_API_KEY first")
self.openai_api_base = self._get("OPENAI_API_BASE")
- openai_proxy = self._get("OPENAI_PROXY") or self.global_proxy
- if openai_proxy:
- openai.proxy = openai_proxy
- openai.api_base = self.openai_api_base
+ self.openai_proxy = self._get("OPENAI_PROXY") or self.global_proxy
self.openai_api_type = self._get("OPENAI_API_TYPE")
self.openai_api_version = self._get("OPENAI_API_VERSION")
self.openai_api_rpm = self._get("RPM", 3)
@@ -83,6 +92,7 @@ class Config(metaclass=Singleton):
logger.warning("LONG_TERM_MEMORY is True")
self.max_budget = self._get("MAX_BUDGET", 10.0)
self.total_cost = 0.0
+ self.code_review_k_times = 2
self.puppeteer_config = self._get("PUPPETEER_CONFIG", "")
self.mmdc = self._get("MMDC", "mmdc")
@@ -93,12 +103,18 @@ class Config(metaclass=Singleton):
self.pyppeteer_executable_path = self._get("PYPPETEER_EXECUTABLE_PATH", "")
self.prompt_format = self._get("PROMPT_FORMAT", "markdown")
+ self.workspace_path = Path(self._get("WORKSPACE_PATH", DEFAULT_WORKSPACE_ROOT))
+ self._ensure_workspace_exists()
- def _init_with_config_files_and_env(self, configs: dict, yaml_file):
+ def _ensure_workspace_exists(self):
+ self.workspace_path.mkdir(parents=True, exist_ok=True)
+ logger.info(f"WORKSPACE_PATH set to {self.workspace_path}")
+
+ def _init_with_config_files_and_env(self, yaml_file):
"""Load from config/key.yaml, config/config.yaml, and env in decreasing order of priority"""
- configs.update(os.environ)
+ configs = dict(os.environ)
- for _yaml_file in [yaml_file, self.key_yaml_file]:
+ for _yaml_file in [yaml_file, self.key_yaml_file, self.home_yaml_file]:
if not _yaml_file.exists():
continue
@@ -107,11 +123,13 @@ class Config(metaclass=Singleton):
yaml_data = yaml.safe_load(file)
if not yaml_data:
continue
- os.environ.update({k: v for k, v in yaml_data.items() if isinstance(v, str)})
configs.update(yaml_data)
+ OPTIONS.set(configs)
- def _get(self, *args, **kwargs):
- return self._configs.get(*args, **kwargs)
+ @staticmethod
+ def _get(*args, **kwargs):
+ m = OPTIONS.get()
+ return m.get(*args, **kwargs)
def get(self, key, *args, **kwargs):
"""Search for a value in config/key.yaml, config/config.yaml, and env; raise an error if not found"""
@@ -120,5 +138,33 @@ class Config(metaclass=Singleton):
raise ValueError(f"Key '{key}' not found in environment variables or in the YAML file")
return value
+ def __setattr__(self, name: str, value: Any) -> None:
+ OPTIONS.get()[name] = value
+
+ def __getattr__(self, name: str) -> Any:
+ m = OPTIONS.get()
+ return m.get(name)
+
+ def set_context(self, options: dict):
+ """Update current config"""
+ if not options:
+ return
+ opts = deepcopy(OPTIONS.get())
+ opts.update(options)
+ OPTIONS.set(opts)
+ self._update()
+
+ @property
+ def options(self):
+ """Return all key-values"""
+ return OPTIONS.get()
+
+ def new_environ(self):
+ """Return a new os.environ object"""
+ env = os.environ.copy()
+ m = self.options
+ env.update({k: v for k, v in m.items() if isinstance(v, str)})
+ return env
+
CONFIG = Config()
diff --git a/metagpt/const.py b/metagpt/const.py
index fa0ccc536..a646cea7a 100644
--- a/metagpt/const.py
+++ b/metagpt/const.py
@@ -4,48 +4,88 @@
@Time : 2023/5/1 11:59
@Author : alexanderwu
@File : const.py
-@Modified By: mashenquan, 2023-11-1. According to Chapter 2.2.1 and 2.2.2 of RFC 116, added key definitions for
+@Modified By: mashenquan, 2023-11-1. According to Section 2.2.1 and 2.2.2 of RFC 116, added key definitions for
common properties in the Message.
+@Modified By: mashenquan, 2023-11-27. Defines file repository paths according to Section 2.2.3.4 of RFC 135.
"""
+import contextvars
+import os
from pathlib import Path
+from loguru import logger
-def get_project_root():
- """Search upwards to find the project root directory."""
- current_path = Path.cwd()
- while True:
- if (
- (current_path / ".git").exists()
- or (current_path / ".project_root").exists()
- or (current_path / ".gitignore").exists()
- ):
- return current_path
- parent_path = current_path.parent
- if parent_path == current_path:
- raise Exception("Project root not found.")
- current_path = parent_path
+import metagpt
+
+OPTIONS = contextvars.ContextVar("OPTIONS")
-PROJECT_ROOT = get_project_root()
-DATA_PATH = PROJECT_ROOT / "data"
-WORKSPACE_ROOT = PROJECT_ROOT / "workspace"
-PROMPT_PATH = PROJECT_ROOT / "metagpt/prompts"
-UT_PATH = PROJECT_ROOT / "data/ut"
-SWAGGER_PATH = UT_PATH / "files/api/"
-UT_PY_PATH = UT_PATH / "files/ut/"
-API_QUESTIONS_PATH = UT_PATH / "files/question/"
-YAPI_URL = "http://yapi.deepwisdomai.com/"
-TMP = PROJECT_ROOT / "tmp"
+def get_metagpt_package_root():
+ """Get the root directory of the installed package."""
+ package_root = Path(metagpt.__file__).parent.parent
+ logger.info(f"Package root set to {str(package_root)}")
+ return package_root
+
+
+def get_metagpt_root():
+ """Get the project root directory."""
+ # Check if a project root is specified in the environment variable
+ project_root_env = os.getenv("METAGPT_PROJECT_ROOT")
+ if project_root_env:
+ project_root = Path(project_root_env)
+ logger.info(f"PROJECT_ROOT set from environment variable to {str(project_root)}")
+ else:
+ # Fallback to package root if no environment variable is set
+ project_root = get_metagpt_package_root()
+ return project_root
+
+
+# METAGPT PROJECT ROOT AND VARS
+
+METAGPT_ROOT = get_metagpt_root()
+DEFAULT_WORKSPACE_ROOT = METAGPT_ROOT / "workspace"
+
+DATA_PATH = METAGPT_ROOT / "data"
RESEARCH_PATH = DATA_PATH / "research"
TUTORIAL_PATH = DATA_PATH / "tutorial_docx"
INVOICE_OCR_TABLE_PATH = DATA_PATH / "invoice_table"
+UT_PATH = DATA_PATH / "ut"
+SWAGGER_PATH = UT_PATH / "files/api/"
+UT_PY_PATH = UT_PATH / "files/ut/"
+API_QUESTIONS_PATH = UT_PATH / "files/question/"
-SKILL_DIRECTORY = PROJECT_ROOT / "metagpt/skills"
+TMP = METAGPT_ROOT / "tmp"
+
+SOURCE_ROOT = METAGPT_ROOT / "metagpt"
+PROMPT_PATH = SOURCE_ROOT / "prompts"
+SKILL_DIRECTORY = SOURCE_ROOT / "skills"
+
+
+# REAL CONSTS
MEM_TTL = 24 * 30 * 3600
+
MESSAGE_ROUTE_FROM = "sent_from"
MESSAGE_ROUTE_TO = "send_to"
MESSAGE_ROUTE_CAUSE_BY = "cause_by"
MESSAGE_META_ROLE = "role"
MESSAGE_ROUTE_TO_ALL = ""
+MESSAGE_ROUTE_TO_NONE = ""
+
+REQUIREMENT_FILENAME = "requirement.txt"
+PACKAGE_REQUIREMENTS_FILENAME = "requirements.txt"
+
+DOCS_FILE_REPO = "docs"
+PRDS_FILE_REPO = "docs/prds"
+SYSTEM_DESIGN_FILE_REPO = "docs/system_design"
+TASK_FILE_REPO = "docs/tasks"
+COMPETITIVE_ANALYSIS_FILE_REPO = "resources/competitive_analysis"
+DATA_API_DESIGN_FILE_REPO = "resources/data_api_design"
+SEQ_FLOW_FILE_REPO = "resources/seq_flow"
+SYSTEM_DESIGN_PDF_FILE_REPO = "resources/system_design"
+PRD_PDF_FILE_REPO = "resources/prd"
+TASK_PDF_FILE_REPO = "resources/api_spec_and_tasks"
+TEST_CODES_FILE_REPO = "tests"
+TEST_OUTPUTS_FILE_REPO = "test_outputs"
+
+YAPI_URL = "http://yapi.deepwisdomai.com/"
diff --git a/metagpt/document.py b/metagpt/document.py
new file mode 100644
index 000000000..6ac4834aa
--- /dev/null
+++ b/metagpt/document.py
@@ -0,0 +1,255 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/6/8 14:03
+@Author : alexanderwu
+@File : document.py
+"""
+from enum import Enum
+from pathlib import Path
+from typing import Optional, Union
+
+import pandas as pd
+from langchain.document_loaders import (
+ TextLoader,
+ UnstructuredPDFLoader,
+ UnstructuredWordDocumentLoader,
+)
+from langchain.text_splitter import CharacterTextSplitter
+from pydantic import BaseModel, Field
+from tqdm import tqdm
+
+from metagpt.config import CONFIG
+from metagpt.logs import logger
+from metagpt.repo_parser import RepoParser
+
+
+def validate_cols(content_col: str, df: pd.DataFrame):
+ if content_col not in df.columns:
+ raise ValueError("Content column not found in DataFrame.")
+
+
+def read_data(data_path: Path):
+ suffix = data_path.suffix
+ if ".xlsx" == suffix:
+ data = pd.read_excel(data_path)
+ elif ".csv" == suffix:
+ data = pd.read_csv(data_path)
+ elif ".json" == suffix:
+ data = pd.read_json(data_path)
+ elif suffix in (".docx", ".doc"):
+ data = UnstructuredWordDocumentLoader(str(data_path), mode="elements").load()
+ elif ".txt" == suffix:
+ data = TextLoader(str(data_path)).load()
+ text_splitter = CharacterTextSplitter(separator="\n", chunk_size=256, chunk_overlap=0)
+ texts = text_splitter.split_documents(data)
+ data = texts
+ elif ".pdf" == suffix:
+ data = UnstructuredPDFLoader(str(data_path), mode="elements").load()
+ else:
+ raise NotImplementedError("File format not supported.")
+ return data
+
+
+class DocumentStatus(Enum):
+ """Indicates document status, a mechanism similar to RFC/PEP"""
+
+ DRAFT = "draft"
+ UNDERREVIEW = "underreview"
+ APPROVED = "approved"
+ DONE = "done"
+
+
+class Document(BaseModel):
+ """
+ Document: Handles operations related to document files.
+ """
+
+ path: Path = Field(default=None)
+ name: str = Field(default="")
+ content: str = Field(default="")
+
+ # metadata? in content perhaps.
+ author: str = Field(default="")
+ status: DocumentStatus = Field(default=DocumentStatus.DRAFT)
+ reviews: list = Field(default_factory=list)
+
+ @classmethod
+ def from_path(cls, path: Path):
+ """
+ Create a Document instance from a file path.
+ """
+ if not path.exists():
+ raise FileNotFoundError(f"File {path} not found.")
+ content = path.read_text()
+ return cls(content=content, path=path)
+
+ @classmethod
+ def from_text(cls, text: str, path: Optional[Path] = None):
+ """
+ Create a Document from a text string.
+ """
+ return cls(content=text, path=path)
+
+ def to_path(self, path: Optional[Path] = None):
+ """
+ Save content to the specified file path.
+ """
+ if path is not None:
+ self.path = path
+
+ if self.path is None:
+ raise ValueError("File path is not set.")
+
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ self.path.write_text(self.content, encoding="utf-8")
+
+ def persist(self):
+ """
+ Persist document to disk.
+ """
+ return self.to_path()
+
+
+class IndexableDocument(Document):
+ """
+ Advanced document handling: For vector databases or search engines.
+ """
+
+ data: Union[pd.DataFrame, list]
+ content_col: Optional[str] = Field(default="")
+ meta_col: Optional[str] = Field(default="")
+
+ class Config:
+ arbitrary_types_allowed = True
+
+ @classmethod
+ def from_path(cls, data_path: Path, content_col="content", meta_col="metadata"):
+ if not data_path.exists():
+ raise FileNotFoundError(f"File {data_path} not found.")
+ data = read_data(data_path)
+ content = data_path.read_text()
+ if isinstance(data, pd.DataFrame):
+ validate_cols(content_col, data)
+ return cls(data=data, content=content, content_col=content_col, meta_col=meta_col)
+
+ def _get_docs_and_metadatas_by_df(self) -> (list, list):
+ df = self.data
+ docs = []
+ metadatas = []
+ for i in tqdm(range(len(df))):
+ docs.append(df[self.content_col].iloc[i])
+ if self.meta_col:
+ metadatas.append({self.meta_col: df[self.meta_col].iloc[i]})
+ else:
+ metadatas.append({})
+ return docs, metadatas
+
+ def _get_docs_and_metadatas_by_langchain(self) -> (list, list):
+ data = self.data
+ docs = [i.page_content for i in data]
+ metadatas = [i.metadata for i in data]
+ return docs, metadatas
+
+ def get_docs_and_metadatas(self) -> (list, list):
+ if isinstance(self.data, pd.DataFrame):
+ return self._get_docs_and_metadatas_by_df()
+ elif isinstance(self.data, list):
+ return self._get_docs_and_metadatas_by_langchain()
+ else:
+ raise NotImplementedError("Data type not supported for metadata extraction.")
+
+
+class RepoMetadata(BaseModel):
+ name: str = Field(default="")
+ n_docs: int = Field(default=0)
+ n_chars: int = Field(default=0)
+ symbols: list = Field(default_factory=list)
+
+
+class Repo(BaseModel):
+ # Name of this repo.
+ name: str = Field(default="")
+ # metadata: RepoMetadata = Field(default=RepoMetadata)
+ docs: dict[Path, Document] = Field(default_factory=dict)
+ codes: dict[Path, Document] = Field(default_factory=dict)
+ assets: dict[Path, Document] = Field(default_factory=dict)
+ path: Path = Field(default=None)
+
+ def _path(self, filename):
+ return self.path / filename
+
+ @classmethod
+ def from_path(cls, path: Path):
+ """Load documents, code, and assets from a repository path."""
+ path.mkdir(parents=True, exist_ok=True)
+ repo = Repo(path=path, name=path.name)
+ for file_path in path.rglob("*"):
+ # FIXME: These judgments are difficult to support multiple programming languages and need to be more general
+ if file_path.is_file() and file_path.suffix in [".json", ".txt", ".md", ".py", ".js", ".css", ".html"]:
+ repo._set(file_path.read_text(), file_path)
+ return repo
+
+ def to_path(self):
+ """Persist all documents, code, and assets to the given repository path."""
+ for doc in self.docs.values():
+ doc.to_path()
+ for code in self.codes.values():
+ code.to_path()
+ for asset in self.assets.values():
+ asset.to_path()
+
+ def _set(self, content: str, path: Path):
+ """Add a document to the appropriate category based on its file extension."""
+ suffix = path.suffix
+ doc = Document(content=content, path=path, name=str(path.relative_to(self.path)))
+
+ # FIXME: These judgments are difficult to support multiple programming languages and need to be more general
+ if suffix.lower() == ".md":
+ self.docs[path] = doc
+ elif suffix.lower() in [".py", ".js", ".css", ".html"]:
+ self.codes[path] = doc
+ else:
+ self.assets[path] = doc
+ return doc
+
+ def set(self, content: str, filename: str):
+ """Set a document and persist it to disk."""
+ path = self._path(filename)
+ doc = self._set(content, path)
+ doc.to_path()
+
+ def get(self, filename: str) -> Optional[Document]:
+ """Get a document by its filename."""
+ path = self._path(filename)
+ return self.docs.get(path) or self.codes.get(path) or self.assets.get(path)
+
+ def get_text_documents(self) -> list[Document]:
+ return list(self.docs.values()) + list(self.codes.values())
+
+ def eda(self) -> RepoMetadata:
+ n_docs = sum(len(i) for i in [self.docs, self.codes, self.assets])
+ n_chars = sum(sum(len(j.content) for j in i.values()) for i in [self.docs, self.codes, self.assets])
+ symbols = RepoParser(base_directory=self.path).generate_symbols()
+ return RepoMetadata(name=self.name, n_docs=n_docs, n_chars=n_chars, symbols=symbols)
+
+
+def set_existing_repo(path=CONFIG.workspace_path / "t1"):
+ repo1 = Repo.from_path(path)
+ repo1.set("wtf content", "doc/wtf_file.md")
+ repo1.set("wtf code", "code/wtf_file.py")
+ logger.info(repo1) # check doc
+
+
+def load_existing_repo(path=CONFIG.workspace_path / "web_tetris"):
+ repo = Repo.from_path(path)
+ logger.info(repo)
+ logger.info(repo.eda())
+
+
+def main():
+ load_existing_repo()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/metagpt/document_store/base_store.py b/metagpt/document_store/base_store.py
index 7d102e00b..5de377d21 100644
--- a/metagpt/document_store/base_store.py
+++ b/metagpt/document_store/base_store.py
@@ -28,20 +28,20 @@ class BaseStore(ABC):
class LocalStore(BaseStore, ABC):
- def __init__(self, raw_data: Path, cache_dir: Path = None):
- if not raw_data:
+ def __init__(self, raw_data_path: Path, cache_dir: Path = None):
+ if not raw_data_path:
raise FileNotFoundError
self.config = Config()
- self.raw_data = raw_data
+ self.raw_data_path = raw_data_path
if not cache_dir:
- cache_dir = raw_data.parent
+ cache_dir = raw_data_path.parent
self.cache_dir = cache_dir
self.store = self._load()
if not self.store:
self.store = self.write()
def _get_index_and_store_fname(self):
- fname = self.raw_data.name.split(".")[0]
+ fname = self.raw_data_path.name.split(".")[0]
index_file = self.cache_dir / f"{fname}.index"
store_file = self.cache_dir / f"{fname}.pkl"
return index_file, store_file
diff --git a/metagpt/document_store/faiss_store.py b/metagpt/document_store/faiss_store.py
index 8ff904cdd..53725c3f9 100644
--- a/metagpt/document_store/faiss_store.py
+++ b/metagpt/document_store/faiss_store.py
@@ -14,16 +14,16 @@ from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from metagpt.const import DATA_PATH
+from metagpt.document import IndexableDocument
from metagpt.document_store.base_store import LocalStore
-from metagpt.document_store.document import Document
from metagpt.logs import logger
class FaissStore(LocalStore):
- def __init__(self, raw_data: Path, cache_dir=None, meta_col="source", content_col="output"):
+ def __init__(self, raw_data_path: Path, cache_dir=None, meta_col="source", content_col="output"):
self.meta_col = meta_col
self.content_col = content_col
- super().__init__(raw_data, cache_dir)
+ super().__init__(raw_data_path, cache_dir)
def _load(self) -> Optional["FaissStore"]:
index_file, store_file = self._get_index_and_store_fname()
@@ -60,9 +60,9 @@ class FaissStore(LocalStore):
def write(self):
"""Initialize the index and library based on the Document (JSON / XLSX, etc.) file provided by the user."""
- if not self.raw_data.exists():
+ if not self.raw_data_path.exists():
raise FileNotFoundError
- doc = Document(self.raw_data, self.content_col, self.meta_col)
+ doc = IndexableDocument.from_path(self.raw_data_path, self.content_col, self.meta_col)
docs, metadatas = doc.get_docs_and_metadatas()
self.store = self._write(docs, metadatas)
diff --git a/metagpt/environment.py b/metagpt/environment.py
index a50dbde9e..02eb3d340 100644
--- a/metagpt/environment.py
+++ b/metagpt/environment.py
@@ -32,7 +32,6 @@ class Environment(BaseModel):
members: dict[Role, Set] = Field(default_factory=dict)
history: str = Field(default="") # For debug
-
class Config:
arbitrary_types_allowed = True
@@ -50,7 +49,6 @@ class Environment(BaseModel):
for role in roles:
self.add_role(role)
-
def publish_message(self, message: Message) -> bool:
"""
Distribute the message to the recipients.
diff --git a/metagpt/llm.py b/metagpt/llm.py
index 410f3dcb5..d8d06c0a1 100644
--- a/metagpt/llm.py
+++ b/metagpt/llm.py
@@ -6,15 +6,28 @@
@File : llm.py
"""
+from metagpt.config import CONFIG
from metagpt.provider.anthropic_api import Claude2 as Claude
-from metagpt.provider.openai_api import OpenAIGPTAPI as LLM
+from metagpt.provider.human_provider import HumanProvider
+from metagpt.provider.openai_api import OpenAIGPTAPI
+from metagpt.provider.spark_api import SparkAPI
+from metagpt.provider.zhipuai_api import ZhiPuAIGPTAPI
-DEFAULT_LLM = LLM()
-CLAUDE_LLM = Claude()
+_ = HumanProvider() # Avoid pre-commit error
-async def ai_func(prompt):
- """使用LLM进行QA
- QA with LLMs
- """
- return await DEFAULT_LLM.aask(prompt)
+def LLM() -> "BaseGPTAPI":
+ """initialize different LLM instance according to the key field existence"""
+ # TODO a little trick, can use registry to initialize LLM instance further
+ if CONFIG.openai_api_key:
+ llm = OpenAIGPTAPI()
+ elif CONFIG.claude_api_key:
+ llm = Claude()
+ elif CONFIG.spark_api_key:
+ llm = SparkAPI()
+ elif CONFIG.zhipuai_api_key:
+ llm = ZhiPuAIGPTAPI()
+ else:
+ raise RuntimeError("You should config a LLM configuration first")
+
+ return llm
diff --git a/metagpt/logs.py b/metagpt/logs.py
index 55d85312f..092b4c476 100644
--- a/metagpt/logs.py
+++ b/metagpt/logs.py
@@ -10,16 +10,14 @@ import sys
from loguru import logger as _logger
-from metagpt.const import PROJECT_ROOT
+from metagpt.const import METAGPT_ROOT
def define_log_level(print_level="INFO", logfile_level="DEBUG"):
- """调整日志级别到level之上
- Adjust the log level to above level
- """
+ """Adjust the log level to above level"""
_logger.remove()
_logger.add(sys.stderr, level=print_level)
- _logger.add(PROJECT_ROOT / "logs/log.txt", level=logfile_level)
+ _logger.add(METAGPT_ROOT / "logs/log.txt", level=logfile_level)
return _logger
diff --git a/metagpt/manager.py b/metagpt/manager.py
index d0b6b101c..a063608be 100644
--- a/metagpt/manager.py
+++ b/metagpt/manager.py
@@ -14,7 +14,7 @@ class Manager:
def __init__(self, llm: LLM = LLM()):
self.llm = llm # Large Language Model
self.role_directions = {
- "BOSS": "Product Manager",
+ "User": "Product Manager",
"Product Manager": "Architect",
"Architect": "Engineer",
"Engineer": "QA Engineer",
diff --git a/metagpt/memory/__init__.py b/metagpt/memory/__init__.py
index 710930626..3d8f679b2 100644
--- a/metagpt/memory/__init__.py
+++ b/metagpt/memory/__init__.py
@@ -7,10 +7,11 @@
"""
from metagpt.memory.memory import Memory
-from metagpt.memory.longterm_memory import LongTermMemory
+
+# from metagpt.memory.longterm_memory import LongTermMemory
__all__ = [
"Memory",
- "LongTermMemory",
+ # "LongTermMemory",
]
diff --git a/metagpt/memory/longterm_memory.py b/metagpt/memory/longterm_memory.py
index 6fc8050ef..22032a86e 100644
--- a/metagpt/memory/longterm_memory.py
+++ b/metagpt/memory/longterm_memory.py
@@ -30,7 +30,7 @@ class LongTermMemory(Memory):
logger.warning(f"It may the first time to run Agent {role_id}, the long-term memory is empty")
else:
logger.warning(
- f"Agent {role_id} has existed memory storage with {len(messages)} messages " f"and has recovered them."
+ f"Agent {role_id} has existing memory storage with {len(messages)} messages " f"and has recovered them."
)
self.msg_from_recover = True
self.add_batch(messages)
diff --git a/metagpt/provider/base_chatbot.py b/metagpt/provider/base_chatbot.py
index 2d4cfe2d9..a6950f144 100644
--- a/metagpt/provider/base_chatbot.py
+++ b/metagpt/provider/base_chatbot.py
@@ -14,6 +14,7 @@ class BaseChatbot(ABC):
"""Abstract GPT class"""
mode: str = "API"
+ use_system_prompt: bool = True
@abstractmethod
def ask(self, msg: str) -> str:
diff --git a/metagpt/provider/base_gpt_api.py b/metagpt/provider/base_gpt_api.py
index adc57c66b..565ae94f7 100644
--- a/metagpt/provider/base_gpt_api.py
+++ b/metagpt/provider/base_gpt_api.py
@@ -5,6 +5,7 @@
@Author : alexanderwu
@File : base_gpt_api.py
"""
+import json
from abc import abstractmethod
from typing import Optional
@@ -33,15 +34,21 @@ class BaseGPTAPI(BaseChatbot):
return self._system_msg(self.system_prompt)
def ask(self, msg: str) -> str:
- message = [self._default_system_msg(), self._user_msg(msg)]
+ message = [self._default_system_msg(), self._user_msg(msg)] if self.use_system_prompt else [self._user_msg(msg)]
rsp = self.completion(message)
return self.get_choice_text(rsp)
async def aask(self, msg: str, system_msgs: Optional[list[str]] = None) -> str:
if system_msgs:
- message = self._system_msgs(system_msgs) + [self._user_msg(msg)]
+ message = (
+ self._system_msgs(system_msgs) + [self._user_msg(msg)]
+ if self.use_system_prompt
+ else [self._user_msg(msg)]
+ )
else:
- message = [self._default_system_msg(), self._user_msg(msg)]
+ message = (
+ [self._default_system_msg(), self._user_msg(msg)] if self.use_system_prompt else [self._user_msg(msg)]
+ )
rsp = await self.acompletion_text(message, stream=True)
logger.debug(message)
# logger.debug(rsp)
@@ -109,6 +116,46 @@ class BaseGPTAPI(BaseChatbot):
"""Required to provide the first text of choice"""
return rsp.get("choices")[0]["message"]["content"]
+ def get_choice_function(self, rsp: dict) -> dict:
+ """Required to provide the first function of choice
+ :param dict rsp: OpenAI chat.comletion respond JSON, Note "message" must include "tool_calls",
+ and "tool_calls" must include "function", for example:
+ {...
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": null,
+ "tool_calls": [
+ {
+ "id": "call_Y5r6Ddr2Qc2ZrqgfwzPX5l72",
+ "type": "function",
+ "function": {
+ "name": "execute",
+ "arguments": "{\n \"language\": \"python\",\n \"code\": \"print('Hello, World!')\"\n}"
+ }
+ }
+ ]
+ },
+ "finish_reason": "stop"
+ }
+ ],
+ ...}
+ :return dict: return first function of choice, for exmaple,
+ {'name': 'execute', 'arguments': '{\n "language": "python",\n "code": "print(\'Hello, World!\')"\n}'}
+ """
+ return rsp.get("choices")[0]["message"]["tool_calls"][0]["function"].to_dict()
+
+ def get_choice_function_arguments(self, rsp: dict) -> dict:
+ """Required to provide the first function arguments of choice.
+
+ :param dict rsp: same as in self.get_choice_function(rsp)
+ :return dict: return the first function arguments of choice, for example,
+ {'language': 'python', 'code': "print('Hello, World!')"}
+ """
+ return json.loads(self.get_choice_function(rsp)["arguments"])
+
def messages_to_prompt(self, messages: list[dict]):
"""[{"role": "user", "content": msg}] to user: etc."""
return "\n".join([f"{i['role']}: {i['content']}" for i in messages])
diff --git a/metagpt/provider/constant.py b/metagpt/provider/constant.py
new file mode 100644
index 000000000..db67847a8
--- /dev/null
+++ b/metagpt/provider/constant.py
@@ -0,0 +1,30 @@
+# function in tools, https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools
+# Reference: https://github.com/KillianLucas/open-interpreter/blob/v0.1.14/interpreter/llm/setup_openai_coding_llm.py
+GENERAL_FUNCTION_SCHEMA = {
+ "name": "execute",
+ "description": "Executes code on the user's machine, **in the users local environment**, and returns the output",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "language": {
+ "type": "string",
+ "description": "The programming language (required parameter to the `execute` function)",
+ "enum": [
+ "python",
+ "R",
+ "shell",
+ "applescript",
+ "javascript",
+ "html",
+ "powershell",
+ ],
+ },
+ "code": {"type": "string", "description": "The code to execute (required)"},
+ },
+ "required": ["language", "code"],
+ },
+}
+
+# tool_choice value for general_function_schema
+# https://platform.openai.com/docs/api-reference/chat/create#chat-create-tool_choice
+GENERAL_TOOL_CHOICE = {"type": "function", "function": {"name": "execute"}}
diff --git a/metagpt/provider/general_api_requestor.py b/metagpt/provider/general_api_requestor.py
new file mode 100644
index 000000000..875122e8b
--- /dev/null
+++ b/metagpt/provider/general_api_requestor.py
@@ -0,0 +1,58 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : General Async API for http-based LLM model
+
+import asyncio
+from typing import AsyncGenerator, Tuple, Union
+
+import aiohttp
+from openai.api_requestor import APIRequestor
+
+from metagpt.logs import logger
+
+
+class GeneralAPIRequestor(APIRequestor):
+ """
+ usage
+ # full_url = "{api_base}{url}"
+ requester = GeneralAPIRequestor(api_base=api_base)
+ result, _, api_key = await requester.arequest(
+ method=method,
+ url=url,
+ headers=headers,
+ stream=stream,
+ params=kwargs,
+ request_timeout=120
+ )
+ """
+
+ def _interpret_response_line(self, rbody: str, rcode: int, rheaders, stream: bool) -> str:
+ # just do nothing to meet the APIRequestor process and return the raw data
+ # due to the openai sdk will convert the data into OpenAIResponse which we don't need in general cases.
+
+ return rbody
+
+ async def _interpret_async_response(
+ self, result: aiohttp.ClientResponse, stream: bool
+ ) -> Tuple[Union[str, AsyncGenerator[str, None]], bool]:
+ if stream and "text/event-stream" in result.headers.get("Content-Type", ""):
+ return (
+ self._interpret_response_line(line, result.status, result.headers, stream=True)
+ async for line in result.content
+ ), True
+ else:
+ try:
+ await result.read()
+ except (aiohttp.ServerTimeoutError, asyncio.TimeoutError) as e:
+ raise TimeoutError("Request timed out") from e
+ except aiohttp.ClientError as exp:
+ logger.warning(f"response: {result.content}, exp: {exp}")
+ return (
+ self._interpret_response_line(
+ await result.read(), # let the caller to decode the msg
+ result.status,
+ result.headers,
+ stream=False,
+ ),
+ False,
+ )
diff --git a/metagpt/provider/human_provider.py b/metagpt/provider/human_provider.py
new file mode 100644
index 000000000..c70a7f1a6
--- /dev/null
+++ b/metagpt/provider/human_provider.py
@@ -0,0 +1,37 @@
+"""
+Filename: MetaGPT/metagpt/provider/human_provider.py
+Created Date: Wednesday, November 8th 2023, 11:55:46 pm
+Author: garylin2099
+"""
+from typing import Optional
+
+from metagpt.logs import logger
+from metagpt.provider.base_gpt_api import BaseGPTAPI
+
+
+class HumanProvider(BaseGPTAPI):
+ """Humans provide themselves as a 'model', which actually takes in human input as its response.
+ This enables replacing LLM anywhere in the framework with a human, thus introducing human interaction
+ """
+
+ def ask(self, msg: str) -> str:
+ logger.info("It's your turn, please type in your response. You may also refer to the context below")
+ rsp = input(msg)
+ if rsp in ["exit", "quit"]:
+ exit()
+ return rsp
+
+ async def aask(self, msg: str, system_msgs: Optional[list[str]] = None) -> str:
+ return self.ask(msg)
+
+ def completion(self, messages: list[dict]):
+ """dummy implementation of abstract method in base"""
+ return []
+
+ async def acompletion(self, messages: list[dict]):
+ """dummy implementation of abstract method in base"""
+ return []
+
+ async def acompletion_text(self, messages: list[dict], stream=False) -> str:
+ """dummy implementation of abstract method in base"""
+ return []
diff --git a/metagpt/provider/openai_api.py b/metagpt/provider/openai_api.py
index ac0edd44f..8ac0c4b21 100644
--- a/metagpt/provider/openai_api.py
+++ b/metagpt/provider/openai_api.py
@@ -21,6 +21,8 @@ from tenacity import (
from metagpt.config import CONFIG
from metagpt.logs import logger
from metagpt.provider.base_gpt_api import BaseGPTAPI
+from metagpt.provider.constant import GENERAL_FUNCTION_SCHEMA, GENERAL_TOOL_CHOICE
+from metagpt.schema import Message
from metagpt.utils.singleton import Singleton
from metagpt.utils.token_counter import (
TOKEN_COSTS,
@@ -155,6 +157,8 @@ class OpenAIGPTAPI(BaseGPTAPI, RateLimiter):
if config.openai_api_type:
openai.api_type = config.openai_api_type
openai.api_version = config.openai_api_version
+ if config.openai_proxy:
+ openai.proxy = config.openai_proxy
self.rpm = int(config.get("RPM", 10))
async def _achat_completion_stream(self, messages: list[dict]) -> str:
@@ -179,7 +183,7 @@ class OpenAIGPTAPI(BaseGPTAPI, RateLimiter):
self._update_costs(usage)
return full_reply_content
- def _cons_kwargs(self, messages: list[dict]) -> dict:
+ def _cons_kwargs(self, messages: list[dict], **configs) -> dict:
kwargs = {
"messages": messages,
"max_tokens": self.get_max_tokens(messages),
@@ -188,6 +192,9 @@ class OpenAIGPTAPI(BaseGPTAPI, RateLimiter):
"temperature": 0.3,
"timeout": 3,
}
+ if configs:
+ kwargs.update(configs)
+
if CONFIG.openai_api_type == "azure":
if CONFIG.deployment_name and CONFIG.deployment_id:
raise ValueError("You can only use one of the `deployment_id` or `deployment_name` model")
@@ -237,6 +244,81 @@ class OpenAIGPTAPI(BaseGPTAPI, RateLimiter):
rsp = await self._achat_completion(messages)
return self.get_choice_text(rsp)
+ def _func_configs(self, messages: list[dict], **kwargs) -> dict:
+ """
+ Note: Keep kwargs consistent with the parameters in the https://platform.openai.com/docs/api-reference/chat/create
+ """
+ if "tools" not in kwargs:
+ configs = {
+ "tools": [{"type": "function", "function": GENERAL_FUNCTION_SCHEMA}],
+ "tool_choice": GENERAL_TOOL_CHOICE,
+ }
+ kwargs.update(configs)
+
+ return self._cons_kwargs(messages, **kwargs)
+
+ def _chat_completion_function(self, messages: list[dict], **kwargs) -> dict:
+ rsp = self.llm.ChatCompletion.create(**self._func_configs(messages, **kwargs))
+ self._update_costs(rsp.get("usage"))
+ return rsp
+
+ async def _achat_completion_function(self, messages: list[dict], **chat_configs) -> dict:
+ rsp = await self.llm.ChatCompletion.acreate(**self._func_configs(messages, **chat_configs))
+ self._update_costs(rsp.get("usage"))
+ return rsp
+
+ def _process_message(self, messages: Union[str, Message, list[dict], list[Message], list[str]]) -> list[dict]:
+ """convert messages to list[dict]."""
+ if isinstance(messages, list):
+ messages = [Message(msg) if isinstance(msg, str) else msg for msg in messages]
+ return [msg if isinstance(msg, dict) else msg.to_dict() for msg in messages]
+
+ if isinstance(messages, Message):
+ messages = [messages.to_dict()]
+ elif isinstance(messages, str):
+ messages = [{"role": "user", "content": messages}]
+ else:
+ raise ValueError(
+ f"Only support messages type are: str, Message, list[dict], but got {type(messages).__name__}!"
+ )
+ return messages
+
+ def ask_code(self, messages: Union[str, Message, list[dict]], **kwargs) -> dict:
+ """Use function of tools to ask a code.
+
+ Note: Keep kwargs consistent with the parameters in the https://platform.openai.com/docs/api-reference/chat/create
+
+ Examples:
+
+ >>> llm = OpenAIGPTAPI()
+ >>> llm.ask_code("Write a python hello world code.")
+ {'language': 'python', 'code': "print('Hello, World!')"}
+ >>> msg = [{'role': 'user', 'content': "Write a python hello world code."}]
+ >>> llm.ask_code(msg)
+ {'language': 'python', 'code': "print('Hello, World!')"}
+ """
+ messages = self._process_message(messages)
+ rsp = self._chat_completion_function(messages, **kwargs)
+ return self.get_choice_function_arguments(rsp)
+
+ async def aask_code(self, messages: Union[str, Message, list[dict]], **kwargs) -> dict:
+ """Use function of tools to ask a code.
+
+ Note: Keep kwargs consistent with the parameters in the https://platform.openai.com/docs/api-reference/chat/create
+
+ Examples:
+
+ >>> llm = OpenAIGPTAPI()
+ >>> rsp = await llm.ask_code("Write a python hello world code.")
+ >>> rsp
+ {'language': 'python', 'code': "print('Hello, World!')"}
+ >>> msg = [{'role': 'user', 'content': "Write a python hello world code."}]
+ >>> rsp = await llm.aask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ """
+ messages = self._process_message(messages)
+ rsp = await self._achat_completion_function(messages, **kwargs)
+ return self.get_choice_function_arguments(rsp)
+
def _calc_usage(self, messages: list[dict], rsp: str) -> dict:
usage = {}
if CONFIG.calc_usage:
diff --git a/metagpt/provider/zhipuai/__init__.py b/metagpt/provider/zhipuai/__init__.py
new file mode 100644
index 000000000..2bcf8efd0
--- /dev/null
+++ b/metagpt/provider/zhipuai/__init__.py
@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc :
diff --git a/metagpt/provider/zhipuai/async_sse_client.py b/metagpt/provider/zhipuai/async_sse_client.py
new file mode 100644
index 000000000..d7168202a
--- /dev/null
+++ b/metagpt/provider/zhipuai/async_sse_client.py
@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : async_sse_client to make keep the use of Event to access response
+# refs to `https://github.com/zhipuai/zhipuai-sdk-python/blob/main/zhipuai/utils/sse_client.py`
+
+from zhipuai.utils.sse_client import _FIELD_SEPARATOR, Event, SSEClient
+
+
+class AsyncSSEClient(SSEClient):
+ async def _aread(self):
+ data = b""
+ async for chunk in self._event_source:
+ for line in chunk.splitlines(True):
+ data += line
+ if data.endswith((b"\r\r", b"\n\n", b"\r\n\r\n")):
+ yield data
+ data = b""
+ if data:
+ yield data
+
+ async def async_events(self):
+ async for chunk in self._aread():
+ event = Event()
+ # Split before decoding so splitlines() only uses \r and \n
+ for line in chunk.splitlines():
+ # Decode the line.
+ line = line.decode(self._char_enc)
+
+ # Lines starting with a separator are comments and are to be
+ # ignored.
+ if not line.strip() or line.startswith(_FIELD_SEPARATOR):
+ continue
+
+ data = line.split(_FIELD_SEPARATOR, 1)
+ field = data[0]
+
+ # Ignore unknown fields.
+ if field not in event.__dict__:
+ self._logger.debug("Saw invalid field %s while parsing " "Server Side Event", field)
+ continue
+
+ if len(data) > 1:
+ # From the spec:
+ # "If value starts with a single U+0020 SPACE character,
+ # remove it from value."
+ if data[1].startswith(" "):
+ value = data[1][1:]
+ else:
+ value = data[1]
+ else:
+ # If no value is present after the separator,
+ # assume an empty value.
+ value = ""
+
+ # The data field may come over multiple lines and their values
+ # are concatenated with each other.
+ if field == "data":
+ event.__dict__[field] += value + "\n"
+ else:
+ event.__dict__[field] = value
+
+ # Events with no data are not dispatched.
+ if not event.data:
+ continue
+
+ # If the data field ends with a newline, remove it.
+ if event.data.endswith("\n"):
+ event.data = event.data[0:-1]
+
+ # Empty event names default to 'message'
+ event.event = event.event or "message"
+
+ # Dispatch the event
+ self._logger.debug("Dispatching %s...", event)
+ yield event
diff --git a/metagpt/provider/zhipuai/zhipu_model_api.py b/metagpt/provider/zhipuai/zhipu_model_api.py
new file mode 100644
index 000000000..23dd7229d
--- /dev/null
+++ b/metagpt/provider/zhipuai/zhipu_model_api.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : zhipu model api to support sync & async for invoke & sse_invoke
+
+import zhipuai
+from zhipuai.model_api.api import InvokeType, ModelAPI
+from zhipuai.utils.http_client import headers as zhipuai_default_headers
+
+from metagpt.provider.general_api_requestor import GeneralAPIRequestor
+from metagpt.provider.zhipuai.async_sse_client import AsyncSSEClient
+
+
+class ZhiPuModelAPI(ModelAPI):
+ @classmethod
+ def get_header(cls) -> dict:
+ token = cls._generate_token()
+ zhipuai_default_headers.update({"Authorization": token})
+ return zhipuai_default_headers
+
+ @classmethod
+ def get_sse_header(cls) -> dict:
+ token = cls._generate_token()
+ headers = {"Authorization": token}
+ return headers
+
+ @classmethod
+ def split_zhipu_api_url(cls, invoke_type: InvokeType, kwargs):
+ # use this method to prevent zhipu api upgrading to different version.
+ # and follow the GeneralAPIRequestor implemented based on openai sdk
+ zhipu_api_url = cls._build_api_url(kwargs, invoke_type)
+ """
+ example:
+ zhipu_api_url: https://open.bigmodel.cn/api/paas/v3/model-api/{model}/{invoke_method}
+ """
+ arr = zhipu_api_url.split("/api/")
+ # ("https://open.bigmodel.cn/api/" , "/paas/v3/model-api/chatglm_turbo/invoke")
+ return f"{arr[0]}/api", f"/{arr[1]}"
+
+ @classmethod
+ async def arequest(cls, invoke_type: InvokeType, stream: bool, method: str, headers: dict, kwargs):
+ # TODO to make the async request to be more generic for models in http mode.
+ assert method in ["post", "get"]
+
+ api_base, url = cls.split_zhipu_api_url(invoke_type, kwargs)
+ requester = GeneralAPIRequestor(api_base=api_base)
+ result, _, api_key = await requester.arequest(
+ method=method,
+ url=url,
+ headers=headers,
+ stream=stream,
+ params=kwargs,
+ request_timeout=zhipuai.api_timeout_seconds,
+ )
+
+ return result
+
+ @classmethod
+ async def ainvoke(cls, **kwargs) -> dict:
+ """async invoke different from raw method `async_invoke` which get the final result by task_id"""
+ headers = cls.get_header()
+ resp = await cls.arequest(
+ invoke_type=InvokeType.SYNC, stream=False, method="post", headers=headers, kwargs=kwargs
+ )
+ return resp
+
+ @classmethod
+ async def asse_invoke(cls, **kwargs) -> AsyncSSEClient:
+ """async sse_invoke"""
+ headers = cls.get_sse_header()
+ return AsyncSSEClient(
+ await cls.arequest(invoke_type=InvokeType.SSE, stream=True, method="post", headers=headers, kwargs=kwargs)
+ )
diff --git a/metagpt/provider/zhipuai_api.py b/metagpt/provider/zhipuai_api.py
new file mode 100644
index 000000000..edd9084e3
--- /dev/null
+++ b/metagpt/provider/zhipuai_api.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : zhipuai LLM from https://open.bigmodel.cn/dev/api#sdk
+
+import json
+from enum import Enum
+
+import openai
+import zhipuai
+from requests import ConnectionError
+from tenacity import (
+ after_log,
+ retry,
+ retry_if_exception_type,
+ stop_after_attempt,
+ wait_fixed,
+)
+
+from metagpt.config import CONFIG
+from metagpt.logs import logger
+from metagpt.provider.base_gpt_api import BaseGPTAPI
+from metagpt.provider.openai_api import CostManager, log_and_reraise
+from metagpt.provider.zhipuai.zhipu_model_api import ZhiPuModelAPI
+
+
+class ZhiPuEvent(Enum):
+ ADD = "add"
+ ERROR = "error"
+ INTERRUPTED = "interrupted"
+ FINISH = "finish"
+
+
+class ZhiPuAIGPTAPI(BaseGPTAPI):
+ """
+ Refs to `https://open.bigmodel.cn/dev/api#chatglm_turbo`
+ From now, there is only one model named `chatglm_turbo`
+ """
+
+ use_system_prompt: bool = False # zhipuai has no system prompt when use api
+
+ def __init__(self):
+ self.__init_zhipuai(CONFIG)
+ self.llm = ZhiPuModelAPI
+ self.model = "chatglm_turbo" # so far only one model, just use it
+ self._cost_manager = CostManager()
+
+ def __init_zhipuai(self, config: CONFIG):
+ assert config.zhipuai_api_key
+ zhipuai.api_key = config.zhipuai_api_key
+ openai.api_key = zhipuai.api_key # due to use openai sdk, set the api_key but it will't be used.
+
+ def _const_kwargs(self, messages: list[dict]) -> dict:
+ kwargs = {"model": self.model, "prompt": messages, "temperature": 0.3}
+ return kwargs
+
+ def _update_costs(self, usage: dict):
+ """update each request's token cost"""
+ if CONFIG.calc_usage:
+ try:
+ prompt_tokens = int(usage.get("prompt_tokens", 0))
+ completion_tokens = int(usage.get("completion_tokens", 0))
+ self._cost_manager.update_cost(prompt_tokens, completion_tokens, self.model)
+ except Exception as e:
+ logger.error("zhipuai updats costs failed!", e)
+
+ def get_choice_text(self, resp: dict) -> str:
+ """get the first text of choice from llm response"""
+ assist_msg = resp.get("data", {}).get("choices", [{"role": "error"}])[-1]
+ assert assist_msg["role"] == "assistant"
+ return assist_msg.get("content")
+
+ def completion(self, messages: list[dict]) -> dict:
+ resp = self.llm.invoke(**self._const_kwargs(messages))
+ usage = resp.get("data").get("usage")
+ self._update_costs(usage)
+ return resp
+
+ async def _achat_completion(self, messages: list[dict]) -> dict:
+ resp = await self.llm.ainvoke(**self._const_kwargs(messages))
+ usage = resp.get("data").get("usage")
+ self._update_costs(usage)
+ return resp
+
+ async def acompletion(self, messages: list[dict]) -> dict:
+ return await self._achat_completion(messages)
+
+ async def _achat_completion_stream(self, messages: list[dict]) -> str:
+ response = await self.llm.asse_invoke(**self._const_kwargs(messages))
+ collected_content = []
+ usage = {}
+ async for event in response.async_events():
+ if event.event == ZhiPuEvent.ADD.value:
+ content = event.data
+ collected_content.append(content)
+ print(content, end="")
+ elif event.event == ZhiPuEvent.ERROR.value or event.event == ZhiPuEvent.INTERRUPTED.value:
+ content = event.data
+ logger.error(f"event error: {content}", end="")
+ collected_content.append([content])
+ elif event.event == ZhiPuEvent.FINISH.value:
+ """
+ event.meta
+ {
+ "task_status":"SUCCESS",
+ "usage":{
+ "completion_tokens":351,
+ "prompt_tokens":595,
+ "total_tokens":946
+ },
+ "task_id":"xx",
+ "request_id":"xxx"
+ }
+ """
+ meta = json.loads(event.meta)
+ usage = meta.get("usage")
+ else:
+ print(f"zhipuapi else event: {event.data}", end="")
+
+ self._update_costs(usage)
+ full_content = "".join(collected_content)
+ return full_content
+
+ @retry(
+ stop=stop_after_attempt(3),
+ wait=wait_fixed(1),
+ after=after_log(logger, logger.level("WARNING").name),
+ retry=retry_if_exception_type(ConnectionError),
+ retry_error_callback=log_and_reraise,
+ )
+ async def acompletion_text(self, messages: list[dict], stream=False) -> str:
+ """response in async with stream or non-stream mode"""
+ if stream:
+ return await self._achat_completion_stream(messages)
+ resp = await self._achat_completion(messages)
+ return self.get_choice_text(resp)
diff --git a/metagpt/repo_parser.py b/metagpt/repo_parser.py
new file mode 100644
index 000000000..b84dbab9a
--- /dev/null
+++ b/metagpt/repo_parser.py
@@ -0,0 +1,94 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/17 17:58
+@Author : alexanderwu
+@File : repo_parser.py
+"""
+import ast
+import json
+from pathlib import Path
+from pprint import pformat
+
+import pandas as pd
+from pydantic import BaseModel, Field
+
+from metagpt.config import CONFIG
+from metagpt.logs import logger
+
+
+class RepoParser(BaseModel):
+ base_directory: Path = Field(default=None)
+
+ def parse_file(self, file_path):
+ """Parse a Python file in the repository."""
+ try:
+ return ast.parse(file_path.read_text()).body
+ except:
+ return []
+
+ def extract_class_and_function_info(self, tree, file_path):
+ """Extract class, function, and global variable information from the AST."""
+ file_info = {
+ "file": str(file_path.relative_to(self.base_directory)),
+ "classes": [],
+ "functions": [],
+ "globals": [],
+ }
+
+ for node in tree:
+ if isinstance(node, ast.ClassDef):
+ class_methods = [m.name for m in node.body if is_func(m)]
+ file_info["classes"].append({"name": node.name, "methods": class_methods})
+ elif is_func(node):
+ file_info["functions"].append(node.name)
+ elif isinstance(node, (ast.Assign, ast.AnnAssign)):
+ for target in node.targets if isinstance(node, ast.Assign) else [node.target]:
+ if isinstance(target, ast.Name):
+ file_info["globals"].append(target.id)
+ return file_info
+
+ def generate_symbols(self):
+ files_classes = []
+ directory = self.base_directory
+ for path in directory.rglob("*.py"):
+ tree = self.parse_file(path)
+ file_info = self.extract_class_and_function_info(tree, path)
+ files_classes.append(file_info)
+
+ return files_classes
+
+ def generate_json_structure(self, output_path):
+ """Generate a JSON file documenting the repository structure."""
+ files_classes = self.generate_symbols()
+ output_path.write_text(json.dumps(files_classes, indent=4))
+
+ def generate_dataframe_structure(self, output_path):
+ """Generate a DataFrame documenting the repository structure and save as CSV."""
+ files_classes = self.generate_symbols()
+ df = pd.DataFrame(files_classes)
+ df.to_csv(output_path, index=False)
+
+ def generate_structure(self, output_path=None, mode="json"):
+ """Generate the structure of the repository as a specified format."""
+ output_file = self.base_directory / f"{self.base_directory.name}-structure.{mode}"
+ output_path = Path(output_path) if output_path else output_file
+
+ if mode == "json":
+ self.generate_json_structure(output_path)
+ elif mode == "csv":
+ self.generate_dataframe_structure(output_path)
+
+
+def is_func(node):
+ return isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+
+
+def main():
+ repo_parser = RepoParser(base_directory=CONFIG.workspace_path / "web_2048")
+ symbols = repo_parser.generate_symbols()
+ logger.info(pformat(symbols))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/metagpt/roles/engineer.py b/metagpt/roles/engineer.py
index ffd96849b..d42835a1b 100644
--- a/metagpt/roles/engineer.py
+++ b/metagpt/roles/engineer.py
@@ -10,46 +10,29 @@
2. Consolidate message reception and processing logic within `_observe`.
3. Fix bug: Add logic for handling asynchronous message processing when messages are not ready.
4. Supplemented the external transmission of internal messages.
+@Modified By: mashenquan, 2023-11-27.
+ 1. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
+ 2. According to the design in Section 2.2.3.5.5 of RFC 135, add incremental iteration functionality.
"""
-import asyncio
-import shutil
-from collections import OrderedDict
-from pathlib import Path
+from __future__ import annotations
-from metagpt.actions import WriteCode, WriteCodeReview, WriteDesign, WriteTasks
-from metagpt.const import WORKSPACE_ROOT
+import json
+from pathlib import Path
+from typing import Set
+
+from metagpt.actions import Action, WriteCode, WriteCodeReview, WriteTasks
+from metagpt.actions.summarize_code import SummarizeCode
+from metagpt.config import CONFIG
+from metagpt.const import MESSAGE_ROUTE_TO_NONE, SYSTEM_DESIGN_FILE_REPO, TASK_FILE_REPO
from metagpt.logs import logger
from metagpt.roles import Role
-from metagpt.schema import Message
-from metagpt.utils.common import CodeParser, any_to_str
-from metagpt.utils.special_tokens import FILENAME_CODE_SEP, MSG_SEP
-
-
-async def gather_ordered_k(coros, k) -> list:
- tasks = OrderedDict()
- results = [None] * len(coros)
- done_queue = asyncio.Queue()
-
- for i, coro in enumerate(coros):
- if len(tasks) >= k:
- done, _ = await asyncio.wait(tasks.keys(), return_when=asyncio.FIRST_COMPLETED)
- for task in done:
- index = tasks.pop(task)
- await done_queue.put((index, task.result()))
- task = asyncio.create_task(coro)
- tasks[task] = i
-
- if tasks:
- done, _ = await asyncio.wait(tasks.keys())
- for task in done:
- index = tasks[task]
- await done_queue.put((index, task.result()))
-
- while not done_queue.empty():
- index, result = await done_queue.get()
- results[index] = result
-
- return results
+from metagpt.schema import (
+ CodeSummarizeContext,
+ CodingContext,
+ Document,
+ Documents,
+ Message,
+)
class Engineer(Role):
@@ -63,7 +46,6 @@ class Engineer(Role):
constraints (str): Constraints for the engineer.
n_borg (int): Number of borgs.
use_code_review (bool): Whether to use code review.
- todos (list): List of tasks.
"""
def __init__(
@@ -77,104 +59,21 @@ class Engineer(Role):
) -> None:
"""Initializes the Engineer role with given attributes."""
super().__init__(name, profile, goal, constraints)
- self._init_actions([WriteCode])
self.use_code_review = use_code_review
- if self.use_code_review:
- self._init_actions([WriteCode, WriteCodeReview])
self._watch([WriteTasks])
- self.todos = []
+ self.code_todos = []
+ self.summarize_todos = []
self.n_borg = n_borg
- @classmethod
- def parse_tasks(self, task_msg: Message) -> list[str]:
- if task_msg.instruct_content:
- return task_msg.instruct_content.dict().get("Task list")
- return CodeParser.parse_file_list(block="Task list", text=task_msg.content)
+ @staticmethod
+ def _parse_tasks(task_msg: Document) -> list[str]:
+ m = json.loads(task_msg.content)
+ return m.get("Task list")
- @classmethod
- def parse_code(self, code_text: str) -> str:
- return CodeParser.parse_code(block="", text=code_text)
-
- @classmethod
- def parse_workspace(cls, system_design_msg: Message) -> str:
- if system_design_msg.instruct_content:
- return system_design_msg.instruct_content.dict().get("Python package name").strip().strip("'").strip('"')
- return CodeParser.parse_str(block="Python package name", text=system_design_msg.content)
-
- def get_workspace(self) -> Path:
- msg = self._rc.memory.get_by_action(WriteDesign)[-1]
- if not msg:
- return WORKSPACE_ROOT / "src"
- workspace = self.parse_workspace(msg)
- # Codes are written in workspace/{package_name}/{package_name}
- return WORKSPACE_ROOT / workspace / workspace
-
- def recreate_workspace(self):
- workspace = self.get_workspace()
- try:
- shutil.rmtree(workspace)
- except FileNotFoundError:
- pass # The folder does not exist, but we don't care
- workspace.mkdir(parents=True, exist_ok=True)
-
- def write_file(self, filename: str, code: str):
- workspace = self.get_workspace()
- filename = filename.replace('"', "").replace("\n", "")
- file = workspace / filename
- file.parent.mkdir(parents=True, exist_ok=True)
- file.write_text(code)
- return file
-
- async def _act_mp(self) -> Message:
- # self.recreate_workspace()
- todo_coros = []
- for todo in self.todos:
- todo_coro = WriteCode().run(
- context=self._rc.memory.get_by_actions([WriteTasks, WriteDesign]),
- filename=todo,
- )
- todo_coros.append(todo_coro)
-
- rsps = await gather_ordered_k(todo_coros, self.n_borg)
- for todo, code_rsp in zip(self.todos, rsps):
- _ = self.parse_code(code_rsp)
- logger.info(todo)
- logger.info(code_rsp)
- # self.write_file(todo, code)
- msg = Message(content=code_rsp, role=self.profile, cause_by=self._rc.todo)
- self._rc.memory.add(msg)
- del self.todos[0]
-
- logger.info(f"Done {self.get_workspace()} generating.")
- msg = Message(content="all done.", role=self.profile, cause_by=self._rc.todo)
- return msg
-
- async def _act_sp(self) -> Message:
- code_msg_all = [] # gather all code info, will pass to qa_engineer for tests later
- for todo in self.todos:
- code = await WriteCode().run(context=self._rc.history, filename=todo)
- # logger.info(todo)
- # logger.info(code_rsp)
- # code = self.parse_code(code_rsp)
- file_path = self.write_file(todo, code)
- msg = Message(content=code, role=self.profile, cause_by=self._rc.todo)
- self._rc.memory.add(msg)
-
- code_msg = todo + FILENAME_CODE_SEP + str(file_path)
- code_msg_all.append(code_msg)
-
- logger.info(f"Done {self.get_workspace()} generating.")
- msg = Message(
- content=MSG_SEP.join(code_msg_all),
- role=self.profile,
- cause_by=self._rc.todo,
- send_to="Edward", # name of QaEngineer
- )
- return msg
-
- async def _act_sp_precision(self) -> Message:
- code_msg_all = [] # gather all code info, will pass to qa_engineer for tests later
- for todo in self.todos:
+ async def _act_sp_with_cr(self, review=False) -> Set[str]:
+ changed_files = set()
+ src_file_repo = CONFIG.git_repo.new_file_repository(CONFIG.src_workspace)
+ for todo in self.code_todos:
"""
# Select essential information from the historical data to reduce the length of the prompt (summarized from human experience):
1. All from Architect
@@ -182,55 +81,153 @@ class Engineer(Role):
3. Do we need other codes (currently needed)?
TODO: The goal is not to need it. After clear task decomposition, based on the design idea, you should be able to write a single file without needing other codes. If you can't, it means you need a clearer definition. This is the key to writing longer code.
"""
- context = []
- msg = self._rc.memory.get_by_actions([WriteDesign, WriteTasks, WriteCode])
- for m in msg:
- context.append(m.content)
- context_str = "\n".join(context)
- # Write code
- code = await WriteCode().run(context=context_str, filename=todo)
+ coding_context = await todo.run()
# Code review
- if self.use_code_review:
- try:
- rewrite_code = await WriteCodeReview().run(context=context_str, code=code, filename=todo)
- code = rewrite_code
- except Exception as e:
- logger.error("code review failed!", e)
- pass
- file_path = self.write_file(todo, code)
- msg = Message(content=code, role=self.profile, cause_by=WriteCode)
+ if review:
+ coding_context = await WriteCodeReview(context=coding_context, llm=self._llm).run()
+ await src_file_repo.save(
+ coding_context.filename,
+ dependencies={coding_context.design_doc.root_relative_path, coding_context.task_doc.root_relative_path},
+ content=coding_context.code_doc.content,
+ )
+ msg = Message(
+ content=coding_context.json(), instruct_content=coding_context, role=self.profile, cause_by=WriteCode
+ )
self._rc.memory.add(msg)
- code_msg = todo + FILENAME_CODE_SEP + str(file_path)
- code_msg_all.append(code_msg)
+ changed_files.add(coding_context.code_doc.filename)
+ if not changed_files:
+ logger.info("Nothing has changed.")
+ return changed_files
- logger.info(f"Done {self.get_workspace()} generating.")
- msg = Message(
- content=MSG_SEP.join(code_msg_all),
- role=self.profile,
- cause_by=self._rc.todo,
- send_to="Edward", # name of QaEngineer
- )
- return msg
-
- async def _act(self) -> Message:
+ async def _act(self) -> Message | None:
"""Determines the mode of action based on whether code review is used."""
- if not self._rc.todo:
+ if self._rc.todo is None:
return None
- if self.use_code_review:
- return await self._act_sp_precision()
- return await self._act_sp()
+ if isinstance(self._rc.todo, WriteCode):
+ changed_files = await self._act_sp_with_cr(review=self.use_code_review)
+ # Unit tests only.
+ if CONFIG.REQA_FILENAME and CONFIG.REQA_FILENAME not in changed_files:
+ changed_files.add(CONFIG.REQA_FILENAME)
+ return Message(
+ content="\n".join(changed_files),
+ role=self.profile,
+ cause_by=WriteCodeReview if self.use_code_review else WriteCode,
+ send_to="Edward", # The name of QaEngineer
+ )
+ if isinstance(self._rc.todo, SummarizeCode):
+ summaries = []
+ for todo in self.summarize_todos:
+ summary = await todo.run()
+ summaries.append(summary.json(ensure_ascii=False))
+ return Message(
+ content="\n".join(summaries),
+ role=self.profile,
+ cause_by=SummarizeCode,
+ send_to=MESSAGE_ROUTE_TO_NONE,
+ )
+ return None
- async def _observe(self) -> int:
- ret = await super(Engineer, self)._observe()
- if ret == 0:
- return ret
+ async def _think(self) -> Action | None:
+ if not CONFIG.src_workspace:
+ CONFIG.src_workspace = CONFIG.git_repo.workdir / CONFIG.git_repo.workdir.name
+ if not self.code_todos:
+ await self._new_code_actions()
+ elif not self.summarize_todos:
+ await self._new_summarize_actions()
+ else:
+ return None
+ return self._rc.todo # For agent store
- # Parse task lists
- for message in self._rc.news:
- if not message.cause_by == any_to_str(WriteTasks):
+ @staticmethod
+ async def _new_coding_context(
+ filename, src_file_repo, task_file_repo, design_file_repo, dependency
+ ) -> CodingContext:
+ old_code_doc = await src_file_repo.get(filename)
+ if not old_code_doc:
+ old_code_doc = Document(root_path=str(src_file_repo.root_path), filename=filename, content="")
+ dependencies = {Path(i) for i in await dependency.get(old_code_doc.root_relative_path)}
+ task_doc = None
+ design_doc = None
+ for i in dependencies:
+ if str(i.parent) == TASK_FILE_REPO:
+ task_doc = task_file_repo.get(i.filename)
+ elif str(i.parent) == SYSTEM_DESIGN_FILE_REPO:
+ design_doc = design_file_repo.get(i.filename)
+ context = CodingContext(filename=filename, design_doc=design_doc, task_doc=task_doc, code_doc=old_code_doc)
+ return context
+
+ @staticmethod
+ async def _new_coding_doc(filename, src_file_repo, task_file_repo, design_file_repo, dependency):
+ context = await Engineer._new_coding_context(
+ filename, src_file_repo, task_file_repo, design_file_repo, dependency
+ )
+ coding_doc = Document(root_path=str(src_file_repo.root_path), filename=filename, content=context.json())
+ return coding_doc
+
+ async def _new_code_actions(self):
+ # Prepare file repos
+ src_file_repo = CONFIG.git_repo.new_file_repository(CONFIG.src_workspace)
+ changed_src_files = src_file_repo.changed_files
+ task_file_repo = CONFIG.git_repo.new_file_repository(TASK_FILE_REPO)
+ changed_task_files = task_file_repo.changed_files
+ design_file_repo = CONFIG.git_repo.new_file_repository(SYSTEM_DESIGN_FILE_REPO)
+
+ changed_files = Documents()
+ # Recode caused by upstream changes.
+ for filename in changed_task_files:
+ design_doc = await design_file_repo.get(filename)
+ task_doc = await task_file_repo.get(filename)
+ task_list = self._parse_tasks(task_doc)
+ for task_filename in task_list:
+ old_code_doc = await src_file_repo.get(task_filename)
+ if not old_code_doc:
+ old_code_doc = Document(root_path=str(src_file_repo.root_path), filename=task_filename, content="")
+ context = CodingContext(
+ filename=task_filename, design_doc=design_doc, task_doc=task_doc, code_doc=old_code_doc
+ )
+ coding_doc = Document(
+ root_path=str(src_file_repo.root_path), filename=task_filename, content=context.json()
+ )
+ if task_filename in changed_files.docs:
+ logger.warning(
+ f"Log to expose potential conflicts: {coding_doc.json()} & "
+ f"{changed_files.docs[task_filename].json()}"
+ )
+ changed_files.docs[task_filename] = coding_doc
+ self.code_todos = [WriteCode(context=i, llm=self._llm) for i in changed_files.docs.values()]
+ # Code directly modified by the user.
+ dependency = await CONFIG.git_repo.get_dependency()
+ for filename in changed_src_files:
+ if filename in changed_files.docs:
continue
- self.todos = self.parse_tasks(message)
- return 1
+ coding_doc = await self._new_coding_doc(
+ filename=filename,
+ src_file_repo=src_file_repo,
+ task_file_repo=task_file_repo,
+ design_file_repo=design_file_repo,
+ dependency=dependency,
+ )
+ changed_files.docs[filename] = coding_doc
+ self.code_todos.append(WriteCode(context=coding_doc, llm=self._llm))
- return 0
+ if self.code_todos:
+ self._rc.todo = self.code_todos[0]
+
+ async def _new_summarize_actions(self):
+ src_file_repo = CONFIG.git_repo.new_file_repository(CONFIG.src_workspace)
+ changed_src_files = src_file_repo.changed_files
+ # Generate a SummarizeCode action for each pair of (system_design_doc, task_doc).
+ summarizations = {}
+ for filename in changed_src_files:
+ dependencies = src_file_repo.get_dependency(filename=filename)
+ ctx = CodeSummarizeContext.loads(filenames=dependencies)
+ if ctx not in summarizations:
+ summarizations[ctx] = set()
+ srcs = summarizations.get(ctx)
+ srcs.add(filename)
+ for ctx, filenames in summarizations.items():
+ ctx.codes_filenames = filenames
+ self.summarize_todos.append(SummarizeCode(context=ctx, llm=self._llm))
+ if self.summarize_todos:
+ self._rc.todo = self.summarize_todos[0]
diff --git a/metagpt/roles/invoice_ocr_assistant.py b/metagpt/roles/invoice_ocr_assistant.py
index 3087a4da7..bf8fc454e 100644
--- a/metagpt/roles/invoice_ocr_assistant.py
+++ b/metagpt/roles/invoice_ocr_assistant.py
@@ -42,17 +42,7 @@ class InvoiceOCRAssistant(Role):
self.filename = ""
self.origin_query = ""
self.orc_data = None
-
- async def _think(self) -> None:
- """Determine the next action to be taken by the role."""
- if self._rc.todo is None:
- self._set_state(0)
- return
-
- if self._rc.state + 1 < len(self._states):
- self._set_state(self._rc.state + 1)
- else:
- self._rc.todo = None
+ self._set_react_mode(react_mode="by_order")
async def _act(self) -> Message:
"""Perform an action as determined by the role.
@@ -94,16 +84,3 @@ class InvoiceOCRAssistant(Role):
msg = Message(content=content, instruct_content=resp)
self._rc.memory.add(msg)
return msg
-
- async def _react(self) -> Message:
- """Execute the invoice ocr assistant's think and actions.
-
- Returns:
- A message containing the final result of the assistant's actions.
- """
- while True:
- await self._think()
- if self._rc.todo is None:
- break
- msg = await self._act()
- return msg
diff --git a/metagpt/roles/product_manager.py b/metagpt/roles/product_manager.py
index a58ea5385..017feade7 100644
--- a/metagpt/roles/product_manager.py
+++ b/metagpt/roles/product_manager.py
@@ -4,8 +4,12 @@
@Time : 2023/5/11 14:43
@Author : alexanderwu
@File : product_manager.py
+@Modified By: mashenquan, 2023/11/27. Add `PrepareDocuments` action according to Section 2.2.3.5.1 of RFC 135.
"""
-from metagpt.actions import BossRequirement, WritePRD
+
+from metagpt.actions import UserRequirement, WritePRD
+from metagpt.actions.prepare_documents import PrepareDocuments
+from metagpt.config import CONFIG
from metagpt.roles import Role
@@ -37,5 +41,17 @@ class ProductManager(Role):
constraints (str): Constraints or limitations for the product manager.
"""
super().__init__(name, profile, goal, constraints)
- self._init_actions([WritePRD])
- self._watch([BossRequirement])
+
+ self._init_actions([PrepareDocuments, WritePRD])
+ self._watch([UserRequirement, PrepareDocuments])
+
+ async def _think(self) -> None:
+ """Decide what to do"""
+ if CONFIG.git_repo:
+ self._set_state(1)
+ else:
+ self._set_state(0)
+ return self._rc.todo
+
+ async def _observe(self, ignore_memory=False) -> int:
+ return await super(ProductManager, self)._observe(ignore_memory=True)
diff --git a/metagpt/roles/qa_engineer.py b/metagpt/roles/qa_engineer.py
index 59a4135b8..41a3213dc 100644
--- a/metagpt/roles/qa_engineer.py
+++ b/metagpt/roles/qa_engineer.py
@@ -6,24 +6,25 @@
@File : qa_engineer.py
@Modified By: mashenquan, 2023-11-1. In accordance with Chapter 2.2.1 and 2.2.2 of RFC 116, modify the data
type of the `cause_by` value in the `Message` to a string, and utilize the new message filtering feature.
+@Modified By: mashenquan, 2023-11-27.
+ 1. Following the think-act principle, solidify the task parameters when creating the
+ WriteTest/RunCode/DebugError object, rather than passing them in when calling the run function.
+ 2. According to Section 2.2.3.5.7 of RFC 135, change the method of transferring files from using the Message
+ to using file references.
"""
-import os
-from pathlib import Path
+from metagpt.actions import DebugError, RunCode, WriteCode, WriteCodeReview, WriteTest
-from metagpt.actions import (
- DebugError,
- RunCode,
- WriteCode,
- WriteCodeReview,
- WriteDesign,
- WriteTest,
+# from metagpt.const import WORKSPACE_ROOT
+from metagpt.config import CONFIG
+from metagpt.const import (
+ MESSAGE_ROUTE_TO_NONE,
+ TEST_CODES_FILE_REPO,
+ TEST_OUTPUTS_FILE_REPO,
)
-from metagpt.const import WORKSPACE_ROOT
from metagpt.logs import logger
from metagpt.roles import Role
-from metagpt.schema import Message
-from metagpt.utils.common import CodeParser, any_to_str_set, parse_recipient
-from metagpt.utils.special_tokens import FILENAME_CODE_SEP, MSG_SEP
+from metagpt.schema import Document, Message, RunCodeContext, TestingContext
+from metagpt.utils.common import any_to_str_set, parse_recipient
class QaEngineer(Role):
@@ -43,113 +44,96 @@ class QaEngineer(Role):
self.test_round = 0
self.test_round_allowed = test_round_allowed
- @classmethod
- def parse_workspace(cls, system_design_msg: Message) -> str:
- if system_design_msg.instruct_content:
- return system_design_msg.instruct_content.dict().get("Python package name")
- return CodeParser.parse_str(block="Python package name", text=system_design_msg.content)
-
- def get_workspace(self, return_proj_dir=True) -> Path:
- msg = self._rc.memory.get_by_action(WriteDesign)[-1]
- if not msg:
- return WORKSPACE_ROOT / "src"
- workspace = self.parse_workspace(msg)
- # project directory: workspace/{package_name}, which contains package source code folder, tests folder, resources folder, etc.
- if return_proj_dir:
- return WORKSPACE_ROOT / workspace
- # development codes directory: workspace/{package_name}/{package_name}
- return WORKSPACE_ROOT / workspace / workspace
-
- def write_file(self, filename: str, code: str):
- workspace = self.get_workspace() / "tests"
- file = workspace / filename
- file.parent.mkdir(parents=True, exist_ok=True)
- file.write_text(code)
-
async def _write_test(self, message: Message) -> None:
- code_msgs = message.content.split(MSG_SEP)
- # result_msg_all = []
- for code_msg in code_msgs:
+ changed_files = message.content.splitlines()
+ src_file_repo = CONFIG.git_repo.new_file_repository(CONFIG.src_workspace)
+ tests_file_repo = CONFIG.git_repo.new_file_repository(TEST_CODES_FILE_REPO)
+ for filename in changed_files:
# write tests
- file_name, file_path = code_msg.split(FILENAME_CODE_SEP)
- code_to_test = open(file_path, "r").read()
- if "test" in file_name:
- continue # Engineer might write some test files, skip testing a test file
- test_file_name = "test_" + file_name
- test_file_path = self.get_workspace() / "tests" / test_file_name
- logger.info(f"Writing {test_file_name}..")
- test_code = await WriteTest().run(
- code_to_test=code_to_test,
- test_file_name=test_file_name,
- # source_file_name=file_name,
- source_file_path=file_path,
- workspace=self.get_workspace(),
+ if not filename or "test" in filename:
+ continue
+ code_doc = await src_file_repo.get(filename)
+ test_doc = await tests_file_repo.get("test_" + code_doc.filename)
+ if not test_doc:
+ test_doc = Document(
+ root_path=str(tests_file_repo.root_path), filename="test_" + code_doc.filename, content=""
+ )
+ logger.info(f"Writing {test_doc.filename}..")
+ context = TestingContext(filename=test_doc.filename, test_doc=test_doc, code_doc=code_doc)
+ context = await WriteTest(context=context, llm=self._llm).run()
+ await tests_file_repo.save(
+ filename=context.test_doc.filename,
+ content=context.test_doc.content,
+ dependencies={context.code_doc.root_relative_path},
)
- self.write_file(test_file_name, test_code)
# prepare context for run tests in next round
- command = ["python", f"tests/{test_file_name}"]
- file_info = {
- "file_name": file_name,
- "file_path": str(file_path),
- "test_file_name": test_file_name,
- "test_file_path": str(test_file_path),
- "command": command,
- }
- msg = Message(
- content=str(file_info),
- role=self.profile,
- cause_by=WriteTest,
- sent_from=self,
- send_to=self,
+ run_code_context = RunCodeContext(
+ command=["python", context.test_doc.root_relative_path],
+ code_filename=context.code_doc.filename,
+ test_filename=context.test_doc.filename,
+ working_directory=str(CONFIG.git_repo.workdir),
+ additional_python_paths=[str(CONFIG.src_workspace)],
+ )
+ self.publish_message(
+ Message(
+ content=run_code_context.json(),
+ role=self.profile,
+ cause_by=WriteTest,
+ sent_from=self,
+ send_to=self,
+ )
)
- self.publish_message(msg)
- logger.info(f"Done {self.get_workspace()}/tests generating.")
+ logger.info(f"Done {str(tests_file_repo.workdir)} generating.")
async def _run_code(self, msg):
- file_info = eval(msg.content)
- development_file_path = file_info["file_path"]
- test_file_path = file_info["test_file_path"]
- if not os.path.exists(development_file_path) or not os.path.exists(test_file_path):
+ run_code_context = RunCodeContext.loads(msg.content)
+ src_doc = await CONFIG.git_repo.new_file_repository(CONFIG.src_workspace).get(run_code_context.code_filename)
+ if not src_doc:
return
-
- development_code = open(development_file_path, "r").read()
- test_code = open(test_file_path, "r").read()
- proj_dir = self.get_workspace()
- development_code_dir = self.get_workspace(return_proj_dir=False)
-
- result_msg = await RunCode().run(
- mode="script",
- code=development_code,
- code_file_name=file_info["file_name"],
- test_code=test_code,
- test_file_name=file_info["test_file_name"],
- command=file_info["command"],
- working_directory=proj_dir, # workspace/package_name, will run tests/test_xxx.py here
- additional_python_paths=[development_code_dir], # workspace/package_name/package_name,
- # import statement inside package code needs this
+ test_doc = await CONFIG.git_repo.new_file_repository(TEST_CODES_FILE_REPO).get(run_code_context.test_filename)
+ if not test_doc:
+ return
+ run_code_context.code = src_doc.content
+ run_code_context.test_code = test_doc.content
+ result = await RunCode(context=run_code_context, llm=self._llm).run()
+ run_code_context.output_filename = run_code_context.test_filename + ".json"
+ await CONFIG.git_repo.new_file_repository(TEST_OUTPUTS_FILE_REPO).save(
+ filename=run_code_context.output_filename,
+ content=result.json(),
+ dependencies={src_doc.root_relative_path, test_doc.root_relative_path},
+ )
+ run_code_context.code = None
+ run_code_context.test_code = None
+ recipient = parse_recipient(result.summary) # the recipient might be Engineer or myself
+ mappings = {"Engineer": "Alex", "QaEngineer": "Edward"}
+ self.publish_message(
+ Message(
+ content=run_code_context.json(),
+ role=self.profile,
+ cause_by=RunCode,
+ sent_from=self,
+ send_to=mappings.get(recipient, MESSAGE_ROUTE_TO_NONE),
+ )
)
- recipient = parse_recipient(result_msg) # the recipient might be Engineer or myself
- content = str(file_info) + FILENAME_CODE_SEP + result_msg
- msg = Message(content=content, role=self.profile, cause_by=RunCode, sent_from=self.profile, send_to=recipient)
- self.publish_message(msg)
-
async def _debug_error(self, msg):
- file_info, context = msg.content.split(FILENAME_CODE_SEP)
- file_name, code = await DebugError().run(context)
- if file_name:
- self.write_file(file_name, code)
- recipient = msg.sent_from # send back to the one who ran the code for another run, might be one's self
- msg = Message(
- content=file_info,
+ run_code_context = RunCodeContext.loads(msg.content)
+ code = await DebugError(context=run_code_context, llm=self._llm).run()
+ await CONFIG.git_repo.new_file_repository(CONFIG.src_workspace).save(
+ filename=run_code_context.code_filename, content=code
+ )
+ run_code_context.output = None
+ self.publish_message(
+ Message(
+ content=run_code_context.json(),
role=self.profile,
cause_by=DebugError,
sent_from=self,
send_to=self,
)
- self.publish_message(msg)
+ )
async def _act(self) -> Message:
if self.test_round > self.test_round_allowed:
@@ -158,7 +142,7 @@ class QaEngineer(Role):
role=self.profile,
cause_by=WriteTest,
sent_from=self.profile,
- send_to=""
+ send_to=MESSAGE_ROUTE_TO_NONE,
)
return result_msg
@@ -178,11 +162,15 @@ class QaEngineer(Role):
# I ran my test code, time to fix bugs, if any
await self._debug_error(msg)
self.test_round += 1
- result_msg = Message(
+ return Message(
content=f"Round {self.test_round} of tests done",
role=self.profile,
cause_by=WriteTest,
sent_from=self.profile,
- send_to=""
+ send_to=MESSAGE_ROUTE_TO_NONE,
)
- return result_msg
+
+ async def _observe(self, ignore_memory=False) -> int:
+ # This role has events that trigger and execute themselves based on conditions, and cannot rely on the
+ # content of memory to activate.
+ return await super(QaEngineer, self)._observe(ignore_memory=True)
diff --git a/metagpt/roles/researcher.py b/metagpt/roles/researcher.py
index 29889b8ec..6a76b1a40 100644
--- a/metagpt/roles/researcher.py
+++ b/metagpt/roles/researcher.py
@@ -36,20 +36,11 @@ class Researcher(Role):
):
super().__init__(name, profile, goal, constraints, **kwargs)
self._init_actions([CollectLinks(name), WebBrowseAndSummarize(name), ConductResearch(name)])
+ self._set_react_mode(react_mode="by_order")
self.language = language
if language not in ("en-us", "zh-cn"):
logger.warning(f"The language `{language}` has not been tested, it may not work.")
- async def _think(self) -> None:
- if self._rc.todo is None:
- self._set_state(0)
- return
-
- if self._rc.state + 1 < len(self._states):
- self._set_state(self._rc.state + 1)
- else:
- self._rc.todo = None
-
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
todo = self._rc.todo
@@ -78,12 +69,8 @@ class Researcher(Role):
self._rc.memory.add(ret)
return ret
- async def _react(self) -> Message:
- while True:
- await self._think()
- if self._rc.todo is None:
- break
- msg = await self._act()
+ async def react(self) -> Message:
+ msg = await super().react()
report = msg.instruct_content
self.write_report(report.topic, report.content)
return msg
diff --git a/metagpt/roles/role.py b/metagpt/roles/role.py
index 424a28c6f..1e99cc1ff 100644
--- a/metagpt/roles/role.py
+++ b/metagpt/roles/role.py
@@ -20,21 +20,21 @@
"""
from __future__ import annotations
+from enum import Enum
from typing import Iterable, Set, Type
from pydantic import BaseModel, Field
from metagpt.actions import Action, ActionOutput
-
from metagpt.config import CONFIG
-from metagpt.llm import LLM
+from metagpt.llm import LLM, HumanProvider
from metagpt.logs import logger
-from metagpt.memory import LongTermMemory, Memory
+from metagpt.memory import Memory
+# from metagpt.memory import LongTermMemory
from metagpt.schema import Message, MessageQueue
from metagpt.utils.common import any_to_str
-
PREFIX_TEMPLATE = """You are a {profile}, named {name}, your goal is {goal}, and the constraint is {constraints}. """
STATE_TEMPLATE = """Here are your conversation records. You can decide which stage you should enter or stay in based on these records.
@@ -43,12 +43,14 @@ Please note that only the text between the first and second "===" is information
{history}
===
-You can now choose one of the following stages to decide the stage you need to go in the next step:
+Your previous stage: {previous_state}
+
+Now choose one of the following stages you need to go to in the next step:
{states}
Just answer a number between 0-{n_states}, choose the most suitable stage according to the understanding of the conversation.
Please note that the answer only needs a number, no need to add any other text.
-If there is no conversation record, choose 0.
+If you think you have completed your goal and don't need to go to any of the stages, return -1.
Do not answer anything else, and do not add any other information in your answer.
"""
@@ -63,6 +65,16 @@ ROLE_TEMPLATE = """Your response should be based on the previous conversation hi
"""
+class RoleReactMode(str, Enum):
+ REACT = "react"
+ BY_ORDER = "by_order"
+ PLAN_AND_ACT = "plan_and_act"
+
+ @classmethod
+ def values(cls):
+ return [item.value for item in cls]
+
+
class RoleSetting(BaseModel):
"""Role Settings"""
@@ -71,6 +83,7 @@ class RoleSetting(BaseModel):
goal: str
constraints: str
desc: str
+ is_human: bool
def __str__(self):
return f"{self.name}({self.profile})"
@@ -85,11 +98,15 @@ class RoleContext(BaseModel):
env: "Environment" = Field(default=None)
msg_buffer: MessageQueue = Field(default_factory=MessageQueue) # Message Buffer with Asynchronous Updates
memory: Memory = Field(default_factory=Memory)
- long_term_memory: LongTermMemory = Field(default_factory=LongTermMemory)
- state: int = Field(default=0)
+ # long_term_memory: LongTermMemory = Field(default_factory=LongTermMemory)
+ state: int = Field(default=-1) # -1 indicates initial or termination state where todo is None
todo: Action = Field(default=None)
watch: set[str] = Field(default_factory=set)
news: list[Type[Message]] = Field(default=[])
+ react_mode: RoleReactMode = (
+ RoleReactMode.REACT
+ ) # see `Role._set_react_mode` for definitions of the following two attributes
+ max_react_loop: int = 1
class Config:
arbitrary_types_allowed = True
@@ -112,9 +129,11 @@ class RoleContext(BaseModel):
class Role:
"""Role/Agent"""
- def __init__(self, name="", profile="", goal="", constraints="", desc=""):
- self._llm = LLM()
- self._setting = RoleSetting(name=name, profile=profile, goal=goal, constraints=constraints, desc=desc)
+ def __init__(self, name="", profile="", goal="", constraints="", desc="", is_human=False):
+ self._llm = LLM() if not is_human else HumanProvider()
+ self._setting = RoleSetting(
+ name=name, profile=profile, goal=goal, constraints=constraints, desc=desc, is_human=is_human
+ )
self._states = []
self._actions = []
self._role_id = str(self._setting)
@@ -129,13 +148,40 @@ class Role:
self._reset()
for idx, action in enumerate(actions):
if not isinstance(action, Action):
- i = action("")
+ i = action("", llm=self._llm)
else:
+ if self._setting.is_human and not isinstance(action.llm, HumanProvider):
+ logger.warning(
+ f"is_human attribute does not take effect,"
+ f"as Role's {str(action)} was initialized using LLM, try passing in Action classes instead of initialized instances"
+ )
i = action
+ i.set_env(self._rc.env)
i.set_prefix(self._get_prefix(), self.profile)
self._actions.append(i)
self._states.append(f"{idx}. {action}")
+ def _set_react_mode(self, react_mode: str, max_react_loop: int = 1):
+ """Set strategy of the Role reacting to observed Message. Variation lies in how
+ this Role elects action to perform during the _think stage, especially if it is capable of multiple Actions.
+
+ Args:
+ react_mode (str): Mode for choosing action during the _think stage, can be one of:
+ "react": standard think-act loop in the ReAct paper, alternating thinking and acting to solve the task, i.e. _think -> _act -> _think -> _act -> ...
+ Use llm to select actions in _think dynamically;
+ "by_order": switch action each time by order defined in _init_actions, i.e. _act (Action1) -> _act (Action2) -> ...;
+ "plan_and_act": first plan, then execute an action sequence, i.e. _think (of a plan) -> _act -> _act -> ...
+ Use llm to come up with the plan dynamically.
+ Defaults to "react".
+ max_react_loop (int): Maximum react cycles to execute, used to prevent the agent from reacting forever.
+ Take effect only when react_mode is react, in which we use llm to choose actions, including termination.
+ Defaults to 1, i.e. _think -> _act (-> return result and end)
+ """
+ assert react_mode in RoleReactMode.values(), f"react_mode must be one of {RoleReactMode.values()}"
+ self._rc.react_mode = react_mode
+ if react_mode == RoleReactMode.REACT:
+ self._rc.max_react_loop = max_react_loop
+
def _watch(self, actions: Iterable[Type[Action]]):
"""Watch Actions of interest. Role will select Messages caused by these Actions from its personal message
buffer during _observe.
@@ -154,11 +200,11 @@ class Role:
if self._rc.env: # According to the routing feature plan in Chapter 2.2.3.2 of RFC 113
self._rc.env.set_subscription(self, self._subscription)
- def _set_state(self, state):
+ def _set_state(self, state: int):
"""Update the current state."""
self._rc.state = state
logger.debug(self._actions)
- self._rc.todo = self._actions[self._rc.state]
+ self._rc.todo = self._actions[self._rc.state] if state >= 0 else None
def set_env(self, env: "Environment"):
"""Set the environment in which the role works. The role can talk to the environment and can also receive
@@ -167,6 +213,22 @@ class Role:
if env:
env.set_subscription(self, self._subscription)
+ # # Replaced by FileRepository.set_file
+ # def set_doc(self, content: str, filename: str):
+ # return self._rc.env.set_doc(content, filename)
+ #
+ # # Replaced by FileRepository.get_file
+ # def get_doc(self, filename: str):
+ # return self._rc.env.get_doc(filename)
+ #
+ # # Replaced by CONFIG.xx
+ # def set(self, k, v):
+ # return self._rc.env.set(k, v)
+ #
+ # # Replaced by CONFIG.xx
+ # def get(self, k):
+ # return self._rc.env.get(k)
+
@property
def profile(self):
"""Get the role description (position)"""
@@ -196,14 +258,22 @@ class Role:
return
prompt = self._get_prefix()
prompt += STATE_TEMPLATE.format(
- history=self._rc.history, states="\n".join(self._states), n_states=len(self._states) - 1
+ history=self._rc.history,
+ states="\n".join(self._states),
+ n_states=len(self._states) - 1,
+ previous_state=self._rc.state,
)
+ # print(prompt)
next_state = await self._llm.aask(prompt)
logger.debug(f"{prompt=}")
- if not next_state.isdigit() or int(next_state) not in range(len(self._states)):
- logger.warning(f"Invalid answer of state, {next_state=}")
- next_state = "0"
- self._set_state(int(next_state))
+ if (not next_state.isdigit() and next_state != "-1") or int(next_state) not in range(-1, len(self._states)):
+ logger.warning(f"Invalid answer of state, {next_state=}, will be set to -1")
+ next_state = -1
+ else:
+ next_state = int(next_state)
+ if next_state == -1:
+ logger.info(f"End actions with {next_state=}")
+ self._set_state(next_state)
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
@@ -221,12 +291,12 @@ class Role:
return msg
- async def _observe(self) -> int:
+ async def _observe(self, ignore_memory=False) -> int:
"""Prepare new messages for processing from the message buffer and other sources."""
# Read unprocessed messages from the msg buffer.
news = self._rc.msg_buffer.pop_all()
# Store the read messages in your own memory to prevent duplicate processing.
- old_messages = self._rc.memory.get()
+ old_messages = [] if ignore_memory else self._rc.memory.get()
self._rc.memory.add_batch(news)
# Filter out messages of interest.
self._rc.news = [n for n in news if n.cause_by in self._rc.watch and n not in old_messages]
@@ -255,10 +325,66 @@ class Role:
self._rc.msg_buffer.push(message)
async def _react(self) -> Message:
- """Think first, then act"""
- await self._think()
- logger.debug(f"{self._setting}: {self._rc.state=}, will do {self._rc.todo}")
- return await self._act()
+ """Think first, then act, until the Role _think it is time to stop and requires no more todo.
+ This is the standard think-act loop in the ReAct paper, which alternates thinking and acting in task solving, i.e. _think -> _act -> _think -> _act -> ...
+ Use llm to select actions in _think dynamically
+ """
+ actions_taken = 0
+ rsp = Message("No actions taken yet") # will be overwritten after Role _act
+ while actions_taken < self._rc.max_react_loop:
+ # think
+ await self._think()
+ if self._rc.todo is None:
+ break
+ # act
+ logger.debug(f"{self._setting}: {self._rc.state=}, will do {self._rc.todo}")
+ rsp = await self._act() # 这个rsp是否需要publish_message？
+ actions_taken += 1
+ return rsp # return output from the last action
+
+ async def _act_by_order(self) -> Message:
+ """switch action each time by order defined in _init_actions, i.e. _act (Action1) -> _act (Action2) -> ..."""
+ for i in range(len(self._states)):
+ self._set_state(i)
+ rsp = await self._act()
+ return rsp # return output from the last action
+
+ async def _plan_and_act(self) -> Message:
+ """first plan, then execute an action sequence, i.e. _think (of a plan) -> _act -> _act -> ... Use llm to come up with the plan dynamically."""
+ # TODO: to be implemented
+ return Message("")
+
+ async def react(self) -> Message:
+ """Entry to one of three strategies by which Role reacts to the observed Message"""
+ if self._rc.react_mode == RoleReactMode.REACT:
+ rsp = await self._react()
+ elif self._rc.react_mode == RoleReactMode.BY_ORDER:
+ rsp = await self._act_by_order()
+ elif self._rc.react_mode == RoleReactMode.PLAN_AND_ACT:
+ rsp = await self._plan_and_act()
+ self._set_state(state=-1) # current reaction is complete, reset state to -1 and todo back to None
+ return rsp
+
+ # # Replaced by run()
+ # def recv(self, message: Message) -> None:
+ # """add message to history."""
+ # # self._history += f"\n{message}"
+ # # self._context = self._history
+ # if message in self._rc.memory.get():
+ # return
+ # self._rc.memory.add(message)
+
+ # # Replaced by run()
+ # async def handle(self, message: Message) -> Message:
+ # """Receive information and reply with actions"""
+ # # logger.debug(f"{self.name=}, {self.profile=}, {message.role=}")
+ # self.recv(message)
+ #
+ # return await self._react()
+
+ def get_memories(self, k=0) -> list[Message]:
+ """A wrapper to return the most recent k memories of this role, return all when k=0"""
+ return self._rc.memory.get(k=k)
async def run(self, with_message=None):
"""Observe, and think and act based on the results of the observation"""
diff --git a/metagpt/roles/sk_agent.py b/metagpt/roles/sk_agent.py
index 2443b8b58..56482ef26 100644
--- a/metagpt/roles/sk_agent.py
+++ b/metagpt/roles/sk_agent.py
@@ -11,7 +11,7 @@ from semantic_kernel.planning import SequentialPlanner
from semantic_kernel.planning.action_planner.action_planner import ActionPlanner
from semantic_kernel.planning.basic_planner import BasicPlanner
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.actions.execute_task import ExecuteTask
from metagpt.logs import logger
from metagpt.roles import Role
@@ -41,7 +41,7 @@ class SkAgent(Role):
"""Initializes the Engineer role with given attributes."""
super().__init__(name, profile, goal, constraints)
self._init_actions([ExecuteTask()])
- self._watch([BossRequirement])
+ self._watch([UserRequirement])
self.kernel = make_sk_kernel()
# how funny the interface is inconsistent
diff --git a/metagpt/schema.py b/metagpt/schema.py
index b40d65d5f..d1174799a 100644
--- a/metagpt/schema.py
+++ b/metagpt/schema.py
@@ -6,22 +6,33 @@
@File : schema.py
@Modified By: mashenquan, 2023-10-31. According to Chapter 2.2.1 of RFC 116:
Replanned the distribution of responsibilities and functional positioning of `Message` class attributes.
+@Modified By: mashenquan, 2023/11/22.
+ 1. Add `Document` and `Documents` for `FileRepository` in Section 2.2.3.4 of RFC 135.
+ 2. Encapsulate the common key-values set to pydantic structures to standardize and unify parameter passing
+ between actions.
+ 3. Add `id` to `Message` according to Section 2.2.3.1.1 of RFC 135.
"""
from __future__ import annotations
import asyncio
import json
+import os.path
+import uuid
from asyncio import Queue, QueueEmpty, wait_for
from json import JSONDecodeError
-from typing import List, Set, TypedDict
+from pathlib import Path
+from typing import Dict, List, Optional, Set, TypedDict
from pydantic import BaseModel, Field
+from metagpt.config import CONFIG
from metagpt.const import (
MESSAGE_ROUTE_CAUSE_BY,
MESSAGE_ROUTE_FROM,
MESSAGE_ROUTE_TO,
MESSAGE_ROUTE_TO_ALL,
+ SYSTEM_DESIGN_FILE_REPO,
+ TASK_FILE_REPO,
)
from metagpt.logs import logger
from metagpt.utils.common import any_to_str, any_to_str_set
@@ -32,9 +43,52 @@ class RawMessage(TypedDict):
role: str
+class Document(BaseModel):
+ """
+ Represents a document.
+ """
+
+ root_path: str
+ filename: str
+ content: Optional[str] = None
+
+ def get_meta(self) -> Document:
+ """Get metadata of the document.
+
+ :return: A new Document instance with the same root path and filename.
+ """
+
+ return Document(root_path=self.root_path, filename=self.filename)
+
+ @property
+ def root_relative_path(self):
+ """Get relative path from root of git repository.
+
+ :return: relative path from root of git repository.
+ """
+ return os.path.join(self.root_path, self.filename)
+
+ @property
+ def full_path(self):
+ if not CONFIG.git_repo:
+ return None
+ return str(CONFIG.git_repo.workdir / self.root_path / self.filename)
+
+
+class Documents(BaseModel):
+ """A class representing a collection of documents.
+
+ Attributes:
+ docs (Dict[str, Document]): A dictionary mapping document names to Document instances.
+ """
+
+ docs: Dict[str, Document] = Field(default_factory=dict)
+
+
class Message(BaseModel):
"""list[: ]"""
+ id: str # According to Section 2.2.3.1.1 of RFC 135
content: str
instruct_content: BaseModel = Field(default=None)
role: str = "user" # system / user / assistant
@@ -62,6 +116,7 @@ class Message(BaseModel):
:param role: Message meta info tells who sent this message.
"""
super().__init__(
+ id=uuid.uuid4().hex,
content=content,
instruct_content=instruct_content,
role=role,
@@ -83,7 +138,6 @@ class Message(BaseModel):
new_val = val
super().__setattr__(key, new_val)
-
def __str__(self):
# prefix = '-'.join([self.role, str(self.cause_by)])
return f"{self.role}: {self.content}"
@@ -110,7 +164,6 @@ class Message(BaseModel):
return None
-
class UserMessage(Message):
"""便于支持OpenAI的消息
Facilitate support for OpenAI messages
@@ -120,7 +173,6 @@ class UserMessage(Message):
super().__init__(content=content, role="user")
-
class SystemMessage(Message):
"""便于支持OpenAI的消息
Facilitate support for OpenAI messages
@@ -204,3 +256,84 @@ class MessageQueue:
return q
+
+class CodingContext(BaseModel):
+ filename: str
+ design_doc: Document
+ task_doc: Document
+ code_doc: Document
+
+ @staticmethod
+ def loads(val: str) -> CodingContext | None:
+ try:
+ m = json.loads(val)
+ return CodingContext(**m)
+ except Exception:
+ return None
+
+
+class TestingContext(BaseModel):
+ filename: str
+ code_doc: Document
+ test_doc: Document
+
+ @staticmethod
+ def loads(val: str) -> TestingContext | None:
+ try:
+ m = json.loads(val)
+ return TestingContext(**m)
+ except Exception:
+ return None
+
+
+class RunCodeContext(BaseModel):
+ mode: str = "script"
+ code: Optional[str]
+ code_filename: str = ""
+ test_code: Optional[str]
+ test_filename: str = ""
+ command: List[str] = Field(default_factory=list)
+ working_directory: str = ""
+ additional_python_paths: List[str] = Field(default_factory=list)
+ output_filename: Optional[str]
+ output: Optional[str]
+
+ @staticmethod
+ def loads(val: str) -> RunCodeContext | None:
+ try:
+ m = json.loads(val)
+ return RunCodeContext(**m)
+ except Exception:
+ return None
+
+
+class RunCodeResult(BaseModel):
+ summary: str
+ stdout: str
+ stderr: str
+
+ @staticmethod
+ def loads(val: str) -> RunCodeResult | None:
+ try:
+ m = json.loads(val)
+ return RunCodeResult(**m)
+ except Exception:
+ return None
+
+
+class CodeSummarizeContext(BaseModel):
+ design_filename: str = ""
+ task_filename: str = ""
+ codes_filenames: Set[str] = Field(default_factory=set)
+
+ @staticmethod
+ def loads(filenames: Set) -> CodeSummarizeContext:
+ ctx = CodeSummarizeContext()
+ for filename in filenames:
+ if Path(filename).is_relative_to(SYSTEM_DESIGN_FILE_REPO):
+ ctx.design_filename = str(filename)
+ continue
+ if Path(filename).is_relative_to(TASK_FILE_REPO):
+ ctx.task_filename = str(filename)
+ continue
+ return ctx
diff --git a/metagpt/startup.py b/metagpt/startup.py
new file mode 100644
index 000000000..78f32d556
--- /dev/null
+++ b/metagpt/startup.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import asyncio
+
+import typer
+
+from metagpt.config import CONFIG
+
+app = typer.Typer()
+
+
+@app.command()
+def startup(
+ idea: str = typer.Argument(..., help="Your innovative idea, such as 'Create a 2048 game.'"),
+ investment: float = typer.Option(default=3.0, help="Dollar amount to invest in the AI company."),
+ n_round: int = typer.Option(default=5, help="Number of rounds for the simulation."),
+ code_review: bool = typer.Option(default=True, help="Whether to use code review."),
+ run_tests: bool = typer.Option(default=False, help="Whether to enable QA for adding & running tests."),
+ implement: bool = typer.Option(default=True, help="Enable or disable code implementation."),
+ project_name: str = typer.Option(default="", help="Unique project name, such as 'game_2048'."),
+ inc: bool = typer.Option(default=False, help="Incremental mode. Use it to coop with existing repo."),
+ project_path: str = typer.Option(
+ default="",
+ help="Specify the directory path of the old version project to fulfill the " "incremental requirements.",
+ ),
+ reqa_file: str = typer.Option(default="", help="Specify the source file name for rewriting the quality test code."),
+):
+ """Run a startup. Be a boss."""
+ from metagpt.roles import (
+ Architect,
+ Engineer,
+ ProductManager,
+ ProjectManager,
+ QaEngineer,
+ )
+ from metagpt.team import Team
+
+ # Use in the PrepareDocuments action according to Section 2.2.3.5.1 of RFC 135.
+ CONFIG.project_name = project_name
+ CONFIG.inc = inc
+ CONFIG.project_path = project_path
+ CONFIG.reqa_file = reqa_file
+
+ company = Team()
+ company.hire(
+ [
+ ProductManager(),
+ Architect(),
+ ProjectManager(),
+ ]
+ )
+
+ if implement or code_review:
+ company.hire([Engineer(n_borg=5, use_code_review=code_review)])
+
+ if run_tests:
+ company.hire([QaEngineer()])
+
+ company.invest(investment)
+ company.run_project(idea)
+ asyncio.run(company.run(n_round=n_round))
+
+
+if __name__ == "__main__":
+ app()
diff --git a/metagpt/software_company.py b/metagpt/team.py
similarity index 56%
rename from metagpt/software_company.py
rename to metagpt/team.py
index d3c2c463b..92f379c97 100644
--- a/metagpt/software_company.py
+++ b/metagpt/team.py
@@ -4,11 +4,14 @@
@Time : 2023/5/12 00:30
@Author : alexanderwu
@File : software_company.py
+@Modified By: mashenquan, 2023/11/27. Add an archiving operation after completing the project, as specified in
+ Section 2.2.3.3 of RFC 135.
"""
from pydantic import BaseModel, Field
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.config import CONFIG
+from metagpt.const import MESSAGE_ROUTE_TO_ALL
from metagpt.environment import Environment
from metagpt.logs import logger
from metagpt.roles import Role
@@ -16,13 +19,13 @@ from metagpt.schema import Message
from metagpt.utils.common import NoMoneyException
-class SoftwareCompany(BaseModel):
+class Team(BaseModel):
"""
- Software Company: Possesses a team, SOP (Standard Operating Procedures), and a platform for instant messaging,
- dedicated to writing executable code.
+ Team: Possesses one or more roles (agents), SOP (Standard Operating Procedures), and a platform for instant messaging,
+ dedicated to perform any multi-agent activity, such as collaboratively writing executable code.
"""
- environment: Environment = Field(default_factory=Environment)
+ env: Environment = Field(default_factory=Environment)
investment: float = Field(default=10.0)
idea: str = Field(default="")
@@ -31,7 +34,7 @@ class SoftwareCompany(BaseModel):
def hire(self, roles: list[Role]):
"""Hire roles to cooperate"""
- self.environment.add_roles(roles)
+ self.env.add_roles(roles)
def invest(self, investment: float):
"""Invest company. raise NoMoneyException when exceed max_budget."""
@@ -43,13 +46,17 @@ class SoftwareCompany(BaseModel):
if CONFIG.total_cost > CONFIG.max_budget:
raise NoMoneyException(CONFIG.total_cost, f"Insufficient funds: {CONFIG.max_budget}")
- def start_project(self, idea):
- """Start a project from publishing boss requirement."""
+ def run_project(self, idea, send_to: str = ""):
+ """Start a project from publishing user requirement."""
self.idea = idea
- self.environment.publish_message(Message(role="BOSS", content=idea, cause_by=BossRequirement))
+
+ # Human requirement.
+ self.env.publish_message(
+ Message(role="Human", content=idea, cause_by=UserRequirement, send_to=send_to or MESSAGE_ROUTE_TO_ALL)
+ )
def _save(self):
- logger.info(self.json())
+ logger.info(self.json(ensure_ascii=False))
async def run(self, n_round=3):
"""Run company until target round or no money"""
@@ -58,5 +65,7 @@ class SoftwareCompany(BaseModel):
n_round -= 1
logger.debug(f"{n_round=}")
self._check_balance()
- await self.environment.run()
- return self.environment.history
+ await self.env.run()
+ if CONFIG.git_repo:
+ CONFIG.git_repo.archive()
+ return self.env.history
diff --git a/metagpt/tools/sd_engine.py b/metagpt/tools/sd_engine.py
index a63dbe5ac..c6676a247 100644
--- a/metagpt/tools/sd_engine.py
+++ b/metagpt/tools/sd_engine.py
@@ -13,11 +13,10 @@ from typing import List
from aiohttp import ClientSession
from PIL import Image, PngImagePlugin
-from metagpt.config import Config
-from metagpt.const import WORKSPACE_ROOT
-from metagpt.logs import logger
+from metagpt.config import CONFIG
-config = Config()
+# from metagpt.const import WORKSPACE_ROOT
+from metagpt.logs import logger
payload = {
"prompt": "",
@@ -56,9 +55,8 @@ default_negative_prompt = "(easynegative:0.8),black, dark,Low resolution"
class SDEngine:
def __init__(self):
# Initialize the SDEngine with configuration
- self.config = Config()
- self.sd_url = self.config.get("SD_URL")
- self.sd_t2i_url = f"{self.sd_url}{self.config.get('SD_T2I_API')}"
+ self.sd_url = CONFIG.get("SD_URL")
+ self.sd_t2i_url = f"{self.sd_url}{CONFIG.get('SD_T2I_API')}"
# Define default payload settings for SD API
self.payload = payload
logger.info(self.sd_t2i_url)
@@ -81,7 +79,7 @@ class SDEngine:
return self.payload
def _save(self, imgs, save_name=""):
- save_dir = WORKSPACE_ROOT / "resources" / "SD_Output"
+ save_dir = CONFIG.workspace_path / "resources" / "SD_Output"
if not os.path.exists(save_dir):
os.makedirs(save_dir, exist_ok=True)
batch_decode_base64_to_image(imgs, save_dir, save_name=save_name)
diff --git a/metagpt/utils/common.py b/metagpt/utils/common.py
index 798acf214..f08519f8e 100644
--- a/metagpt/utils/common.py
+++ b/metagpt/utils/common.py
@@ -6,7 +6,11 @@
@File : common.py
@Modified By: mashenquan, 2023-11-1. According to Chapter 2.2.2 of RFC 116:
Add generic class-to-string and object-to-string conversion functionality.
+@Modified By: mashenquan, 2023/11/27. Bug fix: `parse_recipient` failed to parse the recipient in certain GPT-3.5
+ responses.
"""
+from __future__ import annotations
+
import ast
import contextlib
import inspect
@@ -304,7 +308,13 @@ def print_members(module, indent=0):
def parse_recipient(text):
pattern = r"## Send To:\s*([A-Za-z]+)\s*?" # hard code for now
recipient = re.search(pattern, text)
- return recipient.group(1) if recipient else ""
+ if recipient:
+ return recipient.group(1)
+ pattern = r"Send To:\s*([A-Za-z]+)\s*?"
+ recipient = re.search(pattern, text)
+ if recipient:
+ return recipient.group(1)
+ return ""
def get_class_name(cls) -> str:
diff --git a/metagpt/utils/dependency_file.py b/metagpt/utils/dependency_file.py
new file mode 100644
index 000000000..653e07ef9
--- /dev/null
+++ b/metagpt/utils/dependency_file.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/22
+@Author : mashenquan
+@File : dependency_file.py
+@Desc: Implementation of the dependency file described in Section 2.2.3.2 of RFC 135.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Set
+
+import aiofiles
+
+from metagpt.logs import logger
+
+
+class DependencyFile:
+ """A class representing a DependencyFile for managing dependencies.
+
+ :param workdir: The working directory path for the DependencyFile.
+ """
+
+ def __init__(self, workdir: Path | str):
+ """Initialize a DependencyFile instance.
+
+ :param workdir: The working directory path for the DependencyFile.
+ """
+ self._dependencies = {}
+ self._filename = Path(workdir) / ".dependencies.json"
+
+ async def load(self):
+ """Load dependencies from the file asynchronously."""
+ if not self._filename.exists():
+ return
+ try:
+ async with aiofiles.open(str(self._filename), mode="r") as reader:
+ data = await reader.read()
+ self._dependencies = json.loads(data)
+ except Exception as e:
+ logger.error(f"Failed to load {str(self._filename)}, error:{e}")
+
+ async def save(self):
+ """Save dependencies to the file asynchronously."""
+ try:
+ data = json.dumps(self._dependencies)
+ async with aiofiles.open(str(self._filename), mode="w") as writer:
+ await writer.write(data)
+ except Exception as e:
+ logger.error(f"Failed to save {str(self._filename)}, error:{e}")
+
+ async def update(self, filename: Path | str, dependencies: Set[Path | str], persist=True):
+ """Update dependencies for a file asynchronously.
+
+ :param filename: The filename or path.
+ :param dependencies: The set of dependencies.
+ :param persist: Whether to persist the changes immediately.
+ """
+ if persist:
+ await self.load()
+
+ root = self._filename.parent
+ try:
+ key = Path(filename).relative_to(root)
+ except ValueError:
+ key = filename
+
+ if dependencies:
+ relative_paths = []
+ for i in dependencies:
+ try:
+ relative_paths.append(str(Path(i).relative_to(root)))
+ except ValueError:
+ relative_paths.append(str(i))
+ self._dependencies[str(key)] = relative_paths
+ elif str(key) in self._dependencies:
+ del self._dependencies[str(key)]
+
+ if persist:
+ await self.save()
+
+ async def get(self, filename: Path | str, persist=False):
+ """Get dependencies for a file asynchronously.
+
+ :param filename: The filename or path.
+ :param persist: Whether to load dependencies from the file immediately.
+ :return: A set of dependencies.
+ """
+ if persist:
+ await self.load()
+
+ root = self._filename.parent
+ try:
+ key = Path(filename).relative_to(root)
+ except ValueError:
+ key = filename
+ return set(self._dependencies.get(str(key), {}))
+
+ def delete_file(self):
+ """Delete the dependency file."""
+ self._filename.unlink(missing_ok=True)
+
+ @property
+ def exists(self):
+ """Check if the dependency file exists."""
+ return self._filename.exists()
diff --git a/metagpt/utils/file_repository.py b/metagpt/utils/file_repository.py
new file mode 100644
index 000000000..0815bf90a
--- /dev/null
+++ b/metagpt/utils/file_repository.py
@@ -0,0 +1,261 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : git_repository.py
+@Desc: File repository management. RFC 135 2.2.3.2, 2.2.3.4 and 2.2.3.13.
+"""
+from __future__ import annotations
+
+import json
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, List, Set
+
+import aiofiles
+
+from metagpt.config import CONFIG
+from metagpt.logs import logger
+from metagpt.schema import Document
+from metagpt.utils.json_to_markdown import json_to_markdown
+
+
+class FileRepository:
+ """A class representing a FileRepository associated with a Git repository.
+
+ :param git_repo: The associated GitRepository instance.
+ :param relative_path: The relative path within the Git repository.
+
+ Attributes:
+ _relative_path (Path): The relative path within the Git repository.
+ _git_repo (GitRepository): The associated GitRepository instance.
+ """
+
+ def __init__(self, git_repo, relative_path: Path = Path(".")):
+ """Initialize a FileRepository instance.
+
+ :param git_repo: The associated GitRepository instance.
+ :param relative_path: The relative path within the Git repository.
+ """
+ self._relative_path = relative_path
+ self._git_repo = git_repo
+
+ # Initializing
+ self.workdir.mkdir(parents=True, exist_ok=True)
+
+ async def save(self, filename: Path | str, content, dependencies: List[str] = None):
+ """Save content to a file and update its dependencies.
+
+ :param filename: The filename or path within the repository.
+ :param content: The content to be saved.
+ :param dependencies: List of dependency filenames or paths.
+ """
+ pathname = self.workdir / filename
+ pathname.parent.mkdir(parents=True, exist_ok=True)
+ async with aiofiles.open(str(pathname), mode="w") as writer:
+ await writer.write(content)
+ logger.info(f"save to: {str(pathname)}")
+
+ if dependencies is not None:
+ dependency_file = await self._git_repo.get_dependency()
+ await dependency_file.update(pathname, set(dependencies))
+ logger.info(f"update dependency: {str(pathname)}:{dependencies}")
+
+ async def get_dependency(self, filename: Path | str) -> Set[str]:
+ """Get the dependencies of a file.
+
+ :param filename: The filename or path within the repository.
+ :return: Set of dependency filenames or paths.
+ """
+ pathname = self.workdir / filename
+ dependency_file = await self._git_repo.get_dependency()
+ return await dependency_file.get(pathname)
+
+ async def get_changed_dependency(self, filename: Path | str) -> Set[str]:
+ """Get the dependencies of a file that have changed.
+
+ :param filename: The filename or path within the repository.
+ :return: List of changed dependency filenames or paths.
+ """
+ dependencies = await self.get_dependency(filename=filename)
+ changed_files = self.changed_files
+ changed_dependent_files = set()
+ for df in dependencies:
+ if df in changed_files.keys():
+ changed_dependent_files.add(df)
+ return changed_dependent_files
+
+ async def get(self, filename: Path | str) -> Document | None:
+ """Read the content of a file.
+
+ :param filename: The filename or path within the repository.
+ :return: The content of the file.
+ """
+ doc = Document(root_path=str(self.root_path), filename=str(filename))
+ path_name = self.workdir / filename
+ if not path_name.exists():
+ return None
+ try:
+ async with aiofiles.open(str(path_name), mode="r") as reader:
+ doc.content = await reader.read()
+ except FileNotFoundError as e:
+ logger.info(f"open {str(path_name)} failed:{e}")
+ return None
+ except Exception as e:
+ logger.info(f"open {str(path_name)} failed:{e}")
+ return None
+ return doc
+
+ async def get_all(self) -> List[Document]:
+ """Get the content of all files in the repository.
+
+ :return: List of Document instances representing files.
+ """
+ docs = []
+ for root, dirs, files in os.walk(str(self.workdir)):
+ for file in files:
+ file_path = Path(root) / file
+ relative_path = file_path.relative_to(self.workdir)
+ doc = await self.get(relative_path)
+ docs.append(doc)
+ return docs
+
+ @property
+ def workdir(self):
+ """Return the absolute path to the working directory of the FileRepository.
+
+ :return: The absolute path to the working directory.
+ """
+ return self._git_repo.workdir / self._relative_path
+
+ @property
+ def root_path(self):
+ """Return the relative path from git repository root"""
+ return self._relative_path
+
+ @property
+ def changed_files(self) -> Dict[str, str]:
+ """Return a dictionary of changed files and their change types.
+
+ :return: A dictionary where keys are file paths and values are change types.
+ """
+ files = self._git_repo.changed_files
+ relative_files = {}
+ for p, ct in files.items():
+ try:
+ rf = Path(p).relative_to(self._relative_path)
+ except ValueError:
+ continue
+ relative_files[str(rf)] = ct
+ return relative_files
+
+ def get_change_dir_files(self, dir: Path | str) -> List:
+ """Get the files in a directory that have changed.
+
+ :param dir: The directory path within the repository.
+ :return: List of changed filenames or paths within the directory.
+ """
+ changed_files = self.changed_files
+ children = []
+ for f in changed_files:
+ try:
+ Path(f).relative_to(Path(dir))
+ except ValueError:
+ continue
+ children.append(str(f))
+ return children
+
+ @staticmethod
+ def new_filename():
+ """Generate a new filename based on the current timestamp and a UUID suffix.
+
+ :return: A new filename string.
+ """
+ current_time = datetime.now().strftime("%Y%m%d%H%M%S")
+ return current_time
+ # guid_suffix = str(uuid.uuid4())[:8]
+ # return f"{current_time}x{guid_suffix}"
+
+ async def save_doc(self, doc: Document, with_suffix: str = None, dependencies: List[str] = None):
+ """Save a Document instance as a PDF file.
+
+ This method converts the content of the Document instance to Markdown,
+ saves it to a file with an optional specified suffix, and logs the saved file.
+
+ :param doc: The Document instance to be saved.
+ :type doc: Document
+ :param with_suffix: An optional suffix to append to the saved file's name.
+ :type with_suffix: str, optional
+ :param dependencies: A list of dependencies for the saved file.
+ :type dependencies: List[str], optional
+ """
+ m = json.loads(doc.content)
+ filename = Path(doc.filename).with_suffix(with_suffix) if with_suffix is not None else Path(doc.filename)
+ await self.save(filename=str(filename), content=json_to_markdown(m), dependencies=dependencies)
+ logger.info(f"File Saved: {str(filename)}")
+
+ @staticmethod
+ async def get_file(filename: Path | str, relative_path: Path | str = ".") -> Document | None:
+ """Retrieve a specific file from the file repository.
+
+ :param filename: The name or path of the file to retrieve.
+ :type filename: Path or str
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ :return: The document representing the file, or None if not found.
+ :rtype: Document or None
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.get(filename=filename)
+
+ @staticmethod
+ async def get_all_files(relative_path: Path | str = ".") -> List[Document]:
+ """Retrieve all files from the file repository.
+
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ :return: A list of documents representing all files in the repository.
+ :rtype: List[Document]
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.get_all()
+
+ @staticmethod
+ async def save_file(filename: Path | str, content, dependencies: List[str] = None, relative_path: Path | str = "."):
+ """Save a file to the file repository.
+
+ :param filename: The name or path of the file to save.
+ :type filename: Path or str
+ :param content: The content of the file.
+ :param dependencies: A list of dependencies for the file.
+ :type dependencies: List[str], optional
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.save(filename=filename, content=content, dependencies=dependencies)
+
+ @staticmethod
+ async def save_as(
+ doc: Document, with_suffix: str = None, dependencies: List[str] = None, relative_path: Path | str = "."
+ ):
+ """Save a Document instance with optional modifications.
+
+ This static method creates a new FileRepository, saves the Document instance
+ with optional modifications (such as a suffix), and logs the saved file.
+
+ :param doc: The Document instance to be saved.
+ :type doc: Document
+ :param with_suffix: An optional suffix to append to the saved file's name.
+ :type with_suffix: str, optional
+ :param dependencies: A list of dependencies for the saved file.
+ :type dependencies: List[str], optional
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ :return: A boolean indicating whether the save operation was successful.
+ :rtype: bool
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.save_doc(doc=doc, with_suffix=with_suffix, dependencies=dependencies)
diff --git a/metagpt/utils/git_repository.py b/metagpt/utils/git_repository.py
new file mode 100644
index 000000000..7c9ec645f
--- /dev/null
+++ b/metagpt/utils/git_repository.py
@@ -0,0 +1,217 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : git_repository.py
+@Desc: Git repository management. RFC 135 2.2.3.3.
+"""
+from __future__ import annotations
+
+import shutil
+from enum import Enum
+from pathlib import Path
+from typing import Dict
+
+from git.repo import Repo
+from git.repo.fun import is_git_dir
+
+from metagpt.const import DEFAULT_WORKSPACE_ROOT
+from metagpt.logs import logger
+from metagpt.utils.dependency_file import DependencyFile
+from metagpt.utils.file_repository import FileRepository
+
+
+class ChangeType(Enum):
+ ADDED = "A" # File was added
+ COPIED = "C" # File was copied
+ DELETED = "D" # File was deleted
+ RENAMED = "R" # File was renamed
+ MODIFIED = "M" # File was modified
+ TYPE_CHANGED = "T" # Type of the file was changed
+ UNTRACTED = "U" # File is untracked (not added to version control)
+
+
+class GitRepository:
+ """A class representing a Git repository.
+
+ :param local_path: The local path to the Git repository.
+ :param auto_init: If True, automatically initializes a new Git repository if the provided path is not a Git repository.
+
+ Attributes:
+ _repository (Repo): The GitPython `Repo` object representing the Git repository.
+ """
+
+ def __init__(self, local_path=None, auto_init=True):
+ """Initialize a GitRepository instance.
+
+ :param local_path: The local path to the Git repository.
+ :param auto_init: If True, automatically initializes a new Git repository if the provided path is not a Git repository.
+ """
+ self._repository = None
+ self._dependency = None
+ if local_path:
+ self.open(local_path=local_path, auto_init=auto_init)
+
+ def open(self, local_path: Path, auto_init=False):
+ """Open an existing Git repository or initialize a new one if auto_init is True.
+
+ :param local_path: The local path to the Git repository.
+ :param auto_init: If True, automatically initializes a new Git repository if the provided path is not a Git repository.
+ """
+ local_path = Path(local_path)
+ if self.is_git_dir(local_path):
+ self._repository = Repo(local_path)
+ return
+ if not auto_init:
+ return
+ local_path.mkdir(parents=True, exist_ok=True)
+ return self._init(local_path)
+
+ def _init(self, local_path: Path):
+ """Initialize a new Git repository at the specified path.
+
+ :param local_path: The local path where the new Git repository will be initialized.
+ """
+ self._repository = Repo.init(path=Path(local_path))
+
+ gitignore_filename = Path(local_path) / ".gitignore"
+ ignores = ["__pycache__", "*.pyc"]
+ with open(str(gitignore_filename), mode="w") as writer:
+ writer.write("\n".join(ignores))
+ self._repository.index.add([".gitignore"])
+ self._repository.index.commit("Add .gitignore")
+
+ def add_change(self, files: Dict):
+ """Add or remove files from the staging area based on the provided changes.
+
+ :param files: A dictionary where keys are file paths and values are instances of ChangeType.
+ """
+ if not self.is_valid or not files:
+ return
+
+ for k, v in files.items():
+ self._repository.index.remove(k) if v is ChangeType.DELETED else self._repository.index.add([k])
+
+ def commit(self, comments):
+ """Commit the staged changes with the given comments.
+
+ :param comments: Comments for the commit.
+ """
+ if self.is_valid:
+ self._repository.index.commit(comments)
+
+ def delete_repository(self):
+ """Delete the entire repository directory."""
+ if self.is_valid:
+ shutil.rmtree(self._repository.working_dir)
+
+ @property
+ def changed_files(self) -> Dict[str, str]:
+ """Return a dictionary of changed files and their change types.
+
+ :return: A dictionary where keys are file paths and values are change types.
+ """
+ files = {i: ChangeType.UNTRACTED for i in self._repository.untracked_files}
+ changed_files = {f.a_path: ChangeType(f.change_type) for f in self._repository.index.diff(None)}
+ files.update(changed_files)
+ return files
+
+ @staticmethod
+ def is_git_dir(local_path):
+ """Check if the specified directory is a Git repository.
+
+ :param local_path: The local path to check.
+ :return: True if the directory is a Git repository, False otherwise.
+ """
+ git_dir = Path(local_path) / ".git"
+ if git_dir.exists() and is_git_dir(git_dir):
+ return True
+ return False
+
+ @property
+ def is_valid(self):
+ """Check if the Git repository is valid (exists and is initialized).
+
+ :return: True if the repository is valid, False otherwise.
+ """
+ return bool(self._repository)
+
+ @property
+ def status(self) -> str:
+ """Return the Git repository's status as a string."""
+ if not self.is_valid:
+ return ""
+ return self._repository.git.status()
+
+ @property
+ def workdir(self) -> Path | None:
+ """Return the path to the working directory of the Git repository.
+
+ :return: The path to the working directory or None if the repository is not valid.
+ """
+ if not self.is_valid:
+ return None
+ return Path(self._repository.working_dir)
+
+ def archive(self, comments="Archive"):
+ """Archive the current state of the Git repository.
+
+ :param comments: Comments for the archive commit.
+ """
+ logger.info(f"Archive: {list(self.changed_files.keys())}")
+ self.add_change(self.changed_files)
+ self.commit(comments)
+
+ def new_file_repository(self, relative_path: Path | str = ".") -> FileRepository:
+ """Create a new instance of FileRepository associated with this Git repository.
+
+ :param relative_path: The relative path to the file repository within the Git repository.
+ :return: A new instance of FileRepository.
+ """
+ path = Path(relative_path)
+ try:
+ path = path.relative_to(self.workdir)
+ except ValueError:
+ path = relative_path
+ return FileRepository(git_repo=self, relative_path=Path(path))
+
+ async def get_dependency(self) -> DependencyFile:
+ """Get the dependency file associated with the Git repository.
+
+ :return: An instance of DependencyFile.
+ """
+ if not self._dependency:
+ self._dependency = DependencyFile(workdir=self.workdir)
+ return self._dependency
+
+ def rename_root(self, new_dir_name):
+ """Rename the root directory of the Git repository.
+
+ :param new_dir_name: The new name for the root directory.
+ """
+ if self.workdir.name == new_dir_name:
+ return
+ new_path = self.workdir.parent / new_dir_name
+ if new_path.exists():
+ logger.info(f"Delete directory {str(new_path)}")
+ shutil.rmtree(new_path)
+ self.workdir.rename(new_path)
+ logger.info(f"Rename directory {str(self.workdir)} to {str(new_path)}")
+ self._repository = Repo(new_path)
+
+
+if __name__ == "__main__":
+ path = DEFAULT_WORKSPACE_ROOT / "git"
+ path.mkdir(exist_ok=True, parents=True)
+
+ repo = GitRepository()
+ repo.open(path, auto_init=True)
+
+ changes = repo.changed_files
+ print(changes)
+ repo.add_change(changes)
+ print(repo.status)
+ repo.commit("test")
+ print(repo.status)
+ repo.delete_repository()
diff --git a/metagpt/utils/mermaid.py b/metagpt/utils/mermaid.py
index 5e5b275b0..eb85a3f90 100644
--- a/metagpt/utils/mermaid.py
+++ b/metagpt/utils/mermaid.py
@@ -10,7 +10,7 @@ import os
from pathlib import Path
from metagpt.config import CONFIG
-from metagpt.const import PROJECT_ROOT
+from metagpt.const import METAGPT_ROOT
from metagpt.logs import logger
from metagpt.utils.common import check_cmd_exists
@@ -34,7 +34,10 @@ async def mermaid_to_file(mermaid_code, output_file_without_suffix, width=2048,
engine = CONFIG.mermaid_engine.lower()
if engine == "nodejs":
if check_cmd_exists(CONFIG.mmdc) != 0:
- logger.warning("RUN `npm install -g @mermaid-js/mermaid-cli` to install mmdc")
+ logger.warning(
+ "RUN `npm install -g @mermaid-js/mermaid-cli` to install mmdc,"
+ "or consider changing MERMAID_ENGINE to `playwright`, `pyppeteer`, or `ink`."
+ )
return -1
for suffix in ["pdf", "svg", "png"]:
@@ -66,7 +69,7 @@ async def mermaid_to_file(mermaid_code, output_file_without_suffix, width=2048,
if stdout:
logger.info(stdout.decode())
if stderr:
- logger.error(stderr.decode())
+ logger.warning(stderr.decode())
else:
if engine == "playwright":
from metagpt.utils.mmdc_playwright import mermaid_to_file
@@ -138,6 +141,6 @@ MMC2 = """sequenceDiagram
if __name__ == "__main__":
loop = asyncio.new_event_loop()
- result = loop.run_until_complete(mermaid_to_file(MMC1, PROJECT_ROOT / f"{CONFIG.mermaid_engine}/1"))
- result = loop.run_until_complete(mermaid_to_file(MMC2, PROJECT_ROOT / f"{CONFIG.mermaid_engine}/1"))
+ result = loop.run_until_complete(mermaid_to_file(MMC1, METAGPT_ROOT / f"{CONFIG.mermaid_engine}/1"))
+ result = loop.run_until_complete(mermaid_to_file(MMC2, METAGPT_ROOT / f"{CONFIG.mermaid_engine}/1"))
loop.close()
diff --git a/metagpt/utils/token_counter.py b/metagpt/utils/token_counter.py
index a5a65803a..53d9de345 100644
--- a/metagpt/utils/token_counter.py
+++ b/metagpt/utils/token_counter.py
@@ -21,7 +21,9 @@ TOKEN_COSTS = {
"gpt-4-32k": {"prompt": 0.06, "completion": 0.12},
"gpt-4-32k-0314": {"prompt": 0.06, "completion": 0.12},
"gpt-4-0613": {"prompt": 0.06, "completion": 0.12},
+ "gpt-4-1106-preview": {"prompt": 0.01, "completion": 0.03},
"text-embedding-ada-002": {"prompt": 0.0004, "completion": 0.0},
+ "chatglm_turbo": {"prompt": 0.0, "completion": 0.00069}, # 32k version, prompt + completion tokens=0.005￥/k-tokens
}
@@ -36,7 +38,9 @@ TOKEN_MAX = {
"gpt-4-32k": 32768,
"gpt-4-32k-0314": 32768,
"gpt-4-0613": 8192,
+ "gpt-4-1106-preview": 128000,
"text-embedding-ada-002": 8192,
+ "chatglm_turbo": 32768,
}
@@ -54,21 +58,24 @@ def count_message_tokens(messages, model="gpt-3.5-turbo-0613"):
"gpt-4-32k-0314",
"gpt-4-0613",
"gpt-4-32k-0613",
+ "gpt-4-1106-preview",
}:
tokens_per_message = 3
tokens_per_name = 1
elif model == "gpt-3.5-turbo-0301":
tokens_per_message = 4 # every message follows <|start|>{role/name}\n{content}<|end|>\n
tokens_per_name = -1 # if there's a name, the role is omitted
- elif "gpt-3.5-turbo" in model:
+ elif "gpt-3.5-turbo" == model:
print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
return count_message_tokens(messages, model="gpt-3.5-turbo-0613")
- elif "gpt-4" in model:
+ elif "gpt-4" == model:
print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
return count_message_tokens(messages, model="gpt-4-0613")
else:
raise NotImplementedError(
- f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens."""
+ f"num_tokens_from_messages() is not implemented for model {model}. "
+ f"See https://github.com/openai/openai-python/blob/main/chatml.md "
+ f"for information on how messages are converted to tokens."
)
num_tokens = 0
for message in messages:
diff --git a/requirements.txt b/requirements.txt
index c3b909e77..99f738448 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -6,7 +6,8 @@ channels==4.0.0
# docx==0.2.4
#faiss==1.5.3
faiss_cpu==1.7.4
-fire==0.4.0
+# fire==0.4.0
+typer
# godot==0.1.1
# google_api_python_client==2.93.0
lancedb==0.1.16
@@ -14,7 +15,7 @@ langchain==0.0.231
loguru==0.6.0
meilisearch==0.21.0
numpy==1.24.3
-openai==0.28.1
+openai>=0.28.1
openpyxl
beautifulsoup4==4.12.2
pandas==2.0.3
@@ -44,4 +45,7 @@ ta==0.10.2
semantic-kernel==0.3.13.dev0
wrapt==1.15.0
websocket-client==0.58.0
+aiofiles==23.2.1
+gitpython==3.1.40
+zhipuai==1.0.7
diff --git a/setup.py b/setup.py
index f9ae768e6..6d3708c32 100644
--- a/setup.py
+++ b/setup.py
@@ -30,16 +30,16 @@ with open(path.join(here, "requirements.txt"), encoding="utf-8") as f:
setup(
name="metagpt",
- version="0.1",
- description="The Multi-Role Meta Programming Framework",
+ version="0.3.0",
+ description="The Multi-Agent Framework",
long_description=long_description,
long_description_content_type="text/markdown",
- url="https://gitlab.deepwisdomai.com/pub/metagpt",
+ url="https://github.com/geekan/MetaGPT",
author="Alexander Wu",
author_email="alexanderwu@fuzhi.ai",
- license="Apache 2.0",
- keywords="metagpt multi-role multi-agent programming gpt llm",
- packages=find_packages(exclude=["contrib", "docs", "examples"]),
+ license="MIT",
+ keywords="metagpt multi-role multi-agent programming gpt llm metaprogramming",
+ packages=find_packages(exclude=["contrib", "docs", "examples", "tests*"]),
python_requires=">=3.9",
install_requires=requirements,
extras_require={
@@ -52,4 +52,9 @@ setup(
cmdclass={
"install_mermaid": InstallMermaidCLI,
},
+ entry_points={
+ "console_scripts": [
+ "metagpt=metagpt.startup:app",
+ ],
+ },
)
diff --git a/startup.py b/startup.py
deleted file mode 100644
index e2a903c9b..000000000
--- a/startup.py
+++ /dev/null
@@ -1,72 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-import asyncio
-
-import fire
-
-from metagpt.roles import (
- Architect,
- Engineer,
- ProductManager,
- ProjectManager,
- QaEngineer,
-)
-from metagpt.software_company import SoftwareCompany
-
-
-async def startup(
- idea: str,
- investment: float = 3.0,
- n_round: int = 5,
- code_review: bool = False,
- run_tests: bool = False,
- implement: bool = True,
-):
- """Run a startup. Be a boss."""
- company = SoftwareCompany()
- company.hire(
- [
- ProductManager(),
- Architect(),
- ProjectManager(),
- ]
- )
-
- # if implement or code_review
- if implement or code_review:
- # developing features: implement the idea
- company.hire([Engineer(n_borg=5, use_code_review=code_review)])
-
- if run_tests:
- # developing features: run tests on the spot and identify bugs
- # (bug fixing capability comes soon!)
- company.hire([QaEngineer()])
-
- company.invest(investment)
- company.start_project(idea)
- await company.run(n_round=n_round)
-
-
-def main(
- idea: str,
- investment: float = 3.0,
- n_round: int = 5,
- code_review: bool = True,
- run_tests: bool = False,
- implement: bool = True,
-):
- """
- We are a software startup comprised of AI. By investing in us,
- you are empowering a future filled with limitless possibilities.
- :param idea: Your innovative idea, such as "Creating a snake game."
- :param investment: As an investor, you have the opportunity to contribute
- a certain dollar amount to this AI company.
- :param n_round:
- :param code_review: Whether to use code review.
- :return:
- """
- asyncio.run(startup(idea, investment, n_round, code_review, run_tests, implement))
-
-
-if __name__ == "__main__":
- fire.Fire(main)
diff --git a/tests/metagpt/actions/mock.py b/tests/metagpt/actions/mock.py
index 23d10ccc4..c48913755 100644
--- a/tests/metagpt/actions/mock.py
+++ b/tests/metagpt/actions/mock.py
@@ -90,7 +90,7 @@ Python's in-built data structures like lists and dictionaries will be used exten
For testing, we can use the PyTest framework. This is a mature full-featured Python testing tool that helps you write better programs.
-## Python package name:
+## project_name:
```python
"adventure_game"
```
@@ -100,7 +100,7 @@ For testing, we can use the PyTest framework. This is a mature full-featured Pyt
file_list = ["main.py", "room.py", "player.py", "game.py", "object.py", "puzzle.py", "test_game.py"]
```
-## Data structures and interface definitions:
+## Data structures and interfaces:
```mermaid
classDiagram
class Room{
@@ -209,7 +209,7 @@ Shared knowledge for this project includes understanding the basic principles of
"""
```
-## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
+## Anything UNCLEAR: Provide as Plain text. Try to clarify it. For example, don't forget a main entry. don't forget to init 3rd party libs.
```python
"""
The original requirements did not specify whether the game should have a save/load feature, multiplayer support, or any specific graphical user interface. More information on these aspects could help in further refining the product design and requirements.
diff --git a/tests/metagpt/actions/test_write_prd.py b/tests/metagpt/actions/test_write_prd.py
index 07d701cb9..8f8ef84f5 100644
--- a/tests/metagpt/actions/test_write_prd.py
+++ b/tests/metagpt/actions/test_write_prd.py
@@ -8,7 +8,7 @@
"""
import pytest
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.logs import logger
from metagpt.roles.product_manager import ProductManager
from metagpt.schema import Message
@@ -18,7 +18,7 @@ from metagpt.schema import Message
async def test_write_prd():
product_manager = ProductManager()
requirements = "开发一个基于大语言模型与私有知识库的搜索引擎，希望可以基于大语言模型进行搜索总结"
- prd = await product_manager.run(Message(content=requirements, cause_by=BossRequirement))
+ prd = await product_manager.run(Message(content=requirements, cause_by=UserRequirement))
logger.info(requirements)
logger.info(prd)
diff --git a/tests/metagpt/document_store/test_document.py b/tests/metagpt/document_store/test_document.py
index 5ae357fb1..13c0921a3 100644
--- a/tests/metagpt/document_store/test_document.py
+++ b/tests/metagpt/document_store/test_document.py
@@ -7,22 +7,22 @@
"""
import pytest
-from metagpt.const import DATA_PATH
-from metagpt.document_store.document import Document
+from metagpt.const import METAGPT_ROOT
+from metagpt.document import IndexableDocument
CASES = [
- ("st/faq.xlsx", "Question", "Answer", 1),
- ("cases/faq.csv", "Question", "Answer", 1),
+ ("requirements.txt", None, None, 0),
+ # ("cases/faq.csv", "Question", "Answer", 1),
# ("cases/faq.json", "Question", "Answer", 1),
- ("docx/faq.docx", None, None, 1),
- ("cases/faq.pdf", None, None, 0), # 这是因为pdf默认没有分割段落
- ("cases/faq.txt", None, None, 0), # 这是因为txt按照256分割段落
+ # ("docx/faq.docx", None, None, 1),
+ # ("cases/faq.pdf", None, None, 0), # 这是因为pdf默认没有分割段落
+ # ("cases/faq.txt", None, None, 0), # 这是因为txt按照256分割段落
]
@pytest.mark.parametrize("relative_path, content_col, meta_col, threshold", CASES)
def test_document(relative_path, content_col, meta_col, threshold):
- doc = Document(DATA_PATH / relative_path, content_col, meta_col)
+ doc = IndexableDocument.from_path(METAGPT_ROOT / relative_path, content_col, meta_col)
rsp = doc.get_docs_and_metadatas()
assert len(rsp[0]) > threshold
assert len(rsp[1]) > threshold
diff --git a/tests/metagpt/memory/test_longterm_memory.py b/tests/metagpt/memory/test_longterm_memory.py
index c5b5c6eb1..b6ae0ac79 100644
--- a/tests/metagpt/memory/test_longterm_memory.py
+++ b/tests/metagpt/memory/test_longterm_memory.py
@@ -4,7 +4,7 @@
@Desc : unittest of `metagpt/memory/longterm_memory.py`
"""
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.config import CONFIG
from metagpt.memory import LongTermMemory
from metagpt.roles.role import RoleContext
@@ -17,24 +17,24 @@ def test_ltm_search():
assert len(openai_api_key) > 20
role_id = "UTUserLtm(Product Manager)"
- rc = RoleContext(watch=[BossRequirement])
+ rc = RoleContext(watch=[UserRequirement])
ltm = LongTermMemory()
ltm.recover_memory(role_id, rc)
idea = "Write a cli snake game"
- message = Message(role="BOSS", content=idea, cause_by=BossRequirement)
+ message = Message(role="User", content=idea, cause_by=UserRequirement)
news = ltm.find_news([message])
assert len(news) == 1
ltm.add(message)
sim_idea = "Write a game of cli snake"
- sim_message = Message(role="BOSS", content=sim_idea, cause_by=BossRequirement)
+ sim_message = Message(role="User", content=sim_idea, cause_by=UserRequirement)
news = ltm.find_news([sim_message])
assert len(news) == 0
ltm.add(sim_message)
new_idea = "Write a 2048 web game"
- new_message = Message(role="BOSS", content=new_idea, cause_by=BossRequirement)
+ new_message = Message(role="User", content=new_idea, cause_by=UserRequirement)
news = ltm.find_news([new_message])
assert len(news) == 1
ltm.add(new_message)
@@ -50,7 +50,7 @@ def test_ltm_search():
assert len(news) == 0
new_idea = "Write a Battle City"
- new_message = Message(role="BOSS", content=new_idea, cause_by=BossRequirement)
+ new_message = Message(role="User", content=new_idea, cause_by=UserRequirement)
news = ltm_new.find_news([new_message])
assert len(news) == 1
diff --git a/tests/metagpt/memory/test_memory_storage.py b/tests/metagpt/memory/test_memory_storage.py
index 251c70b02..c67ca689f 100644
--- a/tests/metagpt/memory/test_memory_storage.py
+++ b/tests/metagpt/memory/test_memory_storage.py
@@ -7,7 +7,7 @@
from typing import List
-from metagpt.actions import BossRequirement, WritePRD
+from metagpt.actions import UserRequirement, WritePRD
from metagpt.actions.action_output import ActionOutput
from metagpt.memory.memory_storage import MemoryStorage
from metagpt.schema import Message
@@ -16,7 +16,7 @@ from metagpt.schema import Message
def test_idea_message():
idea = "Write a cli snake game"
role_id = "UTUser1(Product Manager)"
- message = Message(role="BOSS", content=idea, cause_by=BossRequirement)
+ message = Message(role="User", content=idea, cause_by=UserRequirement)
memory_storage: MemoryStorage = MemoryStorage()
messages = memory_storage.recover_memory(role_id)
@@ -26,12 +26,12 @@ def test_idea_message():
assert memory_storage.is_initialized is True
sim_idea = "Write a game of cli snake"
- sim_message = Message(role="BOSS", content=sim_idea, cause_by=BossRequirement)
+ sim_message = Message(role="User", content=sim_idea, cause_by=UserRequirement)
new_messages = memory_storage.search(sim_message)
assert len(new_messages) == 0 # similar, return []
new_idea = "Write a 2048 web game"
- new_message = Message(role="BOSS", content=new_idea, cause_by=BossRequirement)
+ new_message = Message(role="User", content=new_idea, cause_by=UserRequirement)
new_messages = memory_storage.search(new_message)
assert new_messages[0].content == message.content
@@ -45,7 +45,7 @@ def test_actionout_message():
ic_obj = ActionOutput.create_model_class("prd", out_mapping)
role_id = "UTUser2(Architect)"
- content = "The boss has requested the creation of a command-line interface (CLI) snake game"
+ content = "The user has requested the creation of a command-line interface (CLI) snake game"
message = Message(
content=content, instruct_content=ic_obj(**out_data), role="user", cause_by=WritePRD
) # WritePRD as test action
diff --git a/tests/metagpt/planner/test_action_planner.py b/tests/metagpt/planner/test_action_planner.py
index b8d4c1ad9..1bc451db8 100644
--- a/tests/metagpt/planner/test_action_planner.py
+++ b/tests/metagpt/planner/test_action_planner.py
@@ -11,7 +11,7 @@ import pytest
from semantic_kernel.core_skills import FileIOSkill, MathSkill, TextSkill, TimeSkill
from semantic_kernel.planning.action_planner.action_planner import ActionPlanner
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.roles.sk_agent import SkAgent
from metagpt.schema import Message
@@ -25,7 +25,8 @@ async def test_action_planner():
role.import_skill(TimeSkill(), "time")
role.import_skill(TextSkill(), "text")
task = "What is the sum of 110 and 990?"
- role.put_message(Message(content=task, cause_by=BossRequirement))
+
+ role.put_message(Message(content=task, cause_by=UserRequirement))
await role._observe()
await role._think() # it will choose mathskill.Add
assert "1100" == (await role._act()).content
diff --git a/tests/metagpt/planner/test_basic_planner.py b/tests/metagpt/planner/test_basic_planner.py
index 24250a0b0..f406143ee 100644
--- a/tests/metagpt/planner/test_basic_planner.py
+++ b/tests/metagpt/planner/test_basic_planner.py
@@ -10,7 +10,7 @@
import pytest
from semantic_kernel.core_skills import TextSkill
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.const import SKILL_DIRECTORY
from metagpt.roles.sk_agent import SkAgent
from metagpt.schema import Message
@@ -28,7 +28,7 @@ async def test_basic_planner():
role.import_semantic_skill_from_directory(SKILL_DIRECTORY, "WriterSkill")
role.import_skill(TextSkill(), "TextSkill")
# using BasicPlanner
- role.put_message(Message(content=task, cause_by=BossRequirement))
+ role.put_message(Message(content=task, cause_by=UserRequirement))
await role._observe()
await role._think()
# assuming sk_agent will think he needs WriterSkill.Brainstorm and WriterSkill.Translate
diff --git a/tests/metagpt/provider/test_openai.py b/tests/metagpt/provider/test_openai.py
new file mode 100644
index 000000000..2b0af37b5
--- /dev/null
+++ b/tests/metagpt/provider/test_openai.py
@@ -0,0 +1,80 @@
+import pytest
+
+from metagpt.provider.openai_api import OpenAIGPTAPI
+from metagpt.schema import UserMessage
+
+
+@pytest.mark.asyncio
+async def test_aask_code():
+ llm = OpenAIGPTAPI()
+ msg = [{"role": "user", "content": "Write a python hello world code."}]
+ rsp = await llm.aask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+@pytest.mark.asyncio
+async def test_aask_code_str():
+ llm = OpenAIGPTAPI()
+ msg = "Write a python hello world code."
+ rsp = await llm.aask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+@pytest.mark.asyncio
+async def test_aask_code_Message():
+ llm = OpenAIGPTAPI()
+ msg = UserMessage("Write a python hello world code.")
+ rsp = await llm.aask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+def test_ask_code():
+ llm = OpenAIGPTAPI()
+ msg = [{"role": "user", "content": "Write a python hello world code."}]
+ rsp = llm.ask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+def test_ask_code_str():
+ llm = OpenAIGPTAPI()
+ msg = "Write a python hello world code."
+ rsp = llm.ask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+def test_ask_code_Message():
+ llm = OpenAIGPTAPI()
+ msg = UserMessage("Write a python hello world code.")
+ rsp = llm.ask_code(msg) # -> {'language': 'python', 'code': "print('Hello, World!')"}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+def test_ask_code_list_Message():
+ llm = OpenAIGPTAPI()
+ msg = [UserMessage("a=[1,2,5,10,-10]"), UserMessage("写出求a中最大值的代码python")]
+ rsp = llm.ask_code(msg) # -> {'language': 'python', 'code': 'max_value = max(a)\nmax_value'}
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
+
+
+def test_ask_code_list_str():
+ llm = OpenAIGPTAPI()
+ msg = ["a=[1,2,5,10,-10]", "写出求a中最大值的代码python"]
+ rsp = llm.ask_code(msg) # -> {'language': 'python', 'code': 'max_value = max(a)\nmax_value'}
+ print(rsp)
+ assert "language" in rsp
+ assert "code" in rsp
+ assert len(rsp["code"]) > 0
diff --git a/tests/metagpt/provider/test_zhipuai_api.py b/tests/metagpt/provider/test_zhipuai_api.py
new file mode 100644
index 000000000..4684e8887
--- /dev/null
+++ b/tests/metagpt/provider/test_zhipuai_api.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : the unittest of ZhiPuAIGPTAPI
+
+import pytest
+
+from metagpt.provider.zhipuai_api import ZhiPuAIGPTAPI
+
+default_resp = {"code": 200, "data": {"choices": [{"role": "assistant", "content": "I'm chatglm-turbo"}]}}
+
+messages = [{"role": "user", "content": "who are you"}]
+
+
+def mock_llm_ask(self, messages: list[dict]) -> dict:
+ return default_resp
+
+
+def test_zhipuai_completion(mocker):
+ mocker.patch("metagpt.provider.zhipuai_api.ZhiPuAIGPTAPI.completion", mock_llm_ask)
+
+ resp = ZhiPuAIGPTAPI().completion(messages)
+ assert resp["code"] == 200
+ assert "chatglm-turbo" in resp["data"]["choices"][0]["content"]
+
+
+async def mock_llm_aask(self, messgaes: list[dict], stream: bool = False) -> dict:
+ return default_resp
+
+
+@pytest.mark.asyncio
+async def test_zhipuai_acompletion(mocker):
+ mocker.patch("metagpt.provider.zhipuai_api.ZhiPuAIGPTAPI.acompletion_text", mock_llm_aask)
+
+ resp = await ZhiPuAIGPTAPI().acompletion_text(messages, stream=False)
+
+ assert resp["code"] == 200
+ assert "chatglm-turbo" in resp["data"]["choices"][0]["content"]
diff --git a/tests/metagpt/roles/mock.py b/tests/metagpt/roles/mock.py
index 1b02fbaa5..5500b69f7 100644
--- a/tests/metagpt/roles/mock.py
+++ b/tests/metagpt/roles/mock.py
@@ -5,10 +5,10 @@
@Author : alexanderwu
@File : mock.py
"""
-from metagpt.actions import BossRequirement, WriteDesign, WritePRD, WriteTasks
+from metagpt.actions import UserRequirement, WriteDesign, WritePRD, WriteTasks
from metagpt.schema import Message
-BOSS_REQUIREMENT = """开发一个基于大语言模型与私有知识库的搜索引擎，希望可以基于大语言模型进行搜索总结"""
+USER_REQUIREMENT = """开发一个基于大语言模型与私有知识库的搜索引擎，希望可以基于大语言模型进行搜索总结"""
DETAIL_REQUIREMENT = """需求：开发一个基于LLM（大语言模型）与私有知识库的搜索引擎，希望有几点能力
1. 用户可以在私有知识库进行搜索，再根据大语言模型进行总结，输出的结果包括了总结
@@ -71,7 +71,7 @@ PRD = '''## 原始需求
```
'''
-SYSTEM_DESIGN = """## Python package name
+SYSTEM_DESIGN = """## project_name
```python
"smart_search_engine"
```
@@ -94,7 +94,7 @@ SYSTEM_DESIGN = """## Python package name
]
```
-## Data structures and interface definitions
+## Data structures and interfaces
```mermaid
classDiagram
class Main {
@@ -252,7 +252,7 @@ a = 'a'
class MockMessages:
- req = Message(role="Boss", content=BOSS_REQUIREMENT, cause_by=BossRequirement)
+ req = Message(role="User", content=USER_REQUIREMENT, cause_by=UserRequirement)
prd = Message(role="Product Manager", content=PRD, cause_by=WritePRD)
system_design = Message(role="Architect", content=SYSTEM_DESIGN, cause_by=WriteDesign)
tasks = Message(role="Project Manager", content=TASKS, cause_by=WriteTasks)
diff --git a/tests/metagpt/roles/test_ui.py b/tests/metagpt/roles/test_ui.py
index 2d9cb85c9..5904bee8f 100644
--- a/tests/metagpt/roles/test_ui.py
+++ b/tests/metagpt/roles/test_ui.py
@@ -3,7 +3,7 @@
# @Author : stellahong (stellahong@fuzhi.ai)
#
from metagpt.roles import ProductManager
-from metagpt.software_company import SoftwareCompany
+from metagpt.team import Team
from tests.metagpt.roles.ui_role import UI
@@ -14,8 +14,8 @@ def test_add_ui():
async def test_ui_role(idea: str, investment: float = 3.0, n_round: int = 5):
"""Run a startup. Be a boss."""
- company = SoftwareCompany()
+ company = Team()
company.hire([ProductManager(), UI()])
company.invest(investment)
- company.start_project(idea)
+ company.run_project(idea)
await company.run(n_round=n_round)
diff --git a/tests/metagpt/roles/ui_role.py b/tests/metagpt/roles/ui_role.py
index a45a89cde..ee36befbd 100644
--- a/tests/metagpt/roles/ui_role.py
+++ b/tests/metagpt/roles/ui_role.py
@@ -8,7 +8,9 @@ from functools import wraps
from importlib import import_module
from metagpt.actions import Action, ActionOutput, WritePRD
-from metagpt.const import WORKSPACE_ROOT
+
+# from metagpt.const import WORKSPACE_ROOT
+from metagpt.config import CONFIG
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
@@ -29,7 +31,7 @@ Attention: Use '##' to split sections, not '#', and '## ' SHOULD W
## Selected Elements:Provide as Plain text, up to 5 specified elements, clear and simple
## HTML Layout:Provide as Plain text, use standard HTML code
## CSS Styles (styles.css):Provide as Plain text,use standard css code
-## Anything UNCLEAR:Provide as Plain text. Make clear here.
+## Anything UNCLEAR:Provide as Plain text. Try to clarify it.
"""
@@ -214,7 +216,7 @@ class UIDesign(Action):
logger.info("Finish icon design using StableDiffusion API")
async def _save(self, css_content, html_content):
- save_dir = WORKSPACE_ROOT / "resources" / "codes"
+ save_dir = CONFIG.workspace_path / "resources" / "codes"
if not os.path.exists(save_dir):
os.makedirs(save_dir, exist_ok=True)
# Save CSS and HTML content to files
diff --git a/tests/metagpt/test_environment.py b/tests/metagpt/test_environment.py
index a0f1f6257..b27bc3da7 100644
--- a/tests/metagpt/test_environment.py
+++ b/tests/metagpt/test_environment.py
@@ -8,7 +8,7 @@
import pytest
-from metagpt.actions import BossRequirement
+from metagpt.actions import UserRequirement
from metagpt.environment import Environment
from metagpt.logs import logger
from metagpt.manager import Manager
@@ -49,7 +49,7 @@ async def test_publish_and_process_message(env: Environment):
env.add_roles([product_manager, architect])
env.set_manager(Manager())
- env.publish_message(Message(role="BOSS", content="需要一个基于LLM做总结的搜索引擎", cause_by=BossRequirement))
+ env.publish_message(Message(role="User", content="需要一个基于LLM做总结的搜索引擎", cause_by=UserRequirement))
await env.run(k=2)
logger.info(f"{env.history=}")
diff --git a/tests/metagpt/test_gpt.py b/tests/metagpt/test_gpt.py
index 285e8134c..431858d4c 100644
--- a/tests/metagpt/test_gpt.py
+++ b/tests/metagpt/test_gpt.py
@@ -15,6 +15,7 @@ from metagpt.logs import logger
class TestGPT:
def test_llm_api_ask(self, llm_api):
answer = llm_api.ask("hello chatgpt")
+ logger.info(answer)
assert len(answer) > 0
# def test_gptapi_ask_batch(self, llm_api):
@@ -23,16 +24,19 @@ class TestGPT:
def test_llm_api_ask_code(self, llm_api):
answer = llm_api.ask_code(["请扮演一个Google Python专家工程师，如果理解，回复明白", "写一个hello world"])
+ logger.info(answer)
assert len(answer) > 0
@pytest.mark.asyncio
async def test_llm_api_aask(self, llm_api):
answer = await llm_api.aask("hello chatgpt")
+ logger.info(answer)
assert len(answer) > 0
@pytest.mark.asyncio
async def test_llm_api_aask_code(self, llm_api):
answer = await llm_api.aask_code(["请扮演一个Google Python专家工程师，如果理解，回复明白", "写一个hello world"])
+ logger.info(answer)
assert len(answer) > 0
@pytest.mark.asyncio
@@ -41,3 +45,7 @@ class TestGPT:
costs = llm_api.get_costs()
logger.info(costs)
assert costs.total_cost > 0
+
+
+# if __name__ == "__main__":
+# pytest.main([__file__, "-s"])
diff --git a/tests/metagpt/test_llm.py b/tests/metagpt/test_llm.py
index 03341212b..49969a2af 100644
--- a/tests/metagpt/test_llm.py
+++ b/tests/metagpt/test_llm.py
@@ -32,3 +32,6 @@ async def test_llm_acompletion(llm):
assert len(await llm.acompletion(hello_msg)) > 0
assert len(await llm.acompletion_batch([hello_msg])) > 0
assert len(await llm.acompletion_batch_text([hello_msg])) > 0
+
+# if __name__ == "__main__":
+# pytest.main([__file__, "-s"])
diff --git a/tests/metagpt/test_message.py b/tests/metagpt/test_message.py
index f52dfb64c..04d85d9e4 100644
--- a/tests/metagpt/test_message.py
+++ b/tests/metagpt/test_message.py
@@ -39,4 +39,3 @@ def test_raw_message():
if __name__ == "__main__":
pytest.main([__file__, "-s"])
-
diff --git a/tests/metagpt/test_role.py b/tests/metagpt/test_role.py
index f93651303..8fac2503c 100644
--- a/tests/metagpt/test_role.py
+++ b/tests/metagpt/test_role.py
@@ -97,7 +97,5 @@ async def test_msg_to():
assert m.send_to == set({"a", get_class_name(MockRole), get_class_name(Message)})
-
if __name__ == "__main__":
pytest.main([__file__, "-s"])
-
diff --git a/tests/metagpt/test_software_company.py b/tests/metagpt/test_software_company.py
deleted file mode 100644
index 00538442c..000000000
--- a/tests/metagpt/test_software_company.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-"""
-@Time : 2023/5/15 11:40
-@Author : alexanderwu
-@File : test_software_company.py
-"""
-import pytest
-
-from metagpt.logs import logger
-from metagpt.software_company import SoftwareCompany
-
-
-@pytest.mark.asyncio
-async def test_software_company():
- company = SoftwareCompany()
- company.start_project("做一个基础搜索引擎，可以支持知识库")
- history = await company.run(n_round=5)
- logger.info(history)
diff --git a/tests/metagpt/test_startup.py b/tests/metagpt/test_startup.py
new file mode 100644
index 000000000..53d3509ed
--- /dev/null
+++ b/tests/metagpt/test_startup.py
@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/5/15 11:40
+@Author : alexanderwu
+@File : test_startup.py
+"""
+import pytest
+from typer.testing import CliRunner
+
+from metagpt.logs import logger
+from metagpt.team import Team
+
+runner = CliRunner()
+
+
+@pytest.mark.asyncio
+async def test_team():
+ company = Team()
+ company.run_project("做一个基础搜索引擎，可以支持知识库")
+ history = await company.run(n_round=5)
+ logger.info(history)
+
+
+# def test_startup():
+# args = ["Make a 2048 game"]
+# result = runner.invoke(app, args)
diff --git a/tests/metagpt/tools/test_sd_tool.py b/tests/metagpt/tools/test_sd_tool.py
index 4edd8fb3b..edb23df42 100644
--- a/tests/metagpt/tools/test_sd_tool.py
+++ b/tests/metagpt/tools/test_sd_tool.py
@@ -4,7 +4,8 @@
#
import os
-from metagpt.tools.sd_engine import WORKSPACE_ROOT, SDEngine
+from metagpt.config import CONFIG
+from metagpt.tools.sd_engine import SDEngine
def test_sd_engine_init():
@@ -21,5 +22,5 @@ def test_sd_engine_generate_prompt():
async def test_sd_engine_run_t2i():
sd_engine = SDEngine()
await sd_engine.run_t2i(prompts=["test"])
- img_path = WORKSPACE_ROOT / "resources" / "SD_Output" / "output_0.png"
- assert os.path.exists(img_path) == True
+ img_path = CONFIG.workspace_path / "resources" / "SD_Output" / "output_0.png"
+ assert os.path.exists(img_path)
diff --git a/tests/metagpt/utils/test_common.py b/tests/metagpt/utils/test_common.py
index 6474b1233..4bd38db63 100644
--- a/tests/metagpt/utils/test_common.py
+++ b/tests/metagpt/utils/test_common.py
@@ -14,7 +14,7 @@ import pytest
from pydantic import BaseModel
from metagpt.actions import RunCode
-from metagpt.const import get_project_root
+from metagpt.const import get_metagpt_root
from metagpt.roles.tutorial_assistant import TutorialAssistant
from metagpt.schema import Message
from metagpt.utils.common import any_to_str, any_to_str_set
@@ -27,13 +27,13 @@ class TestGetProjectRoot:
os.chdir(abs_root)
def test_get_project_root(self):
- project_root = get_project_root()
- assert project_root.name == "MetaGPT"
+ project_root = get_metagpt_root()
+ assert project_root.name == "metagpt"
def test_get_root_exception(self):
with pytest.raises(Exception) as exc_info:
self.change_etc_dir()
- get_project_root()
+ get_metagpt_root()
assert str(exc_info.value) == "Project root not found."
def test_any_to_str(self):
diff --git a/tests/metagpt/utils/test_dependency_file.py b/tests/metagpt/utils/test_dependency_file.py
new file mode 100644
index 000000000..ae4d40ea5
--- /dev/null
+++ b/tests/metagpt/utils/test_dependency_file.py
@@ -0,0 +1,64 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/22
+@Author : mashenquan
+@File : test_dependency_file.py
+@Desc: Unit tests for dependency_file.py
+"""
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional, Set, Union
+
+import pytest
+from pydantic import BaseModel
+
+from metagpt.utils.dependency_file import DependencyFile
+
+
+@pytest.mark.asyncio
+async def test_dependency_file():
+ class Input(BaseModel):
+ x: Union[Path, str]
+ deps: Optional[Set[Union[Path, str]]]
+ key: Optional[Union[Path, str]]
+ want: Set[str]
+
+ inputs = [
+ Input(x="a/b.txt", deps={"c/e.txt", Path(__file__).parent / "d.txt"}, want={"c/e.txt", "d.txt"}),
+ Input(
+ x=Path(__file__).parent / "x/b.txt",
+ deps={"s/e.txt", Path(__file__).parent / "d.txt"},
+ key="x/b.txt",
+ want={"s/e.txt", "d.txt"},
+ ),
+ Input(x="f.txt", deps=None, want=set()),
+ Input(x="a/b.txt", deps=None, want=set()),
+ ]
+
+ file = DependencyFile(workdir=Path(__file__).parent)
+
+ for i in inputs:
+ await file.update(filename=i.x, dependencies=i.deps)
+ assert await file.get(filename=i.key or i.x) == i.want
+
+ file2 = DependencyFile(workdir=Path(__file__).parent)
+ file2.delete_file()
+ assert not file.exists
+ await file2.update(filename="a/b.txt", dependencies={"c/e.txt", Path(__file__).parent / "d.txt"}, persist=False)
+ assert not file.exists
+ await file2.save()
+ assert file2.exists
+
+ file1 = DependencyFile(workdir=Path(__file__).parent)
+ assert file1.exists
+ assert await file1.get("a/b.txt") == set()
+ await file1.load()
+ assert await file1.get("a/b.txt") == {"c/e.txt", "d.txt"}
+ file1.delete_file()
+ assert not file.exists
+
+
+if __name__ == "__main__":
+ pytest.main([__file__, "-s"])
diff --git a/tests/metagpt/utils/test_file_repository.py b/tests/metagpt/utils/test_file_repository.py
new file mode 100644
index 000000000..a830b58aa
--- /dev/null
+++ b/tests/metagpt/utils/test_file_repository.py
@@ -0,0 +1,51 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : test_file_repository.py
+@Desc: Unit tests for file_repository.py
+"""
+import shutil
+from pathlib import Path
+
+import pytest
+
+from metagpt.utils.git_repository import ChangeType, GitRepository
+from tests.metagpt.utils.test_git_repository import mock_file
+
+
+@pytest.mark.asyncio
+async def test_file_repo():
+ local_path = Path(__file__).parent / "file_repo_git"
+ if local_path.exists():
+ shutil.rmtree(local_path)
+
+ git_repo = GitRepository(local_path=local_path, auto_init=True)
+ assert not git_repo.changed_files
+
+ await mock_file(local_path / "g.txt", "")
+
+ file_repo_path = "file_repo1"
+ full_path = local_path / file_repo_path
+ assert not full_path.exists()
+ file_repo = git_repo.new_file_repository(file_repo_path)
+ assert file_repo.workdir == full_path
+ assert file_repo.workdir.exists()
+ await file_repo.save("a.txt", "AAA")
+ await file_repo.save("b.txt", "BBB", ["a.txt"])
+ doc = await file_repo.get("a.txt")
+ assert "AAA" == doc.content
+ doc = await file_repo.get("b.txt")
+ assert "BBB" == doc.content
+ assert {"a.txt"} == await file_repo.get_dependency("b.txt")
+ assert {"a.txt": ChangeType.UNTRACTED, "b.txt": ChangeType.UNTRACTED} == file_repo.changed_files
+ assert {"a.txt"} == await file_repo.get_changed_dependency("b.txt")
+ await file_repo.save("d/e.txt", "EEE")
+ assert ["d/e.txt"] == file_repo.get_change_dir_files("d")
+
+ git_repo.delete_repository()
+
+
+if __name__ == "__main__":
+ pytest.main([__file__, "-s"])
diff --git a/tests/metagpt/utils/test_git_repository.py b/tests/metagpt/utils/test_git_repository.py
new file mode 100644
index 000000000..23bebba7f
--- /dev/null
+++ b/tests/metagpt/utils/test_git_repository.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : test_git_repository.py
+@Desc: Unit tests for git_repository.py
+"""
+
+import shutil
+from pathlib import Path
+
+import aiofiles
+import pytest
+
+from metagpt.utils.git_repository import GitRepository
+
+
+async def mock_file(filename, content=""):
+ async with aiofiles.open(str(filename), mode="w") as file:
+ await file.write(content)
+
+
+async def mock_repo(local_path) -> (GitRepository, Path):
+ if local_path.exists():
+ shutil.rmtree(local_path)
+ assert not local_path.exists()
+ repo = GitRepository(local_path=local_path, auto_init=True)
+ assert local_path.exists()
+ assert local_path == repo.workdir
+ assert not repo.changed_files
+
+ await mock_file(local_path / "a.txt")
+ await mock_file(local_path / "b.txt")
+ subdir = local_path / "subdir"
+ subdir.mkdir(parents=True, exist_ok=True)
+ await mock_file(subdir / "c.txt")
+ return repo, subdir
+
+
+@pytest.mark.asyncio
+async def test_git():
+ local_path = Path(__file__).parent / "git"
+ repo, subdir = await mock_repo(local_path)
+
+ assert len(repo.changed_files) == 3
+ repo.add_change(repo.changed_files)
+ repo.commit("commit1")
+ assert not repo.changed_files
+
+ await mock_file(local_path / "a.txt", "tests")
+ await mock_file(subdir / "d.txt")
+ rmfile = local_path / "b.txt"
+ rmfile.unlink()
+ assert repo.status
+
+ assert len(repo.changed_files) == 3
+ repo.add_change(repo.changed_files)
+ repo.commit("commit2")
+ assert not repo.changed_files
+
+ assert repo.status
+
+ repo.delete_repository()
+ assert not local_path.exists()
+
+
+@pytest.mark.asyncio
+async def test_git1():
+ local_path = Path(__file__).parent / "git1"
+ await mock_repo(local_path)
+
+ repo1 = GitRepository(local_path=local_path, auto_init=False)
+ assert repo1.changed_files
+
+ repo1.delete_repository()
+ assert not local_path.exists()
+
+
+@pytest.mark.asyncio
+async def test_dependency_file():
+ local_path = Path(__file__).parent / "git2"
+ repo, subdir = await mock_repo(local_path)
+
+ dependancy_file = await repo.get_dependency()
+ assert not dependancy_file.exists
+
+ await dependancy_file.update(filename="a/b.txt", dependencies={"c/d.txt", "e/f.txt"})
+ assert dependancy_file.exists
+
+ repo.delete_repository()
+ assert not dependancy_file.exists
+
+
+if __name__ == "__main__":
+ pytest.main([__file__, "-s"])
diff --git a/tests/metagpt/utils/test_output_parser.py b/tests/metagpt/utils/test_output_parser.py
index 7a3aedbe8..c9f5813d9 100644
--- a/tests/metagpt/utils/test_output_parser.py
+++ b/tests/metagpt/utils/test_output_parser.py
@@ -220,7 +220,7 @@ We need clarification on how the high score should be stored. Should it persist
}
t_text1 = """## Original Requirements:
-The boss wants to create a web-based version of the game "Fly Bird".
+The user wants to create a web-based version of the game "Fly Bird".
## Product Goals:
diff --git a/tests/metagpt/utils/test_read_docx.py b/tests/metagpt/utils/test_read_docx.py
index a7d0774a8..adf473ae7 100644
--- a/tests/metagpt/utils/test_read_docx.py
+++ b/tests/metagpt/utils/test_read_docx.py
@@ -6,12 +6,12 @@
@File : test_read_docx.py
"""
-from metagpt.const import PROJECT_ROOT
+from metagpt.const import METAGPT_ROOT
from metagpt.utils.read_document import read_docx
class TestReadDocx:
def test_read_docx(self):
- docx_sample = PROJECT_ROOT / "tests/data/docx_for_test.docx"
+ docx_sample = METAGPT_ROOT / "tests/data/docx_for_test.docx"
docx = read_docx(docx_sample)
assert len(docx) == 6
-
+
