apunkt/MetaGPT
1
0
Fork 0
mirror of https://github.com/FoundationAgents/MetaGPT.git synced 2026-05-21 14:05:17 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'main' into dev_updated

Browse source
This commit is contained in:
yzlin 2024-01-10 14:10:15 +08:00
commit 853086924a
429 changed files with 24237 additions and 5835 deletions
Download patch file Download diff file Expand all files Collapse all files

41
.devcontainer/README.md
View file

@ -1,39 +1,34 @@
# Dev container
# Dev Container
This project includes a [dev container](https://containers.dev/), which lets you use a container as a full-featured dev environment.
This project includes a [Dev Container](https://containers.dev/), offering you a comprehensive and fully-featured development environment within a container. By leveraging the Dev Container configuration in this folder, you can seamlessly build and initiate MetaGPT locally. For detailed information, please refer to the main README in the home directory.
You can use the dev container configuration in this folder to build and start running MetaGPT locally! For more, refer to the main README under the home directory.
You can use it in [GitHub Codespaces](https://github.com/features/codespaces) or the [VS Code Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
You can utilize this Dev Container in [GitHub Codespaces](https://github.com/features/codespaces) or with the [VS Code Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
## GitHub Codespaces
<a href="https://codespaces.new/geekan/MetaGPT"><img src="https://github.com/codespaces/badge.svg" alt="Open in GitHub Codespaces"></a>
[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/geekan/MetaGPT)
You may use the button above to open this repo in a Codespace
Click the button above to open this repository in a Codespace. For additional information, refer to the [GitHub documentation on creating a Codespace](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
## VS Code Dev Containers
<a href="https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT"><img src="https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode" alt="Open in Dev Containers"></a>
[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT)
Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name:
https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT
Note: Clicking the link above opens the main repository. To open your local cloned repository, replace the URL with your username and cloned repository's name: `https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/<your-username>/<your-repo-name>`
If you have VS Code and Docker installed, use the button above to get started. This will prompt VS Code to install the Dev Containers extension if it's not already installed, clone the source code into a container volume, and set up a dev container for you.
If you already have VS Code and Docker installed, you can use the button above to get started. This will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
Alternatively, follow these steps to open this repository in a container using the VS Code Dev Containers extension:
You can also follow these steps to open this repo in a container using the VS Code Dev Containers extension:
1. For first-time users of a development container, ensure your system meets the prerequisites (e.g., Docker installation) as outlined in the [getting started steps](https://aka.ms/vscode-remote/containers/getting-started).
1. If this is your first time using a development container, please ensure your system meets the pre-reqs (i.e. have Docker installed) in the [getting started steps](https://aka.ms/vscode-remote/containers/getting-started).
2. Open a locally cloned copy of the code:
- Fork and Clone this repository to your local filesystem.
2. To open a locally cloned copy of the code:
- Fork and clone this repository to your local file system.
- Press <kbd>F1</kbd> and select the **Dev Containers: Open Folder in Container...** command.
- Select the cloned copy of this folder, wait for the container to start, and try things out!
- Choose the cloned folder, wait for the container to initialize, and start exploring!
You can learn more in the [Dev Containers documentation](https://code.visualstudio.com/docs/devcontainers/containers).
Learn more in the [VS Code Dev Containers documentation](https://code.visualstudio.com/docs/devcontainers/containers).
## Tips and tricks
## Tips and Tricks
* If you are working with the same repository folder in a container and Windows, you'll want consistent line endings (otherwise you may see hundreds of changes in the SCM view). The `.gitattributes` file in the root of this repo will disable line ending conversion and should prevent this. See [tips and tricks](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files) for more info.
* If you'd like to review the contents of the image used in this dev container, you can check it out in the [devcontainers/images](https://github.com/devcontainers/images/tree/main/src/python) repo.
* When working with the same repository folder in both a container and on Windows, it's crucial to have consistent line endings to avoid numerous changes in the SCM view. The `.gitattributes` file in the root of this repository disables line ending conversion, helping to prevent this issue. For more information, see [resolving git line ending issues in containers](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files).
* If you're curious about the contents of the image used in this Dev Container, you can review it in the [devcontainers/images](https://github.com/devcontainers/images/tree/main/src/python) repository.

2
.devcontainer/postCreateCommand.sh
View file

@ -4,4 +4,4 @@ sudo npm install -g @mermaid-js/mermaid-cli
# Step 2: Ensure that Python 3.9+ is installed on your system. You can check this by using:
python --version
pip install -e.
pip install -e .

1
.dockerignore
View file

@ -1,7 +1,6 @@
workspace
tmp
build
workspace
dist
data
geckodriver.log

27
.gitattributes vendored
View file

@ -1,2 +1,29 @@
# HTML code is incorrectly calculated into statistics, so ignore them
*.html linguist-detectable=false
# Auto detect text files and perform LF normalization
* text=auto eol=lf
# Ensure shell scripts use LF (Linux style) line endings on Windows
*.sh text eol=lf
# Treat specific binary files as binary and prevent line ending conversion
*.png binary
*.jpg binary
*.gif binary
*.ico binary
# Preserve original line endings for specific document files
*.doc text eol=crlf
*.docx text eol=crlf
*.pdf binary
# Ensure source code and script files use LF line endings
*.py text eol=lf
*.js text eol=lf
*.html text eol=lf
*.css text eol=lf
# Specify custom diff driver for specific file types
*.md diff=markdown
*.json diff=json

5
.github/ISSUE_TEMPLATE/config.yaml vendored Normal file
View file

@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: "📑 Read online docs"
url: https://docs.deepwisdom.ai/
about: Find the tutorials, use cases and blogs from the doc site.

14
.github/ISSUE_TEMPLATE/request_new_features.md vendored Normal file
View file

@ -0,0 +1,14 @@
---
name: "🤔 Request new features"
about: There are some ideas or demands want to discuss with the official and hope to be implemented in the future.
title: ''
labels: kind/features
assignees: ''
---
**Feature description**
<!-- Clear and direct description of the functionality of the currently submitted or proposed feature -->
**Your Feature**
<!-- Describe the idea or process of implementing the current feature. Of course, you can also paste the URL address of your Pull Request. -->
<!-- When submitting features, you need to complete the corresponding doc/tests/examples to facilitate verification by reviewers. -->

29
.github/ISSUE_TEMPLATE/show_me_the_bug.md vendored Normal file
View file

@ -0,0 +1,29 @@
---
name: "🪲 Show me the Bug"
about: Something happened when I use MetaGPT, I want to report it and hope to get help from the official and community.
title: ''
labels: kind/bug
assignees: ''
---
**Bug description**
<!-- Clearly and directly describe the current bug -->
**Bug solved method**
<!-- If you solved the bug, describe the idea or process to solve the current bug. Of course, you can also paste the URL address of your Pull Request. -->
<!-- If not, provide more auxiliary information to facilitate our further positioning and investigation -->
**Environment information**
<!-- Environment：System version (like ubuntu 22.04), Python version (conda python 3.7), LLM type and model (OpenAI gpt-4-1106-preview) -->
- LLM type and model name:
- System version:
- Python version:
<!-- Dependent packagess：the packages version cause the bug(like `pydantic 1.10.8`), installation method（like `pip install metagpt` or `pip install from source` or `run in docker`） -->
- packages version:
- installation method:
**Screenshots or logs**
<!-- Screenshots or logs of the bug can help us understand the problem more quickly -->

19
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,19 @@
**Features**
<!-- Clear and direct description of the submit features. -->
<!-- If it's a bug fix, please also paste the issue link. -->
- xx
- yy
**Feature Docs**
<!-- The RFC, tutorial, or use cases about the feature if it's a pretty big update. If not, there is no need to fill. -->
**Influence**
<!-- Tell me the impact of the new feature and I'll focus on it. -->
**Result**
<!-- The screenshot/log of unittest/running result -->
**Other**
<!-- Something else about this PR. -->

30
.github/workflows/pre-commit.yaml vendored Normal file
View file

@ -0,0 +1,30 @@
name: Pre-commit checks
on:
pull_request:
branches:
- '**'
push:
branches:
- '**'
jobs:
pre-commit-check:
runs-on: ubuntu-latest
steps:
- name: Checkout Source Code
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: '3.9.17'
- name: Install pre-commit
run: pip install pre-commit
- name: Initialize pre-commit
run: pre-commit install
- name: Run pre-commit hooks
run: pre-commit run --all-files

79
.github/workflows/unittest.yaml vendored Normal file
View file

@ -0,0 +1,79 @@
name: Unit Tests
on:
workflow_dispatch:
pull_request_target:
push:
branches:
- '*-debugger'
jobs:
build:
runs-on: ubuntu-latest
environment: unittest
strategy:
matrix:
# python-version: ['3.9', '3.10', '3.11']
python-version: ['3.9']
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install dependencies
run: |
sh tests/scripts/run_install_deps.sh
- name: Run reverse proxy script for ssh service
if: contains(github.ref, '-debugger')
continue-on-error: true
env:
FPR_SERVER_ADDR: ${{ secrets.FPR_SERVER_ADDR }}
FPR_TOKEN: ${{ secrets.FPR_TOKEN }}
FPR_SSH_REMOTE_PORT: ${{ secrets.FPR_SSH_REMOTE_PORT }}
RSA_PUB: ${{ secrets.RSA_PUB }}
SSH_PORT: ${{ vars.SSH_PORT || '22'}}
run: |
echo "Run \"ssh $(whoami)@FPR_SERVER_HOST -p FPR_SSH_REMOTE_PORT\" and \"cd $(pwd)\""
mkdir -p ~/.ssh/
echo $RSA_PUB >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
wget https://github.com/fatedier/frp/releases/download/v0.32.1/frp_0.32.1_linux_amd64.tar.gz -O frp.tar.gz
tar xvzf frp.tar.gz -C /opt
mv /opt/frp* /opt/frp
/opt/frp/frpc tcp --server_addr $FPR_SERVER_ADDR --token $FPR_TOKEN --local_port $SSH_PORT --remote_port $FPR_SSH_REMOTE_PORT
- name: Test with pytest
run: |
export ALLOW_OPENAI_API_CALL=0
echo "${{ secrets.METAGPT_KEY_YAML }}" | base64 -d > config/key.yaml
pytest tests/ --doctest-modules --cov=./metagpt/ --cov-report=xml:cov.xml --cov-report=html:htmlcov --durations=20 | tee unittest.txt
- name: Show coverage report
run: |
coverage report -m
- name: Show failed tests and overall summary
run: |
grep -E "FAILED tests|ERROR tests|[0-9]+ passed," unittest.txt
failed_count=$(grep -E "FAILED|ERROR" unittest.txt | wc -l)
if [[ "$failed_count" -gt 0 ]]; then
echo "$failed_count failed lines found! Task failed."
exit 1
fi
- name: Upload pytest test results
uses: actions/upload-artifact@v3
with:
name: pytest-results-${{ matrix.python-version }}
path: |
./unittest.txt
./htmlcov/
./tests/data/rsp_cache_new.json
retention-days: 3
if: ${{ always() }}
- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
if: ${{ always() }}

27
.gitignore vendored
View file

@ -52,6 +52,7 @@ coverage.xml
.hypothesis/
.pytest_cache/
cover/
unittest.txt
# Translations
*.mo
@ -59,6 +60,7 @@ cover/
# Django stuff:
*.log
logs
local_settings.py
db.sqlite3
db.sqlite3-journal
@ -144,24 +146,35 @@ cython_debug/
allure-report
allure-results
# idea
# idea / vscode / macos
.idea
.DS_Store
.vscode
log.txt
docs/scripts/set_env.sh
key.yaml
output.json
data
data/output_add.json
data.ms
examples/nb/
.chroma
*~$*
workspace/*
*.mmd
tmp
output.wav
metagpt/roles/idea_agent.py
.aider*
*.bak
*.bk
# output folder
output
tmp.png
.dependencies.json
tests/metagpt/utils/file_repo_git
*.tmp
*.png
htmlcov
htmlcov.*
*.dot
*.pkl
*-structure.csv
*-structure.json

8
.pre-commit-config.yaml
View file

@ -1,8 +1,9 @@
default_stages: [ commit ]
# Install
# 1. pip install pre-commit
# 2. pre-commit install(the first time you download the repo, it will be cached for future use)
# 1. pip install metagpt[dev]
# 2. pre-commit install
# 3. pre-commit run --all-files # make sure all files are clean
repos:
- repo: https://github.com/pycqa/isort
rev: 5.11.5
@ -19,9 +20,10 @@ repos:
rev: v0.0.284
hooks:
- id: ruff
args: [ --fix ]
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
args: ['--line-length', '120']
args: ['--line-length', '120']

2
Dockerfile
View file

@ -18,7 +18,7 @@ COPY . /app/metagpt
WORKDIR /app/metagpt
RUN mkdir workspace &&\
pip install --no-cache-dir -r requirements.txt &&\
pip install -e.
pip install -e .
# Running with an infinite loop using the tail command
CMD ["sh", "-c", "tail -f /dev/null"]

2
LICENSE
View file

@ -1,6 +1,6 @@
The MIT License
Copyright (c) Chenglin Wu
Copyright (c) 2024 Chenglin Wu
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal

48
README.md
View file

@ -1,3 +1,4 @@
# MetaGPT: The Multi-Agent Framework
<p align="center">
@ -32,8 +33,12 @@ # MetaGPT: The Multi-Agent Framework
<p align="center">Software Company Multi-Role Schematic (Gradually Implementing)</p>
## News
🚀 Jan 03: Here comes [v0.6.0](https://github.com/geekan/MetaGPT/releases/tag/v0.6.0)! In this version, we added serialization and deserialization of important objects and enabled breakpoint recovery. We upgraded OpenAI package to v1.6.0 and supported Gemini, ZhipuAI, Ollama, OpenLLM, etc. Moreover, we provided extremely simple examples where you need only 7 lines to implement a general election [debate](https://github.com/geekan/MetaGPT/blob/main/examples/debate_simple.py). Check out more details [here](https://github.com/geekan/MetaGPT/releases/tag/v0.6.0)!
🚀 Dec 15: [v0.5.0](https://github.com/geekan/MetaGPT/releases/tag/v0.5.0) is released! We introduced **incremental development**, facilitating agents to build up larger projects on top of their previous efforts or existing codebase. We also launched a whole collection of important features, including **multilingual support** (experimental), multiple **programming languages support** (experimental), **incremental development** (experimental), CLI support, pip support, enhanced code review, documentation mechanism, and optimized messaging mechanism!
## Install
### Pip installation
@ -48,21 +53,26 @@ # conda activate metagpt
# Step 2: Clone the repository to your local machine for latest version, and install it.
git clone https://github.com/geekan/MetaGPT.git
cd MetaGPT
pip3 install -e. # or pip3 install metagpt # for stable version
pip3 install -e . # or pip3 install metagpt # for stable version
# Step 3: run the startup.py
# setup your OPENAI_API_KEY in key.yaml copy from config.yaml
python3 startup.py "Write a cli snake game"
# Step 3: setup your OPENAI_API_KEY, or make sure it existed in the env
mkdir ~/.metagpt
cp config/config.yaml ~/.metagpt/config.yaml
vim ~/.metagpt/config.yaml
# Step 4 [Optional]: If you want to save the artifacts like diagrams such as quadrant chart, system designs, sequence flow in the workspace, you can execute the step before Step 3. By default, the framework is compatible, and the entire process can be run completely without executing this step.
# Step 4: run metagpt cli
metagpt "Create a 2048 game in python"
# Step 5 [Optional]: If you want to save the artifacts like diagrams such as quadrant chart, system designs, sequence flow in the workspace, you can execute the step before Step 3. By default, the framework is compatible, and the entire process can be run completely without executing this step.
# If executing, ensure that NPM is installed on your system. Then install mermaid-js. (If you don't have npm in your computer, please go to the Node.js official website to install Node.js https://nodejs.org/ and then you will have npm tool in your computer.)
npm --version
sudo npm install -g @mermaid-js/mermaid-cli
```
detail installation please refer to [cli_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-stable-version)
detail installation please refer to [cli_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-stable-version)
### Docker installation
> Note: In the Windows, you need to replace "/opt/metagpt" with a directory that Docker has permission to create, such as "D:\Users\x\metagpt"
```bash
# Step 1: Download metagpt official image and prepare config.yaml
@ -77,10 +87,10 @@ # Step 2: Run metagpt demo with container
-v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
-v /opt/metagpt/workspace:/app/metagpt/workspace \
metagpt/metagpt:latest \
python startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
```
detail installation please refer to [docker_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-with-docker)
detail installation please refer to [docker_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-with-docker)
### QuickStart & Demo Video
- Try it on [MetaGPT Huggingface Space](https://huggingface.co/spaces/deepwisdom/MetaGPT)
@ -91,19 +101,19 @@ ### QuickStart & Demo Video
## Tutorial
- 🗒 [Online Document](https://docs.deepwisdom.ai/)
- 💻 [Usage](https://docs.deepwisdom.ai/guide/get_started/quickstart.html)
- 🔎 [What can MetaGPT do?](https://docs.deepwisdom.ai/guide/get_started/introduction.html)
- 🗒 [Online Document](https://docs.deepwisdom.ai/main/en/)
- 💻 [Usage](https://docs.deepwisdom.ai/main/en/guide/get_started/quickstart.html)
- 🔎 [What can MetaGPT do?](https://docs.deepwisdom.ai/main/en/guide/get_started/introduction.html)
- 🛠 How to build your own agents?
- [MetaGPT Usage & Development Guide | Agent 101](https://docs.deepwisdom.ai/guide/tutorials/agent_101.html)
- [MetaGPT Usage & Development Guide | MultiAgent 101](https://docs.deepwisdom.ai/guide/tutorials/multi_agent_101.html)
- [MetaGPT Usage & Development Guide | Agent 101](https://docs.deepwisdom.ai/main/en/guide/tutorials/agent_101.html)
- [MetaGPT Usage & Development Guide | MultiAgent 101](https://docs.deepwisdom.ai/main/en/guide/tutorials/multi_agent_101.html)
- 🧑‍💻 Contribution
- [Develop Roadmap](docs/ROADMAP.md)
- 🔖 Use Cases
- [Debate](https://docs.deepwisdom.ai/guide/use_cases/multi_agent/debate.html)
- [Researcher](https://docs.deepwisdom.ai/guide/use_cases/agent/researcher.html)
- [Recepit Assistant](https://docs.deepwisdom.ai/guide/use_cases/agent/receipt_assistant.html)
- ❓ [FAQs](https://docs.deepwisdom.ai/guide/faq.html)
- [Debate](https://docs.deepwisdom.ai/main/en/guide/use_cases/multi_agent/debate.html)
- [Researcher](https://docs.deepwisdom.ai/main/en/guide/use_cases/agent/researcher.html)
- [Recepit Assistant](https://docs.deepwisdom.ai/main/en/guide/use_cases/agent/receipt_assistant.html)
- ❓ [FAQs](https://docs.deepwisdom.ai/main/en/guide/faq.html)
## Support
@ -116,14 +126,14 @@ ### Contact Information
If you have any questions or feedback about this project, please feel free to contact us. We highly appreciate your suggestions!
- **Email:** alexanderwu@fuzhi.ai
- **Email:** alexanderwu@deepwisdom.ai
- **GitHub Issues:** For more technical inquiries, you can also create a new issue in our [GitHub repository](https://github.com/geekan/metagpt/issues).
We will respond to all questions within 2-3 business days.
## Citation
For now, cite the [Arxiv paper](https://arxiv.org/abs/2308.00352):
For now, cite the [arXiv paper](https://arxiv.org/abs/2308.00352):
```bibtex
@misc{hong2023metagpt,

79
config/config.yaml
View file

@ -1,16 +1,21 @@
# DO NOT MODIFY THIS FILE, create a new key.yaml, define OPENAI_API_KEY.
# The configuration of key.yaml has a higher priority and will not enter git
#### Project Path Setting
# WORKSPACE_PATH: "Path for placing output files"
#### if OpenAI
## The official OPENAI_API_BASE is https://api.openai.com/v1
## If the official OPENAI_API_BASE is not available, we recommend using the [openai-forward](https://github.com/beidongjiedeguang/openai-forward).
## Or, you can configure OPENAI_PROXY to access official OPENAI_API_BASE.
OPENAI_API_BASE: "https://api.openai.com/v1"
## The official OPENAI_BASE_URL is https://api.openai.com/v1
## If the official OPENAI_BASE_URL is not available, we recommend using the [openai-forward](https://github.com/beidongjiedeguang/openai-forward).
## Or, you can configure OPENAI_PROXY to access official OPENAI_BASE_URL.
OPENAI_BASE_URL: "https://api.openai.com/v1"
#OPENAI_PROXY: "http://127.0.0.1:8118"
#OPENAI_API_KEY: "YOUR_API_KEY" # set the value to sk-xxx if you host the openai interface for open llm model
OPENAI_API_MODEL: "gpt-4"
MAX_TOKENS: 1500
#OPENAI_API_KEY: "YOUR_API_KEY" # set the value to sk-xxx if you host the openai interface for open llm model
OPENAI_API_MODEL: "gpt-4-1106-preview"
MAX_TOKENS: 4096
RPM: 10
TIMEOUT: 60 # Timeout for llm invocation
#DEFAULT_PROVIDER: openai
#### if Spark
#SPARK_APPID : "YOUR_APPID"
@ -20,20 +25,35 @@ RPM: 10
#SPARK_URL : "ws://spark-api.xf-yun.com/v2.1/chat"
#### if Anthropic
#Anthropic_API_KEY: "YOUR_API_KEY"
#ANTHROPIC_API_KEY: "YOUR_API_KEY"
#### if AZURE, check https://github.com/openai/openai-cookbook/blob/main/examples/azure/chat.ipynb
#### You can use ENGINE or DEPLOYMENT mode
#OPENAI_API_TYPE: "azure"
#OPENAI_API_BASE: "YOUR_AZURE_ENDPOINT"
#OPENAI_BASE_URL: "YOUR_AZURE_ENDPOINT"
#OPENAI_API_KEY: "YOUR_AZURE_API_KEY"
#OPENAI_API_VERSION: "YOUR_AZURE_API_VERSION"
#DEPLOYMENT_NAME: "YOUR_DEPLOYMENT_NAME"
#DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
#### if zhipuai from `https://open.bigmodel.cn`. You can set here or export API_KEY="YOUR_API_KEY"
# ZHIPUAI_API_KEY: "YOUR_API_KEY"
#### if Google Gemini from `https://ai.google.dev/` and API_KEY from `https://makersuite.google.com/app/apikey`.
#### You can set here or export GOOGLE_API_KEY="YOUR_API_KEY"
# GEMINI_API_KEY: "YOUR_API_KEY"
#### if use self-host open llm model with openai-compatible interface
#OPEN_LLM_API_BASE: "http://127.0.0.1:8000/v1"
#OPEN_LLM_API_MODEL: "llama2-13b"
#
##### if use Fireworks api
#FIREWORKS_API_KEY: "YOUR_API_KEY"
#FIREWORKS_API_BASE: "https://api.fireworks.ai/inference/v1"
#FIREWORKS_API_MODEL: "YOUR_LLM_MODEL" # example, accounts/fireworks/models/llama-v2-13b-chat
#### if use self-host open llm model by ollama
# OLLAMA_API_BASE: http://127.0.0.1:11434/api
# OLLAMA_API_MODEL: llama2
#### for Search
## Supported values: serpapi/google/serper/ddg
@ -68,8 +88,8 @@ RPM: 10
#### for Stable Diffusion
## Use SD service, based on https://github.com/AUTOMATIC1111/stable-diffusion-webui
SD_URL: "YOUR_SD_URL"
SD_T2I_API: "/sdapi/v1/txt2img"
#SD_URL: "YOUR_SD_URL"
#SD_T2I_API: "/sdapi/v1/txt2img"
#### for Execution
#LONG_TERM_MEMORY: false
@ -84,8 +104,8 @@ SD_T2I_API: "/sdapi/v1/txt2img"
# CALC_USAGE: false
### for Research
MODEL_FOR_RESEARCHER_SUMMARY: gpt-3.5-turbo
MODEL_FOR_RESEARCHER_REPORT: gpt-3.5-turbo-16k
# MODEL_FOR_RESEARCHER_SUMMARY: gpt-3.5-turbo
# MODEL_FOR_RESEARCHER_REPORT: gpt-3.5-turbo-16k
### choose the engine for mermaid conversion,
# default is nodejs, you can change it to playwright,pyppeteer or ink
@ -94,4 +114,31 @@ MODEL_FOR_RESEARCHER_REPORT: gpt-3.5-turbo-16k
### browser path for pyppeteer engine, support Chrome, Chromium,MS Edge
#PYPPETEER_EXECUTABLE_PATH: "/usr/bin/google-chrome-stable"
PROMPT_FORMAT: json #json or markdown
### for repair non-openai LLM's output when parse json-text if PROMPT_FORMAT=json
### due to non-openai LLM's output will not always follow the instruction, so here activate a post-process
### repair operation on the content extracted from LLM's raw output. Warning, it improves the result but not fix all cases.
# REPAIR_LLM_OUTPUT: false
# PROMPT_FORMAT: json #json or markdown
### Agent configurations
# RAISE_NOT_CONFIG_ERROR: true # "true" if the LLM key is not configured, throw a NotConfiguredException, else "false".
# WORKSPACE_PATH_WITH_UID: false # "true" if using `{workspace}/{uid}` as the workspace path; "false" use `{workspace}`.
### Meta Models
#METAGPT_TEXT_TO_IMAGE_MODEL: MODEL_URL
### S3 config
#S3_ACCESS_KEY: "YOUR_S3_ACCESS_KEY"
#S3_SECRET_KEY: "YOUR_S3_SECRET_KEY"
#S3_ENDPOINT_URL: "YOUR_S3_ENDPOINT_URL"
#S3_SECURE: true # true/false
#S3_BUCKET: "YOUR_S3_BUCKET"
### Redis config
#REDIS_HOST: "YOUR_REDIS_HOST"
#REDIS_PORT: "YOUR_REDIS_PORT"
#REDIS_PASSWORD: "YOUR_REDIS_PASSWORD"
#REDIS_DB: "YOUR_REDIS_DB_INDEX, str, 0-based"
# DISABLE_LLM_PROVIDER_CHECK: false

9
docs/.agent-store-config.yaml.example Normal file
View file

@ -0,0 +1,9 @@
role:
name: Teacher # Referenced the `Teacher` in `metagpt/roles/teacher.py`.
module: metagpt.roles.teacher # Referenced `metagpt/roles/teacher.py`.
skills: # Refer to the skill `name` of the published skill in `docs/.well-known/skills.yaml`.
- name: text_to_speech
description: Text-to-speech
- name: text_to_image
description: Create a drawing based on the text.

639
docs/.pylintrc Normal file
View file

@ -0,0 +1,639 @@
[MAIN]
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Clear in-memory caches upon conclusion of linting. Useful if running pylint
# in a server-like mode.
clear-cache-post-run=no
# Load and enable all available extensions. Use --list-extensions to see a list
# all available extensions.
#enable-all-extensions=
# In error mode, messages with a category besides ERROR or FATAL are
# suppressed, and no reports are done by default. Error mode is compatible with
# disabling specific errors.
#errors-only=
# Always return a 0 (non-error) status code, even if lint errors are found.
# This is primarily useful in continuous integration scripts.
#exit-zero=
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code.
extension-pkg-allow-list=
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code. (This is an alternative name to extension-pkg-allow-list
# for backward compatibility.)
extension-pkg-whitelist=pydantic
# Return non-zero exit code if any of these messages/categories are detected,
# even if score is above --fail-under value. Syntax same as enable. Messages
# specified are enabled, while categories only check already-enabled messages.
fail-on=
# Specify a score threshold under which the program will exit with error.
fail-under=10
# Interpret the stdin as a python script, whose filename needs to be passed as
# the module_or_package argument.
#from-stdin=
# Files or directories to be skipped. They should be base names, not paths.
ignore=CVS
# Add files or directories matching the regular expressions patterns to the
# ignore-list. The regex matches against paths and can be in Posix or Windows
# format. Because '\\' represents the directory delimiter on Windows systems,
# it can't be used as an escape character.
ignore-paths=
# Files or directories matching the regular expression patterns are skipped.
# The regex matches against base names, not paths. The default value ignores
# Emacs file locks
#ignore-patterns=^\.#
ignore-patterns=(.)*_test\.py,test_(.)*\.py
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis). It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use, and will cap the count on Windows to
# avoid hangs.
jobs=1
# Control the amount of potential inferred values when inferring a single
# object. This can help the performance when dealing with large functions or
# complex, nested conditions.
limit-inference-results=120
# List of plugins (as comma separated values of python module names) to load,
# usually to register additional checkers.
load-plugins=
# Pickle collected data for later comparisons.
persistent=yes
# Minimum Python version to use for version dependent checks. Will default to
# the version used to run pylint.
py-version=3.9
# Discover python modules and packages in the file system subtree.
recursive=no
# Add paths to the list of the source roots. Supports globbing patterns. The
# source root is an absolute path or a path relative to the current working
# directory used to determine a package namespace for modules located under the
# source root.
source-roots=
# When enabled, pylint would attempt to guess common misconfiguration and emit
# user-friendly hints instead of false-positive error messages.
suggestion-mode=yes
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no
# In verbose mode, extra non-checker-related info will be displayed.
#verbose=
[BASIC]
# Naming style matching correct argument names.
argument-naming-style=snake_case
# Regular expression matching correct argument names. Overrides argument-
# naming-style. If left empty, argument names will be checked with the set
# naming style.
#argument-rgx=
# Naming style matching correct attribute names.
attr-naming-style=snake_case
# Regular expression matching correct attribute names. Overrides attr-naming-
# style. If left empty, attribute names will be checked with the set naming
# style.
#attr-rgx=
# Bad variable names which should always be refused, separated by a comma.
bad-names=foo,
bar,
baz,
toto,
tutu,
tata
# Bad variable names regexes, separated by a comma. If names match any regex,
# they will always be refused
bad-names-rgxs=
# Naming style matching correct class attribute names.
class-attribute-naming-style=any
# Regular expression matching correct class attribute names. Overrides class-
# attribute-naming-style. If left empty, class attribute names will be checked
# with the set naming style.
#class-attribute-rgx=
# Naming style matching correct class constant names.
class-const-naming-style=UPPER_CASE
# Regular expression matching correct class constant names. Overrides class-
# const-naming-style. If left empty, class constant names will be checked with
# the set naming style.
#class-const-rgx=
# Naming style matching correct class names.
class-naming-style=PascalCase
# Regular expression matching correct class names. Overrides class-naming-
# style. If left empty, class names will be checked with the set naming style.
#class-rgx=
# Naming style matching correct constant names.
const-naming-style=UPPER_CASE
# Regular expression matching correct constant names. Overrides const-naming-
# style. If left empty, constant names will be checked with the set naming
# style.
#const-rgx=
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# Naming style matching correct function names.
function-naming-style=snake_case
# Regular expression matching correct function names. Overrides function-
# naming-style. If left empty, function names will be checked with the set
# naming style.
#function-rgx=
# Good variable names which should always be accepted, separated by a comma.
good-names=i,
j,
k,
v,
e,
d,
m,
df,
ex,
Run,
_
# Good variable names regexes, separated by a comma. If names match any regex,
# they will always be accepted
good-names-rgxs=
# Include a hint for the correct naming format with invalid-name.
include-naming-hint=no
# Naming style matching correct inline iteration names.
inlinevar-naming-style=any
# Regular expression matching correct inline iteration names. Overrides
# inlinevar-naming-style. If left empty, inline iteration names will be checked
# with the set naming style.
#inlinevar-rgx=
# Naming style matching correct method names.
method-naming-style=snake_case
# Regular expression matching correct method names. Overrides method-naming-
# style. If left empty, method names will be checked with the set naming style.
#method-rgx=
# Naming style matching correct module names.
module-naming-style=snake_case
# Regular expression matching correct module names. Overrides module-naming-
# style. If left empty, module names will be checked with the set naming style.
#module-rgx=
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=^_
# List of decorators that produce properties, such as abc.abstractproperty. Add
# to this list to register other decorators that produce valid properties.
# These decorators are taken in consideration only for invalid-name.
property-classes=abc.abstractproperty
# Regular expression matching correct type alias names. If left empty, type
# alias names will be checked with the set naming style.
#typealias-rgx=
# Regular expression matching correct type variable names. If left empty, type
# variable names will be checked with the set naming style.
#typevar-rgx=
# Naming style matching correct variable names.
variable-naming-style=snake_case
# Regular expression matching correct variable names. Overrides variable-
# naming-style. If left empty, variable names will be checked with the set
# naming style.
#variable-rgx=
[CLASSES]
# Warn about protected attribute access inside special methods
check-protected-access-in-special-methods=no
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,
__new__,
setUp,
__post_init__
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,_fields,_replace,_source,_make,os._exit
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs
[DESIGN]
# List of regular expressions of class ancestor names to ignore when counting
# public methods (see R0903)
exclude-too-few-public-methods=
# List of qualified class names to ignore when counting class parents (see
# R0901)
ignored-parents=
# Maximum number of arguments for function / method.
max-args=5
# Maximum number of attributes for a class (see R0902).
max-attributes=7
# Maximum number of boolean expressions in an if statement (see R0916).
max-bool-expr=5
# Maximum number of branch for function / method body.
max-branches=12
# Maximum number of locals for function / method body.
max-locals=15
# Maximum number of parents for a class (see R0901).
max-parents=7
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# Maximum number of return / yield for function / method body.
max-returns=6
# Maximum number of statements in function / method body.
max-statements=50
# Minimum number of public methods for a class (see R0903).
min-public-methods=2
[EXCEPTIONS]
# Exceptions that will emit a warning when caught.
overgeneral-exceptions=builtins.BaseException,builtins.Exception
[FORMAT]
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Maximum number of characters on a single line.
max-line-length=120
# Maximum number of lines in a module.
max-module-lines=1000
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.
single-line-class-stmt=no
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
[IMPORTS]
# List of modules that can be imported at any level, not just the top level
# one.
allow-any-import-level=
# Allow explicit reexports by alias from a package __init__.
allow-reexport-from-package=no
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Deprecated modules which should not be used, separated by a comma.
deprecated-modules=
# Output a graph (.gv or any supported image format) of external dependencies
# to the given file (report RP0402 must not be disabled).
ext-import-graph=
# Output a graph (.gv or any supported image format) of all (i.e. internal and
# external) dependencies to the given file (report RP0402 must not be
# disabled).
import-graph=
# Output a graph (.gv or any supported image format) of internal dependencies
# to the given file (report RP0402 must not be disabled).
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
# Couples of modules and preferred modules, separated by a comma.
preferred-modules=
[LOGGING]
# The type of string formatting that logging methods do. `old` means using %
# formatting, `new` is for `{}` formatting.
logging-format-style=old
# Logging modules to check that the string format arguments are in logging
# function parameter format.
logging-modules=logging
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, CONTROL_FLOW, INFERENCE, INFERENCE_FAILURE,
# UNDEFINED.
confidence=HIGH,
CONTROL_FLOW,
INFERENCE,
INFERENCE_FAILURE,
UNDEFINED
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once). You can also use "--disable=all" to
# disable everything first and then re-enable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use "--disable=all --enable=classes
# --disable=W".
disable=raw-checker-failed,
bad-inline-option,
locally-disabled,
file-ignored,
suppressed-message,
useless-suppression,
deprecated-pragma,
use-symbolic-message-instead,
expression-not-assigned,
pointless-statement
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=c-extension-no-member
[METHOD_ARGS]
# List of qualified names (i.e., library.method) which require a timeout
# parameter e.g. 'requests.api.get,requests.api.post'
timeout-methods=requests.api.delete,requests.api.get,requests.api.head,requests.api.options,requests.api.patch,requests.api.post,requests.api.put,requests.api.request
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,
XXX,
TODO
# Regular expression of note tags to take in consideration.
notes-rgx=
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
# Complete name of functions that never returns. When checking for
# inconsistent-return-statements if a never returning function is called then
# it will be considered as an explicit return statement and no message will be
# printed.
never-returning-functions=sys.exit,argparse.parse_error
[REPORTS]
# Python expression which should return a score less than or equal to 10. You
# have access to the variables 'fatal', 'error', 'warning', 'refactor',
# 'convention', and 'info' which contain the number of messages in each
# category, as well as 'statement' which is the total number of statements
# analyzed. This score is used by the global evaluation report (RP0004).
evaluation=max(0, 0 if fatal else 10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10))
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details.
msg-template=
# Set the output format. Available formats are text, parseable, colorized, json
# and msvs (visual studio). You can also give a reporter class, e.g.
# mypackage.mymodule.MyReporterClass.
#output-format=
# Tells whether to display a full report or only the messages.
reports=no
# Activate the evaluation score.
score=yes
[SIMILARITIES]
# Comments are removed from the similarity computation
ignore-comments=yes
# Docstrings are removed from the similarity computation
ignore-docstrings=yes
# Imports are removed from the similarity computation
ignore-imports=yes
# Signatures are removed from the similarity computation
ignore-signatures=yes
# Minimum lines number of a similarity.
min-similarity-lines=4
[SPELLING]
# Limits count of emitted suggestions for spelling mistakes.
max-spelling-suggestions=4
# Spelling dictionary name. No available dictionaries : You need to install
# both the python package and the system dependency for enchant to work..
spelling-dict=
# List of comma separated words that should be considered directives if they
# appear at the beginning of a comment and should not be checked.
spelling-ignore-comment-directives=fmt: on,fmt: off,noqa:,noqa,nosec,isort:skip,mypy:
# List of comma separated words that should not be checked.
spelling-ignore-words=
# A path to a file that contains the private dictionary; one word per line.
spelling-private-dict-file=
# Tells whether to store unknown words to the private dictionary (see the
# --spelling-private-dict-file option) instead of raising a message.
spelling-store-unknown-words=no
[STRING]
# This flag controls whether inconsistent-quotes generates a warning when the
# character used as a quote delimiter is used inconsistently within a module.
check-quote-consistency=no
# This flag controls whether the implicit-str-concat should generate a warning
# on implicit string concatenation in sequences defined over several lines.
check-str-concat-over-line-jumps=no
[TYPECHECK]
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
# Tells whether to warn about missing members when the owner of the attribute
# is inferred to be None.
ignore-none=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# List of symbolic message names to ignore for Mixin members.
ignored-checks-for-mixins=no-member,
not-async-context-manager,
not-context-manager,
attribute-defined-outside-init
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=optparse.Values,thread._local,_thread._local,argparse.Namespace
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
# Regex pattern to define which classes are considered mixins.
mixin-class-rgx=.*[Mm]ixin
# List of decorators that change the signature of a decorated function.
signature-mutators=
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of names allowed to shadow builtins
allowed-redefined-builtins=
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,
_cb
# A regular expression matching the name of dummy variables (i.e. expected to
# not be used).
dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_
# Argument names that match this expression will be ignored.
ignored-argument-names=_.*|^ignored_|^unused_
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io

18
docs/.well-known/ai-plugin.json Normal file
View file

@ -0,0 +1,18 @@
{
"schema_version": "v1",
"name_for_model": "text processing tools",
"name_for_human": "MetaGPT Text Plugin",
"description_for_model": "Plugins for text processing, including text-to-speech, text-to-image, text-to-embedding, text summarization, text-to-code, vector similarity calculation, web content crawling, and more.",
"description_for_human": "Plugins for text processing, including text-to-speech, text-to-image, text-to-embedding, text summarization, text-to-code, vector similarity calculation, web content crawling, and more.",
"auth": {
"type": "none"
},
"api": {
"type": "openapi",
"url": "https://github.com/iorisa/MetaGPT/blob/feature/assistant_role/.well-known/metagpt_oas3_api.yaml",
"has_user_authentication": false
},
"logo_url": "https://github.com/geekan/MetaGPT/blob/main/docs/resources/MetaGPT-logo.png",
"contact_email": "mashenquan@fuzhi.cn",
"legal_info_url": "https://github.com/geekan/MetaGPT/blob/main/docs/README_CN.md"
}

338
docs/.well-known/metagpt_oas3_api.yaml Normal file
View file

@ -0,0 +1,338 @@
openapi: "3.0.0"
info:
title: "MetaGPT Export OpenAPIs"
version: "1.0"
servers:
- url: "/oas3"
variables:
port:
default: '8080'
description: HTTP service port
paths:
/tts/azsure:
x-prerequisite:
configurations:
AZURE_TTS_SUBSCRIPTION_KEY:
type: string
description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
AZURE_TTS_REGION:
type: string
description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
required:
allOf:
- AZURE_TTS_SUBSCRIPTION_KEY
- AZURE_TTS_REGION
post:
summary: "Convert Text to Base64-encoded .wav File Stream"
description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
operationId: azure_tts.oas3_azsure_tts
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- text
properties:
text:
type: string
description: Text to convert
lang:
type: string
description: The language code or locale, e.g., en-US (English - United States)
default: "zh-CN"
voice:
type: string
description: "Voice style, see: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts), [Voice Gallery](https://speech.microsoft.com/portal/voicegallery)"
default: "zh-CN-XiaomoNeural"
style:
type: string
description: "Speaking style to express different emotions. For more details, checkout: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
default: "affectionate"
role:
type: string
description: "Role to specify age and gender. For more details, checkout: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
default: "Girl"
subscription_key:
type: string
description: "Key used to access Azure AI service API, see: [Azure Portal](https://portal.azure.com/) > `Resource Management` > `Keys and Endpoint`"
default: ""
region:
type: string
description: "Location (or region) of your resource, see: [Azure Portal](https://portal.azure.com/) > `Resource Management` > `Keys and Endpoint`"
default: ""
responses:
'200':
description: "Base64-encoded .wav file data if successful, otherwise an empty string."
content:
application/json:
schema:
type: object
properties:
wav_data:
type: string
format: base64
'400':
description: "Bad Request"
'500':
description: "Internal Server Error"
/tts/iflytek:
x-prerequisite:
configurations:
IFLYTEK_APP_ID:
type: string
description: "Application ID is used to access your iFlyTek service API, see: `https://console.xfyun.cn/services/tts`"
IFLYTEK_API_KEY:
type: string
description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`"
IFLYTEK_API_SECRET:
type: string
description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`"
required:
allOf:
- IFLYTEK_APP_ID
- IFLYTEK_API_KEY
- IFLYTEK_API_SECRET
post:
summary: "Convert Text to Base64-encoded .mp3 File Stream"
description: "For more details, check out: [iFlyTek](https://console.xfyun.cn/services/tts)"
operationId: iflytek_tts.oas3_iflytek_tts
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- text
properties:
text:
type: string
description: Text to convert
voice:
type: string
description: "Voice style, see: [iFlyTek Text-to_Speech](https://www.xfyun.cn/doc/tts/online_tts/API.html#%E6%8E%A5%E5%8F%A3%E8%B0%83%E7%94%A8%E6%B5%81%E7%A8%8B)"
default: "xiaoyan"
app_id:
type: string
description: "Application ID is used to access your iFlyTek service API, see: `https://console.xfyun.cn/services/tts`"
default: ""
api_key:
type: string
description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`"
default: ""
api_secret:
type: string
description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`"
default: ""
responses:
'200':
description: "Base64-encoded .mp3 file data if successful, otherwise an empty string."
content:
application/json:
schema:
type: object
properties:
wav_data:
type: string
format: base64
'400':
description: "Bad Request"
'500':
description: "Internal Server Error"
/txt2img/openai:
x-prerequisite:
configurations:
OPENAI_API_KEY:
type: string
description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`"
required:
allOf:
- OPENAI_API_KEY
post:
summary: "Convert Text to Base64-encoded Image Data Stream"
operationId: openai_text_to_image.oas3_openai_text_to_image
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
text:
type: string
description: "The text used for image conversion."
size_type:
type: string
enum: ["256x256", "512x512", "1024x1024"]
default: "1024x1024"
description: "Size of the generated image."
openai_api_key:
type: string
default: ""
description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`"
responses:
'200':
description: "Base64-encoded image data."
content:
application/json:
schema:
type: object
properties:
image_data:
type: string
format: base64
'400':
description: "Bad Request"
'500':
description: "Internal Server Error"
/txt2embedding/openai:
x-prerequisite:
configurations:
OPENAI_API_KEY:
type: string
description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`"
required:
allOf:
- OPENAI_API_KEY
post:
summary: Text to embedding
operationId: openai_text_to_embedding.oas3_openai_text_to_embedding
description: Retrieve an embedding for the provided text using the OpenAI API.
requestBody:
content:
application/json:
schema:
type: object
properties:
input:
type: string
description: The text used for embedding.
model:
type: string
description: "ID of the model to use. For more details, checkout: [models](https://api.openai.com/v1/models)"
enum:
- text-embedding-ada-002
responses:
"200":
description: Successful response
content:
application/json:
schema:
$ref: "#/components/schemas/ResultEmbedding"
"4XX":
description: Client error
content:
application/json:
schema:
$ref: "#/components/schemas/Error"
"5XX":
description: Server error
content:
application/json:
schema:
$ref: "#/components/schemas/Error"
/txt2image/metagpt:
x-prerequisite:
configurations:
METAGPT_TEXT_TO_IMAGE_MODEL_URL:
type: string
description: "Model url."
required:
allOf:
- METAGPT_TEXT_TO_IMAGE_MODEL_URL
post:
summary: "Text to Image"
description: "Generate an image from the provided text using the MetaGPT Text-to-Image API."
operationId: metagpt_text_to_image.oas3_metagpt_text_to_image
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- text
properties:
text:
type: string
description: "The text used for image conversion."
size_type:
type: string
enum: ["512x512", "512x768"]
default: "512x512"
description: "Size of the generated image."
model_url:
type: string
description: "Model reset API URL for text-to-image."
default: ""
responses:
'200':
description: "Base64-encoded image data."
content:
application/json:
schema:
type: object
properties:
image_data:
type: string
format: base64
'400':
description: "Bad Request"
'500':
description: "Internal Server Error"
components:
schemas:
Embedding:
type: object
description: Represents an embedding vector returned by the embedding endpoint.
properties:
object:
type: string
example: embedding
embedding:
type: array
items:
type: number
example: [0.0023064255, -0.009327292, ...]
index:
type: integer
example: 0
Usage:
type: object
properties:
prompt_tokens:
type: integer
example: 8
total_tokens:
type: integer
example: 8
ResultEmbedding:
type: object
properties:
object:
type: string
example: result_embedding
data:
type: array
items:
$ref: "#/components/schemas/Embedding"
model:
type: string
example: text-embedding-ada-002
usage:
$ref: "#/components/schemas/Usage"
Error:
type: object
properties:
error:
type: string
example: An error occurred

35
docs/.well-known/openapi.yaml Normal file
View file

@ -0,0 +1,35 @@
openapi: "3.0.0"
info:
title: Hello World
version: "1.0"
servers:
- url: /openapi
paths:
/greeting/{name}:
post:
summary: Generate greeting
description: Generates a greeting message.
operationId: openapi_v3_hello.post_greeting
responses:
200:
description: greeting response
content:
text/plain:
schema:
type: string
example: "hello dave!"
parameters:
- name: name
in: path
description: Name of the person to greet.
required: true
schema:
type: string
example: "dave"
requestBody:
content:
application/json:
schema:
type: object

161
docs/.well-known/skills.yaml Normal file
View file

@ -0,0 +1,161 @@
skillapi: "0.1.0"
info:
title: "Agent Skill Specification"
version: "1.0"
entities:
Assistant:
summary: assistant
description: assistant
skills:
- name: text_to_speech
description: Generate a voice file from the input text, text-to-speech
id: text_to_speech.text_to_speech
x-prerequisite:
configurations:
AZURE_TTS_SUBSCRIPTION_KEY:
type: string
description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
AZURE_TTS_REGION:
type: string
description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)"
IFLYTEK_APP_ID:
type: string
description: "Application ID is used to access your iFlyTek service API, see: `https://console.xfyun.cn/services/tts`"
IFLYTEK_API_KEY:
type: string
description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`"
IFLYTEK_API_SECRET:
type: string
description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`"
required:
oneOf:
- allOf:
- AZURE_TTS_SUBSCRIPTION_KEY
- AZURE_TTS_REGION
- allOf:
- IFLYTEK_APP_ID
- IFLYTEK_API_KEY
- IFLYTEK_API_SECRET
parameters:
text:
description: 'The text used for voice conversion.'
required: true
type: string
lang:
description: 'The value can contain a language code such as en (English), or a locale such as en-US (English - United States).'
type: string
enum:
- English
- Chinese
default: Chinese
voice:
description: Name of voice styles
type: string
default: zh-CN-XiaomoNeural
style:
type: string
description: Speaking style to express different emotions like cheerfulness, empathy, and calm.
enum:
- affectionate
- angry
- calm
- cheerful
- depressed
- disgruntled
- embarrassed
- envious
- fearful
- gentle
- sad
- serious
default: affectionate
role:
type: string
description: With roles, the same voice can act as a different age and gender.
enum:
- Girl
- Boy
- OlderAdultFemale
- OlderAdultMale
- SeniorFemale
- SeniorMale
- YoungAdultFemale
- YoungAdultMale
default: Girl
examples:
- ask: 'A girl says "hello world"'
answer: 'text_to_speech(text="hello world", role="Girl")'
- ask: 'A boy affectionate says "hello world"'
answer: 'text_to_speech(text="hello world", role="Boy", style="affectionate")'
- ask: 'A boy says "你好"'
answer: 'text_to_speech(text="你好", role="Boy", lang="Chinese")'
returns:
type: string
format: base64
- name: text_to_image
description: Create a drawing based on the text.
id: text_to_image.text_to_image
x-prerequisite:
configurations:
OPENAI_API_KEY:
type: string
description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`"
METAGPT_TEXT_TO_IMAGE_MODEL_URL:
type: string
description: "Model url."
required:
oneOf:
- OPENAI_API_KEY
- METAGPT_TEXT_TO_IMAGE_MODEL_URL
parameters:
text:
description: 'The text used for image conversion.'
type: string
required: true
size_type:
description: size type
type: string
default: "512x512"
examples:
- ask: 'Draw a girl'
answer: 'text_to_image(text="Draw a girl", size_type="512x512")'
- ask: 'Draw an apple'
answer: 'text_to_image(text="Draw an apple", size_type="512x512")'
returns:
type: string
format: base64
- name: web_search
description: Perform Google searches to provide real-time information.
id: web_search.web_search
x-prerequisite:
configurations:
SEARCH_ENGINE:
type: string
description: "Supported values: serpapi/google/serper/ddg"
SERPER_API_KEY:
type: string
description: "SERPER API KEY, For more details, checkout: `https://serper.dev/api-key`"
required:
allOf:
- SEARCH_ENGINE
- SERPER_API_KEY
parameters:
query:
type: string
description: 'The search query.'
required: true
max_results:
type: number
default: 6
description: 'The number of search results to retrieve.'
examples:
- ask: 'Search for information about artificial intelligence'
answer: 'web_search(query="Search for information about artificial intelligence", max_results=6)'
- ask: 'Find news articles about climate change'
answer: 'web_search(query="Find news articles about climate change", max_results=6)'
returns:
type: string

12
docs/FAQ-EN.md
View file

@ -83,10 +83,10 @@
1. PRD stuck / unable to access/ connection interrupted
1. The official OPENAI_API_BASE address is `https://api.openai.com/v1`
1. If the official OPENAI_API_BASE address is inaccessible in your environment (this can be verified with curl), it's recommended to configure using the reverse proxy OPENAI_API_BASE provided by libraries such as openai-forward. For instance, `OPENAI_API_BASE: "``https://api.openai-forward.com/v1``"`
1. If the official OPENAI_API_BASE address is inaccessible in your environment (again, verifiable via curl), another option is to configure the OPENAI_PROXY parameter. This way, you can access the official OPENAI_API_BASE via a local proxy. If you don't need to access via a proxy, please do not enable this configuration; if accessing through a proxy is required, modify it to the correct proxy address. Note that when OPENAI_PROXY is enabled, don't set OPENAI_API_BASE.
1. Note: OpenAI's default API design ends with a v1. An example of the correct configuration is: `OPENAI_API_BASE: "``https://api.openai.com/v1``"`
1. The official OPENAI_BASE_URL address is `https://api.openai.com/v1`
1. If the official OPENAI_BASE_URL address is inaccessible in your environment (this can be verified with curl), it's recommended to configure using the reverse proxy OPENAI_BASE_URL provided by libraries such as openai-forward. For instance, `OPENAI_BASE_URL: "``https://api.openai-forward.com/v1``"`
1. If the official OPENAI_BASE_URL address is inaccessible in your environment (again, verifiable via curl), another option is to configure the OPENAI_PROXY parameter. This way, you can access the official OPENAI_BASE_URL via a local proxy. If you don't need to access via a proxy, please do not enable this configuration; if accessing through a proxy is required, modify it to the correct proxy address. Note that when OPENAI_PROXY is enabled, don't set OPENAI_BASE_URL.
1. Note: OpenAI's default API design ends with a v1. An example of the correct configuration is: `OPENAI_BASE_URL: "``https://api.openai.com/v1``"`
1. Absolutely! How can I assist you today?
@ -98,7 +98,7 @@
1. How to change the investment amount?
1. You can view all commands by typing `python startup.py --help`
1. You can view all commands by typing `metagpt --help`
1. Which version of Python is more stable?
@ -134,7 +134,7 @@
1. Configuration instructions for SD Skills: The SD interface is currently deployed based on *https://github.com/AUTOMATIC1111/stable-diffusion-webui* **For environmental configurations and model downloads, please refer to the aforementioned GitHub repository. To initiate the SD service that supports API calls, run the command specified in cmd with the parameter nowebui, i.e.,
1. > python webui.py --enable-insecure-extension-access --port xxx --no-gradio-queue --nowebui
1. > python3 webui.py --enable-insecure-extension-access --port xxx --no-gradio-queue --nowebui
1.     Once it runs without errors, the interface will be accessible after approximately 1 minute when the model finishes loading.
1. Configure SD_URL and SD_T2I_API in the config.yaml/key.yaml files.
1. ![](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/065295a67b0b4feea665d1372722d49d~tplv-k3u1fbpfcp-zoom-1.image)

31
docs/README_CN.md
View file

@ -47,9 +47,9 @@ # 第 2 步：克隆最新仓库到您的本地机器，并进行安装。
cd MetaGPT
pip3 install -e. # 或者 pip3 install metagpt # 安装稳定版本
# 第 3 步：执行startup.py
# 第 3 步：执行metagpt
# 拷贝config.yaml为key.yaml，并设置你自己的OPENAI_API_KEY
python3 startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# 第 4 步【可选的】：如果你想在执行过程中保存像象限图、系统设计、序列流程等图表这些产物，可以在第3步前执行该步骤。默认的，框架做了兼容，在不执行该步的情况下，也可以完整跑完整个流程。
# 如果执行，确保您的系统上安装了 NPM。并使用npm安装mermaid-js
@ -60,6 +60,7 @@ # 如果执行，确保您的系统上安装了 NPM。并使用npm安装mermaid-
详细的安装请安装 [cli_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-stable-version)
### Docker安装
> 注意：在Windows中，你需要将 "/opt/metagpt" 替换为Docker具有创建权限的目录，比如"D:\Users\x\metagpt"
```bash
# 步骤1: 下载metagpt官方镜像并准备好config.yaml
@ -74,10 +75,10 @@ # 步骤2: 使用容器运行metagpt演示
-v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
-v /opt/metagpt/workspace:/app/metagpt/workspace \
metagpt/metagpt:latest \
python startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
```
详细的安装请安装 [docker_install](https://docs.deepwisdom.ai/zhcn/guide/get_started/installation.html#%E4%BD%BF%E7%94%A8docker%E5%AE%89%E8%A3%85)
详细的安装请安装 [docker_install](https://docs.deepwisdom.ai/main/zh/guide/get_started/installation.html#%E4%BD%BF%E7%94%A8docker%E5%AE%89%E8%A3%85)
### 快速开始的演示视频
- 在 [MetaGPT Huggingface Space](https://huggingface.co/spaces/deepwisdom/MetaGPT) 上进行体验
@ -87,19 +88,19 @@ ### 快速开始的演示视频
https://github.com/geekan/MetaGPT/assets/34952977/34345016-5d13-489d-b9f9-b82ace413419
## 教程
- 🗒 [在线文档](https://docs.deepwisdom.ai/zhcn/)
- 💻 [如何使用](https://docs.deepwisdom.ai/zhcn/guide/get_started/quickstart.html)
- 🔎 [MetaGPT的能力及应用场景](https://docs.deepwisdom.ai/zhcn/guide/get_started/introduction.html)
- 🗒 [在线文档](https://docs.deepwisdom.ai/main/zh/)
- 💻 [如何使用](https://docs.deepwisdom.ai/main/zh/guide/get_started/quickstart.html)
- 🔎 [MetaGPT的能力及应用场景](https://docs.deepwisdom.ai/main/zh/guide/get_started/introduction.html)
- 🛠 如何构建你自己的智能体？
- [MetaGPT的使用和开发教程 | 智能体入门](https://docs.deepwisdom.ai/zhcn/guide/tutorials/agent_101.html)
- [MetaGPT的使用和开发教程 | 多智能体入门](https://docs.deepwisdom.ai/zhcn/guide/tutorials/multi_agent_101.html)
- [MetaGPT的使用和开发教程 | 智能体入门](https://docs.deepwisdom.ai/main/zh/guide/tutorials/agent_101.html)
- [MetaGPT的使用和开发教程 | 多智能体入门](https://docs.deepwisdom.ai/main/zh/guide/tutorials/multi_agent_101.html)
- 🧑‍💻 贡献
- [开发路线图](ROADMAP.md)
- 🔖 示例
- [辩论](https://docs.deepwisdom.ai/zhcn/guide/use_cases/multi_agent/debate.html)
- [调研员](https://docs.deepwisdom.ai/zhcn/guide/use_cases/agent/researcher.html)
- [票据助手](https://docs.deepwisdom.ai/zhcn/guide/use_cases/agent/receipt_assistant.html)
- ❓ [常见问题解答](https://docs.deepwisdom.ai/zhcn/guide/faq.html)
- [辩论](https://docs.deepwisdom.ai/main/zh/guide/use_cases/multi_agent/debate.html)
- [调研员](https://docs.deepwisdom.ai/main/zh/guide/use_cases/agent/researcher.html)
- [票据助手](https://docs.deepwisdom.ai/main/zh/guide/use_cases/agent/receipt_assistant.html)
- ❓ [常见问题解答](https://docs.deepwisdom.ai/main/zh/guide/faq.html)
## 支持
@ -113,14 +114,14 @@ ### 联系信息
如果您对这个项目有任何问题或反馈，欢迎联系我们。我们非常欢迎您的建议！
- **邮箱：** alexanderwu@fuzhi.ai
- **邮箱：** alexanderwu@deepwisdom.ai
- **GitHub 问题：** 对于更技术性的问题，您也可以在我们的 [GitHub 仓库](https://github.com/geekan/metagpt/issues) 中创建一个新的问题。
我们会在2-3个工作日内回复所有问题。
## 引用
引用 [Arxiv paper](https://arxiv.org/abs/2308.00352):
引用 [arXiv paper](https://arxiv.org/abs/2308.00352):
```bibtex
@misc{hong2023metagpt,

33
docs/README_JA.md
View file

@ -41,7 +41,7 @@ ## MetaGPT の能力
## 例（GPT-4 で完全生成）
例えば、`python startup.py "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます
例えば、`metagpt "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます
![Jinri Toutiao Recsys データと API デザイン](resources/workspace/content_rec_sys/resources/data_api_design.png)
@ -60,16 +60,16 @@ ### 伝統的なインストール
```bash
# ステップ 1: Python 3.9+ がシステムにインストールされていることを確認してください。これを確認するには:
python --version
python3 --version
# ステップ 2: リポジトリをローカルマシンにクローンし、インストールする。
git clone https://github.com/geekan/MetaGPT.git
cd MetaGPT
pip install -e.
# ステップ 3: startup.py を実行する
# ステップ 3: metagpt を実行する
# config.yaml を key.yaml にコピーし、独自の OPENAI_API_KEY を設定します
python3 startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# ステップ 4 [オプション]: 実行中に PRD ファイルなどのアーティファクトを保存する場合は、ステップ 3 の前にこのステップを実行できます。デフォルトでは、フレームワークには互換性があり、この手順を実行しなくてもプロセス全体を完了できます。
# NPM がシステムにインストールされていることを確認してください。次に mermaid-js をインストールします。(お使いのコンピューターに npm がない場合は、Node.js 公式サイトで Node.js https://nodejs.org/ をインストールしてください。）
@ -163,6 +163,7 @@ # NPM がシステムにインストールされていることを確認して
注: この方法は pdf エクスポートに対応していません。
### Docker によるインストール
> Windowsでは、"/opt/metagpt"をDockerが作成する権限を持つディレクトリに置き換える必要があります。例えば、"D:\Users\x\metagpt"などです。
```bash
# ステップ 1: metagpt 公式イメージをダウンロードし、config.yaml を準備する
@ -177,7 +178,7 @@ # ステップ 2: コンテナで metagpt デモを実行する
-v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
-v /opt/metagpt/workspace:/app/metagpt/workspace \
metagpt/metagpt:latest \
python startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# コンテナを起動し、その中でコマンドを実行することもできます
docker run --name metagpt -d \
@ -187,7 +188,7 @@ # コンテナを起動し、その中でコマンドを実行することもで
metagpt/metagpt:latest
docker exec -it metagpt /bin/bash
$ python startup.py "Write a cli snake game"
$ metagpt "Write a cli snake game"
```
コマンド `docker run ...` は以下のことを行います:
@ -195,7 +196,7 @@ # コンテナを起動し、その中でコマンドを実行することもで
- 特権モードで実行し、ブラウザの実行権限を得る
- ホスト設定ファイル `/opt/metagpt/config/key.yaml` をコンテナ `/app/metagpt/config/key.yaml` にマップします
- ホストディレクトリ `/opt/metagpt/workspace` をコンテナディレクトリ `/app/metagpt/workspace` にマップするs
- デモコマンド `python startup.py "Write a cli snake game"` を実行する
- デモコマンド `metagpt "Write a cli snake game"` を実行する
### 自分でイメージをビルドする
@ -218,17 +219,17 @@ # 設定ファイルをコピーし、必要な修正を加える。
| 変数名 | config/key.yaml | env |
| --------------------------------------- | ----------------------------------------- | ----------------------------------------------- |
| OPENAI_API_KEY # 自分のキーに置き換える | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." |
| OPENAI_API_BASE # オプション | OPENAI_API_BASE: "https://<YOUR_SITE>/v1" | export OPENAI_API_BASE="https://<YOUR_SITE>/v1" |
| OPENAI_BASE_URL # オプション | OPENAI_BASE_URL: "https://<YOUR_SITE>/v1" | export OPENAI_BASE_URL="https://<YOUR_SITE>/v1" |
## チュートリアル: スタートアップの開始
```shell
# スクリプトの実行
python startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# プロジェクトの実施にエンジニアを雇わないこと
python startup.py "Write a cli snake game" --implement False
metagpt "Write a cli snake game" --no-implement
# エンジニアを雇い、コードレビューを行う
python startup.py "Write a cli snake game" --code_review True
metagpt "Write a cli snake game" --code_review
```
スクリプトを実行すると、`workspace/` ディレクトリに新しいプロジェクトが見つかります。
@ -238,17 +239,17 @@ ### プラットフォームまたはツールの設定
要件を述べるときに、どのプラットフォームまたはツールを使用するかを指定できます。
```shell
python startup.py "pygame をベースとした cli ヘビゲームを書く"
metagpt "pygame をベースとした cli ヘビゲームを書く"
```
### 使用方法
```
会社名
startup.py - 私たちは AI で構成されたソフトウェア・スタートアップです。私たちに投資することは、無限の可能性に満ちた未来に力を与えることです。
metagpt - 私たちは AI で構成されたソフトウェア・スタートアップです。私たちに投資することは、無限の可能性に満ちた未来に力を与えることです。
シノプシス
startup.py IDEA <flags>
metagpt IDEA <flags>
説明
私たちは AI で構成されたソフトウェア・スタートアップです。私たちに投資することは、無限の可能性に満ちた未来に力を与えることです。
@ -299,7 +300,7 @@ ## クイックスタート
## 引用
現時点では、[Arxiv 論文](https://arxiv.org/abs/2308.00352)を引用してください:
現時点では、[arXiv 論文](https://arxiv.org/abs/2308.00352)を引用してください:
```bibtex
@misc{hong2023metagpt,
@ -316,7 +317,7 @@ ## お問い合わせ先
このプロジェクトに関するご質問やご意見がございましたら、お気軽にお問い合わせください。皆様のご意見をお待ちしております！
- **Email:** alexanderwu@fuzhi.ai
- **Email:** alexanderwu@deepwisdom.ai
- **GitHub Issues:** 技術的なお問い合わせについては、[GitHub リポジトリ](https://github.com/geekan/metagpt/issues) に新しい issue を作成することもできます。
ご質問には 2-3 営業日以内に回答いたします。

44
docs/ROADMAP.md
View file

@ -16,12 +16,12 @@ ### Tasks
To reach version v0.5, approximately 70% of the following tasks need to be completed.
1. Usability
1. Release v0.01 pip package to try to solve issues like npm installation (though not necessarily successfully)
1. ~~Release v0.01 pip package to try to solve issues like npm installation (though not necessarily successfully)~~ (v0.3.0)
2. Support for overall save and recovery of software companies
3. Support human confirmation and modification during the process
3. ~~Support human confirmation and modification during the process~~ (v0.3.0) New: Support human confirmation and modification with fewer constrainsts and a more user-friendly interface
4. Support process caching: Consider carefully whether to add server caching mechanism
5. Resolve occasional failure to follow instruction under current prompts, causing code parsing errors, through stricter system prompts
6. Write documentation, describing the current features and usage at all levels
5. ~~Resolve occasional failure to follow instruction under current prompts, causing code parsing errors, through stricter system prompts~~ (v0.4.0, with function call)
6. Write documentation, describing the current features and usage at all levels (ongoing, continuously adding contents to [documentation site](https://docs.deepwisdom.ai/main/en/guide/get_started/introduction.html))
7. ~~Support Docker~~
2. Features
1. Support a more standard and stable parser (need to analyze the format that the current LLM is better at)
@ -30,31 +30,33 @@ ### Tasks
4. Complete the design and implementation of module breakdown
5. Support various modes of memory: clearly distinguish between long-term and short-term memory
6. Perfect the test role, and carry out necessary interactions with humans
7. Provide full mode instead of the current fast mode, allowing natural communication between roles
8. Implement SkillManager and the process of incremental Skill learning
7. ~~Allowing natural communication between roles~~ (v0.5.0)
8. Implement SkillManager and the process of incremental Skill learning (experimentation done with game agents)
9. Automatically get RPM and configure it by calling the corresponding openai page, so that each key does not need to be manually configured
10. ~~IMPORTANT: Support incremental development~~ (v0.5.0)
3. Strategies
1. Support ReAct strategy
2. Support CoT strategy
1. Support ReAct strategy (experimentation done with game agents)
2. Support CoT strategy (experimentation done with game agents)
3. Support ToT strategy
4. Support Reflection strategy
4. Support Reflection strategy (experimentation done with game agents)
5. Support planning
4. Actions
1. Implementation: Search
1. ~~Implementation: Search~~ (v0.2.1)
2. Implementation: Knowledge search, supporting 10+ data formats
3. Implementation: Data EDA
3. Implementation: Data EDA (expected v0.6.0)
4. Implementation: Review
5. Implementation: Add Document
6. Implementation: Delete Document
5. ~~Implementation~~: Add Document (v0.5.0)
6. ~~Implementation~~: Delete Document (v0.5.0)
7. Implementation: Self-training
8. Implementation: DebugError
8. ~~Implementation: DebugError~~ (v0.2.1)
9. Implementation: Generate reliable unit tests based on YAPI
10. Implementation: Self-evaluation
11. Implementation: AI Invocation
12. Implementation: Learning and using third-party standard libraries
13. Implementation: Data collection
14. Implementation: AI training
15. Implementation: Run code
16. Implementation: Web access
15. ~~Implementation: Run code~~ (v0.2.1)
16. ~~Implementation: Web access~~ (v0.2.1)
5. Plugins: Compatibility with plugin system
6. Tools
1. ~~Support SERPER api~~
@ -64,13 +66,13 @@ ### Tasks
1. Perfect the action pool/skill pool for each role
2. Red Book blogger
3. E-commerce seller
4. Data analyst
4. Data analyst (expected v0.6.0)
5. News observer
6. Institutional researcher
6. ~~Institutional researcher~~ (v0.2.1)
8. Evaluation
1. Support an evaluation on a game dataset
2. Reproduce papers, implement full skill acquisition for a single game role, achieving SOTA results
3. Support an evaluation on a math dataset
1. Support an evaluation on a game dataset (experimentation done with game agents)
2. Reproduce papers, implement full skill acquisition for a single game role, achieving SOTA results (experimentation done with game agents)
3. Support an evaluation on a math dataset (expected v0.6.0)
4. Reproduce papers, achieving SOTA results for current mathematical problem solving process
9. LLM
1. Support Claude underlying API

2
docs/install/cli_install_cn.md
View file

@ -15,7 +15,7 @@ # 第 1 步：确保您的系统上安装了 NPM。并使用npm安装mermaid-js
sudo npm install -g @mermaid-js/mermaid-cli
# 第 2 步：确保您的系统上安装了 Python 3.9+。您可以使用以下命令进行检查：
python --version
python3 --version
# 第 3 步：克隆仓库到您的本地机器，并进行安装。
git clone https://github.com/geekan/MetaGPT.git

6
docs/install/docker_install.md
View file

@ -15,7 +15,7 @@ # Step 2: Run metagpt demo with container
-v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
-v /opt/metagpt/workspace:/app/metagpt/workspace \
metagpt/metagpt:latest \
python3 startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# You can also start a container and execute commands in it
docker run --name metagpt -d \
@ -25,7 +25,7 @@ # You can also start a container and execute commands in it
metagpt/metagpt:latest
docker exec -it metagpt /bin/bash
$ python3 startup.py "Write a cli snake game"
$ metagpt "Write a cli snake game"
```
The command `docker run ...` do the following things:
@ -33,7 +33,7 @@ # You can also start a container and execute commands in it
- Run in privileged mode to have permission to run the browser
- Map host configure file `/opt/metagpt/config/key.yaml` to container `/app/metagpt/config/key.yaml`
- Map host directory `/opt/metagpt/workspace` to container `/app/metagpt/workspace`
- Execute the demo command `python3 startup.py "Write a cli snake game"`
- Execute the demo command `metagpt "Write a cli snake game"`
### Build image by yourself

6
docs/install/docker_install_cn.md
View file

@ -15,7 +15,7 @@ # 步骤2: 使用容器运行metagpt演示
-v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \
-v /opt/metagpt/workspace:/app/metagpt/workspace \
metagpt/metagpt:latest \
python startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# 您也可以启动一个容器并在其中执行命令
docker run --name metagpt -d \
@ -25,7 +25,7 @@ # 您也可以启动一个容器并在其中执行命令
metagpt/metagpt:latest
docker exec -it metagpt /bin/bash
$ python startup.py "Write a cli snake game"
$ metagpt "Write a cli snake game"
```
`docker run ...`做了以下事情:
@ -33,7 +33,7 @@ # 您也可以启动一个容器并在其中执行命令
- 以特权模式运行，有权限运行浏览器
- 将主机文件 `/opt/metagpt/config/key.yaml` 映射到容器文件 `/app/metagpt/config/key.yaml`
- 将主机目录 `/opt/metagpt/workspace` 映射到容器目录 `/app/metagpt/workspace`
- 执行示例命令 `python startup.py "Write a cli snake game"`
- 执行示例命令 `metagpt "Write a cli snake game"`
### 自己构建镜像

2
docs/scripts/coverage.sh
View file

@ -1 +1 @@
coverage run --source ./metagpt -m pytest && coverage report -m && coverage html && open htmlcov/index.html
coverage run --source ./metagpt -m pytest --durations=0 --timeout=100 && coverage report -m && coverage html && open htmlcov/index.html

14
docs/tutorial/usage.md
View file

@ -13,17 +13,17 @@ # Copy the configuration file and make the necessary modifications.
| Variable Name | config/key.yaml | env |
| ------------------------------------------ | ----------------------------------------- | ----------------------------------------------- |
| OPENAI_API_KEY # Replace with your own key | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." |
| OPENAI_API_BASE # Optional | OPENAI_API_BASE: "https://<YOUR_SITE>/v1" | export OPENAI_API_BASE="https://<YOUR_SITE>/v1" |
| OPENAI_BASE_URL # Optional | OPENAI_BASE_URL: "https://<YOUR_SITE>/v1" | export OPENAI_BASE_URL="https://<YOUR_SITE>/v1" |
### Initiating a startup
```shell
# Run the script
python startup.py "Write a cli snake game"
metagpt "Write a cli snake game"
# Do not hire an engineer to implement the project
python startup.py "Write a cli snake game" --implement False
metagpt "Write a cli snake game" --no-implement
# Hire an engineer and perform code reviews
python startup.py "Write a cli snake game" --code_review True
metagpt "Write a cli snake game" --code_review
```
After running the script, you can find your new project in the `workspace/` directory.
@ -33,17 +33,17 @@ ### Preference of Platform or Tool
You can tell which platform or tool you want to use when stating your requirements.
```shell
python startup.py "Write a cli snake game based on pygame"
metagpt "Write a cli snake game based on pygame"
```
### Usage
```
NAME
startup.py - We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities.
metagpt - We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities.
SYNOPSIS
startup.py IDEA <flags>
metagpt IDEA <flags>
DESCRIPTION
We are a software startup comprised of AI. By investing in us, you are empowering a future filled with limitless possibilities.

12
docs/tutorial/usage_cn.md
View file

@ -13,14 +13,14 @@ # 复制配置文件并进行必要的修改
| 变量名 | config/key.yaml | env |
| ----------------------------------- | ----------------------------------------- | ----------------------------------------------- |
| OPENAI_API_KEY # 用您自己的密钥替换 | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." |
| OPENAI_API_BASE # 可选 | OPENAI_API_BASE: "https://<YOUR_SITE>/v1" | export OPENAI_API_BASE="https://<YOUR_SITE>/v1" |
| OPENAI_BASE_URL # 可选 | OPENAI_BASE_URL: "https://<YOUR_SITE>/v1" | export OPENAI_BASE_URL="https://<YOUR_SITE>/v1" |
### 示例：启动一个创业公司
```shell
python startup.py "写一个命令行贪吃蛇"
metagpt "写一个命令行贪吃蛇"
# 开启code review模式会花费更多的金钱, 但是会提升代码质量和成功率
python startup.py "写一个命令行贪吃蛇" --code_review True
metagpt "写一个命令行贪吃蛇" --code_review
```
运行脚本后，您可以在 `workspace/` 目录中找到您的新项目。
@ -29,17 +29,17 @@ ### 平台或工具的倾向性
可以在阐述需求时说明想要使用的平台或工具。
例如：
```shell
python startup.py "写一个基于pygame的命令行贪吃蛇"
metagpt "写一个基于pygame的命令行贪吃蛇"
```
### 使用
```
名称
startup.py - 我们是一家AI软件创业公司。通过投资我们，您将赋能一个充满无限可能的未来。
metagpt - 我们是一家AI软件创业公司。通过投资我们，您将赋能一个充满无限可能的未来。
概要
startup.py IDEA <flags>
metagpt IDEA <flags>
描述
我们是一家AI软件创业公司。通过投资我们，您将赋能一个充满无限可能的未来。

54
examples/agent_creator.py
View file

@ -1,23 +1,23 @@
'''
"""
Filename: MetaGPT/examples/agent_creator.py
Created Date: Tuesday, September 12th 2023, 3:28:37 pm
Author: garylin2099
'''
"""
import re
from metagpt.const import PROJECT_ROOT, WORKSPACE_ROOT
from metagpt.actions import Action
from metagpt.config import CONFIG
from metagpt.const import METAGPT_ROOT
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
from metagpt.logs import logger
with open(PROJECT_ROOT / "examples/build_customized_agent.py", "r") as f:
# use official example script to guide AgentCreator
MULTI_ACTION_AGENT_CODE_EXAMPLE = f.read()
EXAMPLE_CODE_FILE = METAGPT_ROOT / "examples/build_customized_agent.py"
MULTI_ACTION_AGENT_CODE_EXAMPLE = EXAMPLE_CODE_FILE.read_text()
class CreateAgent(Action):
PROMPT_TEMPLATE = """
PROMPT_TEMPLATE: str = """
### BACKGROUND
You are using an agent framework called metagpt to write agents capable of different actions,
the usage of metagpt can be illustrated by the following example:
@ -34,7 +34,6 @@ class CreateAgent(Action):
"""
async def run(self, example: str, instruction: str):
prompt = self.PROMPT_TEMPLATE.format(example=example, instruction=instruction)
# logger.info(prompt)
@ -46,29 +45,28 @@ class CreateAgent(Action):
@staticmethod
def parse_code(rsp):
pattern = r'```python(.*)```'
pattern = r"```python(.*)```"
match = re.search(pattern, rsp, re.DOTALL)
code_text = match.group(1) if match else ""
with open(WORKSPACE_ROOT / "agent_created_agent.py", "w") as f:
f.write(code_text)
CONFIG.workspace_path.mkdir(parents=True, exist_ok=True)
new_file = CONFIG.workspace_path / "agent_created_agent.py"
new_file.write_text(code_text)
return code_text
class AgentCreator(Role):
def __init__(
self,
name: str = "Matrix",
profile: str = "AgentCreator",
agent_template: str = MULTI_ACTION_AGENT_CODE_EXAMPLE,
**kwargs,
):
super().__init__(name, profile, **kwargs)
name: str = "Matrix"
profile: str = "AgentCreator"
agent_template: str = MULTI_ACTION_AGENT_CODE_EXAMPLE
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._init_actions([CreateAgent])
self.agent_template = agent_template
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
todo = self._rc.todo
msg = self._rc.memory.get()[-1]
logger.info(f"{self._setting}: to do {self.rc.todo}({self.rc.todo.name})")
todo = self.rc.todo
msg = self.rc.memory.get()[-1]
instruction = msg.content
code_text = await CreateAgent().run(example=self.agent_template, instruction=instruction)
@ -76,19 +74,15 @@ class AgentCreator(Role):
return msg
if __name__ == "__main__":
import asyncio
async def main():
agent_template = MULTI_ACTION_AGENT_CODE_EXAMPLE
creator = AgentCreator(agent_template=agent_template)
# msg = """Write an agent called SimpleTester that will take any code snippet (str)
# and return a testing code (str) for testing
# the given code snippet. Use pytest as the testing framework."""
msg = """
Write an agent called SimpleTester that will take any code snippet (str) and do the following:
1. write a testing code (str) for testing the given code snippet, save the testing code as a .py file in the current working directory;

68
examples/build_customized_agent.py
View file

@ -1,33 +1,30 @@
'''
"""
Filename: MetaGPT/examples/build_customized_agent.py
Created Date: Tuesday, September 19th 2023, 6:52:25 pm
Author: garylin2099
'''
"""
import asyncio
import re
import subprocess
import asyncio
import fire
from metagpt.llm import LLM
from metagpt.actions import Action
from metagpt.roles import Role
from metagpt.schema import Message
from metagpt.logs import logger
from metagpt.roles.role import Role, RoleReactMode
from metagpt.schema import Message
class SimpleWriteCode(Action):
PROMPT_TEMPLATE = """
PROMPT_TEMPLATE: str = """
Write a python function that can {instruction} and provide two runnnable test cases.
Return ```python your_code_here ``` with NO other texts,
your code:
"""
def __init__(self, name: str = "SimpleWriteCode", context=None, llm: LLM = None):
super().__init__(name, context, llm)
name: str = "SimpleWriteCode"
async def run(self, instruction: str):
prompt = self.PROMPT_TEMPLATE.format(instruction=instruction)
rsp = await self._aask(prompt)
@ -38,15 +35,14 @@ class SimpleWriteCode(Action):
@staticmethod
def parse_code(rsp):
pattern = r'```python(.*)```'
pattern = r"```python(.*)```"
match = re.search(pattern, rsp, re.DOTALL)
code_text = match.group(1) if match else rsp
return code_text
class SimpleRunCode(Action):
def __init__(self, name: str = "SimpleRunCode", context=None, llm: LLM = None):
super().__init__(name, context, llm)
name: str = "SimpleRunCode"
async def run(self, code_text: str):
result = subprocess.run(["python3", "-c", code_text], capture_output=True, text=True)
@ -56,21 +52,18 @@ class SimpleRunCode(Action):
class SimpleCoder(Role):
def __init__(
self,
name: str = "Alice",
profile: str = "SimpleCoder",
**kwargs,
):
super().__init__(name, profile, **kwargs)
name: str = "Alice"
profile: str = "SimpleCoder"
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._init_actions([SimpleWriteCode])
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
todo = self._rc.todo # todo will be SimpleWriteCode()
msg = self.get_memories(k=1)[0] # find the most recent messages
logger.info(f"{self._setting}: to do {self.rc.todo}({self.rc.todo.name})")
todo = self.rc.todo # todo will be SimpleWriteCode()
msg = self.get_memories(k=1)[0] # find the most recent messages
code_text = await todo.run(msg.content)
msg = Message(content=code_text, role=self.profile, cause_by=type(todo))
@ -78,27 +71,25 @@ class SimpleCoder(Role):
class RunnableCoder(Role):
def __init__(
self,
name: str = "Alice",
profile: str = "RunnableCoder",
**kwargs,
):
super().__init__(name, profile, **kwargs)
name: str = "Alice"
profile: str = "RunnableCoder"
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._init_actions([SimpleWriteCode, SimpleRunCode])
self._set_react_mode(react_mode="by_order")
self._set_react_mode(react_mode=RoleReactMode.BY_ORDER.value)
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
logger.info(f"{self._setting}: to do {self.rc.todo}({self.rc.todo.name})")
# By choosing the Action by order under the hood
# todo will be first SimpleWriteCode() then SimpleRunCode()
todo = self._rc.todo
todo = self.rc.todo
msg = self.get_memories(k=1)[0] # find the most k recent messages
msg = self.get_memories(k=1)[0] # find the most k recent messages
result = await todo.run(msg.content)
msg = Message(content=result, role=self.profile, cause_by=type(todo))
self._rc.memory.add(msg)
self.rc.memory.add(msg)
return msg
@ -109,5 +100,6 @@ def main(msg="write a function that calculates the product of a list and run it"
result = asyncio.run(role.run(msg))
logger.info(result)
if __name__ == '__main__':
if __name__ == "__main__":
fire.Fire(main)

94
examples/build_customized_multi_agents.py
View file

@ -1,38 +1,35 @@
'''
"""
Filename: MetaGPT/examples/build_customized_multi_agents.py
Created Date: Wednesday, November 15th 2023, 7:12:39 pm
Author: garylin2099
'''
"""
import re
import asyncio
import fire
from metagpt.llm import LLM
from metagpt.actions import Action, BossRequirement
from metagpt.roles import Role
from metagpt.team import Team
from metagpt.schema import Message
from metagpt.actions import Action, UserRequirement
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
from metagpt.team import Team
def parse_code(rsp):
pattern = r'```python(.*)```'
pattern = r"```python(.*)```"
match = re.search(pattern, rsp, re.DOTALL)
code_text = match.group(1) if match else rsp
return code_text
class SimpleWriteCode(Action):
PROMPT_TEMPLATE = """
class SimpleWriteCode(Action):
PROMPT_TEMPLATE: str = """
Write a python function that can {instruction}.
Return ```python your_code_here ``` with NO other texts,
your code:
"""
def __init__(self, name: str = "SimpleWriteCode", context=None, llm: LLM = None):
super().__init__(name, context, llm)
name: str = "SimpleWriteCode"
async def run(self, instruction: str):
prompt = self.PROMPT_TEMPLATE.format(instruction=instruction)
rsp = await self._aask(prompt)
@ -43,31 +40,26 @@ class SimpleWriteCode(Action):
class SimpleCoder(Role):
def __init__(
self,
name: str = "Alice",
profile: str = "SimpleCoder",
**kwargs,
):
super().__init__(name, profile, **kwargs)
self._watch([BossRequirement])
name: str = "Alice"
profile: str = "SimpleCoder"
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._watch([UserRequirement])
self._init_actions([SimpleWriteCode])
class SimpleWriteTest(Action):
PROMPT_TEMPLATE = """
PROMPT_TEMPLATE: str = """
Context: {context}
Write {k} unit tests using pytest for the given function, assuming you have imported it.
Return ```python your_code_here ``` with NO other texts,
your code:
"""
def __init__(self, name: str = "SimpleWriteTest", context=None, llm: LLM = None):
super().__init__(name, context, llm)
name: str = "SimpleWriteTest"
async def run(self, context: str, k: int = 3):
prompt = self.PROMPT_TEMPLATE.format(context=context, k=k)
rsp = await self._aask(prompt)
@ -78,42 +70,37 @@ class SimpleWriteTest(Action):
class SimpleTester(Role):
def __init__(
self,
name: str = "Bob",
profile: str = "SimpleTester",
**kwargs,
):
super().__init__(name, profile, **kwargs)
name: str = "Bob"
profile: str = "SimpleTester"
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._init_actions([SimpleWriteTest])
# self._watch([SimpleWriteCode])
self._watch([SimpleWriteCode, SimpleWriteReview]) # feel free to try this too
self._watch([SimpleWriteCode, SimpleWriteReview]) # feel free to try this too
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
todo = self._rc.todo
logger.info(f"{self._setting}: to do {self.rc.todo}({self.rc.todo.name})")
todo = self.rc.todo
# context = self.get_memories(k=1)[0].content # use the most recent memory as context
context = self.get_memories() # use all memories as context
context = self.get_memories() # use all memories as context
code_text = await todo.run(context, k=5) # specify arguments
code_text = await todo.run(context, k=5) # specify arguments
msg = Message(content=code_text, role=self.profile, cause_by=type(todo))
return msg
class SimpleWriteReview(Action):
PROMPT_TEMPLATE = """
PROMPT_TEMPLATE: str = """
Context: {context}
Review the test cases and provide one critical comments:
"""
def __init__(self, name: str = "SimpleWriteReview", context=None, llm: LLM = None):
super().__init__(name, context, llm)
name: str = "SimpleWriteReview"
async def run(self, context: str):
prompt = self.PROMPT_TEMPLATE.format(context=context)
rsp = await self._aask(prompt)
@ -122,13 +109,11 @@ class SimpleWriteReview(Action):
class SimpleReviewer(Role):
def __init__(
self,
name: str = "Charlie",
profile: str = "SimpleReviewer",
**kwargs,
):
super().__init__(name, profile, **kwargs)
name: str = "Charlie"
profile: str = "SimpleReviewer"
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._init_actions([SimpleWriteReview])
self._watch([SimpleWriteTest])
@ -151,8 +136,9 @@ async def main(
)
team.invest(investment=investment)
team.start_project(idea)
team.run_project(idea)
await team.run(n_round=n_round)
if __name__ == '__main__':
if __name__ == "__main__":
fire.Fire(main)

61
examples/debate.py
View file

@ -1,22 +1,27 @@
'''
"""
Filename: MetaGPT/examples/debate.py
Created Date: Tuesday, September 19th 2023, 6:52:25 pm
Author: garylin2099
'''
@Modified By: mashenquan, 2023-11-1. In accordance with Chapter 2.1.3 of RFC 116, modify the data type of the `send_to`
value of the `Message` object; modify the argument type of `get_by_actions`.
"""
import asyncio
import platform
from typing import Any
import fire
from metagpt.team import Team
from metagpt.actions import Action, BossRequirement
from metagpt.actions import Action, UserRequirement
from metagpt.logs import logger
from metagpt.roles import Role
from metagpt.schema import Message
from metagpt.logs import logger
from metagpt.team import Team
class SpeakAloud(Action):
"""Action: Speak out aloud in a debate (quarrel)"""
PROMPT_TEMPLATE = """
PROMPT_TEMPLATE: str = """
## BACKGROUND
Suppose you are {name}, you are in a debate with {opponent_name}.
## DEBATE HISTORY
@ -26,12 +31,9 @@ class SpeakAloud(Action):
Now it's your turn, you should closely respond to your opponent's latest argument, state your position, defend your arguments, and attack your opponent's arguments,
craft a strong and emotional response in 80 words, in {name}'s rhetoric and viewpoints, your will argue:
"""
def __init__(self, name="SpeakAloud", context=None, llm=None):
super().__init__(name, context, llm)
name: str = "SpeakAloud"
async def run(self, context: str, name: str, opponent_name: str):
prompt = self.PROMPT_TEMPLATE.format(context=context, name=name, opponent_name=opponent_name)
# logger.info(prompt)
@ -39,29 +41,26 @@ class SpeakAloud(Action):
return rsp
class Debator(Role):
def __init__(
self,
name: str,
profile: str,
opponent_name: str,
**kwargs,
):
super().__init__(name, profile, **kwargs)
name: str = ""
profile: str = ""
opponent_name: str = ""
def __init__(self, **data: Any):
super().__init__(**data)
self._init_actions([SpeakAloud])
self._watch([BossRequirement, SpeakAloud])
self.name = name
self.opponent_name = opponent_name
self._watch([UserRequirement, SpeakAloud])
async def _observe(self) -> int:
await super()._observe()
# accept messages sent (from opponent) to self, disregard own messages from the last round
self._rc.news = [msg for msg in self._rc.news if msg.send_to == self.name]
return len(self._rc.news)
self.rc.news = [msg for msg in self.rc.news if msg.send_to == {self.name}]
return len(self.rc.news)
async def _act(self) -> Message:
logger.info(f"{self._setting}: ready to {self._rc.todo}")
todo = self._rc.todo # An instance of SpeakAloud
logger.info(f"{self._setting}: to do {self.rc.todo}({self.rc.todo.name})")
todo = self.rc.todo # An instance of SpeakAloud
memories = self.get_memories()
context = "\n".join(f"{msg.sent_from}: {msg.content}" for msg in memories)
@ -76,25 +75,25 @@ class Debator(Role):
sent_from=self.name,
send_to=self.opponent_name,
)
self._rc.memory.add(msg)
self.rc.memory.add(msg)
return msg
async def debate(idea: str, investment: float = 3.0, n_round: int = 5):
"""Run a team of presidents and watch they quarrel. :) """
"""Run a team of presidents and watch they quarrel. :)"""
Biden = Debator(name="Biden", profile="Democrat", opponent_name="Trump")
Trump = Debator(name="Trump", profile="Republican", opponent_name="Biden")
team = Team()
team.hire([Biden, Trump])
team.invest(investment)
team.start_project(idea, send_to="Biden") # send debate topic to Biden and let him speak first
team.run_project(idea, send_to="Biden") # send debate topic to Biden and let him speak first
await team.run(n_round=n_round)
def main(idea: str, investment: float = 3.0, n_round: int = 10):
"""
:param idea: Debate topic, such as "Topic: The U.S. should commit more in climate change fighting"
:param idea: Debate topic, such as "Topic: The U.S. should commit more in climate change fighting"
or "Trump: Climate change is a hoax"
:param investment: contribute a certain dollar amount to watch the debate
:param n_round: maximum rounds of the debate
@ -105,5 +104,5 @@ def main(idea: str, investment: float = 3.0, n_round: int = 10):
asyncio.run(debate(idea, investment, n_round))
if __name__ == '__main__':
if __name__ == "__main__":
fire.Fire(main)

22
examples/debate_simple.py Normal file
View file

@ -0,0 +1,22 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/12/22
@Author : alexanderwu
@File : debate_simple.py
"""
import asyncio
from metagpt.actions import Action
from metagpt.environment import Environment
from metagpt.roles import Role
from metagpt.team import Team
action1 = Action(name="AlexSay", instruction="Express your opinion with emotion and don't repeat it")
action2 = Action(name="BobSay", instruction="Express your opinion with emotion and don't repeat it")
alex = Role(name="Alex", profile="Democratic candidate", goal="Win the election", actions=[action1], watch=[action2])
bob = Role(name="Bob", profile="Republican candidate", goal="Win the election", actions=[action2], watch=[action1])
env = Environment(desc="US election live broadcast")
team = Team(investment=10.0, env=env, roles=[alex, bob])
asyncio.run(team.run(idea="Topic: climate change. Under 80 words per message.", send_to="Alex", n_round=5))

BIN
examples/example.faiss Normal file
View file

Binary file not shown.

10
examples/example.json Normal file
View file

@ -0,0 +1,10 @@
[
{
"source": "Which facial cleanser is good for oily skin?",
"output": "ABC cleanser is preferred by many with oily skin."
},
{
"source": "Is L'Oreal good to use?",
"output": "L'Oreal is a popular brand with many positive reviews."
}
]

BIN
examples/example.pkl Normal file
View file

Binary file not shown.

BIN
examples/example.xlsx Normal file
View file

Binary file not shown.

12
examples/invoice_ocr.py
View file

@ -10,7 +10,7 @@
import asyncio
from pathlib import Path
from metagpt.roles.invoice_ocr_assistant import InvoiceOCRAssistant
from metagpt.roles.invoice_ocr_assistant import InvoiceOCRAssistant, InvoicePath
from metagpt.schema import Message
@ -19,19 +19,15 @@ async def main():
Path("../tests/data/invoices/invoice-1.pdf"),
Path("../tests/data/invoices/invoice-2.png"),
Path("../tests/data/invoices/invoice-3.jpg"),
Path("../tests/data/invoices/invoice-4.zip")
Path("../tests/data/invoices/invoice-4.zip"),
]
# The absolute path of the file
absolute_file_paths = [Path.cwd() / path for path in relative_paths]
for path in absolute_file_paths:
role = InvoiceOCRAssistant()
await role.run(Message(
content="Invoicing date",
instruct_content={"file_path": path}
))
await role.run(Message(content="Invoicing date", instruct_content=InvoicePath(file_path=path)))
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

17
examples/llm_hello_world.py
View file

@ -7,25 +7,22 @@
"""
import asyncio
from metagpt.llm import LLM, Claude
from metagpt.llm import LLM
from metagpt.logs import logger
async def main():
llm = LLM()
claude = Claude()
logger.info(await claude.aask('你好，请进行自我介绍'))
logger.info(await llm.aask('hello world'))
logger.info(await llm.aask_batch(['hi', 'write python hello world.']))
logger.info(await llm.aask("hello world"))
logger.info(await llm.aask_batch(["hi", "write python hello world."]))
hello_msg = [{'role': 'user', 'content': 'count from 1 to 10. split by newline.'}]
hello_msg = [{"role": "user", "content": "count from 1 to 10. split by newline."}]
logger.info(await llm.acompletion(hello_msg))
logger.info(await llm.acompletion_batch([hello_msg]))
logger.info(await llm.acompletion_batch_text([hello_msg]))
logger.info(await llm.acompletion_text(hello_msg))
# streaming mode, much slower
await llm.acompletion_text(hello_msg, stream=True)
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

2
examples/research.py
View file

@ -12,5 +12,5 @@ async def main():
print(f"save report to {RESEARCH_PATH / f'{topic}.md'}.")
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

2
examples/search_google.py
View file

@ -15,5 +15,5 @@ async def main():
await Searcher().run("What are some good sun protection products?")
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

24
examples/search_kb.py
View file

@ -2,25 +2,31 @@
# -*- coding: utf-8 -*-
"""
@File : search_kb.py
@Modified By: mashenquan, 2023-12-22. Delete useless codes.
"""
import asyncio
from metagpt.const import DATA_PATH
from langchain.embeddings import OpenAIEmbeddings
from metagpt.config import CONFIG
from metagpt.const import DATA_PATH, EXAMPLE_PATH
from metagpt.document_store import FaissStore
from metagpt.logs import logger
from metagpt.roles import Sales
def get_store():
embedding = OpenAIEmbeddings(openai_api_key=CONFIG.openai_api_key, openai_api_base=CONFIG.openai_base_url)
return FaissStore(DATA_PATH / "example.json", embedding=embedding)
async def search():
store = FaissStore(DATA_PATH / 'example.json')
store = FaissStore(EXAMPLE_PATH / "example.json")
role = Sales(profile="Sales", store=store)
queries = ["Which facial cleanser is good for oily skin?", "Is L'Oreal good to use?"]
for query in queries:
logger.info(f"User: {query}")
result = await role.run(query)
logger.info(result)
query = "Which facial cleanser is good for oily skin?"
result = await role.run(query)
logger.info(result)
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(search())

14
examples/search_with_specific_engine.py
View file

@ -1,3 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
"""
import asyncio
from metagpt.roles import Searcher
@ -5,12 +9,14 @@ from metagpt.tools import SearchEngineType
async def main():
question = "What are the most interesting human facts?"
# Serper API
#await Searcher(engine = SearchEngineType.SERPER_GOOGLE).run(["What are some good sun protection products?","What are some of the best beaches?"])
# await Searcher(engine=SearchEngineType.SERPER_GOOGLE).run(question)
# SerpAPI
#await Searcher(engine=SearchEngineType.SERPAPI_GOOGLE).run("What are the best ski brands for skiers?")
await Searcher(engine=SearchEngineType.SERPAPI_GOOGLE).run(question)
# Google API
await Searcher(engine=SearchEngineType.DIRECT_GOOGLE).run("What are the most interesting human facts?")
# await Searcher(engine=SearchEngineType.DIRECT_GOOGLE).run(question)
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

10
examples/sk_agent.py
View file

@ -13,7 +13,7 @@ from semantic_kernel.planning import SequentialPlanner
# from semantic_kernel.planning import SequentialPlanner
from semantic_kernel.planning.action_planner.action_planner import ActionPlanner
from metagpt.actions import BossRequirement
from metagpt.actions import UserRequirement
from metagpt.const import SKILL_DIRECTORY
from metagpt.roles.sk_agent import SkAgent
from metagpt.schema import Message
@ -39,7 +39,7 @@ async def basic_planner_example():
role.import_semantic_skill_from_directory(SKILL_DIRECTORY, "WriterSkill")
role.import_skill(TextSkill(), "TextSkill")
# using BasicPlanner
await role.run(Message(content=task, cause_by=BossRequirement))
await role.run(Message(content=task, cause_by=UserRequirement))
async def sequential_planner_example():
@ -53,7 +53,7 @@ async def sequential_planner_example():
role.import_semantic_skill_from_directory(SKILL_DIRECTORY, "WriterSkill")
role.import_skill(TextSkill(), "TextSkill")
# using BasicPlanner
await role.run(Message(content=task, cause_by=BossRequirement))
await role.run(Message(content=task, cause_by=UserRequirement))
async def basic_planner_web_search_example():
@ -64,7 +64,7 @@ async def basic_planner_web_search_example():
role.import_skill(SkSearchEngine(), "WebSearchSkill")
# role.import_semantic_skill_from_directory(skills_directory, "QASkill")
await role.run(Message(content=task, cause_by=BossRequirement))
await role.run(Message(content=task, cause_by=UserRequirement))
async def action_planner_example():
@ -75,7 +75,7 @@ async def action_planner_example():
role.import_skill(TimeSkill(), "time")
role.import_skill(TextSkill(), "text")
task = "What is the sum of 110 and 990?"
await role.run(Message(content=task, cause_by=BossRequirement)) # it will choose mathskill.Add
await role.run(Message(content=task, cause_by=UserRequirement)) # it will choose mathskill.Add
if __name__ == "__main__":

10
examples/use_off_the_shelf_agent.py
View file

@ -1,12 +1,13 @@
'''
"""
Filename: MetaGPT/examples/use_off_the_shelf_agent.py
Created Date: Tuesday, September 19th 2023, 6:52:25 pm
Author: garylin2099
'''
"""
import asyncio
from metagpt.roles.product_manager import ProductManager
from metagpt.logs import logger
from metagpt.roles.product_manager import ProductManager
async def main():
msg = "Write a PRD for a snake game"
@ -14,5 +15,6 @@ async def main():
result = await role.run(msg)
logger.info(result.content[:100])
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

5
examples/write_tutorial.py
View file

@ -1,10 +1,12 @@
#!/usr/bin/env python3
# _*_ coding: utf-8 _*_
"""
@Time : 2023/9/4 21:40:57
@Author : Stitch-z
@File : tutorial_assistant.py
"""
import asyncio
from metagpt.roles.tutorial_assistant import TutorialAssistant
@ -16,6 +18,5 @@ async def main():
await role.run(topic)
if __name__ == '__main__':
if __name__ == "__main__":
asyncio.run(main())

9
metagpt/actions/__init__.py
View file

@ -9,12 +9,11 @@ from enum import Enum
from metagpt.actions.action import Action
from metagpt.actions.action_output import ActionOutput
from metagpt.actions.add_requirement import BossRequirement
from metagpt.actions.add_requirement import UserRequirement
from metagpt.actions.debug_error import DebugError
from metagpt.actions.design_api import WriteDesign
from metagpt.actions.design_api_review import DesignReview
from metagpt.actions.design_filenames import DesignFilenames
from metagpt.actions.project_management import AssignTasks, WriteTasks
from metagpt.actions.project_management import WriteTasks
from metagpt.actions.research import CollectLinks, WebBrowseAndSummarize, ConductResearch
from metagpt.actions.run_code import RunCode
from metagpt.actions.search_and_summarize import SearchAndSummarize
@ -31,19 +30,17 @@ from metagpt.actions.write_plan import WritePlan
class ActionType(Enum):
"""All types of Actions, used for indexing."""
ADD_REQUIREMENT = BossRequirement
ADD_REQUIREMENT = UserRequirement
WRITE_PRD = WritePRD
WRITE_PRD_REVIEW = WritePRDReview
WRITE_DESIGN = WriteDesign
DESIGN_REVIEW = DesignReview
DESIGN_FILENAMES = DesignFilenames
WRTIE_CODE = WriteCode
WRITE_CODE_REVIEW = WriteCodeReview
WRITE_TEST = WriteTest
RUN_CODE = RunCode
DEBUG_ERROR = DebugError
WRITE_TASKS = WriteTasks
ASSIGN_TASKS = AssignTasks
SEARCH_AND_SUMMARIZE = SearchAndSummarize
COLLECT_LINKS = CollectLinks
WEB_BROWSE_AND_SUMMARIZE = WebBrowseAndSummarize

109
metagpt/actions/action.py
View file

@ -5,36 +5,56 @@
@Author : alexanderwu
@File : action.py
"""
import re
from abc import ABC
from typing import Optional
from tenacity import retry, stop_after_attempt, wait_fixed
from __future__ import annotations
from metagpt.actions.action_output import ActionOutput
from typing import Optional, Union
from pydantic import ConfigDict, Field, model_validator
from metagpt.actions.action_node import ActionNode
from metagpt.llm import LLM
from metagpt.logs import logger
from metagpt.utils.common import OutputParser
from metagpt.utils.custom_decoder import CustomDecoder
from metagpt.provider.base_llm import BaseLLM
from metagpt.schema import (
CodeSummarizeContext,
CodingContext,
RunCodeContext,
SerializationMixin,
TestingContext,
)
class Action(ABC):
def __init__(self, name: str = "", context=None, llm: LLM = None):
self.name: str = name
if llm is None:
llm = LLM()
self.llm = llm
self.context = context
self.prefix = ""
self.profile = ""
self.desc = ""
self.content = ""
self.instruct_content = None
class Action(SerializationMixin, is_polymorphic_base=True):
model_config = ConfigDict(arbitrary_types_allowed=True, exclude=["llm"])
def set_prefix(self, prefix, profile):
name: str = ""
llm: BaseLLM = Field(default_factory=LLM, exclude=True)
context: Union[dict, CodingContext, CodeSummarizeContext, TestingContext, RunCodeContext, str, None] = ""
prefix: str = "" # aask*时会加上prefix，作为system_message
desc: str = "" # for skill manager
node: ActionNode = Field(default=None, exclude=True)
@model_validator(mode="before")
def set_name_if_empty(cls, values):
if "name" not in values or not values["name"]:
values["name"] = cls.__name__
return values
@model_validator(mode="before")
def _init_with_instruction(cls, values):
if "instruction" in values:
name = values["name"]
i = values["instruction"]
values["node"] = ActionNode(key=name, expected_type=str, instruction=i, example="", schema="raw")
return values
def set_prefix(self, prefix):
"""Set prefix for later usage"""
self.prefix = prefix
self.profile = profile
self.llm.system_prompt = prefix
if self.node:
self.node.llm = self.llm
return self
def __str__(self):
return self.__class__.__name__
@ -44,46 +64,17 @@ class Action(ABC):
async def _aask(self, prompt: str, system_msgs: Optional[list[str]] = None) -> str:
"""Append default prefix"""
if not system_msgs:
system_msgs = []
system_msgs.append(self.prefix)
return await self.llm.aask(prompt, system_msgs)
@retry(stop=stop_after_attempt(3), wait=wait_fixed(1))
async def _aask_v1(
self,
prompt: str,
output_class_name: str,
output_data_mapping: dict,
system_msgs: Optional[list[str]] = None,
format="markdown", # compatible to original format
) -> ActionOutput:
"""Append default prefix"""
if not system_msgs:
system_msgs = []
system_msgs.append(self.prefix)
content = await self.llm.aask(prompt, system_msgs)
logger.debug(content)
output_class = ActionOutput.create_model_class(output_class_name, output_data_mapping)
if format == "json":
pattern = r"\[CONTENT\](\s*\{.*?\}\s*)\[/CONTENT\]"
matches = re.findall(pattern, content, re.DOTALL)
for match in matches:
if match:
content = match
break
parsed_data = CustomDecoder(strict=False).decode(content)
else: # using markdown parser
parsed_data = OutputParser.parse_data_with_mapping(content, output_data_mapping)
logger.debug(parsed_data)
instruct_content = output_class(**parsed_data)
return ActionOutput(content, instruct_content)
async def _run_action_node(self, *args, **kwargs):
"""Run action node"""
msgs = args[0]
context = "## History Messages\n"
context += "\n".join([f"{idx}: {i}" for idx, i in enumerate(reversed(msgs))])
return await self.node.fill(context=context, llm=self.llm)
async def run(self, *args, **kwargs):
"""Run action"""
if self.node:
return await self._run_action_node(*args, **kwargs)
raise NotImplementedError("The run method should be implemented in a subclass.")

349
metagpt/actions/action_node.py Normal file
View file

@ -0,0 +1,349 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/12/11 18:45
@Author : alexanderwu
@File : action_node.py
NOTE: You should use typing.List instead of list to do type annotation. Because in the markdown extraction process,
we can use typing to extract the type of the node, but we cannot use built-in list to extract.
"""
import json
from typing import Any, Dict, List, Optional, Tuple, Type
from pydantic import BaseModel, create_model, model_validator
from tenacity import retry, stop_after_attempt, wait_random_exponential
from metagpt.config import CONFIG
from metagpt.llm import BaseLLM
from metagpt.logs import logger
from metagpt.provider.postprocess.llm_output_postprocess import llm_output_postprocess
from metagpt.utils.common import OutputParser, general_after_log
TAG = "CONTENT"
LANGUAGE_CONSTRAINT = "Language: Please use the same language as Human INPUT."
FORMAT_CONSTRAINT = f"Format: output wrapped inside [{TAG}][/{TAG}] like format example, nothing else."
SIMPLE_TEMPLATE = """
## context
{context}
-----
## format example
{example}
## nodes: "<node>: <type> # <instruction>"
{instruction}
## constraint
{constraint}
## action
Follow instructions of nodes, generate output and make sure it follows the format example.
"""
def dict_to_markdown(d, prefix="- ", kv_sep="\n", postfix="\n"):
markdown_str = ""
for key, value in d.items():
markdown_str += f"{prefix}{key}{kv_sep}{value}{postfix}"
return markdown_str
class ActionNode:
"""ActionNode is a tree of nodes."""
schema: str # raw/json/markdown, default: ""
# Action Context
context: str # all the context, including all necessary info
llm: BaseLLM # LLM with aask interface
children: dict[str, "ActionNode"]
# Action Input
key: str # Product Requirement / File list / Code
expected_type: Type # such as str / int / float etc.
# context: str # everything in the history.
instruction: str # the instructions should be followed.
example: Any # example for In Context-Learning.
# Action Output
content: str
instruct_content: BaseModel
def __init__(
self,
key: str,
expected_type: Type,
instruction: str,
example: Any,
content: str = "",
children: dict[str, "ActionNode"] = None,
schema: str = "",
):
self.key = key
self.expected_type = expected_type
self.instruction = instruction
self.example = example
self.content = content
self.children = children if children is not None else {}
self.schema = schema
def __str__(self):
return (
f"{self.key}, {repr(self.expected_type)}, {self.instruction}, {self.example}"
f", {self.content}, {self.children}"
)
def __repr__(self):
return self.__str__()
def add_child(self, node: "ActionNode"):
"""增加子ActionNode"""
self.children[node.key] = node
def add_children(self, nodes: List["ActionNode"]):
"""批量增加子ActionNode"""
for node in nodes:
self.add_child(node)
@classmethod
def from_children(cls, key, nodes: List["ActionNode"]):
"""直接从一系列的子nodes初始化"""
obj = cls(key, str, "", "")
obj.add_children(nodes)
return obj
def get_children_mapping(self, exclude=None) -> Dict[str, Tuple[Type, Any]]:
"""获得子ActionNode的字典，以key索引"""
exclude = exclude or []
return {k: (v.expected_type, ...) for k, v in self.children.items() if k not in exclude}
def get_self_mapping(self) -> Dict[str, Tuple[Type, Any]]:
"""get self key: type mapping"""
return {self.key: (self.expected_type, ...)}
def get_mapping(self, mode="children", exclude=None) -> Dict[str, Tuple[Type, Any]]:
"""get key: type mapping under mode"""
if mode == "children" or (mode == "auto" and self.children):
return self.get_children_mapping(exclude=exclude)
return {} if exclude and self.key in exclude else self.get_self_mapping()
@classmethod
def create_model_class(cls, class_name: str, mapping: Dict[str, Tuple[Type, Any]]):
"""基于pydantic v1的模型动态生成，用来检验结果类型正确性"""
def check_fields(cls, values):
required_fields = set(mapping.keys())
missing_fields = required_fields - set(values.keys())
if missing_fields:
raise ValueError(f"Missing fields: {missing_fields}")
unrecognized_fields = set(values.keys()) - required_fields
if unrecognized_fields:
logger.warning(f"Unrecognized fields: {unrecognized_fields}")
return values
validators = {"check_missing_fields_validator": model_validator(mode="before")(check_fields)}
new_class = create_model(class_name, __validators__=validators, **mapping)
return new_class
def create_children_class(self, exclude=None):
"""使用object内有的字段直接生成model_class"""
class_name = f"{self.key}_AN"
mapping = self.get_children_mapping(exclude=exclude)
return self.create_model_class(class_name, mapping)
def to_dict(self, format_func=None, mode="auto", exclude=None) -> Dict:
"""将当前节点与子节点都按照node: format的格式组织成字典"""
# 如果没有提供格式化函数，使用默认的格式化方式
if format_func is None:
format_func = lambda node: f"{node.instruction}"
# 使用提供的格式化函数来格式化当前节点的值
formatted_value = format_func(self)
# 创建当前节点的键值对
if mode == "children" or (mode == "auto" and self.children):
node_dict = {}
else:
node_dict = {self.key: formatted_value}
if mode == "root":
return node_dict
# 遍历子节点并递归调用 to_dict 方法
exclude = exclude or []
for _, child_node in self.children.items():
if child_node.key in exclude:
continue
node_dict.update(child_node.to_dict(format_func))
return node_dict
def compile_to(self, i: Dict, schema, kv_sep) -> str:
if schema == "json":
return json.dumps(i, indent=4)
elif schema == "markdown":
return dict_to_markdown(i, kv_sep=kv_sep)
else:
return str(i)
def tagging(self, text, schema, tag="") -> str:
if not tag:
return text
if schema == "json":
return f"[{tag}]\n" + text + f"\n[/{tag}]"
else: # markdown
return f"[{tag}]\n" + text + f"\n[/{tag}]"
def _compile_f(self, schema, mode, tag, format_func, kv_sep, exclude=None) -> str:
nodes = self.to_dict(format_func=format_func, mode=mode, exclude=exclude)
text = self.compile_to(nodes, schema, kv_sep)
return self.tagging(text, schema, tag)
def compile_instruction(self, schema="markdown", mode="children", tag="", exclude=None) -> str:
"""compile to raw/json/markdown template with all/root/children nodes"""
format_func = lambda i: f"{i.expected_type} # {i.instruction}"
return self._compile_f(schema, mode, tag, format_func, kv_sep=": ", exclude=exclude)
def compile_example(self, schema="json", mode="children", tag="", exclude=None) -> str:
"""compile to raw/json/markdown examples with all/root/children nodes"""
# 这里不能使用f-string，因为转译为str后再json.dumps会额外加上引号，无法作为有效的example
# 错误示例："File list": "['main.py', 'const.py', 'game.py']", 注意这里值不是list，而是str
format_func = lambda i: i.example
return self._compile_f(schema, mode, tag, format_func, kv_sep="\n", exclude=exclude)
def compile(self, context, schema="json", mode="children", template=SIMPLE_TEMPLATE, exclude=[]) -> str:
"""
mode: all/root/children
mode="children": 编译所有子节点为一个统一模板，包括instruction与example
mode="all": NotImplemented
mode="root": NotImplemented
schmea: raw/json/markdown
schema="raw": 不编译，context, lang_constaint, instruction
schema="json"：编译context, example(json), instruction(markdown), constraint, action
schema="markdown": 编译context, example(markdown), instruction(markdown), constraint, action
"""
if schema == "raw":
return context + "\n\n## Actions\n" + LANGUAGE_CONSTRAINT + "\n" + self.instruction
# FIXME: json instruction会带来格式问题，如："Project name": "web_2048 # 项目名称使用下划线",
# compile example暂时不支持markdown
instruction = self.compile_instruction(schema="markdown", mode=mode, exclude=exclude)
example = self.compile_example(schema=schema, tag=TAG, mode=mode, exclude=exclude)
# nodes = ", ".join(self.to_dict(mode=mode).keys())
constraints = [LANGUAGE_CONSTRAINT, FORMAT_CONSTRAINT]
constraint = "\n".join(constraints)
prompt = template.format(
context=context,
example=example,
instruction=instruction,
constraint=constraint,
)
return prompt
@retry(
wait=wait_random_exponential(min=1, max=20),
stop=stop_after_attempt(6),
after=general_after_log(logger),
)
async def _aask_v1(
self,
prompt: str,
output_class_name: str,
output_data_mapping: dict,
system_msgs: Optional[list[str]] = None,
schema="markdown", # compatible to original format
timeout=CONFIG.timeout,
) -> (str, BaseModel):
"""Use ActionOutput to wrap the output of aask"""
content = await self.llm.aask(prompt, system_msgs, timeout=timeout)
logger.debug(f"llm raw output:\n{content}")
output_class = self.create_model_class(output_class_name, output_data_mapping)
if schema == "json":
parsed_data = llm_output_postprocess(
output=content, schema=output_class.model_json_schema(), req_key=f"[/{TAG}]"
)
else: # using markdown parser
parsed_data = OutputParser.parse_data_with_mapping(content, output_data_mapping)
logger.debug(f"parsed_data:\n{parsed_data}")
instruct_content = output_class(**parsed_data)
return content, instruct_content
def get(self, key):
return self.instruct_content.model_dump()[key]
def set_recursive(self, name, value):
setattr(self, name, value)
for _, i in self.children.items():
i.set_recursive(name, value)
def set_llm(self, llm):
self.set_recursive("llm", llm)
def set_context(self, context):
self.set_recursive("context", context)
async def simple_fill(self, schema, mode, timeout=CONFIG.timeout, exclude=None):
prompt = self.compile(context=self.context, schema=schema, mode=mode, exclude=exclude)
if schema != "raw":
mapping = self.get_mapping(mode, exclude=exclude)
class_name = f"{self.key}_AN"
content, scontent = await self._aask_v1(prompt, class_name, mapping, schema=schema, timeout=timeout)
self.content = content
self.instruct_content = scontent
else:
self.content = await self.llm.aask(prompt)
self.instruct_content = None
return self
async def fill(self, context, llm, schema="json", mode="auto", strgy="simple", timeout=CONFIG.timeout, exclude=[]):
"""Fill the node(s) with mode.
:param context: Everything we should know when filling node.
:param llm: Large Language Model with pre-defined system message.
:param schema: json/markdown, determine example and output format.
- raw: free form text
- json: it's easy to open source LLM with json format
- markdown: when generating code, markdown is always better
:param mode: auto/children/root
- auto: automated fill children's nodes and gather outputs, if no children, fill itself
- children: fill children's nodes and gather outputs
- root: fill root's node and gather output
:param strgy: simple/complex
- simple: run only once
- complex: run each node
:param timeout: Timeout for llm invocation.
:param exclude: The keys of ActionNode to exclude.
:return: self
"""
self.set_llm(llm)
self.set_context(context)
if self.schema:
schema = self.schema
if strgy == "simple":
return await self.simple_fill(schema=schema, mode=mode, timeout=timeout, exclude=exclude)
elif strgy == "complex":
# 这里隐式假设了拥有children
tmp = {}
for _, i in self.children.items():
if exclude and i.key in exclude:
continue
child = await i.simple_fill(schema=schema, mode=mode, timeout=timeout, exclude=exclude)
tmp.update(child.instruct_content.dict())
cls = self.create_children_class()
self.instruct_content = cls(**tmp)
return self

27
metagpt/actions/action_output.py
View file

@ -6,9 +6,7 @@
@File : action_output
"""
from typing import Dict, Type
from pydantic import BaseModel, create_model, root_validator, validator
from pydantic import BaseModel
class ActionOutput:
@ -18,26 +16,3 @@ class ActionOutput:
def __init__(self, content: str, instruct_content: BaseModel):
self.content = content
self.instruct_content = instruct_content
@classmethod
def create_model_class(cls, class_name: str, mapping: Dict[str, Type]):
new_class = create_model(class_name, **mapping)
@validator('*', allow_reuse=True)
def check_name(v, field):
if field.name not in mapping.keys():
raise ValueError(f'Unrecognized block: {field.name}')
return v
@root_validator(pre=True, allow_reuse=True)
def check_missing_fields(values):
required_fields = set(mapping.keys())
missing_fields = required_fields - set(values.keys())
if missing_fields:
raise ValueError(f'Missing fields: {missing_fields}')
return values
new_class.__validator_check_name = classmethod(check_name)
new_class.__root_validator_check_missing_fields = classmethod(check_missing_fields)
return new_class

6
metagpt/actions/add_requirement.py
View file

@ -8,7 +8,5 @@
from metagpt.actions import Action
class BossRequirement(Action):
"""Boss Requirement without any implementation details"""
async def run(self, *args, **kwargs):
raise NotImplementedError
class UserRequirement(Action):
"""User Requirement without any implementation details"""

37
metagpt/actions/analyze_dep_libs.py
View file

@ -1,37 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/5/19 12:01
@Author : alexanderwu
@File : analyze_dep_libs.py
"""
from metagpt.actions import Action
PROMPT = """You are an AI developer, trying to write a program that generates code for users based on their intentions.
For the user's prompt:
---
The API is: {prompt}
---
We decide the generated files are: {filepaths_string}
Now that we have a file list, we need to understand the shared dependencies they have.
Please list and briefly describe the shared contents between the files we are generating, including exported variables,
data patterns, id names of all DOM elements that javascript functions will use, message names and function names.
Focus only on the names of shared dependencies, do not add any other explanations.
"""
class AnalyzeDepLibs(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
self.desc = "Analyze the runtime dependencies of the program based on the context"
async def run(self, requirement, filepaths_string):
# prompt = f"Below is the product requirement document (PRD):\n\n{prd}\n\n{PROMPT}"
prompt = PROMPT.format(prompt=requirement, filepaths_string=filepaths_string)
design_filenames = await self._aask(prompt)
return design_filenames

53
metagpt/actions/azure_tts.py
View file

@ -1,53 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/6/9 22:22
@Author : Leo Xiao
@File : azure_tts.py
"""
from azure.cognitiveservices.speech import AudioConfig, SpeechConfig, SpeechSynthesizer
from metagpt.actions.action import Action
from metagpt.config import Config
class AzureTTS(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
self.config = Config()
# Parameters reference: https://learn.microsoft.com/zh-cn/azure/cognitive-services/speech-service/language-support?tabs=tts#voice-styles-and-roles
def synthesize_speech(self, lang, voice, role, text, output_file):
subscription_key = self.config.get('AZURE_TTS_SUBSCRIPTION_KEY')
region = self.config.get('AZURE_TTS_REGION')
speech_config = SpeechConfig(
subscription=subscription_key, region=region)
speech_config.speech_synthesis_voice_name = voice
audio_config = AudioConfig(filename=output_file)
synthesizer = SpeechSynthesizer(
speech_config=speech_config,
audio_config=audio_config)
# if voice=="zh-CN-YunxiNeural":
ssml_string = f"""
<speak version='1.0' xmlns='http://www.w3.org/2001/10/synthesis' xml:lang='{lang}' xmlns:mstts='http://www.w3.org/2001/mstts'>
<voice name='{voice}'>
<mstts:express-as style='affectionate' role='{role}'>
{text}
</mstts:express-as>
</voice>
</speak>
"""
synthesizer.speak_ssml_async(ssml_string).get()
if __name__ == "__main__":
azure_tts = AzureTTS("azure_tts")
azure_tts.synthesize_speech(
"zh-CN",
"zh-CN-YunxiNeural",
"Boy",
"Hello, I am Kaka",
"output.wav")

65
metagpt/actions/clone_function.py
View file

@ -1,65 +0,0 @@
from pathlib import Path
import traceback
from metagpt.actions.write_code import WriteCode
from metagpt.logs import logger
from metagpt.schema import Message
from metagpt.utils.highlight import highlight
CLONE_PROMPT = """
*context*
Please convert the function code ```{source_code}``` into the the function format: ```{template_func}```.
*Please Write code based on the following list and context*
1. Write code start with ```, and end with ```.
2. Please implement it in one function if possible, except for import statements. for exmaple:
```python
import pandas as pd
def run(*args) -> pd.DataFrame:
...
```
3. Do not use public member functions that do not exist in your design.
4. The output function name, input parameters and return value must be the same as ```{template_func}```.
5. Make sure the results before and after the code conversion are required to be exactly the same.
6. Don't repeat my context in your replies.
7. Return full results, for example, if the return value has df.head(), please return df.
8. If you must use a third-party package, use the most popular ones, for example: pandas, numpy, ta, ...
"""
class CloneFunction(WriteCode):
def __init__(self, name="CloneFunction", context: list[Message] = None, llm=None):
super().__init__(name, context, llm)
def _save(self, code_path, code):
if isinstance(code_path, str):
code_path = Path(code_path)
code_path.parent.mkdir(parents=True, exist_ok=True)
code_path.write_text(code)
logger.info(f"Saving Code to {code_path}")
async def run(self, template_func: str, source_code: str) -> str:
"""将source_code转换成template_func一样的入参和返回类型"""
prompt = CLONE_PROMPT.format(source_code=source_code, template_func=template_func)
logger.info(f"query for CloneFunction: \n {prompt}")
code = await self.write_code(prompt)
logger.info(f'CloneFunction code is \n {highlight(code)}')
return code
def run_function_code(func_code: str, func_name: str, *args, **kwargs):
"""Run function code from string code."""
try:
locals_ = {}
exec(func_code, locals_)
func = locals_[func_name]
return func(*args, **kwargs), ""
except Exception:
return "", traceback.format_exc()
def run_function_script(code_script_path: str, func_name: str, *args, **kwargs):
"""Run function code from script."""
if isinstance(code_script_path, str):
code_path = Path(code_script_path)
code = code_path.read_text(encoding='utf-8')
return run_function_code(code, func_name, *args, **kwargs)

74
metagpt/actions/debug_error.py
View file

@ -4,12 +4,21 @@
@Time : 2023/5/11 17:46
@Author : alexanderwu
@File : debug_error.py
@Modified By: mashenquan, 2023/11/27.
1. Divide the context into three components: legacy code, unit test code, and console log.
2. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
"""
import re
from metagpt.logs import logger
from pydantic import Field
from metagpt.actions.action import Action
from metagpt.config import CONFIG
from metagpt.const import TEST_CODES_FILE_REPO, TEST_OUTPUTS_FILE_REPO
from metagpt.logs import logger
from metagpt.schema import RunCodeContext, RunCodeResult
from metagpt.utils.common import CodeParser
from metagpt.utils.file_repository import FileRepository
PROMPT_TEMPLATE = """
NOTICE
@ -19,33 +28,56 @@ Based on the message, first, figure out your own role, i.e. Engineer or QaEngine
then rewrite the development code or the test code based on your role, the error, and the summary, such that all bugs are fixed and the code performs well.
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the test case or script and triple quotes.
The message is as follows:
{context}
# Legacy Code
```python
{code}
```
---
# Unit Test Code
```python
{test_code}
```
---
# Console logs
```text
{logs}
```
---
Now you should start rewriting the code:
## file name of the code to rewrite: Write code with triple quoto. Do your best to implement THIS IN ONLY ONE FILE.
## file name of the code to rewrite: Write code with triple quote. Do your best to implement THIS IN ONLY ONE FILE.
"""
class DebugError(Action):
def __init__(self, name="DebugError", context=None, llm=None):
super().__init__(name, context, llm)
name: str = "DebugError"
context: RunCodeContext = Field(default_factory=RunCodeContext)
# async def run(self, code, error):
# prompt = f"Here is a piece of Python code:\n\n{code}\n\nThe following error occurred during execution:" \
# f"\n\n{error}\n\nPlease try to fix the error in this code."
# fixed_code = await self._aask(prompt)
# return fixed_code
async def run(self, context):
if "PASS" in context:
return "", "the original code works fine, no need to debug"
file_name = re.search("## File To Rewrite:\s*(.+\\.py)", context).group(1)
async def run(self, *args, **kwargs) -> str:
output_doc = await FileRepository.get_file(
filename=self.context.output_filename, relative_path=TEST_OUTPUTS_FILE_REPO
)
if not output_doc:
return ""
output_detail = RunCodeResult.loads(output_doc.content)
pattern = r"Ran (\d+) tests in ([\d.]+)s\n\nOK"
matches = re.search(pattern, output_detail.stderr)
if matches:
return ""
logger.info(f"Debug and rewrite {file_name}")
logger.info(f"Debug and rewrite {self.context.test_filename}")
code_doc = await FileRepository.get_file(
filename=self.context.code_filename, relative_path=CONFIG.src_workspace
)
if not code_doc:
return ""
test_doc = await FileRepository.get_file(
filename=self.context.test_filename, relative_path=TEST_CODES_FILE_REPO
)
if not test_doc:
return ""
prompt = PROMPT_TEMPLATE.format(code=code_doc.content, test_code=test_doc.content, logs=output_detail.stderr)
prompt = PROMPT_TEMPLATE.format(context=context)
rsp = await self._aask(prompt)
code = CodeParser.parse_code(block="", text=rsp)
return file_name, code
return code

303
metagpt/actions/design_api.py
View file

@ -4,214 +4,133 @@
@Time : 2023/5/11 19:26
@Author : alexanderwu
@File : design_api.py
@Modified By: mashenquan, 2023/11/27.
1. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
2. According to the design in Section 2.2.3.5.3 of RFC 135, add incremental iteration functionality.
@Modified By: mashenquan, 2023/12/5. Move the generation logic of the project name to WritePRD.
"""
import shutil
import json
from pathlib import Path
from typing import List
from typing import Optional
from metagpt.actions import Action, ActionOutput
from metagpt.actions.design_api_an import DESIGN_API_NODE
from metagpt.config import CONFIG
from metagpt.const import WORKSPACE_ROOT
from metagpt.const import (
DATA_API_DESIGN_FILE_REPO,
PRDS_FILE_REPO,
SEQ_FLOW_FILE_REPO,
SYSTEM_DESIGN_FILE_REPO,
SYSTEM_DESIGN_PDF_FILE_REPO,
)
from metagpt.logs import logger
from metagpt.utils.common import CodeParser
from metagpt.utils.get_template import get_template
from metagpt.utils.json_to_markdown import json_to_markdown
from metagpt.schema import Document, Documents, Message
from metagpt.utils.file_repository import FileRepository
from metagpt.utils.mermaid import mermaid_to_file
templates = {
"json": {
"PROMPT_TEMPLATE": """
# Context
NEW_REQ_TEMPLATE = """
### Legacy Content
{old_design}
### New Requirements
{context}
## Format example
{format_example}
-----
Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools
Requirement: Fill in the following missing information based on the context, each section name is a key in json
Max Output: 8192 chars or 2048 tokens. Try to use them up.
## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
## Python package name: Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
## File list: Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
## Data structures and interface definitions: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
## Anything UNCLEAR: Provide as Plain text. Make clear here.
output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like format example,
and only output the json inside this tag, nothing else
""",
"FORMAT_EXAMPLE": """
[CONTENT]
{
"Implementation approach": "We will ...",
"Python package name": "snake_game",
"File list": ["main.py"],
"Data structures and interface definitions": '
classDiagram
class Game{
+int score
}
...
Game "1" -- "1" Food: has
',
"Program call flow": '
sequenceDiagram
participant M as Main
...
G->>M: end game
',
"Anything UNCLEAR": "The requirement is clear to me."
}
[/CONTENT]
""",
},
"markdown": {
"PROMPT_TEMPLATE": """
# Context
{context}
## Format example
{format_example}
-----
Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools
Requirement: Fill in the following missing information based on the context, note that all sections are response with code form separately
Max Output: 8192 chars or 2048 tokens. Try to use them up.
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
## Python package name: Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
## File list: Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
## Data structures and interface definitions: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
## Anything UNCLEAR: Provide as Plain text. Make clear here.
""",
"FORMAT_EXAMPLE": """
---
## Implementation approach
We will ...
## Python package name
```python
"snake_game"
```
## File list
```python
[
"main.py",
]
```
## Data structures and interface definitions
```mermaid
classDiagram
class Game{
+int score
}
...
Game "1" -- "1" Food: has
```
## Program call flow
```mermaid
sequenceDiagram
participant M as Main
...
G->>M: end game
```
## Anything UNCLEAR
The requirement is clear to me.
---
""",
},
}
OUTPUT_MAPPING = {
"Implementation approach": (str, ...),
"Python package name": (str, ...),
"File list": (List[str], ...),
"Data structures and interface definitions": (str, ...),
"Program call flow": (str, ...),
"Anything UNCLEAR": (str, ...),
}
"""
class WriteDesign(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
self.desc = (
"Based on the PRD, think about the system design, and design the corresponding APIs, "
"data structures, library tables, processes, and paths. Please provide your design, feedback "
"clearly and in detail."
)
name: str = ""
context: Optional[str] = None
desc: str = (
"Based on the PRD, think about the system design, and design the corresponding APIs, "
"data structures, library tables, processes, and paths. Please provide your design, feedback "
"clearly and in detail."
)
def recreate_workspace(self, workspace: Path):
try:
shutil.rmtree(workspace)
except FileNotFoundError:
pass # Folder does not exist, but we don't care
workspace.mkdir(parents=True, exist_ok=True)
async def run(self, with_messages: Message, schema: str = CONFIG.prompt_schema):
# Use `git status` to identify which PRD documents have been modified in the `docs/prds` directory.
prds_file_repo = CONFIG.git_repo.new_file_repository(PRDS_FILE_REPO)
changed_prds = prds_file_repo.changed_files
# Use `git status` to identify which design documents in the `docs/system_designs` directory have undergone
# changes.
system_design_file_repo = CONFIG.git_repo.new_file_repository(SYSTEM_DESIGN_FILE_REPO)
changed_system_designs = system_design_file_repo.changed_files
async def _save_prd(self, docs_path, resources_path, context):
prd_file = docs_path / "prd.md"
if context[-1].instruct_content and context[-1].instruct_content.dict()["Competitive Quadrant Chart"]:
quadrant_chart = context[-1].instruct_content.dict()["Competitive Quadrant Chart"]
await mermaid_to_file(quadrant_chart, resources_path / "competitive_analysis")
# For those PRDs and design documents that have undergone changes, regenerate the design content.
changed_files = Documents()
for filename in changed_prds.keys():
doc = await self._update_system_design(
filename=filename, prds_file_repo=prds_file_repo, system_design_file_repo=system_design_file_repo
)
changed_files.docs[filename] = doc
if context[-1].instruct_content:
logger.info(f"Saving PRD to {prd_file}")
prd_file.write_text(json_to_markdown(context[-1].instruct_content.dict()))
for filename in changed_system_designs.keys():
if filename in changed_files.docs:
continue
doc = await self._update_system_design(
filename=filename, prds_file_repo=prds_file_repo, system_design_file_repo=system_design_file_repo
)
changed_files.docs[filename] = doc
if not changed_files.docs:
logger.info("Nothing has changed.")
# Wait until all files under `docs/system_designs/` are processed before sending the publish message,
# leaving room for global optimization in subsequent steps.
return ActionOutput(content=changed_files.model_dump_json(), instruct_content=changed_files)
async def _save_system_design(self, docs_path, resources_path, system_design):
data_api_design = system_design.instruct_content.dict()[
"Data structures and interface definitions"
] # CodeParser.parse_code(block="Data structures and interface definitions", text=content)
seq_flow = system_design.instruct_content.dict()[
"Program call flow"
] # CodeParser.parse_code(block="Program call flow", text=content)
await mermaid_to_file(data_api_design, resources_path / "data_api_design")
await mermaid_to_file(seq_flow, resources_path / "seq_flow")
system_design_file = docs_path / "system_design.md"
logger.info(f"Saving System Designs to {system_design_file}")
system_design_file.write_text((json_to_markdown(system_design.instruct_content.dict())))
async def _new_system_design(self, context, schema=CONFIG.prompt_schema):
node = await DESIGN_API_NODE.fill(context=context, llm=self.llm, schema=schema)
return node
async def _save(self, context, system_design):
if isinstance(system_design, ActionOutput):
ws_name = system_design.instruct_content.dict()["Python package name"]
async def _merge(self, prd_doc, system_design_doc, schema=CONFIG.prompt_schema):
context = NEW_REQ_TEMPLATE.format(old_design=system_design_doc.content, context=prd_doc.content)
node = await DESIGN_API_NODE.fill(context=context, llm=self.llm, schema=schema)
system_design_doc.content = node.instruct_content.model_dump_json()
return system_design_doc
async def _update_system_design(self, filename, prds_file_repo, system_design_file_repo) -> Document:
prd = await prds_file_repo.get(filename)
old_system_design_doc = await system_design_file_repo.get(filename)
if not old_system_design_doc:
system_design = await self._new_system_design(context=prd.content)
doc = Document(
root_path=SYSTEM_DESIGN_FILE_REPO,
filename=filename,
content=system_design.instruct_content.model_dump_json(),
)
else:
ws_name = CodeParser.parse_str(block="Python package name", text=system_design)
workspace = WORKSPACE_ROOT / ws_name
self.recreate_workspace(workspace)
docs_path = workspace / "docs"
resources_path = workspace / "resources"
docs_path.mkdir(parents=True, exist_ok=True)
resources_path.mkdir(parents=True, exist_ok=True)
await self._save_prd(docs_path, resources_path, context)
await self._save_system_design(docs_path, resources_path, system_design)
async def run(self, context, format=CONFIG.prompt_format):
prompt_template, format_example = get_template(templates, format)
prompt = prompt_template.format(context=context, format_example=format_example)
# system_design = await self._aask(prompt)
system_design = await self._aask_v1(prompt, "system_design", OUTPUT_MAPPING, format=format)
# fix Python package name, we can't system_design.instruct_content.python_package_name = "xxx" since "Python package name" contain space, have to use setattr
setattr(
system_design.instruct_content,
"Python package name",
system_design.instruct_content.dict()["Python package name"].strip().strip("'").strip('"'),
doc = await self._merge(prd_doc=prd, system_design_doc=old_system_design_doc)
await system_design_file_repo.save(
filename=filename, content=doc.content, dependencies={prd.root_relative_path}
)
await self._save(context, system_design)
return system_design
await self._save_data_api_design(doc)
await self._save_seq_flow(doc)
await self._save_pdf(doc)
return doc
@staticmethod
async def _save_data_api_design(design_doc):
m = json.loads(design_doc.content)
data_api_design = m.get("Data structures and interfaces")
if not data_api_design:
return
pathname = CONFIG.git_repo.workdir / DATA_API_DESIGN_FILE_REPO / Path(design_doc.filename).with_suffix("")
await WriteDesign._save_mermaid_file(data_api_design, pathname)
logger.info(f"Save class view to {str(pathname)}")
@staticmethod
async def _save_seq_flow(design_doc):
m = json.loads(design_doc.content)
seq_flow = m.get("Program call flow")
if not seq_flow:
return
pathname = CONFIG.git_repo.workdir / Path(SEQ_FLOW_FILE_REPO) / Path(design_doc.filename).with_suffix("")
await WriteDesign._save_mermaid_file(seq_flow, pathname)
logger.info(f"Saving sequence flow to {str(pathname)}")
@staticmethod
async def _save_pdf(design_doc):
await FileRepository.save_as(doc=design_doc, with_suffix=".md", relative_path=SYSTEM_DESIGN_PDF_FILE_REPO)
@staticmethod
async def _save_mermaid_file(data: str, pathname: Path):
pathname.parent.mkdir(parents=True, exist_ok=True)
await mermaid_to_file(data, pathname)

64
metagpt/actions/design_api_an.py Normal file
View file

@ -0,0 +1,64 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/12/12 22:24
@Author : alexanderwu
@File : design_api_an.py
"""
from typing import List
from metagpt.actions.action_node import ActionNode
from metagpt.utils.mermaid import MMC1, MMC2
IMPLEMENTATION_APPROACH = ActionNode(
key="Implementation approach",
expected_type=str,
instruction="Analyze the difficult points of the requirements, select the appropriate open-source framework",
example="We will ...",
)
PROJECT_NAME = ActionNode(
key="Project name", expected_type=str, instruction="The project name with underline", example="game_2048"
)
FILE_LIST = ActionNode(
key="File list",
expected_type=List[str],
instruction="Only need relative paths. ALWAYS write a main.py or app.py here",
example=["main.py", "game.py"],
)
DATA_STRUCTURES_AND_INTERFACES = ActionNode(
key="Data structures and interfaces",
expected_type=str,
instruction="Use mermaid classDiagram code syntax, including classes, method(__init__ etc.) and functions with type"
" annotations, CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. "
"The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.",
example=MMC1,
)
PROGRAM_CALL_FLOW = ActionNode(
key="Program call flow",
expected_type=str,
instruction="Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE "
"accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.",
example=MMC2,
)
ANYTHING_UNCLEAR = ActionNode(
key="Anything UNCLEAR",
expected_type=str,
instruction="Mention unclear project aspects, then try to clarify it.",
example="Clarification needed on third-party API integration, ...",
)
NODES = [
IMPLEMENTATION_APPROACH,
# PROJECT_NAME,
FILE_LIST,
DATA_STRUCTURES_AND_INTERFACES,
PROGRAM_CALL_FLOW,
ANYTHING_UNCLEAR,
]
DESIGN_API_NODE = ActionNode.from_children("DesignAPI", NODES)

16
metagpt/actions/design_api_review.py
View file

@ -5,18 +5,22 @@
@Author : alexanderwu
@File : design_api_review.py
"""
from typing import Optional
from metagpt.actions.action import Action
class DesignReview(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
name: str = "DesignReview"
context: Optional[str] = None
async def run(self, prd, api_design):
prompt = f"Here is the Product Requirement Document (PRD):\n\n{prd}\n\nHere is the list of APIs designed " \
f"based on this PRD:\n\n{api_design}\n\nPlease review whether this API design meets the requirements" \
f" of the PRD, and whether it complies with good design practices."
prompt = (
f"Here is the Product Requirement Document (PRD):\n\n{prd}\n\nHere is the list of APIs designed "
f"based on this PRD:\n\n{api_design}\n\nPlease review whether this API design meets the requirements"
f" of the PRD, and whether it complies with good design practices."
)
api_review = await self._aask(prompt)
return api_review

29
metagpt/actions/design_filenames.py
View file

@ -1,29 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/5/19 11:50
@Author : alexanderwu
@File : design_filenames.py
"""
from metagpt.actions import Action
from metagpt.logs import logger
PROMPT = """You are an AI developer, trying to write a program that generates code for users based on their intentions.
When given their intentions, provide a complete and exhaustive list of file paths needed to write the program for the user.
Only list the file paths you will write and return them as a Python string list.
Do not add any other explanations, just return a Python string list."""
class DesignFilenames(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
self.desc = "Based on the PRD, consider system design, and carry out the basic design of the corresponding " \
"APIs, data structures, and database tables. Please give your design, feedback clearly and in detail."
async def run(self, prd):
prompt = f"The following is the Product Requirement Document (PRD):\n\n{prd}\n\n{PROMPT}"
design_filenames = await self._aask(prompt)
logger.debug(prompt)
logger.debug(design_filenames)
return design_filenames

52
metagpt/actions/detail_mining.py
View file

@ -1,52 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/9/12 17:45
@Author : fisherdeng
@File : detail_mining.py
"""
from metagpt.actions import Action, ActionOutput
from metagpt.logs import logger
PROMPT_TEMPLATE = """
##TOPIC
{topic}
##RECORD
{record}
##Format example
{format_example}
-----
Task: Refer to the "##TOPIC" (discussion objectives) and "##RECORD" (discussion records) to further inquire about the details that interest you, within a word limit of 150 words.
Special Note 1: Your intention is solely to ask questions without endorsing or negating any individual's viewpoints.
Special Note 2: This output should only include the topic "##OUTPUT". Do not add, remove, or modify the topic. Begin the output with '##OUTPUT', followed by an immediate line break, and then proceed to provide the content in the specified format as outlined in the "##Format example" section.
Special Note 3: The output should be in the same language as the input.
"""
FORMAT_EXAMPLE = """
##
##OUTPUT
...(Please provide the specific details you would like to inquire about here.)
##
##
"""
OUTPUT_MAPPING = {
"OUTPUT": (str, ...),
}
class DetailMining(Action):
"""This class allows LLM to further mine noteworthy details based on specific "##TOPIC"(discussion topic) and "##RECORD" (discussion records), thereby deepening the discussion.
"""
def __init__(self, name="", context=None, llm=None):
super().__init__(name, context, llm)
async def run(self, topic, record) -> ActionOutput:
prompt = PROMPT_TEMPLATE.format(topic=topic, record=record, format_example=FORMAT_EXAMPLE)
rsp = await self._aask_v1(prompt, "detail_mining", OUTPUT_MAPPING)
return rsp

8
metagpt/actions/execute_task.py
View file

@ -5,13 +5,15 @@
@Author : femto Zheng
@File : execute_task.py
"""
from metagpt.actions import Action
from metagpt.schema import Message
class ExecuteTask(Action):
def __init__(self, name="ExecuteTask", context: list[Message] = None, llm=None):
super().__init__(name, context, llm)
name: str = "ExecuteTask"
context: list[Message] = []
def run(self, *args, **kwargs):
async def run(self, *args, **kwargs):
pass

13
metagpt/actions/fix_bug.py Normal file
View file

@ -0,0 +1,13 @@
# -*- coding: utf-8 -*-
"""
@Time : 2023-12-12
@Author : mashenquan
@File : fix_bug.py
"""
from metagpt.actions import Action
class FixBug(Action):
"""Fix bug action without any implementation details"""
name: str = "FixBug"

27
metagpt/actions/generate_questions.py Normal file
View file

@ -0,0 +1,27 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/9/12 17:45
@Author : fisherdeng
@File : generate_questions.py
"""
from metagpt.actions import Action
from metagpt.actions.action_node import ActionNode
QUESTIONS = ActionNode(
key="Questions",
expected_type=list[str],
instruction="Task: Refer to the context to further inquire about the details that interest you, within a word limit"
" of 150 words. Please provide the specific details you would like to inquire about here",
example=["1. What ...", "2. How ...", "3. ..."],
)
class GenerateQuestions(Action):
"""This class allows LLM to further mine noteworthy details based on specific "##TOPIC"(discussion topic) and
"##RECORD" (discussion records), thereby deepening the discussion."""
name: str = "GenerateQuestions"
async def run(self, context):
return await QUESTIONS.fill(context=context, llm=self.llm)

32
metagpt/actions/invoice_ocr.py
View file

@ -10,16 +10,23 @@
import os
import zipfile
from pathlib import Path
from datetime import datetime
from pathlib import Path
from typing import Optional
import pandas as pd
from paddleocr import PaddleOCR
from pydantic import Field
from metagpt.actions import Action
from metagpt.const import INVOICE_OCR_TABLE_PATH
from metagpt.llm import LLM
from metagpt.logs import logger
from metagpt.prompts.invoice_ocr import EXTRACT_OCR_MAIN_INFO_PROMPT, REPLY_OCR_QUESTION_PROMPT
from metagpt.prompts.invoice_ocr import (
EXTRACT_OCR_MAIN_INFO_PROMPT,
REPLY_OCR_QUESTION_PROMPT,
)
from metagpt.provider.base_llm import BaseLLM
from metagpt.utils.common import OutputParser
from metagpt.utils.file import File
@ -33,8 +40,8 @@ class InvoiceOCR(Action):
"""
def __init__(self, name: str = "", *args, **kwargs):
super().__init__(name, *args, **kwargs)
name: str = "InvoiceOCR"
context: Optional[str] = None
@staticmethod
async def _check_file_type(file_path: Path) -> str:
@ -81,6 +88,8 @@ class InvoiceOCR(Action):
async def _ocr(invoice_file_path: Path):
ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=1)
ocr_result = ocr.ocr(str(invoice_file_path), cls=True)
for result in ocr_result[0]:
result[1] = (result[1][0], round(result[1][1], 2)) # round long confidence scores to reduce token costs
return ocr_result
async def run(self, file_path: Path, *args, **kwargs) -> list:
@ -122,9 +131,10 @@ class GenerateTable(Action):
"""
def __init__(self, name: str = "", language: str = "ch", *args, **kwargs):
super().__init__(name, *args, **kwargs)
self.language = language
name: str = "GenerateTable"
context: Optional[str] = None
llm: BaseLLM = Field(default_factory=LLM)
language: str = "ch"
async def run(self, ocr_results: list, filename: str, *args, **kwargs) -> dict[str, str]:
"""Processes OCR results, extracts invoice information, generates a table, and saves it as an Excel file.
@ -166,9 +176,10 @@ class ReplyQuestion(Action):
"""
def __init__(self, name: str = "", language: str = "ch", *args, **kwargs):
super().__init__(name, *args, **kwargs)
self.language = language
name: str = "ReplyQuestion"
context: Optional[str] = None
llm: BaseLLM = Field(default_factory=LLM)
language: str = "ch"
async def run(self, query: str, ocr_result: list, *args, **kwargs) -> str:
"""Reply to questions based on ocr results.
@ -183,4 +194,3 @@ class ReplyQuestion(Action):
prompt = REPLY_OCR_QUESTION_PROMPT.format(query=query, ocr_result=ocr_result, language=self.language)
resp = await self._aask(prompt=prompt)
return resp

50
metagpt/actions/prepare_documents.py Normal file
View file

@ -0,0 +1,50 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/11/20
@Author : mashenquan
@File : prepare_documents.py
@Desc: PrepareDocuments Action: initialize project folder and add new requirements to docs/requirements.txt.
RFC 135 2.2.3.5.1.
"""
import shutil
from pathlib import Path
from typing import Optional
from metagpt.actions import Action, ActionOutput
from metagpt.config import CONFIG
from metagpt.const import DOCS_FILE_REPO, REQUIREMENT_FILENAME
from metagpt.schema import Document
from metagpt.utils.file_repository import FileRepository
from metagpt.utils.git_repository import GitRepository
class PrepareDocuments(Action):
"""PrepareDocuments Action: initialize project folder and add new requirements to docs/requirements.txt."""
name: str = "PrepareDocuments"
context: Optional[str] = None
def _init_repo(self):
"""Initialize the Git environment."""
if not CONFIG.project_path:
name = CONFIG.project_name or FileRepository.new_filename()
path = Path(CONFIG.workspace_path) / name
else:
path = Path(CONFIG.project_path)
if path.exists() and not CONFIG.inc:
shutil.rmtree(path)
CONFIG.project_path = path
CONFIG.git_repo = GitRepository(local_path=path, auto_init=True)
async def run(self, with_messages, **kwargs):
"""Create and initialize the workspace folder, initialize the Git environment."""
self._init_repo()
# Write the newly added requirements from the main parameter idea to `docs/requirement.txt`.
doc = Document(root_path=DOCS_FILE_REPO, filename=REQUIREMENT_FILENAME, content=with_messages[0].content)
await FileRepository.save_file(filename=REQUIREMENT_FILENAME, content=doc.content, relative_path=DOCS_FILE_REPO)
# Send a Message notification to the WritePRD action, instructing it to process requirements using
# `docs/requirement.txt` and `docs/prds/`.
return ActionOutput(content=doc.content, instruct_content=doc)

36
metagpt/actions/prepare_interview.py
View file

@ -6,36 +6,20 @@
@File : prepare_interview.py
"""
from metagpt.actions import Action
from metagpt.actions.action_node import ActionNode
PROMPT_TEMPLATE = """
# Context
{context}
## Format example
---
Q1: question 1 here
References:
- point 1
- point 2
Q2: question 2 here...
---
-----
Role: You are an interviewer of our company who is well-knonwn in frontend or backend develop;
QUESTIONS = ActionNode(
key="Questions",
expected_type=list[str],
instruction="""Role: You are an interviewer of our company who is well-knonwn in frontend or backend develop;
Requirement: Provide a list of questions for the interviewer to ask the interviewee, by reading the resume of the interviewee in the context.
Attention: Provide as markdown block as the format above, at least 10 questions.
"""
# prepare for a interview
Attention: Provide as markdown block as the format above, at least 10 questions.""",
example=["1. What ...", "2. How ..."],
)
class PrepareInterview(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
name: str = "PrepareInterview"
async def run(self, context):
prompt = PROMPT_TEMPLATE.format(context=context)
question_list = await self._aask_v1(prompt)
return question_list
return await QUESTIONS.fill(context=context, llm=self.llm)

265
metagpt/actions/project_management.py
View file

@ -4,189 +4,114 @@
@Time : 2023/5/11 19:12
@Author : alexanderwu
@File : project_management.py
@Modified By: mashenquan, 2023/11/27.
1. Divide the context into three components: legacy code, unit test code, and console log.
2. Move the document storage operations related to WritePRD from the save operation of WriteDesign.
3. According to the design in Section 2.2.3.5.4 of RFC 135, add incremental iteration functionality.
"""
from typing import List
import json
from typing import Optional
from metagpt.actions import ActionOutput
from metagpt.actions.action import Action
from metagpt.actions.project_management_an import PM_NODE
from metagpt.config import CONFIG
from metagpt.const import WORKSPACE_ROOT
from metagpt.utils.common import CodeParser
from metagpt.utils.get_template import get_template
from metagpt.utils.json_to_markdown import json_to_markdown
from metagpt.const import (
PACKAGE_REQUIREMENTS_FILENAME,
SYSTEM_DESIGN_FILE_REPO,
TASK_FILE_REPO,
TASK_PDF_FILE_REPO,
)
from metagpt.logs import logger
from metagpt.schema import Document, Documents
from metagpt.utils.file_repository import FileRepository
templates = {
"json": {
"PROMPT_TEMPLATE": """
# Context
NEW_REQ_TEMPLATE = """
### Legacy Content
{old_tasks}
### New Requirements
{context}
## Format example
{format_example}
-----
Role: You are a project manager; the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
Requirements: Based on the context, fill in the following missing information, each section name is a key in json. Here the granularity of the task is a file, if there are any missing files, you can supplement them
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
## Required Python third-party packages: Provided in requirements.txt format
## Required Other language third-party packages: Provided in requirements.txt format
## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first.
## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like format example,
and only output the json inside this tag, nothing else
""",
"FORMAT_EXAMPLE": '''
{
"Required Python third-party packages": [
"flask==1.1.2",
"bcrypt==3.2.0"
],
"Required Other language third-party packages": [
"No third-party ..."
],
"Full API spec": """
openapi: 3.0.0
...
description: A JSON object ...
""",
"Logic Analysis": [
["game.py","Contains..."]
],
"Task list": [
"game.py"
],
"Shared Knowledge": """
'game.py' contains ...
""",
"Anything UNCLEAR": "We need ... how to start."
}
''',
},
"markdown": {
"PROMPT_TEMPLATE": """
# Context
{context}
## Format example
{format_example}
-----
Role: You are a project manager; the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
Requirements: Based on the context, fill in the following missing information, note that all sections are returned in Python code triple quote form seperatedly. Here the granularity of the task is a file, if there are any missing files, you can supplement them
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
## Required Python third-party packages: Provided in requirements.txt format
## Required Other language third-party packages: Provided in requirements.txt format
## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first.
## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
""",
"FORMAT_EXAMPLE": '''
---
## Required Python third-party packages
```python
"""
flask==1.1.2
bcrypt==3.2.0
"""
```
## Required Other language third-party packages
```python
"""
No third-party ...
"""
```
## Full API spec
```python
"""
openapi: 3.0.0
...
description: A JSON object ...
"""
```
## Logic Analysis
```python
[
["game.py", "Contains ..."],
]
```
## Task list
```python
[
"game.py",
]
```
## Shared Knowledge
```python
"""
'game.py' contains ...
"""
```
## Anything UNCLEAR
We need ... how to start.
---
''',
},
}
OUTPUT_MAPPING = {
"Required Python third-party packages": (List[str], ...),
"Required Other language third-party packages": (List[str], ...),
"Full API spec": (str, ...),
"Logic Analysis": (List[List[str]], ...),
"Task list": (List[str], ...),
"Shared Knowledge": (str, ...),
"Anything UNCLEAR": (str, ...),
}
class WriteTasks(Action):
def __init__(self, name="CreateTasks", context=None, llm=None):
super().__init__(name, context, llm)
name: str = "CreateTasks"
context: Optional[str] = None
def _save(self, context, rsp):
if context[-1].instruct_content:
ws_name = context[-1].instruct_content.dict()["Python package name"]
async def run(self, with_messages, schema=CONFIG.prompt_schema):
system_design_file_repo = CONFIG.git_repo.new_file_repository(SYSTEM_DESIGN_FILE_REPO)
changed_system_designs = system_design_file_repo.changed_files
tasks_file_repo = CONFIG.git_repo.new_file_repository(TASK_FILE_REPO)
changed_tasks = tasks_file_repo.changed_files
change_files = Documents()
# Rewrite the system designs that have undergone changes based on the git head diff under
# `docs/system_designs/`.
for filename in changed_system_designs:
task_doc = await self._update_tasks(
filename=filename, system_design_file_repo=system_design_file_repo, tasks_file_repo=tasks_file_repo
)
change_files.docs[filename] = task_doc
# Rewrite the task files that have undergone changes based on the git head diff under `docs/tasks/`.
for filename in changed_tasks:
if filename in change_files.docs:
continue
task_doc = await self._update_tasks(
filename=filename, system_design_file_repo=system_design_file_repo, tasks_file_repo=tasks_file_repo
)
change_files.docs[filename] = task_doc
if not change_files.docs:
logger.info("Nothing has changed.")
# Wait until all files under `docs/tasks/` are processed before sending the publish_message, leaving room for
# global optimization in subsequent steps.
return ActionOutput(content=change_files.model_dump_json(), instruct_content=change_files)
async def _update_tasks(self, filename, system_design_file_repo, tasks_file_repo):
system_design_doc = await system_design_file_repo.get(filename)
task_doc = await tasks_file_repo.get(filename)
if task_doc:
task_doc = await self._merge(system_design_doc=system_design_doc, task_doc=task_doc)
else:
ws_name = CodeParser.parse_str(block="Python package name", text=context[-1].content)
file_path = WORKSPACE_ROOT / ws_name / "docs/api_spec_and_tasks.md"
file_path.write_text(json_to_markdown(rsp.instruct_content.dict()))
rsp = await self._run_new_tasks(context=system_design_doc.content)
task_doc = Document(
root_path=TASK_FILE_REPO, filename=filename, content=rsp.instruct_content.model_dump_json()
)
await tasks_file_repo.save(
filename=filename, content=task_doc.content, dependencies={system_design_doc.root_relative_path}
)
await self._update_requirements(task_doc)
await self._save_pdf(task_doc=task_doc)
return task_doc
# Write requirements.txt
requirements_path = WORKSPACE_ROOT / ws_name / "requirements.txt"
requirements_path.write_text("\n".join(rsp.instruct_content.dict().get("Required Python third-party packages")))
async def _run_new_tasks(self, context, schema=CONFIG.prompt_schema):
node = await PM_NODE.fill(context, self.llm, schema)
return node
async def run(self, context, format=CONFIG.prompt_format):
prompt_template, format_example = get_template(templates, format)
prompt = prompt_template.format(context=context, format_example=format_example)
rsp = await self._aask_v1(prompt, "task", OUTPUT_MAPPING, format=format)
self._save(context, rsp)
return rsp
async def _merge(self, system_design_doc, task_doc, schema=CONFIG.prompt_schema) -> Document:
context = NEW_REQ_TEMPLATE.format(context=system_design_doc.content, old_tasks=task_doc.content)
node = await PM_NODE.fill(context, self.llm, schema)
task_doc.content = node.instruct_content.model_dump_json()
return task_doc
@staticmethod
async def _update_requirements(doc):
m = json.loads(doc.content)
packages = set(m.get("Required Python third-party packages", set()))
file_repo = CONFIG.git_repo.new_file_repository()
requirement_doc = await file_repo.get(filename=PACKAGE_REQUIREMENTS_FILENAME)
if not requirement_doc:
requirement_doc = Document(filename=PACKAGE_REQUIREMENTS_FILENAME, root_path=".", content="")
lines = requirement_doc.content.splitlines()
for pkg in lines:
if pkg == "":
continue
packages.add(pkg)
await file_repo.save(PACKAGE_REQUIREMENTS_FILENAME, content="\n".join(packages))
class AssignTasks(Action):
async def run(self, *args, **kwargs):
# Here you should implement the actual action
pass
@staticmethod
async def _save_pdf(task_doc):
await FileRepository.save_as(doc=task_doc, with_suffix=".md", relative_path=TASK_PDF_FILE_REPO)

87
metagpt/actions/project_management_an.py Normal file
View file

@ -0,0 +1,87 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/12/14 15:28
@Author : alexanderwu
@File : project_management_an.py
"""
from typing import List
from metagpt.actions.action_node import ActionNode
from metagpt.logs import logger
REQUIRED_PYTHON_PACKAGES = ActionNode(
key="Required Python packages",
expected_type=List[str],
instruction="Provide required Python packages in requirements.txt format.",
example=["flask==1.1.2", "bcrypt==3.2.0"],
)
REQUIRED_OTHER_LANGUAGE_PACKAGES = ActionNode(
key="Required Other language third-party packages",
expected_type=List[str],
instruction="List down the required packages for languages other than Python.",
example=["No third-party dependencies required"],
)
LOGIC_ANALYSIS = ActionNode(
key="Logic Analysis",
expected_type=List[List[str]],
instruction="Provide a list of files with the classes/methods/functions to be implemented, "
"including dependency analysis and imports.",
example=[
["game.py", "Contains Game class and ... functions"],
["main.py", "Contains main function, from game import Game"],
],
)
TASK_LIST = ActionNode(
key="Task list",
expected_type=List[str],
instruction="Break down the tasks into a list of filenames, prioritized by dependency order.",
example=["game.py", "main.py"],
)
FULL_API_SPEC = ActionNode(
key="Full API spec",
expected_type=str,
instruction="Describe all APIs using OpenAPI 3.0 spec that may be used by both frontend and backend. If front-end "
"and back-end communication is not required, leave it blank.",
example="openapi: 3.0.0 ...",
)
SHARED_KNOWLEDGE = ActionNode(
key="Shared Knowledge",
expected_type=str,
instruction="Detail any shared knowledge, like common utility functions or configuration variables.",
example="'game.py' contains functions shared across the project.",
)
ANYTHING_UNCLEAR_PM = ActionNode(
key="Anything UNCLEAR",
expected_type=str,
instruction="Mention any unclear aspects in the project management context and try to clarify them.",
example="Clarification needed on how to start and initialize third-party libraries.",
)
NODES = [
REQUIRED_PYTHON_PACKAGES,
REQUIRED_OTHER_LANGUAGE_PACKAGES,
LOGIC_ANALYSIS,
TASK_LIST,
FULL_API_SPEC,
SHARED_KNOWLEDGE,
ANYTHING_UNCLEAR_PM,
]
PM_NODE = ActionNode.from_children("PM_NODE", NODES)
def main():
prompt = PM_NODE.compile(context="")
logger.info(prompt)
if __name__ == "__main__":
main()

217
metagpt/actions/rebuild_class_view.py Normal file
View file

@ -0,0 +1,217 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/12/19
@Author : mashenquan
@File : rebuild_class_view.py
@Desc : Rebuild class view info
"""
import re
from pathlib import Path
import aiofiles
from metagpt.actions import Action
from metagpt.config import CONFIG
from metagpt.const import (
AGGREGATION,
COMPOSITION,
DATA_API_DESIGN_FILE_REPO,
GENERALIZATION,
GRAPH_REPO_FILE_REPO,
)
from metagpt.logs import logger
from metagpt.repo_parser import RepoParser
from metagpt.schema import ClassAttribute, ClassMethod, ClassView
from metagpt.utils.common import split_namespace
from metagpt.utils.di_graph_repository import DiGraphRepository
from metagpt.utils.graph_repository import GraphKeyword, GraphRepository
class RebuildClassView(Action):
async def run(self, with_messages=None, format=CONFIG.prompt_schema):
graph_repo_pathname = CONFIG.git_repo.workdir / GRAPH_REPO_FILE_REPO / CONFIG.git_repo.workdir.name
graph_db = await DiGraphRepository.load_from(str(graph_repo_pathname.with_suffix(".json")))
repo_parser = RepoParser(base_directory=Path(self.context))
# use pylint
class_views, relationship_views, package_root = await repo_parser.rebuild_class_views(path=Path(self.context))
await GraphRepository.update_graph_db_with_class_views(graph_db, class_views)
await GraphRepository.update_graph_db_with_class_relationship_views(graph_db, relationship_views)
# use ast
direction, diff_path = self._diff_path(path_root=Path(self.context).resolve(), package_root=package_root)
symbols = repo_parser.generate_symbols()
for file_info in symbols:
# Align to the same root directory in accordance with `class_views`.
file_info.file = self._align_root(file_info.file, direction, diff_path)
await GraphRepository.update_graph_db_with_file_info(graph_db, file_info)
await self._create_mermaid_class_views(graph_db=graph_db)
await graph_db.save()
async def _create_mermaid_class_views(self, graph_db):
path = Path(CONFIG.git_repo.workdir) / DATA_API_DESIGN_FILE_REPO
path.mkdir(parents=True, exist_ok=True)
pathname = path / CONFIG.git_repo.workdir.name
async with aiofiles.open(str(pathname.with_suffix(".mmd")), mode="w", encoding="utf-8") as writer:
content = "classDiagram\n"
logger.debug(content)
await writer.write(content)
# class names
rows = await graph_db.select(predicate=GraphKeyword.IS, object_=GraphKeyword.CLASS)
class_distinct = set()
relationship_distinct = set()
for r in rows:
await RebuildClassView._create_mermaid_class(r.subject, graph_db, writer, class_distinct)
for r in rows:
await RebuildClassView._create_mermaid_relationship(r.subject, graph_db, writer, relationship_distinct)
@staticmethod
async def _create_mermaid_class(ns_class_name, graph_db, file_writer, distinct):
fields = split_namespace(ns_class_name)
if len(fields) > 2:
# Ignore sub-class
return
class_view = ClassView(name=fields[1])
rows = await graph_db.select(subject=ns_class_name)
for r in rows:
name = split_namespace(r.object_)[-1]
name, visibility, abstraction = RebuildClassView._parse_name(name=name, language="python")
if r.predicate == GraphKeyword.HAS_CLASS_PROPERTY:
var_type = await RebuildClassView._parse_variable_type(r.object_, graph_db)
attribute = ClassAttribute(
name=name, visibility=visibility, abstraction=bool(abstraction), value_type=var_type
)
class_view.attributes.append(attribute)
elif r.predicate == GraphKeyword.HAS_CLASS_FUNCTION:
method = ClassMethod(name=name, visibility=visibility, abstraction=bool(abstraction))
await RebuildClassView._parse_function_args(method, r.object_, graph_db)
class_view.methods.append(method)
# update graph db
await graph_db.insert(ns_class_name, GraphKeyword.HAS_CLASS_VIEW, class_view.model_dump_json())
content = class_view.get_mermaid(align=1)
logger.debug(content)
await file_writer.write(content)
distinct.add(ns_class_name)
@staticmethod
async def _create_mermaid_relationship(ns_class_name, graph_db, file_writer, distinct):
s_fields = split_namespace(ns_class_name)
if len(s_fields) > 2:
# Ignore sub-class
return
predicates = {GraphKeyword.IS + v + GraphKeyword.OF: v for v in [GENERALIZATION, COMPOSITION, AGGREGATION]}
mappings = {
GENERALIZATION: " <|-- ",
COMPOSITION: " *-- ",
AGGREGATION: " o-- ",
}
content = ""
for p, v in predicates.items():
rows = await graph_db.select(subject=ns_class_name, predicate=p)
for r in rows:
o_fields = split_namespace(r.object_)
if len(o_fields) > 2:
# Ignore sub-class
continue
relationship = mappings.get(v, " .. ")
link = f"{o_fields[1]}{relationship}{s_fields[1]}"
distinct.add(link)
content += f"\t{link}\n"
if content:
logger.debug(content)
await file_writer.write(content)
@staticmethod
def _parse_name(name: str, language="python"):
pattern = re.compile(r"<I>(.*?)<\/I>")
result = re.search(pattern, name)
abstraction = ""
if result:
name = result.group(1)
abstraction = "*"
if name.startswith("__"):
visibility = "-"
elif name.startswith("_"):
visibility = "#"
else:
visibility = "+"
return name, visibility, abstraction
@staticmethod
async def _parse_variable_type(ns_name, graph_db) -> str:
rows = await graph_db.select(subject=ns_name, predicate=GraphKeyword.HAS_TYPE_DESC)
if not rows:
return ""
vals = rows[0].object_.replace("'", "").split(":")
if len(vals) == 1:
return ""
val = vals[-1].strip()
return "" if val == "NoneType" else val + " "
@staticmethod
async def _parse_function_args(method: ClassMethod, ns_name: str, graph_db: GraphRepository):
rows = await graph_db.select(subject=ns_name, predicate=GraphKeyword.HAS_ARGS_DESC)
if not rows:
return
info = rows[0].object_.replace("'", "")
fs_tag = "("
ix = info.find(fs_tag)
fe_tag = "):"
eix = info.rfind(fe_tag)
if eix < 0:
fe_tag = ")"
eix = info.rfind(fe_tag)
args_info = info[ix + len(fs_tag) : eix].strip()
method.return_type = info[eix + len(fe_tag) :].strip()
if method.return_type == "None":
method.return_type = ""
if "(" in method.return_type:
method.return_type = method.return_type.replace("(", "Tuple[").replace(")", "]")
# parse args
if not args_info:
return
splitter_ixs = []
cost = 0
for i in range(len(args_info)):
if args_info[i] == "[":
cost += 1
elif args_info[i] == "]":
cost -= 1
if args_info[i] == "," and cost == 0:
splitter_ixs.append(i)
splitter_ixs.append(len(args_info))
args = []
ix = 0
for eix in splitter_ixs:
args.append(args_info[ix:eix])
ix = eix + 1
for arg in args:
parts = arg.strip().split(":")
if len(parts) == 1:
method.args.append(ClassAttribute(name=parts[0].strip()))
continue
method.args.append(ClassAttribute(name=parts[0].strip(), value_type=parts[-1].strip()))
@staticmethod
def _diff_path(path_root: Path, package_root: Path) -> (str, str):
if len(str(path_root)) > len(str(package_root)):
return "+", str(path_root.relative_to(package_root))
if len(str(path_root)) < len(str(package_root)):
return "-", str(package_root.relative_to(path_root))
return "=", "."
@staticmethod
def _align_root(path: str, direction: str, diff_path: str):
if direction == "=":
return path
if direction == "+":
return diff_path + "/" + path
else:
return path[len(diff_path) + 1 :]

60
metagpt/actions/rebuild_sequence_view.py Normal file
View file

@ -0,0 +1,60 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2024/1/4
@Author : mashenquan
@File : rebuild_sequence_view.py
@Desc : Rebuild sequence view info
"""
from __future__ import annotations
from pathlib import Path
from typing import List
from metagpt.actions import Action
from metagpt.config import CONFIG
from metagpt.const import GRAPH_REPO_FILE_REPO
from metagpt.logs import logger
from metagpt.utils.common import aread, list_files
from metagpt.utils.di_graph_repository import DiGraphRepository
from metagpt.utils.graph_repository import GraphKeyword
class RebuildSequenceView(Action):
async def run(self, with_messages=None, format=CONFIG.prompt_schema):
graph_repo_pathname = CONFIG.git_repo.workdir / GRAPH_REPO_FILE_REPO / CONFIG.git_repo.workdir.name
graph_db = await DiGraphRepository.load_from(str(graph_repo_pathname.with_suffix(".json")))
entries = await RebuildSequenceView._search_main_entry(graph_db)
for entry in entries:
await self._rebuild_sequence_view(entry, graph_db)
await graph_db.save()
@staticmethod
async def _search_main_entry(graph_db) -> List:
rows = await graph_db.select(predicate=GraphKeyword.HAS_PAGE_INFO)
tag = "__name__:__main__"
entries = []
for r in rows:
if tag in r.subject or tag in r.object_:
entries.append(r)
return entries
async def _rebuild_sequence_view(self, entry, graph_db):
filename = entry.subject.split(":", 1)[0]
src_filename = RebuildSequenceView._get_full_filename(root=self.context, pathname=filename)
content = await aread(filename=src_filename, encoding="utf-8")
content = f"```python\n{content}\n```\n\n---\nTranslate the code above into Mermaid Sequence Diagram."
data = await self.llm.aask(
msg=content, system_msgs=["You are a python code to Mermaid Sequence Diagram translator in function detail"]
)
await graph_db.insert(subject=filename, predicate=GraphKeyword.HAS_SEQUENCE_VIEW, object_=data)
logger.info(data)
@staticmethod
def _get_full_filename(root: str | Path, pathname: str | Path) -> Path | None:
files = list_files(root=root)
postfix = "/" + str(pathname)
for i in files:
if str(i).endswith(postfix):
return i
return None

16
metagpt/actions/rebuild_sequence_view_an.py Normal file
View file

@ -0,0 +1,16 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2024/1/4
@Author : mashenquan
@File : rebuild_sequence_view_an.py
"""
from metagpt.actions.action_node import ActionNode
from metagpt.utils.mermaid import MMC2
CODE_2_MERMAID_SEQUENCE_DIAGRAM = ActionNode(
key="Program call flow",
expected_type=str,
instruction='Translate the "context" content into "format example" format.',
example=MMC2,
)

83
metagpt/actions/research.py
View file

@ -3,14 +3,15 @@
from __future__ import annotations
import asyncio
import json
from typing import Callable
from typing import Callable, Optional, Union
from pydantic import parse_obj_as
from pydantic import Field, parse_obj_as
from metagpt.actions import Action
from metagpt.config import CONFIG
from metagpt.llm import LLM
from metagpt.logs import logger
from metagpt.provider.base_llm import BaseLLM
from metagpt.tools.search_engine import SearchEngine
from metagpt.tools.web_browser_engine import WebBrowserEngine, WebBrowserEngineType
from metagpt.utils.common import OutputParser
@ -49,7 +50,7 @@ based on the link credibility. If two results have equal credibility, prioritize
ranked results' indices in JSON format, like [0, 1, 3, 4, ...], without including other words.
"""
WEB_BROWSE_AND_SUMMARIZE_PROMPT = '''### Requirements
WEB_BROWSE_AND_SUMMARIZE_PROMPT = """### Requirements
1. Utilize the text in the "Reference Information" section to respond to the question "{query}".
2. If the question cannot be directly answered using the text, but the text is related to the research topic, please provide \
a comprehensive summary of the text.
@ -58,10 +59,10 @@ a comprehensive summary of the text.
### Reference Information
{content}
'''
"""
CONDUCT_RESEARCH_PROMPT = '''### Reference Information
CONDUCT_RESEARCH_PROMPT = """### Reference Information
{content}
### Requirements
@ -73,22 +74,18 @@ above. The report must meet the following requirements:
- Present data and findings in an intuitive manner, utilizing feature comparative tables, if applicable.
- The report should have a minimum word count of 2,000 and be formatted with Markdown syntax following APA style guidelines.
- Include all source URLs in APA format at the end of the report.
'''
"""
class CollectLinks(Action):
"""Action class to collect links from a search engine."""
def __init__(
self,
name: str = "",
*args,
rank_func: Callable[[list[str]], None] | None = None,
**kwargs,
):
super().__init__(name, *args, **kwargs)
self.desc = "Collect links from a search engine."
self.search_engine = SearchEngine()
self.rank_func = rank_func
name: str = "CollectLinks"
context: Optional[str] = None
desc: str = "Collect links from a search engine."
search_engine: SearchEngine = Field(default_factory=SearchEngine)
rank_func: Optional[Callable[[list[str]], None]] = None
async def run(
self,
@ -114,20 +111,26 @@ class CollectLinks(Action):
keywords = OutputParser.extract_struct(keywords, list)
keywords = parse_obj_as(list[str], keywords)
except Exception as e:
logger.exception(f"fail to get keywords related to the research topic \"{topic}\" for {e}")
logger.exception(f"fail to get keywords related to the research topic '{topic}' for {e}")
keywords = [topic]
results = await asyncio.gather(*(self.search_engine.run(i, as_string=False) for i in keywords))
def gen_msg():
while True:
search_results = "\n".join(f"#### Keyword: {i}\n Search Result: {j}\n" for (i, j) in zip(keywords, results))
prompt = SUMMARIZE_SEARCH_PROMPT.format(decomposition_nums=decomposition_nums, search_results=search_results)
search_results = "\n".join(
f"#### Keyword: {i}\n Search Result: {j}\n" for (i, j) in zip(keywords, results)
)
prompt = SUMMARIZE_SEARCH_PROMPT.format(
decomposition_nums=decomposition_nums, search_results=search_results
)
yield prompt
remove = max(results, key=len)
remove.pop()
if len(remove) == 0:
break
prompt = reduce_message_length(gen_msg(), self.llm.model, system_text, CONFIG.max_tokens_rsp)
model_name = CONFIG.get_model_name(CONFIG.get_default_llm_provider_enum())
prompt = reduce_message_length(gen_msg(), model_name, system_text, CONFIG.max_tokens_rsp)
logger.debug(prompt)
queries = await self._aask(prompt, [system_text])
try:
@ -172,20 +175,23 @@ class CollectLinks(Action):
class WebBrowseAndSummarize(Action):
"""Action class to explore the web and provide summaries of articles and webpages."""
def __init__(
self,
*args,
browse_func: Callable[[list[str]], None] | None = None,
**kwargs,
):
super().__init__(*args, **kwargs)
name: str = "WebBrowseAndSummarize"
context: Optional[str] = None
llm: BaseLLM = Field(default_factory=LLM)
desc: str = "Explore the web and provide summaries of articles and webpages."
browse_func: Union[Callable[[list[str]], None], None] = None
web_browser_engine: Optional[WebBrowserEngine] = None
def __init__(self, **kwargs):
super().__init__(**kwargs)
if CONFIG.model_for_researcher_summary:
self.llm.model = CONFIG.model_for_researcher_summary
self.web_browser_engine = WebBrowserEngine(
engine=WebBrowserEngineType.CUSTOM if browse_func else None,
run_func=browse_func,
engine=WebBrowserEngineType.CUSTOM if self.browse_func else None,
run_func=self.browse_func,
)
self.desc = "Explore the web and provide summaries of articles and webpages."
async def run(
self,
@ -214,7 +220,9 @@ class WebBrowseAndSummarize(Action):
for u, content in zip([url, *urls], contents):
content = content.inner_text
chunk_summaries = []
for prompt in generate_prompt_chunk(content, prompt_template, self.llm.model, system_text, CONFIG.max_tokens_rsp):
for prompt in generate_prompt_chunk(
content, prompt_template, self.llm.model, system_text, CONFIG.max_tokens_rsp
):
logger.debug(prompt)
summary = await self._aask(prompt, [system_text])
if summary == "Not relevant.":
@ -238,8 +246,13 @@ class WebBrowseAndSummarize(Action):
class ConductResearch(Action):
"""Action class to conduct research and generate a research report."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
name: str = "ConductResearch"
context: Optional[str] = None
llm: BaseLLM = Field(default_factory=LLM)
def __init__(self, **kwargs):
super().__init__(**kwargs)
if CONFIG.model_for_researcher_report:
self.llm.model = CONFIG.model_for_researcher_report

86
metagpt/actions/run_code.py
View file

@ -4,14 +4,27 @@
@Time : 2023/5/11 17:46
@Author : alexanderwu
@File : run_code.py
@Modified By: mashenquan, 2023/11/27.
1. Mark the location of Console logs in the PROMPT_TEMPLATE with markdown code-block formatting to enhance
the understanding for the LLM.
2. Fix bug: Add the "install dependency" operation.
3. Encapsulate the input of RunCode into RunCodeContext and encapsulate the output of RunCode into
RunCodeResult to standardize and unify parameter passing between WriteCode, RunCode, and DebugError.
4. According to section 2.2.3.5.7 of RFC 135, change the method of transferring file content
(code files, unit test files, log files) from using the message to using the file name.
5. Merged the `Config` class of send18:dev branch to take over the set/get operations of the Environment
class.
"""
import os
import subprocess
import traceback
from typing import Tuple
from pydantic import Field
from metagpt.actions.action import Action
from metagpt.config import CONFIG
from metagpt.logs import logger
from metagpt.schema import RunCodeContext, RunCodeResult
from metagpt.utils.exceptions import handle_exception
PROMPT_TEMPLATE = """
Role: You are a senior development and qa engineer, your role is summarize the code running result.
@ -51,14 +64,20 @@ CONTEXT = """
## Running Command
{command}
## Running Output
standard output: {outs};
standard errors: {errs};
standard output:
```text
{outs}
```
standard errors:
```text
{errs}
```
"""
class RunCode(Action):
def __init__(self, name="RunCode", context=None, llm=None):
super().__init__(name, context, llm)
name: str = "RunCode"
context: RunCodeContext = Field(default_factory=RunCodeContext)
@classmethod
async def run_text(cls, code) -> Tuple[str, str]:
@ -66,10 +85,9 @@ class RunCode(Action):
# We will document_store the result in this dictionary
namespace = {}
exec(code, namespace)
return namespace.get("result", ""), ""
except Exception:
# If there is an error in the code, return the error message
return "", traceback.format_exc()
except Exception as e:
return "", str(e)
return namespace.get("result", ""), ""
@classmethod
async def run_script(cls, working_directory, additional_python_paths=[], command=[]) -> Tuple[str, str]:
@ -77,17 +95,19 @@ class RunCode(Action):
additional_python_paths = [str(path) for path in additional_python_paths]
# Copy the current environment variables
env = os.environ.copy()
env = CONFIG.new_environ()
# Modify the PYTHONPATH environment variable
additional_python_paths = [working_directory] + additional_python_paths
additional_python_paths = ":".join(additional_python_paths)
env["PYTHONPATH"] = additional_python_paths + ":" + env.get("PYTHONPATH", "")
RunCode._install_dependencies(working_directory=working_directory, env=env)
# Start the subprocess
process = subprocess.Popen(
command, cwd=working_directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env
)
logger.info(" ".join(command))
try:
# Wait for the process to complete, with a timeout
@ -98,31 +118,45 @@ class RunCode(Action):
stdout, stderr = process.communicate()
return stdout.decode("utf-8"), stderr.decode("utf-8")
async def run(
self, code, mode="script", code_file_name="", test_code="", test_file_name="", command=[], **kwargs
) -> str:
logger.info(f"Running {' '.join(command)}")
if mode == "script":
outs, errs = await self.run_script(command=command, **kwargs)
elif mode == "text":
outs, errs = await self.run_text(code=code)
async def run(self, *args, **kwargs) -> RunCodeResult:
logger.info(f"Running {' '.join(self.context.command)}")
if self.context.mode == "script":
outs, errs = await self.run_script(
command=self.context.command,
working_directory=self.context.working_directory,
additional_python_paths=self.context.additional_python_paths,
)
elif self.context.mode == "text":
outs, errs = await self.run_text(code=self.context.code)
logger.info(f"{outs=}")
logger.info(f"{errs=}")
context = CONTEXT.format(
code=code,
code_file_name=code_file_name,
test_code=test_code,
test_file_name=test_file_name,
command=" ".join(command),
code=self.context.code,
code_file_name=self.context.code_filename,
test_code=self.context.test_code,
test_file_name=self.context.test_filename,
command=" ".join(self.context.command),
outs=outs[:500], # outs might be long but they are not important, truncate them to avoid token overflow
errs=errs[:10000], # truncate errors to avoid token overflow
)
prompt = PROMPT_TEMPLATE.format(context=context)
rsp = await self._aask(prompt)
return RunCodeResult(summary=rsp, stdout=outs, stderr=errs)
result = context + rsp
@staticmethod
@handle_exception(exception_type=subprocess.CalledProcessError)
def _install_via_subprocess(cmd, check, cwd, env):
return subprocess.run(cmd, check=check, cwd=cwd, env=env)
return result
@staticmethod
def _install_dependencies(working_directory, env):
install_command = ["python", "-m", "pip", "install", "-r", "requirements.txt"]
logger.info(" ".join(install_command))
RunCode._install_via_subprocess(install_command, check=True, cwd=working_directory, env=env)
install_pytest_command = ["python", "-m", "pip", "install", "pytest"]
logger.info(" ".join(install_pytest_command))
RunCode._install_via_subprocess(install_pytest_command, check=True, cwd=working_directory, env=env)

39
metagpt/actions/search_and_summarize.py
View file

@ -5,12 +5,16 @@
@Author : alexanderwu
@File : search_google.py
"""
from typing import Any, Optional
import pydantic
from pydantic import Field, model_validator
from metagpt.actions import Action
from metagpt.config import Config
from metagpt.config import CONFIG, Config
from metagpt.logs import logger
from metagpt.schema import Message
from metagpt.tools import SearchEngineType
from metagpt.tools.search_engine import SearchEngine
SEARCH_AND_SUMMARIZE_SYSTEM = """### Requirements
@ -54,7 +58,6 @@ SEARCH_AND_SUMMARIZE_PROMPT = """
"""
SEARCH_AND_SUMMARIZE_SALES_SYSTEM = """## Requirements
1. Please summarize the latest dialogue based on the reference information (secondary) and dialogue history (primary). Do not include text that is irrelevant to the conversation.
- The context is for reference only. If it is irrelevant to the user's search request history, please reduce its reference and usage.
@ -100,18 +103,32 @@ You are a member of a professional butler team and will provide helpful suggesti
"""
# TOTEST
class SearchAndSummarize(Action):
def __init__(self, name="", context=None, llm=None, engine=None, search_func=None):
self.config = Config()
self.engine = engine or self.config.search_engine
name: str = ""
content: Optional[str] = None
config: None = Field(default_factory=Config)
engine: Optional[SearchEngineType] = CONFIG.search_engine
search_func: Optional[Any] = None
search_engine: SearchEngine = None
result: str = ""
@model_validator(mode="before")
@classmethod
def validate_engine_and_run_func(cls, values):
engine = values.get("engine")
search_func = values.get("search_func")
config = Config()
if engine is None:
engine = config.search_engine
try:
self.search_engine = SearchEngine(self.engine, run_func=search_func)
search_engine = SearchEngine(engine=engine, run_func=search_func)
except pydantic.ValidationError:
self.search_engine = None
search_engine = None
self.result = ""
super().__init__(name, context, llm)
values["search_engine"] = search_engine
return values
async def run(self, context: list[Message], system_text=SEARCH_AND_SUMMARIZE_SYSTEM) -> str:
if self.search_engine is None:
@ -130,8 +147,7 @@ class SearchAndSummarize(Action):
system_prompt = [system_text]
prompt = SEARCH_AND_SUMMARIZE_PROMPT.format(
# PREFIX = self.prefix,
ROLE=self.profile,
ROLE=self.prefix,
CONTEXT=rsp,
QUERY_HISTORY="\n".join([str(i) for i in context[:-1]]),
QUERY=str(context[-1]),
@ -140,4 +156,3 @@ class SearchAndSummarize(Action):
logger.debug(prompt)
logger.debug(result)
return result

111
metagpt/actions/skill_action.py Normal file
View file

@ -0,0 +1,111 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/8/28
@Author : mashenquan
@File : skill_action.py
@Desc : Call learned skill
"""
from __future__ import annotations
import ast
import importlib
import traceback
from copy import deepcopy
from typing import Dict, Optional
from metagpt.actions import Action
from metagpt.learn.skill_loader import Skill
from metagpt.logs import logger
from metagpt.schema import Message
# TOTEST
class ArgumentsParingAction(Action):
skill: Skill
ask: str
rsp: Optional[Message] = None
args: Optional[Dict] = None
@property
def prompt(self):
prompt = "You are a function parser. You can convert spoken words into function parameters.\n"
prompt += "\n---\n"
prompt += f"{self.skill.name} function parameters description:\n"
for k, v in self.skill.arguments.items():
prompt += f"parameter `{k}`: {v}\n"
prompt += "\n---\n"
prompt += "Examples:\n"
for e in self.skill.examples:
prompt += f"If want you to do `{e.ask}`, return `{e.answer}` brief and clear.\n"
prompt += "\n---\n"
prompt += (
f"\nRefer to the `{self.skill.name}` function description, and fill in the function parameters according "
'to the example "I want you to do xx" in the Examples section.'
f"\nNow I want you to do `{self.ask}`, return function parameters in Examples format above, brief and "
"clear."
)
return prompt
async def run(self, with_message=None, **kwargs) -> Message:
prompt = self.prompt
rsp = await self.llm.aask(msg=prompt, system_msgs=[])
logger.debug(f"SKILL:{prompt}\n, RESULT:{rsp}")
self.args = ArgumentsParingAction.parse_arguments(skill_name=self.skill.name, txt=rsp)
self.rsp = Message(content=rsp, role="assistant", instruct_content=self.args, cause_by=self)
return self.rsp
@staticmethod
def parse_arguments(skill_name, txt) -> dict:
prefix = skill_name + "("
if prefix not in txt:
logger.error(f"{skill_name} not in {txt}")
return None
if ")" not in txt:
logger.error(f"')' not in {txt}")
return None
begin_ix = txt.find(prefix)
end_ix = txt.rfind(")")
args_txt = txt[begin_ix + len(prefix) : end_ix]
logger.info(args_txt)
fake_expression = f"dict({args_txt})"
parsed_expression = ast.parse(fake_expression, mode="eval")
args = {}
for keyword in parsed_expression.body.keywords:
key = keyword.arg
value = ast.literal_eval(keyword.value)
args[key] = value
return args
class SkillAction(Action):
skill: Skill
args: Dict
rsp: Optional[Message] = None
async def run(self, with_message=None, **kwargs) -> Message:
"""Run action"""
options = deepcopy(kwargs)
if self.args:
for k in self.args.keys():
if k in options:
options.pop(k)
try:
rsp = await self.find_and_call_function(self.skill.name, args=self.args, **options)
self.rsp = Message(content=rsp, role="assistant", cause_by=self)
except Exception as e:
logger.exception(f"{e}, traceback:{traceback.format_exc()}")
self.rsp = Message(content=f"Error: {e}", role="assistant", cause_by=self)
return self.rsp
@staticmethod
async def find_and_call_function(function_name, args, **kwargs) -> str:
try:
module = importlib.import_module("metagpt.learn")
function = getattr(module, function_name)
# Invoke function and return result
result = await function(**args, **kwargs)
return result
except (ModuleNotFoundError, AttributeError):
logger.error(f"{function_name} not found")
raise ValueError(f"{function_name} not found")

123
metagpt/actions/summarize_code.py Normal file
View file

@ -0,0 +1,123 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Author : alexanderwu
@File : summarize_code.py
@Modified By: mashenquan, 2023/12/5. Archive the summarization content of issue discovery for use in WriteCode.
"""
from pathlib import Path
from pydantic import Field
from tenacity import retry, stop_after_attempt, wait_random_exponential
from metagpt.actions.action import Action
from metagpt.config import CONFIG
from metagpt.const import SYSTEM_DESIGN_FILE_REPO, TASK_FILE_REPO
from metagpt.logs import logger
from metagpt.schema import CodeSummarizeContext
from metagpt.utils.file_repository import FileRepository
PROMPT_TEMPLATE = """
NOTICE
Role: You are a professional software engineer, and your main task is to review the code.
Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
-----
# System Design
```text
{system_design}
```
-----
# Tasks
```text
{tasks}
```
-----
{code_blocks}
## Code Review All: Please read all historical files and find possible bugs in the files, such as unimplemented functions, calling errors, unreferences, etc.
## Call flow: mermaid code, based on the implemented function, use mermaid to draw a complete call chain
## Summary: Summary based on the implementation of historical files
## TODOs: Python dict[str, str], write down the list of files that need to be modified and the reasons. We will modify them later.
"""
FORMAT_EXAMPLE = """
## Code Review All
### a.py
- It fulfills less of xxx requirements...
- Field yyy is not given...
-...
### b.py
...
### c.py
...
## Call flow
```mermaid
flowchart TB
c1-->a2
subgraph one
a1-->a2
end
subgraph two
b1-->b2
end
subgraph three
c1-->c2
end
```
## Summary
- a.py:...
- b.py:...
- c.py:...
- ...
## TODOs
{
"a.py": "implement requirement xxx...",
}
"""
# TOTEST
class SummarizeCode(Action):
name: str = "SummarizeCode"
context: CodeSummarizeContext = Field(default_factory=CodeSummarizeContext)
@retry(stop=stop_after_attempt(2), wait=wait_random_exponential(min=1, max=60))
async def summarize_code(self, prompt):
code_rsp = await self._aask(prompt)
return code_rsp
async def run(self):
design_pathname = Path(self.context.design_filename)
design_doc = await FileRepository.get_file(filename=design_pathname.name, relative_path=SYSTEM_DESIGN_FILE_REPO)
task_pathname = Path(self.context.task_filename)
task_doc = await FileRepository.get_file(filename=task_pathname.name, relative_path=TASK_FILE_REPO)
src_file_repo = CONFIG.git_repo.new_file_repository(relative_path=CONFIG.src_workspace)
code_blocks = []
for filename in self.context.codes_filenames:
code_doc = await src_file_repo.get(filename)
code_block = f"```python\n{code_doc.content}\n```\n-----"
code_blocks.append(code_block)
format_example = FORMAT_EXAMPLE
prompt = PROMPT_TEMPLATE.format(
system_design=design_doc.content,
tasks=task_doc.content,
code_blocks="\n".join(code_blocks),
format_example=format_example,
)
logger.info("Summarize code..")
rsp = await self.summarize_code(prompt)
return rsp

163
metagpt/actions/talk_action.py Normal file
View file

@ -0,0 +1,163 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/8/28
@Author : mashenquan
@File : talk_action.py
@Desc : Act as it’s a talk
"""
from typing import Optional
from metagpt.actions import Action
from metagpt.config import CONFIG
from metagpt.const import DEFAULT_LANGUAGE
from metagpt.logs import logger
from metagpt.schema import Message
# TOTEST
class TalkAction(Action):
context: str
history_summary: str = ""
knowledge: str = ""
rsp: Optional[Message] = None
@property
def prompt(self):
prompt = ""
if CONFIG.agent_description:
prompt = (
f"You are {CONFIG.agent_description}. Your responses should align with the role-play agreement, "
f"maintaining the character's persona and habits. When faced with unrelated questions, playfully "
f"decline to answer without revealing your AI nature to preserve the character's image.\n\n"
)
prompt += f"Knowledge:\n{self.knowledge}\n\n" if self.knowledge else ""
prompt += f"{self.history_summary}\n\n"
prompt += (
"If the information is insufficient, you can search in the historical conversation or knowledge above.\n"
)
language = CONFIG.language or DEFAULT_LANGUAGE
prompt += (
f"Answer the following questions strictly in {language}, and the answers must follow the Markdown format.\n "
f"{self.context}"
)
logger.debug(f"PROMPT: {prompt}")
return prompt
@property
def prompt_gpt4(self):
kvs = {
"{role}": CONFIG.agent_description or "",
"{history}": self.history_summary or "",
"{knowledge}": self.knowledge or "",
"{language}": CONFIG.language or DEFAULT_LANGUAGE,
"{ask}": self.context,
}
prompt = TalkActionPrompt.FORMATION_LOOSE
for k, v in kvs.items():
prompt = prompt.replace(k, v)
logger.info(f"PROMPT: {prompt}")
return prompt
# async def run_old(self, *args, **kwargs) -> ActionOutput:
# prompt = self.prompt
# rsp = await self.llm.aask(msg=prompt, system_msgs=[])
# logger.debug(f"PROMPT:{prompt}\nRESULT:{rsp}\n")
# self._rsp = ActionOutput(content=rsp)
# return self._rsp
@property
def aask_args(self):
language = CONFIG.language or DEFAULT_LANGUAGE
system_msgs = [
f"You are {CONFIG.agent_description}.",
"Your responses should align with the role-play agreement, "
"maintaining the character's persona and habits. When faced with unrelated questions, playfully "
"decline to answer without revealing your AI nature to preserve the character's image.",
"If the information is insufficient, you can search in the context or knowledge.",
f"Answer the following questions strictly in {language}, and the answers must follow the Markdown format.",
]
format_msgs = []
if self.knowledge:
format_msgs.append({"role": "assistant", "content": self.knowledge})
if self.history_summary:
format_msgs.append({"role": "assistant", "content": self.history_summary})
return self.context, format_msgs, system_msgs
async def run(self, with_message=None, **kwargs) -> Message:
msg, format_msgs, system_msgs = self.aask_args
rsp = await self.llm.aask(msg=msg, format_msgs=format_msgs, system_msgs=system_msgs)
self.rsp = Message(content=rsp, role="assistant", cause_by=self)
return self.rsp
class TalkActionPrompt:
FORMATION = """Formation: "Capacity and role" defines the role you are currently playing;
"[HISTORY_BEGIN]" and "[HISTORY_END]" tags enclose the historical conversation;
"[KNOWLEDGE_BEGIN]" and "[KNOWLEDGE_END]" tags enclose the knowledge may help for your responses;
"Statement" defines the work detail you need to complete at this stage;
"[ASK_BEGIN]" and [ASK_END] tags enclose the questions;
"Constraint" defines the conditions that your responses must comply with.
"Personality" defines your language style。
"Insight" provides a deeper understanding of the characters' inner traits.
"Initial" defines the initial setup of a character.
Capacity and role: {role}
Statement: Your responses should align with the role-play agreement, maintaining the
character's persona and habits. When faced with unrelated questions, playfully decline to answer without revealing
your AI nature to preserve the character's image.
[HISTORY_BEGIN]
{history}
[HISTORY_END]
[KNOWLEDGE_BEGIN]
{knowledge}
[KNOWLEDGE_END]
Statement: If the information is insufficient, you can search in the historical conversation or knowledge.
Statement: Unless you are a language professional, answer the following questions strictly in {language}
, and the answers must follow the Markdown format. Strictly excluding any tag likes "[HISTORY_BEGIN]"
, "[HISTORY_END]", "[KNOWLEDGE_BEGIN]", "[KNOWLEDGE_END]" in responses.
{ask}
"""
FORMATION_LOOSE = """Formation: "Capacity and role" defines the role you are currently playing;
"[HISTORY_BEGIN]" and "[HISTORY_END]" tags enclose the historical conversation;
"[KNOWLEDGE_BEGIN]" and "[KNOWLEDGE_END]" tags enclose the knowledge may help for your responses;
"Statement" defines the work detail you need to complete at this stage;
"Constraint" defines the conditions that your responses must comply with.
"Personality" defines your language style。
"Insight" provides a deeper understanding of the characters' inner traits.
"Initial" defines the initial setup of a character.
Capacity and role: {role}
Statement: Your responses should maintaining the character's persona and habits. When faced with unrelated questions
, playfully decline to answer without revealing your AI nature to preserve the character's image.
[HISTORY_BEGIN]
{history}
[HISTORY_END]
[KNOWLEDGE_BEGIN]
{knowledge}
[KNOWLEDGE_END]
Statement: If the information is insufficient, you can search in the historical conversation or knowledge.
Statement: Unless you are a language professional, answer the following questions strictly in {language}
, and the answers must follow the Markdown format. Strictly excluding any tag likes "[HISTORY_BEGIN]"
, "[HISTORY_END]", "[KNOWLEDGE_BEGIN]", "[KNOWLEDGE_END]" in responses.
{ask}
"""

174
metagpt/actions/write_code.py
View file

@ -4,79 +4,151 @@
@Time : 2023/5/11 17:45
@Author : alexanderwu
@File : write_code.py
@Modified By: mashenquan, 2023-11-1. In accordance with Chapter 2.1.3 of RFC 116, modify the data type of the `cause_by`
value of the `Message` object.
@Modified By: mashenquan, 2023-11-27.
1. Mark the location of Design, Tasks, Legacy Code and Debug logs in the PROMPT_TEMPLATE with markdown
code-block formatting to enhance the understanding for the LLM.
2. Following the think-act principle, solidify the task parameters when creating the WriteCode object, rather
than passing them in when calling the run function.
3. Encapsulate the input of RunCode into RunCodeContext and encapsulate the output of RunCode into
RunCodeResult to standardize and unify parameter passing between WriteCode, RunCode, and DebugError.
"""
from metagpt.actions import WriteDesign
import json
from pydantic import Field
from tenacity import retry, stop_after_attempt, wait_random_exponential
from metagpt.actions.action import Action
from metagpt.const import WORKSPACE_ROOT
from metagpt.config import CONFIG
from metagpt.const import (
BUGFIX_FILENAME,
CODE_SUMMARIES_FILE_REPO,
DOCS_FILE_REPO,
TASK_FILE_REPO,
TEST_OUTPUTS_FILE_REPO,
)
from metagpt.logs import logger
from metagpt.schema import Message
from metagpt.schema import CodingContext, Document, RunCodeResult
from metagpt.utils.common import CodeParser
from tenacity import retry, stop_after_attempt, wait_fixed
from metagpt.utils.file_repository import FileRepository
PROMPT_TEMPLATE = """
NOTICE
Role: You are a professional engineer; the main goal is to write PEP8 compliant, elegant, modular, easy to read and maintain Python 3.9 code (but you can also use other programming language)
Role: You are a professional engineer; the main goal is to write google-style, elegant, modular, easy to read and maintain code
Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
## Code: {filename} Write code with triple quoto, based on the following list and context.
1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT.
2. Requirement: Based on the context, implement one following code file, note to return only in code form, your code will be part of the entire project, so please implement complete, reliable, reusable code snippets
3. Attention1: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
4. Attention2: YOU MUST FOLLOW "Data structures and interface definitions". DONT CHANGE ANY DESIGN.
5. Think before writing: What should be implemented and provided in this document?
6. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
7. Do not use public member functions that do not exist in your design.
-----
# Context
{context}
-----
## Format example
-----
## Design
{design}
## Tasks
{tasks}
## Legacy Code
```Code
{code}
```
## Debug logs
```text
{logs}
{summary_log}
```
## Bug Feedback logs
```text
{feedback}
```
# Format example
## Code: {filename}
```python
## {filename}
...
```
-----
# Instruction: Based on the context, follow "Format example", write code.
## Code: {filename}. Write code with triple quoto, based on the following attentions and context.
1. Only One file: do your best to implement THIS ONLY ONE FILE.
2. COMPLETE CODE: Your code will be part of the entire project, so please implement complete, reliable, reusable code snippets.
3. Set default value: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE. AVOID circular import.
4. Follow design: YOU MUST FOLLOW "Data structures and interfaces". DONT CHANGE ANY DESIGN. Do not use public member functions that do not exist in your design.
5. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
6. Before using a external variable/module, make sure you import it first.
7. Write out EVERY CODE DETAIL, DON'T LEAVE TODO.
"""
class WriteCode(Action):
def __init__(self, name="WriteCode", context: list[Message] = None, llm=None):
super().__init__(name, context, llm)
name: str = "WriteCode"
context: Document = Field(default_factory=Document)
def _is_invalid(self, filename):
return any(i in filename for i in ["mp3", "wav"])
def _save(self, context, filename, code):
# logger.info(filename)
# logger.info(code_rsp)
if self._is_invalid(filename):
return
design = [i for i in context if i.cause_by == WriteDesign][0]
ws_name = CodeParser.parse_str(block="Python package name", text=design.content)
ws_path = WORKSPACE_ROOT / ws_name
if f"{ws_name}/" not in filename and all(i not in filename for i in ["requirements.txt", ".md"]):
ws_path = ws_path / ws_name
code_path = ws_path / filename
code_path.parent.mkdir(parents=True, exist_ok=True)
code_path.write_text(code)
logger.info(f"Saving Code to {code_path}")
@retry(stop=stop_after_attempt(2), wait=wait_fixed(1))
async def write_code(self, prompt):
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
async def write_code(self, prompt) -> str:
code_rsp = await self._aask(prompt)
code = CodeParser.parse_code(block="", text=code_rsp)
return code
async def run(self, context, filename):
prompt = PROMPT_TEMPLATE.format(context=context, filename=filename)
logger.info(f'Writing {filename}..')
async def run(self, *args, **kwargs) -> CodingContext:
bug_feedback = await FileRepository.get_file(filename=BUGFIX_FILENAME, relative_path=DOCS_FILE_REPO)
coding_context = CodingContext.loads(self.context.content)
test_doc = await FileRepository.get_file(
filename="test_" + coding_context.filename + ".json", relative_path=TEST_OUTPUTS_FILE_REPO
)
summary_doc = None
if coding_context.design_doc and coding_context.design_doc.filename:
summary_doc = await FileRepository.get_file(
filename=coding_context.design_doc.filename, relative_path=CODE_SUMMARIES_FILE_REPO
)
logs = ""
if test_doc:
test_detail = RunCodeResult.loads(test_doc.content)
logs = test_detail.stderr
if bug_feedback:
code_context = coding_context.code_doc.content
else:
code_context = await self.get_codes(coding_context.task_doc, exclude=self.context.filename)
prompt = PROMPT_TEMPLATE.format(
design=coding_context.design_doc.content if coding_context.design_doc else "",
tasks=coding_context.task_doc.content if coding_context.task_doc else "",
code=code_context,
logs=logs,
feedback=bug_feedback.content if bug_feedback else "",
filename=self.context.filename,
summary_log=summary_doc.content if summary_doc else "",
)
logger.info(f"Writing {coding_context.filename}..")
code = await self.write_code(prompt)
# code_rsp = await self._aask_v1(prompt, "code_rsp", OUTPUT_MAPPING)
# self._save(context, filename, code)
return code
if not coding_context.code_doc:
# avoid root_path pydantic ValidationError if use WriteCode alone
root_path = CONFIG.src_workspace if CONFIG.src_workspace else ""
coding_context.code_doc = Document(filename=coding_context.filename, root_path=str(root_path))
coding_context.code_doc.content = code
return coding_context
@staticmethod
async def get_codes(task_doc, exclude) -> str:
if not task_doc:
return ""
if not task_doc.content:
task_doc.content = FileRepository.get_file(filename=task_doc.filename, relative_path=TASK_FILE_REPO)
m = json.loads(task_doc.content)
code_filenames = m.get("Task list", [])
codes = []
src_file_repo = CONFIG.git_repo.new_file_repository(relative_path=CONFIG.src_workspace)
for filename in code_filenames:
if filename == exclude:
continue
doc = await src_file_repo.get(filename=filename)
if not doc:
continue
codes.append(f"----- {filename}\n" + doc.content)
return "\n".join(codes)

591
metagpt/actions/write_code_an_draft.py Normal file
View file

@ -0,0 +1,591 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Author : alexanderwu
@File : write_review.py
"""
import asyncio
from typing import List
from metagpt.actions import Action
from metagpt.actions.action_node import ActionNode
REVIEW = ActionNode(
key="Review",
expected_type=List[str],
instruction="Act as an experienced reviewer and critically assess the given output. Provide specific and"
" constructive feedback, highlighting areas for improvement and suggesting changes.",
example=[
"The logic in the function `calculate_total` seems flawed. Shouldn't it consider the discount rate as well?",
"The TODO function is not implemented yet? Should we implement it before commit?",
],
)
LGTM = ActionNode(
key="LGTM",
expected_type=str,
instruction="LGTM/LBTM. If the code is fully implemented, "
"give a LGTM (Looks Good To Me), otherwise provide a LBTM (Looks Bad To Me).",
example="LBTM",
)
ACTIONS = ActionNode(
key="Actions",
expected_type=str,
instruction="Based on the code review outcome, suggest actionable steps. This can include code changes, "
"refactoring suggestions, or any follow-up tasks.",
example="""1. Refactor the `process_data` method to improve readability and efficiency.
2. Cover edge cases in the `validate_user` function.
3. Implement a the TODO in the `calculate_total` function.
4. Fix the `handle_events` method to update the game state only if a move is successful.
```python
def handle_events(self):
for event in pygame.event.get():
if event.type == pygame.QUIT:
return False
if event.type == pygame.KEYDOWN:
moved = False
if event.key == pygame.K_UP:
moved = self.game.move('UP')
elif event.key == pygame.K_DOWN:
moved = self.game.move('DOWN')
elif event.key == pygame.K_LEFT:
moved = self.game.move('LEFT')
elif event.key == pygame.K_RIGHT:
moved = self.game.move('RIGHT')
if moved:
# Update the game state only if a move was successful
self.render()
return True
```
""",
)
WRITE_DRAFT = ActionNode(
key="WriteDraft",
expected_type=str,
instruction="Could you write draft code for move function in order to implement it?",
example="Draft: ...",
)
WRITE_MOVE_FUNCTION = ActionNode(
key="WriteFunction",
expected_type=str,
instruction="write code for the function not implemented.",
example="""
```Code
...
```
""",
)
REWRITE_CODE = ActionNode(
key="RewriteCode",
expected_type=str,
instruction="""rewrite code based on the Review and Actions""",
example="""
```python
## example.py
def calculate_total(price, quantity):
total = price * quantity
```
""",
)
CODE_REVIEW_CONTEXT = """
# System
Role: You are a professional software engineer, and your main task is to review and revise the code. You need to ensure that the code conforms to the google-style standards, is elegantly designed and modularized, easy to read and maintain.
Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
# Context
## System Design
{"Implementation approach": "我们将使用HTML、CSS和JavaScript来实现这个单机的响应式2048游戏。为了确保游戏性能流畅和响应式设计，我们会选择使用Vue.js框架，因为它易于上手且适合构建交互式界面。我们还将使用localStorage来记录玩家的最高分。", "File list": ["index.html", "styles.css", "main.js", "game.js", "storage.js"], "Data structures and interfaces": "classDiagram\
class Game {\
-board Array\
-score Number\
-bestScore Number\
+constructor()\
+startGame()\
+move(direction: String)\
+getBoard() Array\
+getScore() Number\
+getBestScore() Number\
+setBestScore(score: Number)\
}\
class Storage {\
+getBestScore() Number\
+setBestScore(score: Number)\
}\
class Main {\
+init()\
+bindEvents()\
}\
Game --> Storage : uses\
Main --> Game : uses", "Program call flow": "sequenceDiagram\
participant M as Main\
participant G as Game\
participant S as Storage\
M->>G: init()\
G->>S: getBestScore()\
S-->>G: return bestScore\
M->>G: bindEvents()\
M->>G: startGame()\
loop Game Loop\
M->>G: move(direction)\
G->>S: setBestScore(score)\
S-->>G: return\
end", "Anything UNCLEAR": "目前项目要求明确，没有不清楚的地方。"}
## Tasks
{"Required Python packages": ["无需Python包"], "Required Other language third-party packages": ["vue.js"], "Logic Analysis": [["index.html", "作为游戏的入口文件和主要的HTML结构"], ["styles.css", "包含所有的CSS样式，确保游戏界面美观"], ["main.js", "包含Main类，负责初始化游戏和绑定事件"], ["game.js", "包含Game类，负责游戏逻辑，如开始游戏、移动方块等"], ["storage.js", "包含Storage类，用于获取和设置玩家的最高分"]], "Task list": ["index.html", "styles.css", "storage.js", "game.js", "main.js"], "Full API spec": "", "Shared Knowledge": "\'game.js\' 包含游戏逻辑相关的函数，被 \'main.js\' 调用。", "Anything UNCLEAR": "目前项目要求明确，没有不清楚的地方。"}
## Code Files
----- index.html
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>2048游戏</title>
<link rel="stylesheet" href="styles.css">
<script src="https://cdn.jsdelivr.net/npm/vue@2.6.14/dist/vue.js"></script>
</head>
<body>
<div id="app">
<h1>2048</h1>
<div class="scores-container">
<div class="score-container">
<div class="score-header">分数</div>
<div>{{ score }}</div>
</div>
<div class="best-container">
<div class="best-header">最高分</div>
<div>{{ bestScore }}</div>
</div>
</div>
<div class="game-container">
<div v-for="(row, rowIndex) in board" :key="rowIndex" class="grid-row">
<div v-for="(cell, cellIndex) in row" :key="cellIndex" class="grid-cell" :class="\'number-cell-\' + cell">
{{ cell !== 0 ? cell : \'\' }}
</div>
</div>
</div>
<button @click="startGame" aria-label="开始新游戏">新游戏</button>
</div>
<script src="storage.js"></script>
<script src="game.js"></script>
<script src="main.js"></script>
<script src="app.js"></script>
</body>
</html>
----- styles.css
/* styles.css */
body, html {
margin: 0;
padding: 0;
font-family: \'Arial\', sans-serif;
}
#app {
text-align: center;
font-size: 18px;
color: #776e65;
}
h1 {
color: #776e65;
font-size: 72px;
font-weight: bold;
margin: 20px 0;
}
.scores-container {
display: flex;
justify-content: center;
margin-bottom: 20px;
}
.score-container, .best-container {
background: #bbada0;
padding: 10px;
border-radius: 5px;
margin: 0 10px;
min-width: 100px;
text-align: center;
}
.score-header, .best-header {
color: #eee4da;
font-size: 18px;
margin-bottom: 5px;
}
.game-container {
max-width: 500px;
margin: 0 auto 20px;
background: #bbada0;
padding: 15px;
border-radius: 10px;
position: relative;
}
.grid-row {
display: flex;
}
.grid-cell {
background: #cdc1b4;
width: 100px;
height: 100px;
margin: 5px;
display: flex;
justify-content: center;
align-items: center;
font-size: 35px;
font-weight: bold;
color: #776e65;
border-radius: 3px;
}
/* Dynamic classes for different number cells */
.number-cell-2 {
background: #eee4da;
}
.number-cell-4 {
background: #ede0c8;
}
.number-cell-8 {
background: #f2b179;
color: #f9f6f2;
}
.number-cell-16 {
background: #f59563;
color: #f9f6f2;
}
.number-cell-32 {
background: #f67c5f;
color: #f9f6f2;
}
.number-cell-64 {
background: #f65e3b;
color: #f9f6f2;
}
.number-cell-128 {
background: #edcf72;
color: #f9f6f2;
}
.number-cell-256 {
background: #edcc61;
color: #f9f6f2;
}
.number-cell-512 {
background: #edc850;
color: #f9f6f2;
}
.number-cell-1024 {
background: #edc53f;
color: #f9f6f2;
}
.number-cell-2048 {
background: #edc22e;
color: #f9f6f2;
}
/* Larger numbers need smaller font sizes */
.number-cell-1024, .number-cell-2048 {
font-size: 30px;
}
button {
background-color: #8f7a66;
color: #f9f6f2;
border: none;
border-radius: 3px;
padding: 10px 20px;
font-size: 18px;
cursor: pointer;
outline: none;
}
button:hover {
background-color: #9f8b76;
}
----- storage.js
## storage.js
class Storage {
// 获取最高分
getBestScore() {
// 尝试从localStorage中获取最高分，如果不存在则默认为0
const bestScore = localStorage.getItem(\'bestScore\');
return bestScore ? Number(bestScore) : 0;
}
// 设置最高分
setBestScore(score) {
// 将最高分设置到localStorage中
localStorage.setItem(\'bestScore\', score.toString());
}
}
## Code to be Reviewed: game.js
```Code
## game.js
class Game {
constructor() {
this.board = this.createEmptyBoard();
this.score = 0;
this.bestScore = 0;
}
createEmptyBoard() {
const board = [];
for (let i = 0; i < 4; i++) {
board[i] = [0, 0, 0, 0];
}
return board;
}
startGame() {
this.board = this.createEmptyBoard();
this.score = 0;
this.addRandomTile();
this.addRandomTile();
}
addRandomTile() {
let emptyCells = [];
for (let r = 0; r < 4; r++) {
for (let c = 0; c < 4; c++) {
if (this.board[r][c] === 0) {
emptyCells.push({ r, c });
}
}
}
if (emptyCells.length > 0) {
let randomCell = emptyCells[Math.floor(Math.random() * emptyCells.length)];
this.board[randomCell.r][randomCell.c] = Math.random() < 0.9 ? 2 : 4;
}
}
move(direction) {
// This function will handle the logic for moving tiles
// in the specified direction and merging them
// It will also update the score and add a new random tile if the move is successful
// The actual implementation of this function is complex and would require
// a significant amount of code to handle all the cases for moving and merging tiles
// For the purposes of this example, we will not implement the full logic
// Instead, we will just call addRandomTile to simulate a move
this.addRandomTile();
}
getBoard() {
return this.board;
}
getScore() {
return this.score;
}
getBestScore() {
return this.bestScore;
}
setBestScore(score) {
this.bestScore = score;
}
}
```
"""
CODE_REVIEW_SMALLEST_CONTEXT = """
## Code to be Reviewed: game.js
```Code
// game.js
class Game {
constructor() {
this.board = this.createEmptyBoard();
this.score = 0;
this.bestScore = 0;
}
createEmptyBoard() {
const board = [];
for (let i = 0; i < 4; i++) {
board[i] = [0, 0, 0, 0];
}
return board;
}
startGame() {
this.board = this.createEmptyBoard();
this.score = 0;
this.addRandomTile();
this.addRandomTile();
}
addRandomTile() {
let emptyCells = [];
for (let r = 0; r < 4; r++) {
for (let c = 0; c < 4; c++) {
if (this.board[r][c] === 0) {
emptyCells.push({ r, c });
}
}
}
if (emptyCells.length > 0) {
let randomCell = emptyCells[Math.floor(Math.random() * emptyCells.length)];
this.board[randomCell.r][randomCell.c] = Math.random() < 0.9 ? 2 : 4;
}
}
move(direction) {
// This function will handle the logic for moving tiles
// in the specified direction and merging them
// It will also update the score and add a new random tile if the move is successful
// The actual implementation of this function is complex and would require
// a significant amount of code to handle all the cases for moving and merging tiles
// For the purposes of this example, we will not implement the full logic
// Instead, we will just call addRandomTile to simulate a move
this.addRandomTile();
}
getBoard() {
return this.board;
}
getScore() {
return this.score;
}
getBestScore() {
return this.bestScore;
}
setBestScore(score) {
this.bestScore = score;
}
}
```
"""
CODE_REVIEW_SAMPLE = """
## Code Review: game.js
1. The code partially implements the requirements. The `Game` class is missing the full implementation of the `move` method, which is crucial for the game\'s functionality.
2. The code logic is not completely correct. The `move` method is not implemented, which means the game cannot process player moves.
3. The existing code follows the "Data structures and interfaces" in terms of class structure but lacks full method implementations.
4. Not all functions are implemented. The `move` method is incomplete and does not handle the logic for moving and merging tiles.
5. All necessary pre-dependencies seem to be imported since the code does not indicate the need for additional imports.
6. The methods from other files (such as `Storage`) are not being used in the provided code snippet, but the class structure suggests that they will be used correctly.
## Actions
1. Implement the `move` method to handle tile movements and merging. This is a complex task that requires careful consideration of the game\'s rules and logic. Here is a simplified version of how one might begin to implement the `move` method:
```javascript
move(direction) {
// Simplified logic for moving tiles up
if (direction === \'up\') {
for (let col = 0; col < 4; col++) {
let tiles = this.board.map(row => row[col]).filter(val => val !== 0);
let merged = [];
for (let i = 0; i < tiles.length; i++) {
if (tiles[i] === tiles[i + 1]) {
tiles[i] *= 2;
this.score += tiles[i];
tiles[i + 1] = 0;
merged.push(i);
}
}
tiles = tiles.filter(val => val !== 0);
while (tiles.length < 4) {
tiles.push(0);
}
for (let row = 0; row < 4; row++) {
this.board[row][col] = tiles[row];
}
}
}
// Additional logic needed for \'down\', \'left\', \'right\'
// ...
this.addRandomTile();
}
```
2. Integrate the `Storage` class methods to handle the best score. This means updating the `startGame` and `setBestScore` methods to use `Storage` for retrieving and setting the best score:
```javascript
startGame() {
this.board = this.createEmptyBoard();
this.score = 0;
this.bestScore = new Storage().getBestScore(); // Retrieve the best score from storage
this.addRandomTile();
this.addRandomTile();
}
setBestScore(score) {
if (score > this.bestScore) {
this.bestScore = score;
new Storage().setBestScore(score); // Set the new best score in storage
}
}
```
## Code Review Result
LBTM
```
"""
WRITE_CODE_NODE = ActionNode.from_children("WRITE_REVIEW_NODE", [REVIEW, LGTM, ACTIONS])
WRITE_MOVE_NODE = ActionNode.from_children("WRITE_MOVE_NODE", [WRITE_DRAFT, WRITE_MOVE_FUNCTION])
CR_FOR_MOVE_FUNCTION_BY_3 = """
The move function implementation provided appears to be well-structured and follows a clear logic for moving and merging tiles in the specified direction. However, there are a few potential improvements that could be made to enhance the code:
1. Encapsulation: The logic for moving and merging tiles could be encapsulated into smaller, reusable functions to improve readability and maintainability.
2. Magic Numbers: There are some magic numbers (e.g., 4, 3) used in the loops that could be replaced with named constants for improved readability and easier maintenance.
3. Comments: Adding comments to explain the logic and purpose of each section of the code can improve understanding for future developers who may need to work on or maintain the code.
4. Error Handling: It's important to consider error handling for unexpected input or edge cases to ensure the function behaves as expected in all scenarios.
Overall, the code could benefit from refactoring to improve readability, maintainability, and extensibility. If you would like, I can provide a refactored version of the move function that addresses these considerations.
"""
class WriteCodeAN(Action):
"""Write a code review for the context."""
async def run(self, context):
self.llm.system_prompt = "You are an outstanding engineer and can implement any code"
return await WRITE_MOVE_FUNCTION.fill(context=context, llm=self.llm, schema="json")
# return await WRITE_CODE_NODE.fill(context=context, llm=self.llm, schema="markdown")
async def main():
await WriteCodeAN().run(CODE_REVIEW_SMALLEST_CONTEXT)
if __name__ == "__main__":
asyncio.run(main())

174
metagpt/actions/write_code_review.py
View file

@ -4,57 +4,114 @@
@Time : 2023/5/11 17:45
@Author : alexanderwu
@File : write_code_review.py
@Modified By: mashenquan, 2023/11/27. Following the think-act principle, solidify the task parameters when creating the
WriteCode object, rather than passing them in when calling the run function.
"""
from pydantic import Field
from tenacity import retry, stop_after_attempt, wait_random_exponential
from metagpt.actions import WriteCode
from metagpt.actions.action import Action
from metagpt.config import CONFIG
from metagpt.logs import logger
from metagpt.schema import Message
from metagpt.schema import CodingContext
from metagpt.utils.common import CodeParser
from tenacity import retry, stop_after_attempt, wait_fixed
PROMPT_TEMPLATE = """
NOTICE
Role: You are a professional software engineer, and your main task is to review the code. You need to ensure that the code conforms to the PEP8 standards, is elegantly designed and modularized, easy to read and maintain, and is written in Python 3.9 (or in another programming language).
# System
Role: You are a professional software engineer, and your main task is to review and revise the code. You need to ensure that the code conforms to the google-style standards, is elegantly designed and modularized, easy to read and maintain.
Language: Please use the same language as the user requirement, but the title and code should be still in English. For example, if the user speaks Chinese, the specific text of your answer should also be in Chinese.
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
## Code Review: Based on the following context and code, and following the check list, Provide key, clear, concise, and specific code modification suggestions, up to 5.
```
1. Check 0: Is the code implemented as per the requirements?
2. Check 1: Are there any issues with the code logic?
3. Check 2: Does the existing code follow the "Data structures and interface definitions"?
4. Check 3: Is there a function in the code that is omitted or not fully implemented that needs to be implemented?
5. Check 4: Does the code have unnecessary or lack dependencies?
```
## Rewrite Code: {filename} Base on "Code Review" and the source code, rewrite code with triple quotes. Do your utmost to optimize THIS SINGLE FILE.
-----
# Context
{context}
## Code: {filename}
```
## Code to be Reviewed: {filename}
```Code
{code}
```
-----
"""
EXAMPLE_AND_INSTRUCTION = """
## Format example
-----
{format_example}
-----
# Instruction: Based on the actual code situation, follow one of the "Format example". Return only 1 file under review.
## Code Review: Ordered List. Based on the "Code to be Reviewed", provide key, clear, concise, and specific answer. If any answer is no, explain how to fix it step by step.
1. Is the code implemented as per the requirements? If not, how to achieve it? Analyse it step by step.
2. Is the code logic completely correct? If there are errors, please indicate how to correct them.
3. Does the existing code follow the "Data structures and interfaces"?
4. Are all functions implemented? If there is no implementation, please indicate how to achieve it step by step.
5. Have all necessary pre-dependencies been imported? If not, indicate which ones need to be imported
6. Are methods from other files being reused correctly?
## Actions: Ordered List. Things that should be done after CR, such as implementing class A and function B
## Code Review Result: str. If the code doesn't have bugs, we don't need to rewrite it, so answer LGTM and stop. ONLY ANSWER LGTM/LBTM.
LGTM/LBTM
"""
FORMAT_EXAMPLE = """
## Code Review
1. The code ...
# Format example 1
## Code Review: {filename}
1. No, we should fix the logic of class A due to ...
2. ...
3. ...
4. ...
4. No, function B is not implemented, ...
5. ...
6. ...
## Rewrite Code: {filename}
```python
## Actions
1. Fix the `handle_events` method to update the game state only if a move is successful.
```python
def handle_events(self):
for event in pygame.event.get():
if event.type == pygame.QUIT:
return False
if event.type == pygame.KEYDOWN:
moved = False
if event.key == pygame.K_UP:
moved = self.game.move('UP')
elif event.key == pygame.K_DOWN:
moved = self.game.move('DOWN')
elif event.key == pygame.K_LEFT:
moved = self.game.move('LEFT')
elif event.key == pygame.K_RIGHT:
moved = self.game.move('RIGHT')
if moved:
# Update the game state only if a move was successful
self.render()
return True
```
2. Implement function B
## Code Review Result
LBTM
# Format example 2
## Code Review: {filename}
1. Yes.
2. Yes.
3. Yes.
4. Yes.
5. Yes.
6. Yes.
## Actions
pass
## Code Review Result
LGTM
"""
REWRITE_CODE_TEMPLATE = """
# Instruction: rewrite code based on the Code Review and Actions
## Rewrite Code: CodeBlock. If it still has some bugs, rewrite {filename} with triple quotes. Do your utmost to optimize THIS SINGLE FILE. Return all completed codes and prohibit the return of unfinished codes.
```Code
## {filename}
...
```
@ -62,21 +119,58 @@ FORMAT_EXAMPLE = """
class WriteCodeReview(Action):
def __init__(self, name="WriteCodeReview", context: list[Message] = None, llm=None):
super().__init__(name, context, llm)
name: str = "WriteCodeReview"
context: CodingContext = Field(default_factory=CodingContext)
@retry(stop=stop_after_attempt(2), wait=wait_fixed(1))
async def write_code(self, prompt):
code_rsp = await self._aask(prompt)
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
async def write_code_review_and_rewrite(self, context_prompt, cr_prompt, filename):
cr_rsp = await self._aask(context_prompt + cr_prompt)
result = CodeParser.parse_block("Code Review Result", cr_rsp)
if "LGTM" in result:
return result, None
# if LBTM, rewrite code
rewrite_prompt = f"{context_prompt}\n{cr_rsp}\n{REWRITE_CODE_TEMPLATE.format(filename=filename)}"
code_rsp = await self._aask(rewrite_prompt)
code = CodeParser.parse_code(block="", text=code_rsp)
return code
return result, code
async def run(self, context, code, filename):
format_example = FORMAT_EXAMPLE.format(filename=filename)
prompt = PROMPT_TEMPLATE.format(context=context, code=code, filename=filename, format_example=format_example)
logger.info(f'Code review {filename}..')
code = await self.write_code(prompt)
async def run(self, *args, **kwargs) -> CodingContext:
iterative_code = self.context.code_doc.content
k = CONFIG.code_review_k_times or 1
for i in range(k):
format_example = FORMAT_EXAMPLE.format(filename=self.context.code_doc.filename)
task_content = self.context.task_doc.content if self.context.task_doc else ""
code_context = await WriteCode.get_codes(self.context.task_doc, exclude=self.context.filename)
context = "\n".join(
[
"## System Design\n" + str(self.context.design_doc) + "\n",
"## Tasks\n" + task_content + "\n",
"## Code Files\n" + code_context + "\n",
]
)
context_prompt = PROMPT_TEMPLATE.format(
context=context,
code=iterative_code,
filename=self.context.code_doc.filename,
)
cr_prompt = EXAMPLE_AND_INSTRUCTION.format(
format_example=format_example,
)
logger.info(
f"Code review and rewrite {self.context.code_doc.filename}: {i + 1}/{k} | {len(iterative_code)=}, "
f"{len(self.context.code_doc.content)=}"
)
result, rewrited_code = await self.write_code_review_and_rewrite(
context_prompt, cr_prompt, self.context.code_doc.filename
)
if "LBTM" in result:
iterative_code = rewrited_code
elif "LGTM" in result:
self.context.code_doc.content = iterative_code
return self.context
# code_rsp = await self._aask_v1(prompt, "code_rsp", OUTPUT_MAPPING)
# self._save(context, filename, code)
return code
# 如果rewrited_code是None（原code perfect），那么直接返回code
self.context.code_doc.content = iterative_code
return self.context

42
metagpt/actions/write_docstring.py
View file

@ -16,19 +16,22 @@ Options:
Default: 'google'
Example:
python3 -m metagpt.actions.write_docstring startup.py --overwrite False --style=numpy
python3 -m metagpt.actions.write_docstring ./metagpt/startup.py --overwrite False --style=numpy
This script uses the 'fire' library to create a command-line interface. It generates docstrings for the given Python code using
the specified docstring style and adds them to the code.
"""
from __future__ import annotations
import ast
from typing import Literal
from pathlib import Path
from typing import Literal, Optional
from metagpt.actions.action import Action
from metagpt.utils.common import OutputParser
from metagpt.utils.common import OutputParser, aread, awrite
from metagpt.utils.pycst import merge_docstring
PYTHON_DOCSTRING_SYSTEM = '''### Requirements
PYTHON_DOCSTRING_SYSTEM = """### Requirements
1. Add docstrings to the given code following the {style} style.
2. Replace the function body with an Ellipsis object(...) to reduce output.
3. If the types are already annotated, there is no need to include them in the docstring.
@ -48,7 +51,7 @@ class ExampleError(Exception):
```python
{example}
```
'''
"""
# https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html
@ -157,12 +160,12 @@ class WriteDocstring(Action):
desc: A string describing the action.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.desc = "Write docstring for code."
desc: str = "Write docstring for code."
context: Optional[str] = None
async def run(
self, code: str,
self,
code: str,
system_text: str = PYTHON_DOCSTRING_SYSTEM,
style: Literal["google", "numpy", "sphinx"] = "google",
) -> str:
@ -182,6 +185,16 @@ class WriteDocstring(Action):
documented_code = OutputParser.parse_python_code(documented_code)
return merge_docstring(code, documented_code)
@staticmethod
async def write_docstring(
filename: str | Path, overwrite: bool = False, style: Literal["google", "numpy", "sphinx"] = "google"
) -> str:
data = await aread(str(filename))
code = await WriteDocstring().run(data, style=style)
if overwrite:
await awrite(filename, code)
return code
def _simplify_python_code(code: str) -> None:
"""Simplifies the given Python code by removing expressions and the last if statement.
@ -202,13 +215,4 @@ def _simplify_python_code(code: str) -> None:
if __name__ == "__main__":
import fire
async def run(filename: str, overwrite: bool = False, style: Literal["google", "numpy", "sphinx"] = "google"):
with open(filename) as f:
code = f.read()
code = await WriteDocstring().run(code, style=style)
if overwrite:
with open(filename, "w") as f:
f.write(code)
return code
fire.Fire(run)
fire.Fire(WriteDocstring.write_docstring)

392
metagpt/actions/write_prd.py
View file

@ -4,238 +4,194 @@
@Time : 2023/5/11 17:45
@Author : alexanderwu
@File : write_prd.py
@Modified By: mashenquan, 2023/11/27.
1. According to Section 2.2.3.1 of RFC 135, replace file data in the message with the file name.
2. According to the design in Section 2.2.3.5.2 of RFC 135, add incremental iteration functionality.
3. Move the document storage operations related to WritePRD from the save operation of WriteDesign.
@Modified By: mashenquan, 2023/12/5. Move the generation logic of the project name to WritePRD.
"""
from typing import List
from __future__ import annotations
import json
import uuid
from pathlib import Path
from typing import Optional
from metagpt.actions import Action, ActionOutput
from metagpt.actions.search_and_summarize import SearchAndSummarize
from metagpt.actions.action_node import ActionNode
from metagpt.actions.fix_bug import FixBug
from metagpt.actions.write_prd_an import (
PROJECT_NAME,
WP_IS_RELATIVE_NODE,
WP_ISSUE_TYPE_NODE,
WRITE_PRD_NODE,
)
from metagpt.config import CONFIG
from metagpt.const import (
BUGFIX_FILENAME,
COMPETITIVE_ANALYSIS_FILE_REPO,
DOCS_FILE_REPO,
PRD_PDF_FILE_REPO,
PRDS_FILE_REPO,
REQUIREMENT_FILENAME,
)
from metagpt.logs import logger
from metagpt.utils.get_template import get_template
from metagpt.schema import BugFixContext, Document, Documents, Message
from metagpt.utils.common import CodeParser
from metagpt.utils.file_repository import FileRepository
from metagpt.utils.mermaid import mermaid_to_file
templates = {
"json": {
"PROMPT_TEMPLATE": """
# Context
## Original Requirements
CONTEXT_TEMPLATE = """
### Project Name
{project_name}
### Original Requirements
{requirements}
## Search Information
{search_information}
### Search Information
-
"""
## mermaid quadrantChart code syntax example. DONT USE QUOTO IN CODE DUE TO INVALID SYNTAX. Replace the <Campain X> with REAL COMPETITOR NAME
```mermaid
quadrantChart
title Reach and engagement of campaigns
x-axis Low Reach --> High Reach
y-axis Low Engagement --> High Engagement
quadrant-1 We should expand
quadrant-2 Need to promote
quadrant-3 Re-evaluate
quadrant-4 May be improved
"Campaign: A": [0.3, 0.6]
"Campaign B": [0.45, 0.23]
"Campaign C": [0.57, 0.69]
"Campaign D": [0.78, 0.34]
"Campaign E": [0.40, 0.34]
"Campaign F": [0.35, 0.78]
"Our Target Product": [0.5, 0.6]
```
NEW_REQ_TEMPLATE = """
### Legacy Content
{old_prd}
## Format example
{format_example}
-----
Role: You are a professional product manager; the goal is to design a concise, usable, efficient product
Requirements: According to the context, fill in the following missing information, each section name is a key in json ,If the requirements are unclear, ensure minimum viability and avoid excessive design
## Original Requirements: Provide as Plain text, place the polished complete original requirements here
## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals. If the requirement itself is simple, the goal should also be simple
## User Stories: Provided as Python list[str], up to 5 scenario-based user stories, If the requirement itself is simple, the user stories should also be less
## Competitive Analysis: Provided as Python list[str], up to 7 competitive product analyses, consider as similar competitors as possible
## Competitive Quadrant Chart: Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
## Requirement Analysis: Provide as Plain text. Be simple. LESS IS MORE. Make your requirements less dumb. Delete the parts unnessasery.
## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards; no more than 5 requirements and consider to make its difficulty lower
## UI Design draft: Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
## Anything UNCLEAR: Provide as Plain text. Make clear here.
output a properly formatted JSON, wrapped inside [CONTENT][/CONTENT] like format example,
and only output the json inside this tag, nothing else
""",
"FORMAT_EXAMPLE": """
[CONTENT]
{
"Original Requirements": "",
"Search Information": "",
"Requirements": "",
"Product Goals": [],
"User Stories": [],
"Competitive Analysis": [],
"Competitive Quadrant Chart": "quadrantChart
title Reach and engagement of campaigns
x-axis Low Reach --> High Reach
y-axis Low Engagement --> High Engagement
quadrant-1 We should expand
quadrant-2 Need to promote
quadrant-3 Re-evaluate
quadrant-4 May be improved
Campaign A: [0.3, 0.6]
Campaign B: [0.45, 0.23]
Campaign C: [0.57, 0.69]
Campaign D: [0.78, 0.34]
Campaign E: [0.40, 0.34]
Campaign F: [0.35, 0.78]",
"Requirement Analysis": "",
"Requirement Pool": [["P0","P0 requirement"],["P1","P1 requirement"]],
"UI Design draft": "",
"Anything UNCLEAR": "",
}
[/CONTENT]
""",
},
"markdown": {
"PROMPT_TEMPLATE": """
# Context
## Original Requirements
### New Requirements
{requirements}
## Search Information
{search_information}
## mermaid quadrantChart code syntax example. DONT USE QUOTO IN CODE DUE TO INVALID SYNTAX. Replace the <Campain X> with REAL COMPETITOR NAME
```mermaid
quadrantChart
title Reach and engagement of campaigns
x-axis Low Reach --> High Reach
y-axis Low Engagement --> High Engagement
quadrant-1 We should expand
quadrant-2 Need to promote
quadrant-3 Re-evaluate
quadrant-4 May be improved
"Campaign: A": [0.3, 0.6]
"Campaign B": [0.45, 0.23]
"Campaign C": [0.57, 0.69]
"Campaign D": [0.78, 0.34]
"Campaign E": [0.40, 0.34]
"Campaign F": [0.35, 0.78]
"Our Target Product": [0.5, 0.6]
```
## Format example
{format_example}
-----
Role: You are a professional product manager; the goal is to design a concise, usable, efficient product
Requirements: According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly. If the requirements are unclear, ensure minimum viability and avoid excessive design
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. AND '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote. Output carefully referenced "Format example" in format.
## Original Requirements: Provide as Plain text, place the polished complete original requirements here
## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals. If the requirement itself is simple, the goal should also be simple
## User Stories: Provided as Python list[str], up to 5 scenario-based user stories, If the requirement itself is simple, the user stories should also be less
## Competitive Analysis: Provided as Python list[str], up to 7 competitive product analyses, consider as similar competitors as possible
## Competitive Quadrant Chart: Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
## Requirement Analysis: Provide as Plain text. Be simple. LESS IS MORE. Make your requirements less dumb. Delete the parts unnessasery.
## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards; no more than 5 requirements and consider to make its difficulty lower
## UI Design draft: Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
## Anything UNCLEAR: Provide as Plain text. Make clear here.
""",
"FORMAT_EXAMPLE": """
---
## Original Requirements
The boss ...
## Product Goals
```python
[
"Create a ...",
]
```
## User Stories
```python
[
"As a user, ...",
]
```
## Competitive Analysis
```python
[
"Python Snake Game: ...",
]
```
## Competitive Quadrant Chart
```mermaid
quadrantChart
title Reach and engagement of campaigns
...
"Our Target Product": [0.6, 0.7]
```
## Requirement Analysis
The product should be a ...
## Requirement Pool
```python
[
["End game ...", "P0"]
]
```
## UI Design draft
Give a basic function description, and a draft
## Anything UNCLEAR
There are no unclear points.
---
""",
},
}
OUTPUT_MAPPING = {
"Original Requirements": (str, ...),
"Product Goals": (List[str], ...),
"User Stories": (List[str], ...),
"Competitive Analysis": (List[str], ...),
"Competitive Quadrant Chart": (str, ...),
"Requirement Analysis": (str, ...),
"Requirement Pool": (List[List[str]], ...),
"UI Design draft": (str, ...),
"Anything UNCLEAR": (str, ...),
}
"""
class WritePRD(Action):
def __init__(self, name="", context=None, llm=None):
super().__init__(name, context, llm)
name: str = "WritePRD"
content: Optional[str] = None
async def run(self, requirements, format=CONFIG.prompt_format, *args, **kwargs) -> ActionOutput:
sas = SearchAndSummarize()
# rsp = await sas.run(context=requirements, system_text=SEARCH_AND_SUMMARIZE_SYSTEM_EN_US)
rsp = ""
info = f"### Search Results\n{sas.result}\n\n### Search Summary\n{rsp}"
if sas.result:
logger.info(sas.result)
logger.info(rsp)
async def run(self, with_messages, schema=CONFIG.prompt_schema, *args, **kwargs) -> ActionOutput | Message:
# Determine which requirement documents need to be rewritten: Use LLM to assess whether new requirements are
# related to the PRD. If they are related, rewrite the PRD.
docs_file_repo = CONFIG.git_repo.new_file_repository(relative_path=DOCS_FILE_REPO)
requirement_doc = await docs_file_repo.get(filename=REQUIREMENT_FILENAME)
if requirement_doc and await self._is_bugfix(requirement_doc.content):
await docs_file_repo.save(filename=BUGFIX_FILENAME, content=requirement_doc.content)
await docs_file_repo.save(filename=REQUIREMENT_FILENAME, content="")
bug_fix = BugFixContext(filename=BUGFIX_FILENAME)
return Message(
content=bug_fix.model_dump_json(),
instruct_content=bug_fix,
role="",
cause_by=FixBug,
sent_from=self,
send_to="Alex", # the name of Engineer
)
else:
await docs_file_repo.delete(filename=BUGFIX_FILENAME)
prompt_template, format_example = get_template(templates, format)
prompt = prompt_template.format(
requirements=requirements, search_information=info, format_example=format_example
prds_file_repo = CONFIG.git_repo.new_file_repository(PRDS_FILE_REPO)
prd_docs = await prds_file_repo.get_all()
change_files = Documents()
for prd_doc in prd_docs:
prd_doc = await self._update_prd(
requirement_doc=requirement_doc, prd_doc=prd_doc, prds_file_repo=prds_file_repo, *args, **kwargs
)
if not prd_doc:
continue
change_files.docs[prd_doc.filename] = prd_doc
logger.info(f"rewrite prd: {prd_doc.filename}")
# If there is no existing PRD, generate one using 'docs/requirement.txt'.
if not change_files.docs:
prd_doc = await self._update_prd(
requirement_doc=requirement_doc, prd_doc=None, prds_file_repo=prds_file_repo, *args, **kwargs
)
if prd_doc:
change_files.docs[prd_doc.filename] = prd_doc
logger.debug(f"new prd: {prd_doc.filename}")
# Once all files under 'docs/prds/' have been compared with the newly added requirements, trigger the
# 'publish' message to transition the workflow to the next stage. This design allows room for global
# optimization in subsequent steps.
return ActionOutput(content=change_files.model_dump_json(), instruct_content=change_files)
async def _run_new_requirement(self, requirements, schema=CONFIG.prompt_schema) -> ActionOutput:
# sas = SearchAndSummarize()
# # rsp = await sas.run(context=requirements, system_text=SEARCH_AND_SUMMARIZE_SYSTEM_EN_US)
# rsp = ""
# info = f"### Search Results\n{sas.result}\n\n### Search Summary\n{rsp}"
# if sas.result:
# logger.info(sas.result)
# logger.info(rsp)
project_name = CONFIG.project_name or ""
context = CONTEXT_TEMPLATE.format(requirements=requirements, project_name=project_name)
exclude = [PROJECT_NAME.key] if project_name else []
node = await WRITE_PRD_NODE.fill(context=context, llm=self.llm, exclude=exclude) # schema=schema
await self._rename_workspace(node)
return node
async def _is_relative(self, new_requirement_doc, old_prd_doc) -> bool:
context = NEW_REQ_TEMPLATE.format(old_prd=old_prd_doc.content, requirements=new_requirement_doc.content)
node = await WP_IS_RELATIVE_NODE.fill(context, self.llm)
return node.get("is_relative") == "YES"
async def _merge(self, new_requirement_doc, prd_doc, schema=CONFIG.prompt_schema) -> Document:
if not CONFIG.project_name:
CONFIG.project_name = Path(CONFIG.project_path).name
prompt = NEW_REQ_TEMPLATE.format(requirements=new_requirement_doc.content, old_prd=prd_doc.content)
node = await WRITE_PRD_NODE.fill(context=prompt, llm=self.llm, schema=schema)
prd_doc.content = node.instruct_content.model_dump_json()
await self._rename_workspace(node)
return prd_doc
async def _update_prd(self, requirement_doc, prd_doc, prds_file_repo, *args, **kwargs) -> Document | None:
if not prd_doc:
prd = await self._run_new_requirement(
requirements=[requirement_doc.content if requirement_doc else ""], *args, **kwargs
)
new_prd_doc = Document(
root_path=PRDS_FILE_REPO,
filename=FileRepository.new_filename() + ".json",
content=prd.instruct_content.model_dump_json(),
)
elif await self._is_relative(requirement_doc, prd_doc):
new_prd_doc = await self._merge(requirement_doc, prd_doc)
else:
return None
await prds_file_repo.save(filename=new_prd_doc.filename, content=new_prd_doc.content)
await self._save_competitive_analysis(new_prd_doc)
await self._save_pdf(new_prd_doc)
return new_prd_doc
@staticmethod
async def _save_competitive_analysis(prd_doc):
m = json.loads(prd_doc.content)
quadrant_chart = m.get("Competitive Quadrant Chart")
if not quadrant_chart:
return
pathname = (
CONFIG.git_repo.workdir / Path(COMPETITIVE_ANALYSIS_FILE_REPO) / Path(prd_doc.filename).with_suffix("")
)
logger.debug(prompt)
# prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING)
prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING, format=format)
return prd
if not pathname.parent.exists():
pathname.parent.mkdir(parents=True, exist_ok=True)
await mermaid_to_file(quadrant_chart, pathname)
@staticmethod
async def _save_pdf(prd_doc):
await FileRepository.save_as(doc=prd_doc, with_suffix=".md", relative_path=PRD_PDF_FILE_REPO)
@staticmethod
async def _rename_workspace(prd):
if not CONFIG.project_name:
if isinstance(prd, (ActionOutput, ActionNode)):
ws_name = prd.instruct_content.model_dump()["Project Name"]
else:
ws_name = CodeParser.parse_str(block="Project Name", text=prd)
if ws_name:
CONFIG.project_name = ws_name
if not CONFIG.project_name: # The LLM failed to provide a project name, and the user didn't provide one either.
CONFIG.project_name = "app" + uuid.uuid4().hex[:16]
CONFIG.git_repo.rename_root(CONFIG.project_name)
async def _is_bugfix(self, context) -> bool:
src_workspace_path = CONFIG.git_repo.workdir / CONFIG.git_repo.workdir.name
code_files = CONFIG.git_repo.get_files(relative_path=src_workspace_path)
if not code_files:
return False
node = await WP_ISSUE_TYPE_NODE.fill(context, self.llm)
return node.get("issue_type") == "BUG"

166
metagpt/actions/write_prd_an.py Normal file
View file

@ -0,0 +1,166 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/12/14 11:40
@Author : alexanderwu
@File : write_prd_an.py
"""
from typing import List
from metagpt.actions.action_node import ActionNode
from metagpt.logs import logger
LANGUAGE = ActionNode(
key="Language",
expected_type=str,
instruction="Provide the language used in the project, typically matching the user's requirement language.",
example="en_us",
)
PROGRAMMING_LANGUAGE = ActionNode(
key="Programming Language",
expected_type=str,
instruction="Python/JavaScript or other mainstream programming language.",
example="Python",
)
ORIGINAL_REQUIREMENTS = ActionNode(
key="Original Requirements",
expected_type=str,
instruction="Place the original user's requirements here.",
example="Create a 2048 game",
)
PROJECT_NAME = ActionNode(
key="Project Name",
expected_type=str,
instruction="According to the content of \"Original Requirements,\" name the project using snake case style , like 'game_2048' or 'simple_crm.",
example="game_2048",
)
PRODUCT_GOALS = ActionNode(
key="Product Goals",
expected_type=List[str],
instruction="Provide up to three clear, orthogonal product goals.",
example=["Create an engaging user experience", "Improve accessibility, be responsive", "More beautiful UI"],
)
USER_STORIES = ActionNode(
key="User Stories",
expected_type=List[str],
instruction="Provide up to 3 to 5 scenario-based user stories.",
example=[
"As a player, I want to be able to choose difficulty levels",
"As a player, I want to see my score after each game",
"As a player, I want to get restart button when I lose",
"As a player, I want to see beautiful UI that make me feel good",
"As a player, I want to play game via mobile phone",
],
)
COMPETITIVE_ANALYSIS = ActionNode(
key="Competitive Analysis",
expected_type=List[str],
instruction="Provide 5 to 7 competitive products.",
example=[
"2048 Game A: Simple interface, lacks responsive features",
"play2048.co: Beautiful and responsive UI with my best score shown",
"2048game.com: Responsive UI with my best score shown, but many ads",
],
)
COMPETITIVE_QUADRANT_CHART = ActionNode(
key="Competitive Quadrant Chart",
expected_type=str,
instruction="Use mermaid quadrantChart syntax. Distribute scores evenly between 0 and 1",
example="""quadrantChart
title "Reach and engagement of campaigns"
x-axis "Low Reach" --> "High Reach"
y-axis "Low Engagement" --> "High Engagement"
quadrant-1 "We should expand"
quadrant-2 "Need to promote"
quadrant-3 "Re-evaluate"
quadrant-4 "May be improved"
"Campaign A": [0.3, 0.6]
"Campaign B": [0.45, 0.23]
"Campaign C": [0.57, 0.69]
"Campaign D": [0.78, 0.34]
"Campaign E": [0.40, 0.34]
"Campaign F": [0.35, 0.78]
"Our Target Product": [0.5, 0.6]""",
)
REQUIREMENT_ANALYSIS = ActionNode(
key="Requirement Analysis",
expected_type=str,
instruction="Provide a detailed analysis of the requirements.",
example="",
)
REQUIREMENT_POOL = ActionNode(
key="Requirement Pool",
expected_type=List[List[str]],
instruction="List down the top-5 requirements with their priority (P0, P1, P2).",
example=[["P0", "The main code ..."], ["P0", "The game algorithm ..."]],
)
UI_DESIGN_DRAFT = ActionNode(
key="UI Design draft",
expected_type=str,
instruction="Provide a simple description of UI elements, functions, style, and layout.",
example="Basic function description with a simple style and layout.",
)
ANYTHING_UNCLEAR = ActionNode(
key="Anything UNCLEAR",
expected_type=str,
instruction="Mention any aspects of the project that are unclear and try to clarify them.",
example="",
)
ISSUE_TYPE = ActionNode(
key="issue_type",
expected_type=str,
instruction="Answer BUG/REQUIREMENT. If it is a bugfix, answer BUG, otherwise answer Requirement",
example="BUG",
)
IS_RELATIVE = ActionNode(
key="is_relative",
expected_type=str,
instruction="Answer YES/NO. If the requirement is related to the old PRD, answer YES, otherwise NO",
example="YES",
)
REASON = ActionNode(
key="reason", expected_type=str, instruction="Explain the reasoning process from question to answer", example="..."
)
NODES = [
LANGUAGE,
PROGRAMMING_LANGUAGE,
ORIGINAL_REQUIREMENTS,
PROJECT_NAME,
PRODUCT_GOALS,
USER_STORIES,
COMPETITIVE_ANALYSIS,
COMPETITIVE_QUADRANT_CHART,
REQUIREMENT_ANALYSIS,
REQUIREMENT_POOL,
UI_DESIGN_DRAFT,
ANYTHING_UNCLEAR,
]
WRITE_PRD_NODE = ActionNode.from_children("WritePRD", NODES)
WP_ISSUE_TYPE_NODE = ActionNode.from_children("WP_ISSUE_TYPE", [ISSUE_TYPE, REASON])
WP_IS_RELATIVE_NODE = ActionNode.from_children("WP_IS_RELATIVE", [IS_RELATIVE, REASON])
def main():
prompt = WRITE_PRD_NODE.compile(context="")
logger.info(prompt)
if __name__ == "__main__":
main()

23
metagpt/actions/write_prd_review.py
View file

@ -5,24 +5,27 @@
@Author : alexanderwu
@File : write_prd_review.py
"""
from typing import Optional
from metagpt.actions.action import Action
class WritePRDReview(Action):
def __init__(self, name, context=None, llm=None):
super().__init__(name, context, llm)
self.prd = None
self.desc = "Based on the PRD, conduct a PRD Review, providing clear and detailed feedback"
self.prd_review_prompt_template = """
Given the following Product Requirement Document (PRD):
{prd}
name: str = ""
context: Optional[str] = None
As a project manager, please review it and provide your feedback and suggestions.
"""
prd: Optional[str] = None
desc: str = "Based on the PRD, conduct a PRD Review, providing clear and detailed feedback"
prd_review_prompt_template: str = """
Given the following Product Requirement Document (PRD):
{prd}
As a project manager, please review it and provide your feedback and suggestions.
"""
async def run(self, prd):
self.prd = prd
prompt = self.prd_review_prompt_template.format(prd=self.prd)
review = await self._aask(prompt)
return review

39
metagpt/actions/write_review.py Normal file
View file

@ -0,0 +1,39 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Author : alexanderwu
@File : write_review.py
"""
from typing import List
from metagpt.actions import Action
from metagpt.actions.action_node import ActionNode
REVIEW = ActionNode(
key="Review",
expected_type=List[str],
instruction="Act as an experienced Reviewer and review the given output. Ask a series of critical questions, "
"concisely and clearly, to help the writer improve their work.",
example=[
"This is a good PRD, but I think it can be improved by adding more details.",
],
)
LGTM = ActionNode(
key="LGTM",
expected_type=str,
instruction="LGTM/LBTM. If the output is good enough, give a LGTM (Looks Good To Me) to the writer, "
"else LBTM (Looks Bad To Me).",
example="LGTM",
)
WRITE_REVIEW_NODE = ActionNode.from_children("WRITE_REVIEW_NODE", [REVIEW, LGTM])
class WriteReview(Action):
"""Write a review for the given context."""
name: str = "WriteReview"
async def run(self, context):
return await WRITE_REVIEW_NODE.fill(context=context, llm=self.llm, schema="json")

188
metagpt/actions/write_teaching_plan.py Normal file
View file

@ -0,0 +1,188 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/7/27
@Author : mashenquan
@File : write_teaching_plan.py
"""
from typing import Optional
from metagpt.actions import Action
from metagpt.config import CONFIG
from metagpt.logs import logger
class WriteTeachingPlanPart(Action):
"""Write Teaching Plan Part"""
context: Optional[str] = None
topic: str = ""
language: str = "Chinese"
rsp: Optional[str] = None
async def run(self, with_message=None, **kwargs):
statement_patterns = TeachingPlanBlock.TOPIC_STATEMENTS.get(self.topic, [])
statements = []
for p in statement_patterns:
s = self.format_value(p)
statements.append(s)
formatter = (
TeachingPlanBlock.PROMPT_TITLE_TEMPLATE
if self.topic == TeachingPlanBlock.COURSE_TITLE
else TeachingPlanBlock.PROMPT_TEMPLATE
)
prompt = formatter.format(
formation=TeachingPlanBlock.FORMATION,
role=self.prefix,
statements="\n".join(statements),
lesson=self.context,
topic=self.topic,
language=self.language,
)
logger.debug(prompt)
rsp = await self._aask(prompt=prompt)
logger.debug(rsp)
self._set_result(rsp)
return self.rsp
def _set_result(self, rsp):
if TeachingPlanBlock.DATA_BEGIN_TAG in rsp:
ix = rsp.index(TeachingPlanBlock.DATA_BEGIN_TAG)
rsp = rsp[ix + len(TeachingPlanBlock.DATA_BEGIN_TAG) :]
if TeachingPlanBlock.DATA_END_TAG in rsp:
ix = rsp.index(TeachingPlanBlock.DATA_END_TAG)
rsp = rsp[0:ix]
self.rsp = rsp.strip()
if self.topic != TeachingPlanBlock.COURSE_TITLE:
return
if "#" not in self.rsp or self.rsp.index("#") != 0:
self.rsp = "# " + self.rsp
def __str__(self):
"""Return `topic` value when str()"""
return self.topic
def __repr__(self):
"""Show `topic` value when debug"""
return self.topic
@staticmethod
def format_value(value):
"""Fill parameters inside `value` with `options`."""
if not isinstance(value, str):
return value
if "{" not in value:
return value
merged_opts = CONFIG.options or {}
try:
return value.format(**merged_opts)
except KeyError as e:
logger.warning(f"Parameter is missing:{e}")
for k, v in merged_opts.items():
value = value.replace("{" + f"{k}" + "}", str(v))
return value
class TeachingPlanBlock:
FORMATION = (
'"Capacity and role" defines the role you are currently playing;\n'
'\t"[LESSON_BEGIN]" and "[LESSON_END]" tags enclose the content of textbook;\n'
'\t"Statement" defines the work detail you need to complete at this stage;\n'
'\t"Answer options" defines the format requirements for your responses;\n'
'\t"Constraint" defines the conditions that your responses must comply with.'
)
COURSE_TITLE = "Title"
TOPICS = [
COURSE_TITLE,
"Teaching Hours",
"Teaching Objectives",
"Teaching Content",
"Teaching Methods and Strategies",
"Learning Activities",
"Teaching Time Allocation",
"Assessment and Feedback",
"Teaching Summary and Improvement",
"Vocabulary Cloze",
"Choice Questions",
"Grammar Questions",
"Translation Questions",
]
TOPIC_STATEMENTS = {
COURSE_TITLE: [
"Statement: Find and return the title of the lesson only in markdown first-level header format, "
"without anything else."
],
"Teaching Content": [
'Statement: "Teaching Content" must include vocabulary, analysis, and examples of various grammar '
"structures that appear in the textbook, as well as the listening materials and key points.",
'Statement: "Teaching Content" must include more examples.',
],
"Teaching Time Allocation": [
'Statement: "Teaching Time Allocation" must include how much time is allocated to each '
"part of the textbook content."
],
"Teaching Methods and Strategies": [
'Statement: "Teaching Methods and Strategies" must include teaching focus, difficulties, materials, '
"procedures, in detail."
],
"Vocabulary Cloze": [
'Statement: Based on the content of the textbook enclosed by "[LESSON_BEGIN]" and "[LESSON_END]", '
"create vocabulary cloze. The cloze should include 10 {language} questions with {teaching_language} "
"answers, and it should also include 10 {teaching_language} questions with {language} answers. "
"The key-related vocabulary and phrases in the textbook content must all be included in the exercises.",
],
"Grammar Questions": [
'Statement: Based on the content of the textbook enclosed by "[LESSON_BEGIN]" and "[LESSON_END]", '
"create grammar questions. 10 questions."
],
"Choice Questions": [
'Statement: Based on the content of the textbook enclosed by "[LESSON_BEGIN]" and "[LESSON_END]", '
"create choice questions. 10 questions."
],
"Translation Questions": [
'Statement: Based on the content of the textbook enclosed by "[LESSON_BEGIN]" and "[LESSON_END]", '
"create translation questions. The translation should include 10 {language} questions with "
"{teaching_language} answers, and it should also include 10 {teaching_language} questions with "
"{language} answers."
],
}
# Teaching plan title
PROMPT_TITLE_TEMPLATE = (
"Do not refer to the context of the previous conversation records, "
"start the conversation anew.\n\n"
"Formation: {formation}\n\n"
"{statements}\n"
"Constraint: Writing in {language}.\n"
'Answer options: Encloses the lesson title with "[TEACHING_PLAN_BEGIN]" '
'and "[TEACHING_PLAN_END]" tags.\n'
"[LESSON_BEGIN]\n"
"{lesson}\n"
"[LESSON_END]"
)
# Teaching plan parts:
PROMPT_TEMPLATE = (
"Do not refer to the context of the previous conversation records, "
"start the conversation anew.\n\n"
"Formation: {formation}\n\n"
"Capacity and role: {role}\n"
'Statement: Write the "{topic}" part of teaching plan, '
'WITHOUT ANY content unrelated to "{topic}"!!\n'
"{statements}\n"
'Answer options: Enclose the teaching plan content with "[TEACHING_PLAN_BEGIN]" '
'and "[TEACHING_PLAN_END]" tags.\n'
"Answer options: Using proper markdown format from second-level header format.\n"
"Constraint: Writing in {language}.\n"
"[LESSON_BEGIN]\n"
"{lesson}\n"
"[LESSON_END]"
)
DATA_BEGIN_TAG = "[TEACHING_PLAN_BEGIN]"
DATA_END_TAG = "[TEACHING_PLAN_END]"

36
metagpt/actions/write_test.py
View file

@ -3,10 +3,17 @@
"""
@Time : 2023/5/11 22:12
@Author : alexanderwu
@File : environment.py
@File : write_test.py
@Modified By: mashenquan, 2023-11-27. Following the think-act principle, solidify the task parameters when creating the
WriteTest object, rather than passing them in when calling the run function.
"""
from typing import Optional
from metagpt.actions.action import Action
from metagpt.const import TEST_CODES_FILE_REPO
from metagpt.logs import logger
from metagpt.schema import Document, TestingContext
from metagpt.utils.common import CodeParser
PROMPT_TEMPLATE = """
@ -15,7 +22,7 @@ NOTICE
2. Requirement: Based on the context, develop a comprehensive test suite that adequately covers all relevant aspects of the code file under review. Your test suite will be part of the overall project QA, so please develop complete, robust, and reusable test cases.
3. Attention1: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the test case or script.
4. Attention2: If there are any settings in your tests, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
5. Attention3: YOU MUST FOLLOW "Data structures and interface definitions". DO NOT CHANGE ANY DESIGN. Make sure your tests respect the existing design and ensure its validity.
5. Attention3: YOU MUST FOLLOW "Data structures and interfaces". DO NOT CHANGE ANY DESIGN. Make sure your tests respect the existing design and ensure its validity.
6. Think before writing: What should be tested and validated in this document? What edge cases could exist? What might fail?
7. CAREFULLY CHECK THAT YOU DON'T MISS ANY NECESSARY TEST CASES/SCRIPTS IN THIS FILE.
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the test case or script and triple quotes.
@ -26,13 +33,13 @@ Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD W
```
Note that the code to test is at {source_file_path}, we will put your test code at {workspace}/tests/{test_file_name}, and run your test code from {workspace},
you should correctly import the necessary classes based on these file locations!
## {test_file_name}: Write test code with triple quoto. Do your best to implement THIS ONLY ONE FILE.
## {test_file_name}: Write test code with triple quote. Do your best to implement THIS ONLY ONE FILE.
"""
class WriteTest(Action):
def __init__(self, name="WriteTest", context=None, llm=None):
super().__init__(name, context, llm)
name: str = "WriteTest"
context: Optional[TestingContext] = None
async def write_code(self, prompt):
code_rsp = await self._aask(prompt)
@ -47,12 +54,17 @@ class WriteTest(Action):
code = code_rsp
return code
async def run(self, code_to_test, test_file_name, source_file_path, workspace):
async def run(self, *args, **kwargs) -> TestingContext:
if not self.context.test_doc:
self.context.test_doc = Document(
filename="test_" + self.context.code_doc.filename, root_path=TEST_CODES_FILE_REPO
)
fake_root = "/data"
prompt = PROMPT_TEMPLATE.format(
code_to_test=code_to_test,
test_file_name=test_file_name,
source_file_path=source_file_path,
workspace=workspace,
code_to_test=self.context.code_doc.content,
test_file_name=self.context.test_doc.filename,
source_file_path=fake_root + "/" + self.context.code_doc.root_relative_path,
workspace=fake_root,
)
code = await self.write_code(prompt)
return code
self.context.test_doc.content = await self.write_code(prompt)
return self.context

15
metagpt/actions/write_tutorial.py
View file

@ -10,7 +10,7 @@
from typing import Dict
from metagpt.actions import Action
from metagpt.prompts.tutorial_assistant import DIRECTORY_PROMPT, CONTENT_PROMPT
from metagpt.prompts.tutorial_assistant import CONTENT_PROMPT, DIRECTORY_PROMPT
from metagpt.utils.common import OutputParser
@ -22,9 +22,8 @@ class WriteDirectory(Action):
language: The language to output, default is "Chinese".
"""
def __init__(self, name: str = "", language: str = "Chinese", *args, **kwargs):
super().__init__(name, *args, **kwargs)
self.language = language
name: str = "WriteDirectory"
language: str = "Chinese"
async def run(self, topic: str, *args, **kwargs) -> Dict:
"""Execute the action to generate a tutorial directory according to the topic.
@ -49,10 +48,9 @@ class WriteContent(Action):
language: The language to output, default is "Chinese".
"""
def __init__(self, name: str = "", directory: str = "", language: str = "Chinese", *args, **kwargs):
super().__init__(name, *args, **kwargs)
self.language = language
self.directory = directory
name: str = "WriteContent"
directory: dict = dict()
language: str = "Chinese"
async def run(self, topic: str, *args, **kwargs) -> str:
"""Execute the action to write document content according to the directory and topic.
@ -65,4 +63,3 @@ class WriteContent(Action):
"""
prompt = CONTENT_PROMPT.format(topic=topic, language=self.language, directory=self.directory)
return await self._aask(prompt=prompt)

228
metagpt/config.py
View file

@ -2,15 +2,27 @@
# -*- coding: utf-8 -*-
"""
Provide configuration, singleton
@Modified By: mashenquan, 2023/11/27.
1. According to Section 2.2.3.11 of RFC 135, add git repository support.
2. Add the parameter `src_workspace` for the old version project path.
"""
import datetime
import json
import os
import warnings
from copy import deepcopy
from enum import Enum
from pathlib import Path
from typing import Any
from uuid import uuid4
import openai
import yaml
from metagpt.const import PROJECT_ROOT
from metagpt.const import DEFAULT_WORKSPACE_ROOT, METAGPT_ROOT, OPTIONS
from metagpt.logs import logger
from metagpt.tools import SearchEngineType, WebBrowserEngineType
from metagpt.utils.common import require_python_version
from metagpt.utils.cost_manager import CostManager
from metagpt.utils.singleton import Singleton
@ -26,6 +38,22 @@ class NotConfiguredException(Exception):
super().__init__(self.message)
class LLMProviderEnum(Enum):
OPENAI = "openai"
ANTHROPIC = "anthropic"
SPARK = "spark"
ZHIPUAI = "zhipuai"
FIREWORKS = "fireworks"
OPEN_LLM = "open_llm"
GEMINI = "gemini"
METAGPT = "metagpt"
AZURE_OPENAI = "azure_openai"
OLLAMA = "ollama"
def __missing__(self, key):
return self.OPENAI
class Config(metaclass=Singleton):
"""
Regular usage method:
@ -35,33 +63,105 @@ class Config(metaclass=Singleton):
"""
_instance = None
key_yaml_file = PROJECT_ROOT / "config/key.yaml"
default_yaml_file = PROJECT_ROOT / "config/config.yaml"
home_yaml_file = Path.home() / ".metagpt/config.yaml"
key_yaml_file = METAGPT_ROOT / "config/key.yaml"
default_yaml_file = METAGPT_ROOT / "config/config.yaml"
def __init__(self, yaml_file=default_yaml_file):
self._configs = {}
self._init_with_config_files_and_env(self._configs, yaml_file)
logger.info("Config loading done.")
def __init__(self, yaml_file=default_yaml_file, cost_data=""):
global_options = OPTIONS.get()
# cli paras
self.project_path = ""
self.project_name = ""
self.inc = False
self.reqa_file = ""
self.max_auto_summarize_code = 0
self.git_reinit = False
self._init_with_config_files_and_env(yaml_file)
# The agent needs to be billed per user, so billing information cannot be destroyed when the session ends.
self.cost_manager = CostManager(**json.loads(cost_data)) if cost_data else CostManager()
self._update()
global_options.update(OPTIONS.get())
logger.debug("Config loading done.")
def get_default_llm_provider_enum(self) -> LLMProviderEnum:
"""Get first valid LLM provider enum"""
mappings = {
LLMProviderEnum.OPENAI: bool(
self._is_valid_llm_key(self.OPENAI_API_KEY) and not self.OPENAI_API_TYPE and self.OPENAI_API_MODEL
),
LLMProviderEnum.ANTHROPIC: self._is_valid_llm_key(self.ANTHROPIC_API_KEY),
LLMProviderEnum.ZHIPUAI: self._is_valid_llm_key(self.ZHIPUAI_API_KEY),
LLMProviderEnum.FIREWORKS: self._is_valid_llm_key(self.FIREWORKS_API_KEY),
LLMProviderEnum.OPEN_LLM: self._is_valid_llm_key(self.OPEN_LLM_API_BASE),
LLMProviderEnum.GEMINI: self._is_valid_llm_key(self.GEMINI_API_KEY),
LLMProviderEnum.METAGPT: bool(
self._is_valid_llm_key(self.OPENAI_API_KEY) and self.OPENAI_API_TYPE == "metagpt"
),
LLMProviderEnum.AZURE_OPENAI: bool(
self._is_valid_llm_key(self.OPENAI_API_KEY)
and self.OPENAI_API_TYPE == "azure"
and self.DEPLOYMENT_NAME
and self.OPENAI_API_VERSION
),
LLMProviderEnum.OLLAMA: self._is_valid_llm_key(self.OLLAMA_API_BASE),
}
provider = None
for k, v in mappings.items():
if v:
provider = k
break
if provider is None:
if self.DEFAULT_PROVIDER:
provider = LLMProviderEnum(self.DEFAULT_PROVIDER)
else:
raise NotConfiguredException("You should config a LLM configuration first")
if provider is LLMProviderEnum.GEMINI and not require_python_version(req_version=(3, 10)):
warnings.warn("Use Gemini requires Python >= 3.10")
model_name = self.get_model_name(provider=provider)
if model_name:
logger.info(f"{provider} Model: {model_name}")
if provider:
logger.info(f"API: {provider}")
return provider
def get_model_name(self, provider=None) -> str:
provider = provider or self.get_default_llm_provider_enum()
model_mappings = {
LLMProviderEnum.OPENAI: self.OPENAI_API_MODEL,
LLMProviderEnum.AZURE_OPENAI: self.DEPLOYMENT_NAME,
}
return model_mappings.get(provider, "")
@staticmethod
def _is_valid_llm_key(k: str) -> bool:
return bool(k and k != "YOUR_API_KEY")
def _update(self):
self.global_proxy = self._get("GLOBAL_PROXY")
self.openai_api_key = self._get("OPENAI_API_KEY")
self.anthropic_api_key = self._get("Anthropic_API_KEY")
self.anthropic_api_key = self._get("ANTHROPIC_API_KEY")
self.zhipuai_api_key = self._get("ZHIPUAI_API_KEY")
if (not self.openai_api_key or "YOUR_API_KEY" == self.openai_api_key) and \
(not self.anthropic_api_key or "YOUR_API_KEY" == self.anthropic_api_key) and \
(not self.zhipuai_api_key or "YOUR_API_KEY" == self.zhipuai_api_key):
raise NotConfiguredException("Set OPENAI_API_KEY or Anthropic_API_KEY or ZHIPUAI_API_KEY first")
self.openai_api_base = self._get("OPENAI_API_BASE")
openai_proxy = self._get("OPENAI_PROXY") or self.global_proxy
if openai_proxy:
openai.proxy = openai_proxy
openai.api_base = self.openai_api_base
self.open_llm_api_base = self._get("OPEN_LLM_API_BASE")
self.open_llm_api_model = self._get("OPEN_LLM_API_MODEL")
self.fireworks_api_key = self._get("FIREWORKS_API_KEY")
self.gemini_api_key = self._get("GEMINI_API_KEY")
self.ollama_api_base = self._get("OLLAMA_API_BASE")
self.ollama_api_model = self._get("OLLAMA_API_MODEL")
if not self._get("DISABLE_LLM_PROVIDER_CHECK"):
_ = self.get_default_llm_provider_enum()
self.openai_base_url = self._get("OPENAI_BASE_URL")
self.openai_proxy = self._get("OPENAI_PROXY") or self.global_proxy
self.openai_api_type = self._get("OPENAI_API_TYPE")
self.openai_api_version = self._get("OPENAI_API_VERSION")
self.openai_api_rpm = self._get("RPM", 3)
self.openai_api_model = self._get("OPENAI_API_MODEL", "gpt-4")
self.openai_api_model = self._get("OPENAI_API_MODEL", "gpt-4-1106-preview")
self.max_tokens_rsp = self._get("MAX_TOKENS", 2048)
self.deployment_name = self._get("DEPLOYMENT_NAME")
self.deployment_id = self._get("DEPLOYMENT_ID")
self.deployment_name = self._get("DEPLOYMENT_NAME", "gpt-4")
self.spark_appid = self._get("SPARK_APPID")
self.spark_api_secret = self._get("SPARK_API_SECRET")
@ -69,7 +169,10 @@ class Config(metaclass=Singleton):
self.domain = self._get("DOMAIN")
self.spark_url = self._get("SPARK_URL")
self.claude_api_key = self._get("Anthropic_API_KEY")
self.fireworks_api_base = self._get("FIREWORKS_API_BASE")
self.fireworks_api_model = self._get("FIREWORKS_API_MODEL")
self.claude_api_key = self._get("ANTHROPIC_API_KEY")
self.serpapi_api_key = self._get("SERPAPI_API_KEY")
self.serper_api_key = self._get("SERPER_API_KEY")
self.google_api_key = self._get("GOOGLE_API_KEY")
@ -82,8 +185,8 @@ class Config(metaclass=Singleton):
self.long_term_memory = self._get("LONG_TERM_MEMORY", False)
if self.long_term_memory:
logger.warning("LONG_TERM_MEMORY is True")
self.max_budget = self._get("MAX_BUDGET", 10.0)
self.total_cost = 0.0
self.cost_manager.max_budget = self._get("MAX_BUDGET", 10.0)
self.code_review_k_times = 2
self.puppeteer_config = self._get("PUPPETEER_CONFIG", "")
self.mmdc = self._get("MMDC", "mmdc")
@ -93,16 +196,44 @@ class Config(metaclass=Singleton):
self.mermaid_engine = self._get("MERMAID_ENGINE", "nodejs")
self.pyppeteer_executable_path = self._get("PYPPETEER_EXECUTABLE_PATH", "")
self.prompt_format = self._get("PROMPT_FORMAT", "markdown")
workspace_uid = (
self._get("WORKSPACE_UID") or f"{datetime.datetime.now().strftime('%Y%m%d%H%M%S')}-{uuid4().hex[-8:]}"
)
self.repair_llm_output = self._get("REPAIR_LLM_OUTPUT", False)
self.prompt_schema = self._get("PROMPT_FORMAT", "json")
self.workspace_path = Path(self._get("WORKSPACE_PATH", DEFAULT_WORKSPACE_ROOT))
val = self._get("WORKSPACE_PATH_WITH_UID")
if val and val.lower() == "true": # for agent
self.workspace_path = self.workspace_path / workspace_uid
self._ensure_workspace_exists()
self.max_auto_summarize_code = self.max_auto_summarize_code or self._get("MAX_AUTO_SUMMARIZE_CODE", 1)
self.timeout = int(self._get("TIMEOUT", 3))
self.kaggle_username = self._get("KAGGLE_USERNAME", "")
self.kaggle_key = self._get("KAGGLE_KEY", "")
def _init_with_config_files_and_env(self, configs: dict, yaml_file):
"""Load from config/key.yaml, config/config.yaml, and env in decreasing order of priority"""
configs.update(os.environ)
def update_via_cli(self, project_path, project_name, inc, reqa_file, max_auto_summarize_code):
"""update config via cli"""
for _yaml_file in [yaml_file, self.key_yaml_file]:
# Use in the PrepareDocuments action according to Section 2.2.3.5.1 of RFC 135.
if project_path:
inc = True
project_name = project_name or Path(project_path).name
self.project_path = project_path
self.project_name = project_name
self.inc = inc
self.reqa_file = reqa_file
self.max_auto_summarize_code = max_auto_summarize_code
def _ensure_workspace_exists(self):
self.workspace_path.mkdir(parents=True, exist_ok=True)
logger.debug(f"WORKSPACE_PATH set to {self.workspace_path}")
def _init_with_config_files_and_env(self, yaml_file):
"""Load from config/key.yaml, config/config.yaml, and env in decreasing order of priority"""
configs = dict(os.environ)
for _yaml_file in [yaml_file, self.key_yaml_file, self.home_yaml_file]:
if not _yaml_file.exists():
continue
@ -111,18 +242,49 @@ class Config(metaclass=Singleton):
yaml_data = yaml.safe_load(file)
if not yaml_data:
continue
os.environ.update({k: v for k, v in yaml_data.items() if isinstance(v, str)})
configs.update(yaml_data)
OPTIONS.set(configs)
def _get(self, *args, **kwargs):
return self._configs.get(*args, **kwargs)
@staticmethod
def _get(*args, **kwargs):
i = OPTIONS.get()
return i.get(*args, **kwargs)
def get(self, key, *args, **kwargs):
"""Search for a value in config/key.yaml, config/config.yaml, and env; raise an error if not found"""
"""Retrieve values from config/key.yaml, config/config.yaml, and environment variables.
Throw an error if not found."""
value = self._get(key, *args, **kwargs)
if value is None:
raise ValueError(f"Key '{key}' not found in environment variables or in the YAML file")
return value
def __setattr__(self, name: str, value: Any) -> None:
OPTIONS.get()[name] = value
def __getattr__(self, name: str) -> Any:
i = OPTIONS.get()
return i.get(name)
def set_context(self, options: dict):
"""Update current config"""
if not options:
return
opts = deepcopy(OPTIONS.get())
opts.update(options)
OPTIONS.set(opts)
self._update()
@property
def options(self):
"""Return all key-values"""
return OPTIONS.get()
def new_environ(self):
"""Return a new os.environ object"""
env = os.environ.copy()
i = self.options
env.update({k: v for k, v in i.items() if isinstance(v, str)})
return env
CONFIG = Config()

145
metagpt/const.py
View file

@ -4,45 +4,130 @@
@Time : 2023/5/1 11:59
@Author : alexanderwu
@File : const.py
@Modified By: mashenquan, 2023-11-1. According to Section 2.2.1 and 2.2.2 of RFC 116, added key definitions for
common properties in the Message.
@Modified By: mashenquan, 2023-11-27. Defines file repository paths according to Section 2.2.3.4 of RFC 135.
@Modified By: mashenquan, 2023/12/5. Add directories for code summarization..
"""
import contextvars
import os
from pathlib import Path
from loguru import logger
def get_project_root():
"""Search upwards to find the project root directory."""
current_path = Path.cwd()
while True:
if (
(current_path / ".git").exists()
or (current_path / ".project_root").exists()
or (current_path / ".gitignore").exists()
):
# use metagpt with git clone will land here
logger.info(f"PROJECT_ROOT set to {str(current_path)}")
return current_path
parent_path = current_path.parent
if parent_path == current_path:
# use metagpt with pip install will land here
cwd = Path.cwd()
logger.info(f"PROJECT_ROOT set to current working directory: {str(cwd)}")
return cwd
current_path = parent_path
import metagpt
OPTIONS = contextvars.ContextVar("OPTIONS", default={})
PROJECT_ROOT = get_project_root()
DATA_PATH = PROJECT_ROOT / "data"
WORKSPACE_ROOT = PROJECT_ROOT / "workspace"
PROMPT_PATH = PROJECT_ROOT / "metagpt/prompts"
UT_PATH = PROJECT_ROOT / "data/ut"
SWAGGER_PATH = UT_PATH / "files/api/"
UT_PY_PATH = UT_PATH / "files/ut/"
API_QUESTIONS_PATH = UT_PATH / "files/question/"
YAPI_URL = "http://yapi.deepwisdomai.com/"
TMP = PROJECT_ROOT / "tmp"
def get_metagpt_package_root():
"""Get the root directory of the installed package."""
package_root = Path(metagpt.__file__).parent.parent
for i in (".git", ".project_root", ".gitignore"):
if (package_root / i).exists():
break
else:
package_root = Path.cwd()
logger.info(f"Package root set to {str(package_root)}")
return package_root
def get_metagpt_root():
"""Get the project root directory."""
# Check if a project root is specified in the environment variable
project_root_env = os.getenv("METAGPT_PROJECT_ROOT")
if project_root_env:
project_root = Path(project_root_env)
logger.info(f"PROJECT_ROOT set from environment variable to {str(project_root)}")
else:
# Fallback to package root if no environment variable is set
project_root = get_metagpt_package_root()
return project_root
# METAGPT PROJECT ROOT AND VARS
METAGPT_ROOT = get_metagpt_root() # Dependent on METAGPT_PROJECT_ROOT
DEFAULT_WORKSPACE_ROOT = METAGPT_ROOT / "workspace"
EXAMPLE_PATH = METAGPT_ROOT / "examples"
DATA_PATH = METAGPT_ROOT / "data"
TEST_DATA_PATH = METAGPT_ROOT / "tests/data"
RESEARCH_PATH = DATA_PATH / "research"
TUTORIAL_PATH = DATA_PATH / "tutorial_docx"
INVOICE_OCR_TABLE_PATH = DATA_PATH / "invoice_table"
SKILL_DIRECTORY = PROJECT_ROOT / "metagpt/skills"
UT_PATH = DATA_PATH / "ut"
SWAGGER_PATH = UT_PATH / "files/api/"
UT_PY_PATH = UT_PATH / "files/ut/"
API_QUESTIONS_PATH = UT_PATH / "files/question/"
SERDESER_PATH = DEFAULT_WORKSPACE_ROOT / "storage" # TODO to store `storage` under the individual generated project
TMP = METAGPT_ROOT / "tmp"
SOURCE_ROOT = METAGPT_ROOT / "metagpt"
PROMPT_PATH = SOURCE_ROOT / "prompts"
SKILL_DIRECTORY = SOURCE_ROOT / "skills"
# REAL CONSTS
MEM_TTL = 24 * 30 * 3600
MESSAGE_ROUTE_FROM = "sent_from"
MESSAGE_ROUTE_TO = "send_to"
MESSAGE_ROUTE_CAUSE_BY = "cause_by"
MESSAGE_META_ROLE = "role"
MESSAGE_ROUTE_TO_ALL = "<all>"
MESSAGE_ROUTE_TO_NONE = "<none>"
REQUIREMENT_FILENAME = "requirement.txt"
BUGFIX_FILENAME = "bugfix.txt"
PACKAGE_REQUIREMENTS_FILENAME = "requirements.txt"
DOCS_FILE_REPO = "docs"
PRDS_FILE_REPO = "docs/prds"
SYSTEM_DESIGN_FILE_REPO = "docs/system_design"
TASK_FILE_REPO = "docs/tasks"
COMPETITIVE_ANALYSIS_FILE_REPO = "resources/competitive_analysis"
DATA_API_DESIGN_FILE_REPO = "resources/data_api_design"
SEQ_FLOW_FILE_REPO = "resources/seq_flow"
SYSTEM_DESIGN_PDF_FILE_REPO = "resources/system_design"
PRD_PDF_FILE_REPO = "resources/prd"
TASK_PDF_FILE_REPO = "resources/api_spec_and_tasks"
TEST_CODES_FILE_REPO = "tests"
TEST_OUTPUTS_FILE_REPO = "test_outputs"
CODE_SUMMARIES_FILE_REPO = "docs/code_summaries"
CODE_SUMMARIES_PDF_FILE_REPO = "resources/code_summaries"
RESOURCES_FILE_REPO = "resources"
SD_OUTPUT_FILE_REPO = "resources/SD_Output"
GRAPH_REPO_FILE_REPO = "docs/graph_repo"
CLASS_VIEW_FILE_REPO = "docs/class_views"
YAPI_URL = "http://yapi.deepwisdomai.com/"
DEFAULT_LANGUAGE = "English"
DEFAULT_MAX_TOKENS = 1500
COMMAND_TOKENS = 500
BRAIN_MEMORY = "BRAIN_MEMORY"
SKILL_PATH = "SKILL_PATH"
SERPER_API_KEY = "SERPER_API_KEY"
DEFAULT_TOKEN_SIZE = 500
# format
BASE64_FORMAT = "base64"
# REDIS
REDIS_KEY = "REDIS_KEY"
LLM_API_TIMEOUT = 300
# Message id
IGNORED_MESSAGE_ID = "0"
# Class Relationship
GENERALIZATION = "Generalize"
COMPOSITION = "Composite"
AGGREGATION = "Aggregate"

235
metagpt/document.py Normal file
View file

@ -0,0 +1,235 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/6/8 14:03
@Author : alexanderwu
@File : document.py
@Desc : Classes and Operations Related to Files in the File System.
"""
from enum import Enum
from pathlib import Path
from typing import Optional, Union
import pandas as pd
from langchain.document_loaders import (
TextLoader,
UnstructuredPDFLoader,
UnstructuredWordDocumentLoader,
)
from langchain.text_splitter import CharacterTextSplitter
from pydantic import BaseModel, ConfigDict, Field
from tqdm import tqdm
from metagpt.repo_parser import RepoParser
def validate_cols(content_col: str, df: pd.DataFrame):
if content_col not in df.columns:
raise ValueError("Content column not found in DataFrame.")
def read_data(data_path: Path):
suffix = data_path.suffix
if ".xlsx" == suffix:
data = pd.read_excel(data_path)
elif ".csv" == suffix:
data = pd.read_csv(data_path)
elif ".json" == suffix:
data = pd.read_json(data_path)
elif suffix in (".docx", ".doc"):
data = UnstructuredWordDocumentLoader(str(data_path), mode="elements").load()
elif ".txt" == suffix:
data = TextLoader(str(data_path)).load()
text_splitter = CharacterTextSplitter(separator="\n", chunk_size=256, chunk_overlap=0)
texts = text_splitter.split_documents(data)
data = texts
elif ".pdf" == suffix:
data = UnstructuredPDFLoader(str(data_path), mode="elements").load()
else:
raise NotImplementedError("File format not supported.")
return data
class DocumentStatus(Enum):
"""Indicates document status, a mechanism similar to RFC/PEP"""
DRAFT = "draft"
UNDERREVIEW = "underreview"
APPROVED = "approved"
DONE = "done"
class Document(BaseModel):
"""
Document: Handles operations related to document files.
"""
path: Path = Field(default=None)
name: str = Field(default="")
content: str = Field(default="")
# metadata? in content perhaps.
author: str = Field(default="")
status: DocumentStatus = Field(default=DocumentStatus.DRAFT)
reviews: list = Field(default_factory=list)
@classmethod
def from_path(cls, path: Path):
"""
Create a Document instance from a file path.
"""
if not path.exists():
raise FileNotFoundError(f"File {path} not found.")
content = path.read_text()
return cls(content=content, path=path)
@classmethod
def from_text(cls, text: str, path: Optional[Path] = None):
"""
Create a Document from a text string.
"""
return cls(content=text, path=path)
def to_path(self, path: Optional[Path] = None):
"""
Save content to the specified file path.
"""
if path is not None:
self.path = path
if self.path is None:
raise ValueError("File path is not set.")
self.path.parent.mkdir(parents=True, exist_ok=True)
# TODO: excel, csv, json, etc.
self.path.write_text(self.content, encoding="utf-8")
def persist(self):
"""
Persist document to disk.
"""
return self.to_path()
class IndexableDocument(Document):
"""
Advanced document handling: For vector databases or search engines.
"""
model_config = ConfigDict(arbitrary_types_allowed=True)
data: Union[pd.DataFrame, list]
content_col: Optional[str] = Field(default="")
meta_col: Optional[str] = Field(default="")
@classmethod
def from_path(cls, data_path: Path, content_col="content", meta_col="metadata"):
if not data_path.exists():
raise FileNotFoundError(f"File {data_path} not found.")
data = read_data(data_path)
if isinstance(data, pd.DataFrame):
validate_cols(content_col, data)
return cls(data=data, content=str(data), content_col=content_col, meta_col=meta_col)
else:
content = data_path.read_text()
return cls(data=data, content=content, content_col=content_col, meta_col=meta_col)
def _get_docs_and_metadatas_by_df(self) -> (list, list):
df = self.data
docs = []
metadatas = []
for i in tqdm(range(len(df))):
docs.append(df[self.content_col].iloc[i])
if self.meta_col:
metadatas.append({self.meta_col: df[self.meta_col].iloc[i]})
else:
metadatas.append({})
return docs, metadatas
def _get_docs_and_metadatas_by_langchain(self) -> (list, list):
data = self.data
docs = [i.page_content for i in data]
metadatas = [i.metadata for i in data]
return docs, metadatas
def get_docs_and_metadatas(self) -> (list, list):
if isinstance(self.data, pd.DataFrame):
return self._get_docs_and_metadatas_by_df()
elif isinstance(self.data, list):
return self._get_docs_and_metadatas_by_langchain()
else:
raise NotImplementedError("Data type not supported for metadata extraction.")
class RepoMetadata(BaseModel):
name: str = Field(default="")
n_docs: int = Field(default=0)
n_chars: int = Field(default=0)
symbols: list = Field(default_factory=list)
class Repo(BaseModel):
# Name of this repo.
name: str = Field(default="")
# metadata: RepoMetadata = Field(default=RepoMetadata)
docs: dict[Path, Document] = Field(default_factory=dict)
codes: dict[Path, Document] = Field(default_factory=dict)
assets: dict[Path, Document] = Field(default_factory=dict)
path: Path = Field(default=None)
def _path(self, filename):
return self.path / filename
@classmethod
def from_path(cls, path: Path):
"""Load documents, code, and assets from a repository path."""
path.mkdir(parents=True, exist_ok=True)
repo = Repo(path=path, name=path.name)
for file_path in path.rglob("*"):
# FIXME: These judgments are difficult to support multiple programming languages and need to be more general
if file_path.is_file() and file_path.suffix in [".json", ".txt", ".md", ".py", ".js", ".css", ".html"]:
repo._set(file_path.read_text(), file_path)
return repo
def to_path(self):
"""Persist all documents, code, and assets to the given repository path."""
for doc in self.docs.values():
doc.to_path()
for code in self.codes.values():
code.to_path()
for asset in self.assets.values():
asset.to_path()
def _set(self, content: str, path: Path):
"""Add a document to the appropriate category based on its file extension."""
suffix = path.suffix
doc = Document(content=content, path=path, name=str(path.relative_to(self.path)))
# FIXME: These judgments are difficult to support multiple programming languages and need to be more general
if suffix.lower() == ".md":
self.docs[path] = doc
elif suffix.lower() in [".py", ".js", ".css", ".html"]:
self.codes[path] = doc
else:
self.assets[path] = doc
return doc
def set(self, filename: str, content: str):
"""Set a document and persist it to disk."""
path = self._path(filename)
doc = self._set(content, path)
doc.to_path()
def get(self, filename: str) -> Optional[Document]:
"""Get a document by its filename."""
path = self._path(filename)
return self.docs.get(path) or self.codes.get(path) or self.assets.get(path)
def get_text_documents(self) -> list[Document]:
return list(self.docs.values()) + list(self.codes.values())
def eda(self) -> RepoMetadata:
n_docs = sum(len(i) for i in [self.docs, self.codes, self.assets])
n_chars = sum(sum(len(j.content) for j in i.values()) for i in [self.docs, self.codes, self.assets])
symbols = RepoParser(base_directory=self.path).generate_symbols()
return RepoMetadata(name=self.name, n_docs=n_docs, n_chars=n_chars, symbols=symbols)

17
metagpt/document_store/base_store.py
View file

@ -28,22 +28,22 @@ class BaseStore(ABC):
class LocalStore(BaseStore, ABC):
def __init__(self, raw_data: Path, cache_dir: Path = None):
if not raw_data:
def __init__(self, raw_data_path: Path, cache_dir: Path = None):
if not raw_data_path:
raise FileNotFoundError
self.config = Config()
self.raw_data = raw_data
self.raw_data_path = raw_data_path
self.fname = self.raw_data_path.stem
if not cache_dir:
cache_dir = raw_data.parent
cache_dir = raw_data_path.parent
self.cache_dir = cache_dir
self.store = self._load()
if not self.store:
self.store = self.write()
def _get_index_and_store_fname(self):
fname = self.raw_data.name.split('.')[0]
index_file = self.cache_dir / f"{fname}.index"
store_file = self.cache_dir / f"{fname}.pkl"
def _get_index_and_store_fname(self, index_ext=".index", pkl_ext=".pkl"):
index_file = self.cache_dir / f"{self.fname}{index_ext}"
store_file = self.cache_dir / f"{self.fname}{pkl_ext}"
return index_file, store_file
@abstractmethod
@ -53,4 +53,3 @@ class LocalStore(BaseStore, ABC):
@abstractmethod
def _write(self, docs, metadatas):
raise NotImplementedError

3
metagpt/document_store/chromadb_store.py
View file

@ -10,6 +10,7 @@ import chromadb
class ChromaStore:
"""If inherited from BaseStore, or importing other modules from metagpt, a Python exception occurs, which is strange."""
def __init__(self, name):
client = chromadb.Client()
collection = client.create_collection(name)
@ -22,7 +23,7 @@ class ChromaStore:
query_texts=[query],
n_results=n_results,
where=metadata_filter, # optional filter
where_document=document_filter # optional filter
where_document=document_filter, # optional filter
)
return results

82
metagpt/document_store/document.py
View file

@ -1,82 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/6/8 14:03
@Author : alexanderwu
@File : document.py
"""
from pathlib import Path
import pandas as pd
from langchain.document_loaders import (
TextLoader,
UnstructuredPDFLoader,
UnstructuredWordDocumentLoader,
)
from langchain.text_splitter import CharacterTextSplitter
from tqdm import tqdm
def validate_cols(content_col: str, df: pd.DataFrame):
if content_col not in df.columns:
raise ValueError
def read_data(data_path: Path):
suffix = data_path.suffix
if '.xlsx' == suffix:
data = pd.read_excel(data_path)
elif '.csv' == suffix:
data = pd.read_csv(data_path)
elif '.json' == suffix:
data = pd.read_json(data_path)
elif suffix in ('.docx', '.doc'):
data = UnstructuredWordDocumentLoader(str(data_path), mode='elements').load()
elif '.txt' == suffix:
data = TextLoader(str(data_path)).load()
text_splitter = CharacterTextSplitter(separator='\n', chunk_size=256, chunk_overlap=0)
texts = text_splitter.split_documents(data)
data = texts
elif '.pdf' == suffix:
data = UnstructuredPDFLoader(str(data_path), mode="elements").load()
else:
raise NotImplementedError
return data
class Document:
def __init__(self, data_path, content_col='content', meta_col='metadata'):
self.data = read_data(data_path)
if isinstance(self.data, pd.DataFrame):
validate_cols(content_col, self.data)
self.content_col = content_col
self.meta_col = meta_col
def _get_docs_and_metadatas_by_df(self) -> (list, list):
df = self.data
docs = []
metadatas = []
for i in tqdm(range(len(df))):
docs.append(df[self.content_col].iloc[i])
if self.meta_col:
metadatas.append({self.meta_col: df[self.meta_col].iloc[i]})
else:
metadatas.append({})
return docs, metadatas
def _get_docs_and_metadatas_by_langchain(self) -> (list, list):
data = self.data
docs = [i.page_content for i in data]
metadatas = [i.metadata for i in data]
return docs, metadatas
def get_docs_and_metadatas(self) -> (list, list):
if isinstance(self.data, pd.DataFrame):
return self._get_docs_and_metadatas_by_df()
elif isinstance(self.data, list):
return self._get_docs_and_metadatas_by_langchain()
else:
raise NotImplementedError

52
metagpt/document_store/faiss_store.py
View file

@ -5,52 +5,48 @@
@Author : alexanderwu
@File : faiss_store.py
"""
import pickle
import asyncio
from pathlib import Path
from typing import Optional
import faiss
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain_core.embeddings import Embeddings
from metagpt.const import DATA_PATH
from metagpt.config import CONFIG
from metagpt.document import IndexableDocument
from metagpt.document_store.base_store import LocalStore
from metagpt.document_store.document import Document
from metagpt.logs import logger
class FaissStore(LocalStore):
def __init__(self, raw_data: Path, cache_dir=None, meta_col='source', content_col='output'):
def __init__(
self, raw_data: Path, cache_dir=None, meta_col="source", content_col="output", embedding: Embeddings = None
):
self.meta_col = meta_col
self.content_col = content_col
self.embedding = embedding or OpenAIEmbeddings(
openai_api_key=CONFIG.openai_api_key, openai_api_base=CONFIG.openai_base_url
)
super().__init__(raw_data, cache_dir)
def _load(self) -> Optional["FaissStore"]:
index_file, store_file = self._get_index_and_store_fname()
index_file, store_file = self._get_index_and_store_fname(index_ext=".faiss") # langchain FAISS using .faiss
if not (index_file.exists() and store_file.exists()):
logger.info("Missing at least one of index_file/store_file, load failed and return None")
return None
index = faiss.read_index(str(index_file))
with open(str(store_file), "rb") as f:
store = pickle.load(f)
store.index = index
return store
return FAISS.load_local(self.raw_data_path.parent, self.embedding, self.fname)
def _write(self, docs, metadatas):
store = FAISS.from_texts(docs, OpenAIEmbeddings(openai_api_version="2020-11-07"), metadatas=metadatas)
store = FAISS.from_texts(docs, self.embedding, metadatas=metadatas)
return store
def persist(self):
index_file, store_file = self._get_index_and_store_fname()
store = self.store
index = self.store.index
faiss.write_index(store.index, str(index_file))
store.index = None
with open(store_file, "wb") as f:
pickle.dump(store, f)
store.index = index
self.store.save_local(self.raw_data_path.parent, self.fname)
def search(self, query, expand_cols=False, sep='\n', *args, k=5, **kwargs):
def search(self, query, expand_cols=False, sep="\n", *args, k=5, **kwargs):
rsp = self.store.similarity_search(query, k=k, **kwargs)
logger.debug(rsp)
if expand_cols:
@ -58,11 +54,14 @@ class FaissStore(LocalStore):
else:
return str(sep.join([f"{x.page_content}" for x in rsp]))
async def asearch(self, *args, **kwargs):
return await asyncio.to_thread(self.search, *args, **kwargs)
def write(self):
"""Initialize the index and library based on the Document (JSON / XLSX, etc.) file provided by the user."""
if not self.raw_data.exists():
if not self.raw_data_path.exists():
raise FileNotFoundError
doc = Document(self.raw_data, self.content_col, self.meta_col)
doc = IndexableDocument.from_path(self.raw_data_path, self.content_col, self.meta_col)
docs, metadatas = doc.get_docs_and_metadatas()
self.store = self._write(docs, metadatas)
@ -76,10 +75,3 @@ class FaissStore(LocalStore):
def delete(self, *args, **kwargs):
"""Currently, langchain does not provide a delete interface."""
raise NotImplementedError
if __name__ == '__main__':
faiss_store = FaissStore(DATA_PATH / 'qcs/qcs_4w.json')
logger.info(faiss_store.search('Oily Skin Facial Cleanser'))
faiss_store.add([f'Oily Skin Facial Cleanser-{i}' for i in range(3)])
logger.info(faiss_store.search('Oily Skin Facial Cleanser'))

122
metagpt/document_store/milvus_store.py
View file

@ -1,122 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Time : 2023/5/28 00:00
@Author : alexanderwu
@File : milvus_store.py
"""
from typing import TypedDict
import numpy as np
from pymilvus import Collection, CollectionSchema, DataType, FieldSchema, connections
from metagpt.document_store.base_store import BaseStore
type_mapping = {
int: DataType.INT64,
str: DataType.VARCHAR,
float: DataType.DOUBLE,
np.ndarray: DataType.FLOAT_VECTOR
}
def columns_to_milvus_schema(columns: dict, primary_col_name: str = "", desc: str = ""):
"""Assume the structure of columns is str: regular type"""
fields = []
for col, ctype in columns.items():
if ctype == str:
mcol = FieldSchema(name=col, dtype=type_mapping[ctype], max_length=100)
elif ctype == np.ndarray:
mcol = FieldSchema(name=col, dtype=type_mapping[ctype], dim=2)
else:
mcol = FieldSchema(name=col, dtype=type_mapping[ctype], is_primary=(col == primary_col_name))
fields.append(mcol)
schema = CollectionSchema(fields, description=desc)
return schema
class MilvusConnection(TypedDict):
alias: str
host: str
port: str
class MilvusStore(BaseStore):
"""
FIXME: ADD TESTS
https://milvus.io/docs/v2.0.x/create_collection.md
"""
def __init__(self, connection):
connections.connect(**connection)
self.collection = None
def _create_collection(self, name, schema):
collection = Collection(
name=name,
schema=schema,
using='default',
shards_num=2,
consistency_level="Strong"
)
return collection
def create_collection(self, name, columns):
schema = columns_to_milvus_schema(columns, 'idx')
self.collection = self._create_collection(name, schema)
return self.collection
def drop(self, name):
Collection(name).drop()
def load_collection(self):
self.collection.load()
def build_index(self, field='emb'):
self.collection.create_index(field, {"index_type": "FLAT", "metric_type": "L2", "params": {}})
def search(self, query: list[list[float]], *args, **kwargs):
"""
FIXME: ADD TESTS
https://milvus.io/docs/v2.0.x/search.md
All search and query operations within Milvus are executed in memory. Load the collection to memory before conducting a vector similarity search.
Note the above description, is this logic serious? This should take a long time, right?
"""
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
results = self.collection.search(
data=query,
anns_field=kwargs.get('field', 'emb'),
param=search_params,
limit=10,
expr=None,
consistency_level="Strong"
)
# FIXME: results contain id, but to get the actual value from the id, we still need to call the query interface
return results
def write(self, name, schema, *args, **kwargs):
"""
FIXME: ADD TESTS
https://milvus.io/docs/v2.0.x/create_collection.md
:param args:
:param kwargs:
:return:
"""
raise NotImplementedError
def add(self, data, *args, **kwargs):
"""
FIXME: ADD TESTS
https://milvus.io/docs/v2.0.x/insert_data.md
import random
data = [
[i for i in range(2000)],
[i for i in range(10000, 12000)],
[[random.random() for _ in range(2)] for _ in range(2000)],
]
:param args:
:param kwargs:
:return:
"""
self.collection.insert(data)

Some files were not shown because too many files have changed in this diff Show more