+[](https://codespaces.new/geekan/MetaGPT)
-You may use the button above to open this repo in a Codespace
+Click the button above to open this repository in a Codespace. For additional information, refer to the [GitHub documentation on creating a Codespace](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
-For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
-
## VS Code Dev Containers
-
+[](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT)
-Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name:
-https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT
+Note: Clicking the link above opens the main repository. To open your local cloned repository, replace the URL with your username and cloned repository's name: `https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/@@ -32,8 +33,12 @@ # MetaGPT: The Multi-Agent Framework
Software Company Multi-Role Schematic (Gradually Implementing)
+## News +🚀 Jan 03: Here comes [v0.6.0](https://github.com/geekan/MetaGPT/releases/tag/v0.6.0)! In this version, we added serialization and deserialization of important objects and enabled breakpoint recovery. We upgraded OpenAI package to v1.6.0 and supported Gemini, ZhipuAI, Ollama, OpenLLM, etc. Moreover, we provided extremely simple examples where you need only 7 lines to implement a general election [debate](https://github.com/geekan/MetaGPT/blob/main/examples/debate_simple.py). Check out more details [here](https://github.com/geekan/MetaGPT/releases/tag/v0.6.0)! +🚀 Dec 15: [v0.5.0](https://github.com/geekan/MetaGPT/releases/tag/v0.5.0) is released! We introduced **incremental development**, facilitating agents to build up larger projects on top of their previous efforts or existing codebase. We also launched a whole collection of important features, including **multilingual support** (experimental), multiple **programming languages support** (experimental), **incremental development** (experimental), CLI support, pip support, enhanced code review, documentation mechanism, and optimized messaging mechanism! + ## Install ### Pip installation @@ -48,21 +53,26 @@ # conda activate metagpt # Step 2: Clone the repository to your local machine for latest version, and install it. git clone https://github.com/geekan/MetaGPT.git cd MetaGPT -pip3 install -e. # or pip3 install metagpt # for stable version +pip3 install -e . # or pip3 install metagpt # for stable version -# Step 3: run the startup.py -# setup your OPENAI_API_KEY in key.yaml copy from config.yaml -python3 startup.py "Write a cli snake game" +# Step 3: setup your OPENAI_API_KEY, or make sure it existed in the env +mkdir ~/.metagpt +cp config/config.yaml ~/.metagpt/config.yaml +vim ~/.metagpt/config.yaml -# Step 4 [Optional]: If you want to save the artifacts like diagrams such as quadrant chart, system designs, sequence flow in the workspace, you can execute the step before Step 3. By default, the framework is compatible, and the entire process can be run completely without executing this step. +# Step 4: run metagpt cli +metagpt "Create a 2048 game in python" + +# Step 5 [Optional]: If you want to save the artifacts like diagrams such as quadrant chart, system designs, sequence flow in the workspace, you can execute the step before Step 3. By default, the framework is compatible, and the entire process can be run completely without executing this step. # If executing, ensure that NPM is installed on your system. Then install mermaid-js. (If you don't have npm in your computer, please go to the Node.js official website to install Node.js https://nodejs.org/ and then you will have npm tool in your computer.) npm --version sudo npm install -g @mermaid-js/mermaid-cli ``` -detail installation please refer to [cli_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-stable-version) +detail installation please refer to [cli_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-stable-version) ### Docker installation +> Note: In the Windows, you need to replace "/opt/metagpt" with a directory that Docker has permission to create, such as "D:\Users\x\metagpt" ```bash # Step 1: Download metagpt official image and prepare config.yaml @@ -77,10 +87,10 @@ # Step 2: Run metagpt demo with container -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" ``` -detail installation please refer to [docker_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-with-docker) +detail installation please refer to [docker_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-with-docker) ### QuickStart & Demo Video - Try it on [MetaGPT Huggingface Space](https://huggingface.co/spaces/deepwisdom/MetaGPT) @@ -91,19 +101,19 @@ ### QuickStart & Demo Video ## Tutorial -- 🗒 [Online Document](https://docs.deepwisdom.ai/) -- 💻 [Usage](https://docs.deepwisdom.ai/guide/get_started/quickstart.html) -- 🔎 [What can MetaGPT do?](https://docs.deepwisdom.ai/guide/get_started/introduction.html) +- 🗒 [Online Document](https://docs.deepwisdom.ai/main/en/) +- 💻 [Usage](https://docs.deepwisdom.ai/main/en/guide/get_started/quickstart.html) +- 🔎 [What can MetaGPT do?](https://docs.deepwisdom.ai/main/en/guide/get_started/introduction.html) - 🛠 How to build your own agents? - - [MetaGPT Usage & Development Guide | Agent 101](https://docs.deepwisdom.ai/guide/tutorials/agent_101.html) - - [MetaGPT Usage & Development Guide | MultiAgent 101](https://docs.deepwisdom.ai/guide/tutorials/multi_agent_101.html) + - [MetaGPT Usage & Development Guide | Agent 101](https://docs.deepwisdom.ai/main/en/guide/tutorials/agent_101.html) + - [MetaGPT Usage & Development Guide | MultiAgent 101](https://docs.deepwisdom.ai/main/en/guide/tutorials/multi_agent_101.html) - 🧑💻 Contribution - [Develop Roadmap](docs/ROADMAP.md) - 🔖 Use Cases - - [Debate](https://docs.deepwisdom.ai/guide/use_cases/multi_agent/debate.html) - - [Researcher](https://docs.deepwisdom.ai/guide/use_cases/agent/researcher.html) - - [Recepit Assistant](https://docs.deepwisdom.ai/guide/use_cases/agent/receipt_assistant.html) -- ❓ [FAQs](https://docs.deepwisdom.ai/guide/faq.html) + - [Debate](https://docs.deepwisdom.ai/main/en/guide/use_cases/multi_agent/debate.html) + - [Researcher](https://docs.deepwisdom.ai/main/en/guide/use_cases/agent/researcher.html) + - [Recepit Assistant](https://docs.deepwisdom.ai/main/en/guide/use_cases/agent/receipt_assistant.html) +- ❓ [FAQs](https://docs.deepwisdom.ai/main/en/guide/faq.html) ## Support @@ -116,14 +126,14 @@ ### Contact Information If you have any questions or feedback about this project, please feel free to contact us. We highly appreciate your suggestions! -- **Email:** alexanderwu@fuzhi.ai +- **Email:** alexanderwu@deepwisdom.ai - **GitHub Issues:** For more technical inquiries, you can also create a new issue in our [GitHub repository](https://github.com/geekan/metagpt/issues). We will respond to all questions within 2-3 business days. ## Citation -For now, cite the [Arxiv paper](https://arxiv.org/abs/2308.00352): +For now, cite the [arXiv paper](https://arxiv.org/abs/2308.00352): ```bibtex @misc{hong2023metagpt, diff --git a/config/config.yaml b/config/config.yaml index 17605307a..79ebae863 100644 --- a/config/config.yaml +++ b/config/config.yaml @@ -1,16 +1,21 @@ # DO NOT MODIFY THIS FILE, create a new key.yaml, define OPENAI_API_KEY. # The configuration of key.yaml has a higher priority and will not enter git +#### Project Path Setting +# WORKSPACE_PATH: "Path for placing output files" + #### if OpenAI -## The official OPENAI_API_BASE is https://api.openai.com/v1 -## If the official OPENAI_API_BASE is not available, we recommend using the [openai-forward](https://github.com/beidongjiedeguang/openai-forward). -## Or, you can configure OPENAI_PROXY to access official OPENAI_API_BASE. -OPENAI_API_BASE: "https://api.openai.com/v1" +## The official OPENAI_BASE_URL is https://api.openai.com/v1 +## If the official OPENAI_BASE_URL is not available, we recommend using the [openai-forward](https://github.com/beidongjiedeguang/openai-forward). +## Or, you can configure OPENAI_PROXY to access official OPENAI_BASE_URL. +OPENAI_BASE_URL: "https://api.openai.com/v1" #OPENAI_PROXY: "http://127.0.0.1:8118" -#OPENAI_API_KEY: "YOUR_API_KEY" # set the value to sk-xxx if you host the openai interface for open llm model -OPENAI_API_MODEL: "gpt-4" -MAX_TOKENS: 1500 +#OPENAI_API_KEY: "YOUR_API_KEY" # set the value to sk-xxx if you host the openai interface for open llm model +OPENAI_API_MODEL: "gpt-4-1106-preview" +MAX_TOKENS: 4096 RPM: 10 +TIMEOUT: 60 # Timeout for llm invocation +#DEFAULT_PROVIDER: openai #### if Spark #SPARK_APPID : "YOUR_APPID" @@ -20,20 +25,35 @@ RPM: 10 #SPARK_URL : "ws://spark-api.xf-yun.com/v2.1/chat" #### if Anthropic -#Anthropic_API_KEY: "YOUR_API_KEY" +#ANTHROPIC_API_KEY: "YOUR_API_KEY" #### if AZURE, check https://github.com/openai/openai-cookbook/blob/main/examples/azure/chat.ipynb -#### You can use ENGINE or DEPLOYMENT mode #OPENAI_API_TYPE: "azure" -#OPENAI_API_BASE: "YOUR_AZURE_ENDPOINT" +#OPENAI_BASE_URL: "YOUR_AZURE_ENDPOINT" #OPENAI_API_KEY: "YOUR_AZURE_API_KEY" #OPENAI_API_VERSION: "YOUR_AZURE_API_VERSION" #DEPLOYMENT_NAME: "YOUR_DEPLOYMENT_NAME" -#DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID" #### if zhipuai from `https://open.bigmodel.cn`. You can set here or export API_KEY="YOUR_API_KEY" # ZHIPUAI_API_KEY: "YOUR_API_KEY" +#### if Google Gemini from `https://ai.google.dev/` and API_KEY from `https://makersuite.google.com/app/apikey`. +#### You can set here or export GOOGLE_API_KEY="YOUR_API_KEY" +# GEMINI_API_KEY: "YOUR_API_KEY" + +#### if use self-host open llm model with openai-compatible interface +#OPEN_LLM_API_BASE: "http://127.0.0.1:8000/v1" +#OPEN_LLM_API_MODEL: "llama2-13b" +# +##### if use Fireworks api +#FIREWORKS_API_KEY: "YOUR_API_KEY" +#FIREWORKS_API_BASE: "https://api.fireworks.ai/inference/v1" +#FIREWORKS_API_MODEL: "YOUR_LLM_MODEL" # example, accounts/fireworks/models/llama-v2-13b-chat + +#### if use self-host open llm model by ollama +# OLLAMA_API_BASE: http://127.0.0.1:11434/api +# OLLAMA_API_MODEL: llama2 + #### for Search ## Supported values: serpapi/google/serper/ddg @@ -68,8 +88,8 @@ RPM: 10 #### for Stable Diffusion ## Use SD service, based on https://github.com/AUTOMATIC1111/stable-diffusion-webui -SD_URL: "YOUR_SD_URL" -SD_T2I_API: "/sdapi/v1/txt2img" +#SD_URL: "YOUR_SD_URL" +#SD_T2I_API: "/sdapi/v1/txt2img" #### for Execution #LONG_TERM_MEMORY: false @@ -84,8 +104,8 @@ SD_T2I_API: "/sdapi/v1/txt2img" # CALC_USAGE: false ### for Research -MODEL_FOR_RESEARCHER_SUMMARY: gpt-3.5-turbo -MODEL_FOR_RESEARCHER_REPORT: gpt-3.5-turbo-16k +# MODEL_FOR_RESEARCHER_SUMMARY: gpt-3.5-turbo +# MODEL_FOR_RESEARCHER_REPORT: gpt-3.5-turbo-16k ### choose the engine for mermaid conversion, # default is nodejs, you can change it to playwright,pyppeteer or ink @@ -94,4 +114,31 @@ MODEL_FOR_RESEARCHER_REPORT: gpt-3.5-turbo-16k ### browser path for pyppeteer engine, support Chrome, Chromium,MS Edge #PYPPETEER_EXECUTABLE_PATH: "/usr/bin/google-chrome-stable" -PROMPT_FORMAT: json #json or markdown \ No newline at end of file +### for repair non-openai LLM's output when parse json-text if PROMPT_FORMAT=json +### due to non-openai LLM's output will not always follow the instruction, so here activate a post-process +### repair operation on the content extracted from LLM's raw output. Warning, it improves the result but not fix all cases. +# REPAIR_LLM_OUTPUT: false + +# PROMPT_FORMAT: json #json or markdown + +### Agent configurations +# RAISE_NOT_CONFIG_ERROR: true # "true" if the LLM key is not configured, throw a NotConfiguredException, else "false". +# WORKSPACE_PATH_WITH_UID: false # "true" if using `{workspace}/{uid}` as the workspace path; "false" use `{workspace}`. + +### Meta Models +#METAGPT_TEXT_TO_IMAGE_MODEL: MODEL_URL + +### S3 config +#S3_ACCESS_KEY: "YOUR_S3_ACCESS_KEY" +#S3_SECRET_KEY: "YOUR_S3_SECRET_KEY" +#S3_ENDPOINT_URL: "YOUR_S3_ENDPOINT_URL" +#S3_SECURE: true # true/false +#S3_BUCKET: "YOUR_S3_BUCKET" + +### Redis config +#REDIS_HOST: "YOUR_REDIS_HOST" +#REDIS_PORT: "YOUR_REDIS_PORT" +#REDIS_PASSWORD: "YOUR_REDIS_PASSWORD" +#REDIS_DB: "YOUR_REDIS_DB_INDEX, str, 0-based" + +# DISABLE_LLM_PROVIDER_CHECK: false diff --git a/docs/.agent-store-config.yaml.example b/docs/.agent-store-config.yaml.example new file mode 100644 index 000000000..bec0dd170 --- /dev/null +++ b/docs/.agent-store-config.yaml.example @@ -0,0 +1,9 @@ +role: + name: Teacher # Referenced the `Teacher` in `metagpt/roles/teacher.py`. + module: metagpt.roles.teacher # Referenced `metagpt/roles/teacher.py`. + skills: # Refer to the skill `name` of the published skill in `docs/.well-known/skills.yaml`. + - name: text_to_speech + description: Text-to-speech + - name: text_to_image + description: Create a drawing based on the text. + diff --git a/docs/.pylintrc b/docs/.pylintrc new file mode 100644 index 000000000..9e8488bc7 --- /dev/null +++ b/docs/.pylintrc @@ -0,0 +1,639 @@ +[MAIN] + +# Analyse import fallback blocks. This can be used to support both Python 2 and +# 3 compatible code, which means that the block might have code that exists +# only in one or another interpreter, leading to false positives when analysed. +analyse-fallback-blocks=no + +# Clear in-memory caches upon conclusion of linting. Useful if running pylint +# in a server-like mode. +clear-cache-post-run=no + +# Load and enable all available extensions. Use --list-extensions to see a list +# all available extensions. +#enable-all-extensions= + +# In error mode, messages with a category besides ERROR or FATAL are +# suppressed, and no reports are done by default. Error mode is compatible with +# disabling specific errors. +#errors-only= + +# Always return a 0 (non-error) status code, even if lint errors are found. +# This is primarily useful in continuous integration scripts. +#exit-zero= + +# A comma-separated list of package or module names from where C extensions may +# be loaded. Extensions are loading into the active Python interpreter and may +# run arbitrary code. +extension-pkg-allow-list= + +# A comma-separated list of package or module names from where C extensions may +# be loaded. Extensions are loading into the active Python interpreter and may +# run arbitrary code. (This is an alternative name to extension-pkg-allow-list +# for backward compatibility.) +extension-pkg-whitelist=pydantic + +# Return non-zero exit code if any of these messages/categories are detected, +# even if score is above --fail-under value. Syntax same as enable. Messages +# specified are enabled, while categories only check already-enabled messages. +fail-on= + +# Specify a score threshold under which the program will exit with error. +fail-under=10 + +# Interpret the stdin as a python script, whose filename needs to be passed as +# the module_or_package argument. +#from-stdin= + +# Files or directories to be skipped. They should be base names, not paths. +ignore=CVS + +# Add files or directories matching the regular expressions patterns to the +# ignore-list. The regex matches against paths and can be in Posix or Windows +# format. Because '\\' represents the directory delimiter on Windows systems, +# it can't be used as an escape character. +ignore-paths= + +# Files or directories matching the regular expression patterns are skipped. +# The regex matches against base names, not paths. The default value ignores +# Emacs file locks +#ignore-patterns=^\.# +ignore-patterns=(.)*_test\.py,test_(.)*\.py + + +# List of module names for which member attributes should not be checked +# (useful for modules/projects where namespaces are manipulated during runtime +# and thus existing member attributes cannot be deduced by static analysis). It +# supports qualified module names, as well as Unix pattern matching. +ignored-modules= + +# Python code to execute, usually for sys.path manipulation such as +# pygtk.require(). +#init-hook= + +# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the +# number of processors available to use, and will cap the count on Windows to +# avoid hangs. +jobs=1 + +# Control the amount of potential inferred values when inferring a single +# object. This can help the performance when dealing with large functions or +# complex, nested conditions. +limit-inference-results=120 + +# List of plugins (as comma separated values of python module names) to load, +# usually to register additional checkers. +load-plugins= + +# Pickle collected data for later comparisons. +persistent=yes + +# Minimum Python version to use for version dependent checks. Will default to +# the version used to run pylint. +py-version=3.9 + +# Discover python modules and packages in the file system subtree. +recursive=no + +# Add paths to the list of the source roots. Supports globbing patterns. The +# source root is an absolute path or a path relative to the current working +# directory used to determine a package namespace for modules located under the +# source root. +source-roots= + +# When enabled, pylint would attempt to guess common misconfiguration and emit +# user-friendly hints instead of false-positive error messages. +suggestion-mode=yes + +# Allow loading of arbitrary C extensions. Extensions are imported into the +# active Python interpreter and may run arbitrary code. +unsafe-load-any-extension=no + +# In verbose mode, extra non-checker-related info will be displayed. +#verbose= + + +[BASIC] + +# Naming style matching correct argument names. +argument-naming-style=snake_case + +# Regular expression matching correct argument names. Overrides argument- +# naming-style. If left empty, argument names will be checked with the set +# naming style. +#argument-rgx= + +# Naming style matching correct attribute names. +attr-naming-style=snake_case + +# Regular expression matching correct attribute names. Overrides attr-naming- +# style. If left empty, attribute names will be checked with the set naming +# style. +#attr-rgx= + +# Bad variable names which should always be refused, separated by a comma. +bad-names=foo, + bar, + baz, + toto, + tutu, + tata + +# Bad variable names regexes, separated by a comma. If names match any regex, +# they will always be refused +bad-names-rgxs= + +# Naming style matching correct class attribute names. +class-attribute-naming-style=any + +# Regular expression matching correct class attribute names. Overrides class- +# attribute-naming-style. If left empty, class attribute names will be checked +# with the set naming style. +#class-attribute-rgx= + +# Naming style matching correct class constant names. +class-const-naming-style=UPPER_CASE + +# Regular expression matching correct class constant names. Overrides class- +# const-naming-style. If left empty, class constant names will be checked with +# the set naming style. +#class-const-rgx= + +# Naming style matching correct class names. +class-naming-style=PascalCase + +# Regular expression matching correct class names. Overrides class-naming- +# style. If left empty, class names will be checked with the set naming style. +#class-rgx= + +# Naming style matching correct constant names. +const-naming-style=UPPER_CASE + +# Regular expression matching correct constant names. Overrides const-naming- +# style. If left empty, constant names will be checked with the set naming +# style. +#const-rgx= + +# Minimum line length for functions/classes that require docstrings, shorter +# ones are exempt. +docstring-min-length=-1 + +# Naming style matching correct function names. +function-naming-style=snake_case + +# Regular expression matching correct function names. Overrides function- +# naming-style. If left empty, function names will be checked with the set +# naming style. +#function-rgx= + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + v, + e, + d, + m, + df, + ex, + Run, + _ + +# Good variable names regexes, separated by a comma. If names match any regex, +# they will always be accepted +good-names-rgxs= + +# Include a hint for the correct naming format with invalid-name. +include-naming-hint=no + +# Naming style matching correct inline iteration names. +inlinevar-naming-style=any + +# Regular expression matching correct inline iteration names. Overrides +# inlinevar-naming-style. If left empty, inline iteration names will be checked +# with the set naming style. +#inlinevar-rgx= + +# Naming style matching correct method names. +method-naming-style=snake_case + +# Regular expression matching correct method names. Overrides method-naming- +# style. If left empty, method names will be checked with the set naming style. +#method-rgx= + +# Naming style matching correct module names. +module-naming-style=snake_case + +# Regular expression matching correct module names. Overrides module-naming- +# style. If left empty, module names will be checked with the set naming style. +#module-rgx= + +# Colon-delimited sets of names that determine each other's naming style when +# the name regexes allow several styles. +name-group= + +# Regular expression which should only match function or class names that do +# not require a docstring. +no-docstring-rgx=^_ + +# List of decorators that produce properties, such as abc.abstractproperty. Add +# to this list to register other decorators that produce valid properties. +# These decorators are taken in consideration only for invalid-name. +property-classes=abc.abstractproperty + +# Regular expression matching correct type alias names. If left empty, type +# alias names will be checked with the set naming style. +#typealias-rgx= + +# Regular expression matching correct type variable names. If left empty, type +# variable names will be checked with the set naming style. +#typevar-rgx= + +# Naming style matching correct variable names. +variable-naming-style=snake_case + +# Regular expression matching correct variable names. Overrides variable- +# naming-style. If left empty, variable names will be checked with the set +# naming style. +#variable-rgx= + + +[CLASSES] + +# Warn about protected attribute access inside special methods +check-protected-access-in-special-methods=no + +# List of method names used to declare (i.e. assign) instance attributes. +defining-attr-methods=__init__, + __new__, + setUp, + __post_init__ + +# List of member names, which should be excluded from the protected access +# warning. +exclude-protected=_asdict,_fields,_replace,_source,_make,os._exit + +# List of valid names for the first argument in a class method. +valid-classmethod-first-arg=cls + +# List of valid names for the first argument in a metaclass class method. +valid-metaclass-classmethod-first-arg=mcs + + +[DESIGN] + +# List of regular expressions of class ancestor names to ignore when counting +# public methods (see R0903) +exclude-too-few-public-methods= + +# List of qualified class names to ignore when counting class parents (see +# R0901) +ignored-parents= + +# Maximum number of arguments for function / method. +max-args=5 + +# Maximum number of attributes for a class (see R0902). +max-attributes=7 + +# Maximum number of boolean expressions in an if statement (see R0916). +max-bool-expr=5 + +# Maximum number of branch for function / method body. +max-branches=12 + +# Maximum number of locals for function / method body. +max-locals=15 + +# Maximum number of parents for a class (see R0901). +max-parents=7 + +# Maximum number of public methods for a class (see R0904). +max-public-methods=20 + +# Maximum number of return / yield for function / method body. +max-returns=6 + +# Maximum number of statements in function / method body. +max-statements=50 + +# Minimum number of public methods for a class (see R0903). +min-public-methods=2 + + +[EXCEPTIONS] + +# Exceptions that will emit a warning when caught. +overgeneral-exceptions=builtins.BaseException,builtins.Exception + + +[FORMAT] + +# Expected format of line ending, e.g. empty (any line ending), LF or CRLF. +expected-line-ending-format= + +# Regexp for a line that is allowed to be longer than the limit. +ignore-long-lines=^\s*(# )??$ + +# Number of spaces of indent required inside a hanging or continued line. +indent-after-paren=4 + +# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 +# tab). +indent-string=' ' + +# Maximum number of characters on a single line. +max-line-length=120 + +# Maximum number of lines in a module. +max-module-lines=1000 + +# Allow the body of a class to be on the same line as the declaration if body +# contains single statement. +single-line-class-stmt=no + +# Allow the body of an if to be on the same line as the test if there is no +# else. +single-line-if-stmt=no + + +[IMPORTS] + +# List of modules that can be imported at any level, not just the top level +# one. +allow-any-import-level= + +# Allow explicit reexports by alias from a package __init__. +allow-reexport-from-package=no + +# Allow wildcard imports from modules that define __all__. +allow-wildcard-with-all=no + +# Deprecated modules which should not be used, separated by a comma. +deprecated-modules= + +# Output a graph (.gv or any supported image format) of external dependencies +# to the given file (report RP0402 must not be disabled). +ext-import-graph= + +# Output a graph (.gv or any supported image format) of all (i.e. internal and +# external) dependencies to the given file (report RP0402 must not be +# disabled). +import-graph= + +# Output a graph (.gv or any supported image format) of internal dependencies +# to the given file (report RP0402 must not be disabled). +int-import-graph= + +# Force import order to recognize a module as part of the standard +# compatibility libraries. +known-standard-library= + +# Force import order to recognize a module as part of a third party library. +known-third-party=enchant + +# Couples of modules and preferred modules, separated by a comma. +preferred-modules= + + +[LOGGING] + +# The type of string formatting that logging methods do. `old` means using % +# formatting, `new` is for `{}` formatting. +logging-format-style=old + +# Logging modules to check that the string format arguments are in logging +# function parameter format. +logging-modules=logging + + +[MESSAGES CONTROL] + +# Only show warnings with the listed confidence levels. Leave empty to show +# all. Valid levels: HIGH, CONTROL_FLOW, INFERENCE, INFERENCE_FAILURE, +# UNDEFINED. +confidence=HIGH, + CONTROL_FLOW, + INFERENCE, + INFERENCE_FAILURE, + UNDEFINED + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then re-enable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=raw-checker-failed, + bad-inline-option, + locally-disabled, + file-ignored, + suppressed-message, + useless-suppression, + deprecated-pragma, + use-symbolic-message-instead, + expression-not-assigned, + pointless-statement + +# Enable the message, report, category or checker with the given id(s). You can +# either give multiple identifier separated by comma (,) or put this option +# multiple time (only on the command line, not in the configuration file where +# it should appear only once). See also the "--disable" option for examples. +enable=c-extension-no-member + + +[METHOD_ARGS] + +# List of qualified names (i.e., library.method) which require a timeout +# parameter e.g. 'requests.api.get,requests.api.post' +timeout-methods=requests.api.delete,requests.api.get,requests.api.head,requests.api.options,requests.api.patch,requests.api.post,requests.api.put,requests.api.request + + +[MISCELLANEOUS] + +# List of note tags to take in consideration, separated by a comma. +notes=FIXME, + XXX, + TODO + +# Regular expression of note tags to take in consideration. +notes-rgx= + + +[REFACTORING] + +# Maximum number of nested blocks for function / method body +max-nested-blocks=5 + +# Complete name of functions that never returns. When checking for +# inconsistent-return-statements if a never returning function is called then +# it will be considered as an explicit return statement and no message will be +# printed. +never-returning-functions=sys.exit,argparse.parse_error + + +[REPORTS] + +# Python expression which should return a score less than or equal to 10. You +# have access to the variables 'fatal', 'error', 'warning', 'refactor', +# 'convention', and 'info' which contain the number of messages in each +# category, as well as 'statement' which is the total number of statements +# analyzed. This score is used by the global evaluation report (RP0004). +evaluation=max(0, 0 if fatal else 10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)) + +# Template used to display messages. This is a python new-style format string +# used to format the message information. See doc for all details. +msg-template= + +# Set the output format. Available formats are text, parseable, colorized, json +# and msvs (visual studio). You can also give a reporter class, e.g. +# mypackage.mymodule.MyReporterClass. +#output-format= + +# Tells whether to display a full report or only the messages. +reports=no + +# Activate the evaluation score. +score=yes + + +[SIMILARITIES] + +# Comments are removed from the similarity computation +ignore-comments=yes + +# Docstrings are removed from the similarity computation +ignore-docstrings=yes + +# Imports are removed from the similarity computation +ignore-imports=yes + +# Signatures are removed from the similarity computation +ignore-signatures=yes + +# Minimum lines number of a similarity. +min-similarity-lines=4 + + +[SPELLING] + +# Limits count of emitted suggestions for spelling mistakes. +max-spelling-suggestions=4 + +# Spelling dictionary name. No available dictionaries : You need to install +# both the python package and the system dependency for enchant to work.. +spelling-dict= + +# List of comma separated words that should be considered directives if they +# appear at the beginning of a comment and should not be checked. +spelling-ignore-comment-directives=fmt: on,fmt: off,noqa:,noqa,nosec,isort:skip,mypy: + +# List of comma separated words that should not be checked. +spelling-ignore-words= + +# A path to a file that contains the private dictionary; one word per line. +spelling-private-dict-file= + +# Tells whether to store unknown words to the private dictionary (see the +# --spelling-private-dict-file option) instead of raising a message. +spelling-store-unknown-words=no + + +[STRING] + +# This flag controls whether inconsistent-quotes generates a warning when the +# character used as a quote delimiter is used inconsistently within a module. +check-quote-consistency=no + +# This flag controls whether the implicit-str-concat should generate a warning +# on implicit string concatenation in sequences defined over several lines. +check-str-concat-over-line-jumps=no + + +[TYPECHECK] + +# List of decorators that produce context managers, such as +# contextlib.contextmanager. Add to this list to register other decorators that +# produce valid context managers. +contextmanager-decorators=contextlib.contextmanager + +# List of members which are set dynamically and missed by pylint inference +# system, and so shouldn't trigger E1101 when accessed. Python regular +# expressions are accepted. +generated-members= + +# Tells whether to warn about missing members when the owner of the attribute +# is inferred to be None. +ignore-none=yes + +# This flag controls whether pylint should warn about no-member and similar +# checks whenever an opaque object is returned when inferring. The inference +# can return multiple potential results while evaluating a Python object, but +# some branches might not be evaluated, which results in partial inference. In +# that case, it might be useful to still emit no-member and other checks for +# the rest of the inferred objects. +ignore-on-opaque-inference=yes + +# List of symbolic message names to ignore for Mixin members. +ignored-checks-for-mixins=no-member, + not-async-context-manager, + not-context-manager, + attribute-defined-outside-init + +# List of class names for which member attributes should not be checked (useful +# for classes with dynamically set attributes). This supports the use of +# qualified names. +ignored-classes=optparse.Values,thread._local,_thread._local,argparse.Namespace + +# Show a hint with possible names when a member name was not found. The aspect +# of finding the hint is based on edit distance. +missing-member-hint=yes + +# The minimum edit distance a name should have in order to be considered a +# similar match for a missing member name. +missing-member-hint-distance=1 + +# The total number of similar names that should be taken in consideration when +# showing a hint for a missing member. +missing-member-max-choices=1 + +# Regex pattern to define which classes are considered mixins. +mixin-class-rgx=.*[Mm]ixin + +# List of decorators that change the signature of a decorated function. +signature-mutators= + + +[VARIABLES] + +# List of additional names supposed to be defined in builtins. Remember that +# you should avoid defining new builtins when possible. +additional-builtins= + +# Tells whether unused global variables should be treated as a violation. +allow-global-unused-variables=yes + +# List of names allowed to shadow builtins +allowed-redefined-builtins= + +# List of strings which can identify a callback function by name. A callback +# name must start or end with one of those strings. +callbacks=cb_, + _cb + +# A regular expression matching the name of dummy variables (i.e. expected to +# not be used). +dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_ + +# Argument names that match this expression will be ignored. +ignored-argument-names=_.*|^ignored_|^unused_ + +# Tells whether we should check for unused import in __init__ files. +init-import=no + +# List of qualified module names which can have objects that can redefine +# builtins. +redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io diff --git a/docs/.well-known/ai-plugin.json b/docs/.well-known/ai-plugin.json new file mode 100644 index 000000000..ac0178fd0 --- /dev/null +++ b/docs/.well-known/ai-plugin.json @@ -0,0 +1,18 @@ +{ + "schema_version": "v1", + "name_for_model": "text processing tools", + "name_for_human": "MetaGPT Text Plugin", + "description_for_model": "Plugins for text processing, including text-to-speech, text-to-image, text-to-embedding, text summarization, text-to-code, vector similarity calculation, web content crawling, and more.", + "description_for_human": "Plugins for text processing, including text-to-speech, text-to-image, text-to-embedding, text summarization, text-to-code, vector similarity calculation, web content crawling, and more.", + "auth": { + "type": "none" + }, + "api": { + "type": "openapi", + "url": "https://github.com/iorisa/MetaGPT/blob/feature/assistant_role/.well-known/metagpt_oas3_api.yaml", + "has_user_authentication": false + }, + "logo_url": "https://github.com/geekan/MetaGPT/blob/main/docs/resources/MetaGPT-logo.png", + "contact_email": "mashenquan@fuzhi.cn", + "legal_info_url": "https://github.com/geekan/MetaGPT/blob/main/docs/README_CN.md" +} \ No newline at end of file diff --git a/docs/.well-known/metagpt_oas3_api.yaml b/docs/.well-known/metagpt_oas3_api.yaml new file mode 100644 index 000000000..0a702e8b6 --- /dev/null +++ b/docs/.well-known/metagpt_oas3_api.yaml @@ -0,0 +1,338 @@ +openapi: "3.0.0" + +info: + title: "MetaGPT Export OpenAPIs" + version: "1.0" +servers: + - url: "/oas3" + variables: + port: + default: '8080' + description: HTTP service port + +paths: + /tts/azsure: + x-prerequisite: + configurations: + AZURE_TTS_SUBSCRIPTION_KEY: + type: string + description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + AZURE_TTS_REGION: + type: string + description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + required: + allOf: + - AZURE_TTS_SUBSCRIPTION_KEY + - AZURE_TTS_REGION + post: + summary: "Convert Text to Base64-encoded .wav File Stream" + description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + operationId: azure_tts.oas3_azsure_tts + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - text + properties: + text: + type: string + description: Text to convert + lang: + type: string + description: The language code or locale, e.g., en-US (English - United States) + default: "zh-CN" + voice: + type: string + description: "Voice style, see: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts), [Voice Gallery](https://speech.microsoft.com/portal/voicegallery)" + default: "zh-CN-XiaomoNeural" + style: + type: string + description: "Speaking style to express different emotions. For more details, checkout: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + default: "affectionate" + role: + type: string + description: "Role to specify age and gender. For more details, checkout: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + default: "Girl" + subscription_key: + type: string + description: "Key used to access Azure AI service API, see: [Azure Portal](https://portal.azure.com/) > `Resource Management` > `Keys and Endpoint`" + default: "" + region: + type: string + description: "Location (or region) of your resource, see: [Azure Portal](https://portal.azure.com/) > `Resource Management` > `Keys and Endpoint`" + default: "" + responses: + '200': + description: "Base64-encoded .wav file data if successful, otherwise an empty string." + content: + application/json: + schema: + type: object + properties: + wav_data: + type: string + format: base64 + '400': + description: "Bad Request" + '500': + description: "Internal Server Error" + + /tts/iflytek: + x-prerequisite: + configurations: + IFLYTEK_APP_ID: + type: string + description: "Application ID is used to access your iFlyTek service API, see: `https://console.xfyun.cn/services/tts`" + IFLYTEK_API_KEY: + type: string + description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`" + IFLYTEK_API_SECRET: + type: string + description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`" + required: + allOf: + - IFLYTEK_APP_ID + - IFLYTEK_API_KEY + - IFLYTEK_API_SECRET + post: + summary: "Convert Text to Base64-encoded .mp3 File Stream" + description: "For more details, check out: [iFlyTek](https://console.xfyun.cn/services/tts)" + operationId: iflytek_tts.oas3_iflytek_tts + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - text + properties: + text: + type: string + description: Text to convert + voice: + type: string + description: "Voice style, see: [iFlyTek Text-to_Speech](https://www.xfyun.cn/doc/tts/online_tts/API.html#%E6%8E%A5%E5%8F%A3%E8%B0%83%E7%94%A8%E6%B5%81%E7%A8%8B)" + default: "xiaoyan" + app_id: + type: string + description: "Application ID is used to access your iFlyTek service API, see: `https://console.xfyun.cn/services/tts`" + default: "" + api_key: + type: string + description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`" + default: "" + api_secret: + type: string + description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`" + default: "" + responses: + '200': + description: "Base64-encoded .mp3 file data if successful, otherwise an empty string." + content: + application/json: + schema: + type: object + properties: + wav_data: + type: string + format: base64 + '400': + description: "Bad Request" + '500': + description: "Internal Server Error" + + + /txt2img/openai: + x-prerequisite: + configurations: + OPENAI_API_KEY: + type: string + description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`" + required: + allOf: + - OPENAI_API_KEY + post: + summary: "Convert Text to Base64-encoded Image Data Stream" + operationId: openai_text_to_image.oas3_openai_text_to_image + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + text: + type: string + description: "The text used for image conversion." + size_type: + type: string + enum: ["256x256", "512x512", "1024x1024"] + default: "1024x1024" + description: "Size of the generated image." + openai_api_key: + type: string + default: "" + description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`" + responses: + '200': + description: "Base64-encoded image data." + content: + application/json: + schema: + type: object + properties: + image_data: + type: string + format: base64 + '400': + description: "Bad Request" + '500': + description: "Internal Server Error" + /txt2embedding/openai: + x-prerequisite: + configurations: + OPENAI_API_KEY: + type: string + description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`" + required: + allOf: + - OPENAI_API_KEY + post: + summary: Text to embedding + operationId: openai_text_to_embedding.oas3_openai_text_to_embedding + description: Retrieve an embedding for the provided text using the OpenAI API. + requestBody: + content: + application/json: + schema: + type: object + properties: + input: + type: string + description: The text used for embedding. + model: + type: string + description: "ID of the model to use. For more details, checkout: [models](https://api.openai.com/v1/models)" + enum: + - text-embedding-ada-002 + responses: + "200": + description: Successful response + content: + application/json: + schema: + $ref: "#/components/schemas/ResultEmbedding" + "4XX": + description: Client error + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + "5XX": + description: Server error + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + + /txt2image/metagpt: + x-prerequisite: + configurations: + METAGPT_TEXT_TO_IMAGE_MODEL_URL: + type: string + description: "Model url." + required: + allOf: + - METAGPT_TEXT_TO_IMAGE_MODEL_URL + post: + summary: "Text to Image" + description: "Generate an image from the provided text using the MetaGPT Text-to-Image API." + operationId: metagpt_text_to_image.oas3_metagpt_text_to_image + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - text + properties: + text: + type: string + description: "The text used for image conversion." + size_type: + type: string + enum: ["512x512", "512x768"] + default: "512x512" + description: "Size of the generated image." + model_url: + type: string + description: "Model reset API URL for text-to-image." + default: "" + responses: + '200': + description: "Base64-encoded image data." + content: + application/json: + schema: + type: object + properties: + image_data: + type: string + format: base64 + '400': + description: "Bad Request" + '500': + description: "Internal Server Error" + +components: + schemas: + Embedding: + type: object + description: Represents an embedding vector returned by the embedding endpoint. + properties: + object: + type: string + example: embedding + embedding: + type: array + items: + type: number + example: [0.0023064255, -0.009327292, ...] + index: + type: integer + example: 0 + Usage: + type: object + properties: + prompt_tokens: + type: integer + example: 8 + total_tokens: + type: integer + example: 8 + ResultEmbedding: + type: object + properties: + object: + type: string + example: result_embedding + data: + type: array + items: + $ref: "#/components/schemas/Embedding" + model: + type: string + example: text-embedding-ada-002 + usage: + $ref: "#/components/schemas/Usage" + Error: + type: object + properties: + error: + type: string + example: An error occurred \ No newline at end of file diff --git a/docs/.well-known/openapi.yaml b/docs/.well-known/openapi.yaml new file mode 100644 index 000000000..47ca04b23 --- /dev/null +++ b/docs/.well-known/openapi.yaml @@ -0,0 +1,35 @@ +openapi: "3.0.0" + +info: + title: Hello World + version: "1.0" +servers: + - url: /openapi + +paths: + /greeting/{name}: + post: + summary: Generate greeting + description: Generates a greeting message. + operationId: openapi_v3_hello.post_greeting + responses: + 200: + description: greeting response + content: + text/plain: + schema: + type: string + example: "hello dave!" + parameters: + - name: name + in: path + description: Name of the person to greet. + required: true + schema: + type: string + example: "dave" + requestBody: + content: + application/json: + schema: + type: object \ No newline at end of file diff --git a/docs/.well-known/skills.yaml b/docs/.well-known/skills.yaml new file mode 100644 index 000000000..c19a9501e --- /dev/null +++ b/docs/.well-known/skills.yaml @@ -0,0 +1,161 @@ +skillapi: "0.1.0" + +info: + title: "Agent Skill Specification" + version: "1.0" + +entities: + Assistant: + summary: assistant + description: assistant + skills: + - name: text_to_speech + description: Generate a voice file from the input text, text-to-speech + id: text_to_speech.text_to_speech + x-prerequisite: + configurations: + AZURE_TTS_SUBSCRIPTION_KEY: + type: string + description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + AZURE_TTS_REGION: + type: string + description: "For more details, check out: [Azure Text-to_Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)" + IFLYTEK_APP_ID: + type: string + description: "Application ID is used to access your iFlyTek service API, see: `https://console.xfyun.cn/services/tts`" + IFLYTEK_API_KEY: + type: string + description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`" + IFLYTEK_API_SECRET: + type: string + description: "WebAPI argument, see: `https://console.xfyun.cn/services/tts`" + required: + oneOf: + - allOf: + - AZURE_TTS_SUBSCRIPTION_KEY + - AZURE_TTS_REGION + - allOf: + - IFLYTEK_APP_ID + - IFLYTEK_API_KEY + - IFLYTEK_API_SECRET + parameters: + text: + description: 'The text used for voice conversion.' + required: true + type: string + lang: + description: 'The value can contain a language code such as en (English), or a locale such as en-US (English - United States).' + type: string + enum: + - English + - Chinese + default: Chinese + voice: + description: Name of voice styles + type: string + default: zh-CN-XiaomoNeural + style: + type: string + description: Speaking style to express different emotions like cheerfulness, empathy, and calm. + enum: + - affectionate + - angry + - calm + - cheerful + - depressed + - disgruntled + - embarrassed + - envious + - fearful + - gentle + - sad + - serious + default: affectionate + role: + type: string + description: With roles, the same voice can act as a different age and gender. + enum: + - Girl + - Boy + - OlderAdultFemale + - OlderAdultMale + - SeniorFemale + - SeniorMale + - YoungAdultFemale + - YoungAdultMale + default: Girl + examples: + - ask: 'A girl says "hello world"' + answer: 'text_to_speech(text="hello world", role="Girl")' + - ask: 'A boy affectionate says "hello world"' + answer: 'text_to_speech(text="hello world", role="Boy", style="affectionate")' + - ask: 'A boy says "你好"' + answer: 'text_to_speech(text="你好", role="Boy", lang="Chinese")' + returns: + type: string + format: base64 + + - name: text_to_image + description: Create a drawing based on the text. + id: text_to_image.text_to_image + x-prerequisite: + configurations: + OPENAI_API_KEY: + type: string + description: "OpenAI API key, For more details, checkout: `https://platform.openai.com/account/api-keys`" + METAGPT_TEXT_TO_IMAGE_MODEL_URL: + type: string + description: "Model url." + required: + oneOf: + - OPENAI_API_KEY + - METAGPT_TEXT_TO_IMAGE_MODEL_URL + parameters: + text: + description: 'The text used for image conversion.' + type: string + required: true + size_type: + description: size type + type: string + default: "512x512" + examples: + - ask: 'Draw a girl' + answer: 'text_to_image(text="Draw a girl", size_type="512x512")' + - ask: 'Draw an apple' + answer: 'text_to_image(text="Draw an apple", size_type="512x512")' + returns: + type: string + format: base64 + + - name: web_search + description: Perform Google searches to provide real-time information. + id: web_search.web_search + x-prerequisite: + configurations: + SEARCH_ENGINE: + type: string + description: "Supported values: serpapi/google/serper/ddg" + SERPER_API_KEY: + type: string + description: "SERPER API KEY, For more details, checkout: `https://serper.dev/api-key`" + required: + allOf: + - SEARCH_ENGINE + - SERPER_API_KEY + parameters: + query: + type: string + description: 'The search query.' + required: true + max_results: + type: number + default: 6 + description: 'The number of search results to retrieve.' + examples: + - ask: 'Search for information about artificial intelligence' + answer: 'web_search(query="Search for information about artificial intelligence", max_results=6)' + - ask: 'Find news articles about climate change' + answer: 'web_search(query="Find news articles about climate change", max_results=6)' + returns: + type: string diff --git a/docs/FAQ-EN.md b/docs/FAQ-EN.md index f9df50caf..d4a9f6097 100644 --- a/docs/FAQ-EN.md +++ b/docs/FAQ-EN.md @@ -83,10 +83,10 @@ 1. PRD stuck / unable to access/ connection interrupted - 1. The official OPENAI_API_BASE address is `https://api.openai.com/v1` - 1. If the official OPENAI_API_BASE address is inaccessible in your environment (this can be verified with curl), it's recommended to configure using the reverse proxy OPENAI_API_BASE provided by libraries such as openai-forward. For instance, `OPENAI_API_BASE: "``https://api.openai-forward.com/v1``"` - 1. If the official OPENAI_API_BASE address is inaccessible in your environment (again, verifiable via curl), another option is to configure the OPENAI_PROXY parameter. This way, you can access the official OPENAI_API_BASE via a local proxy. If you don't need to access via a proxy, please do not enable this configuration; if accessing through a proxy is required, modify it to the correct proxy address. Note that when OPENAI_PROXY is enabled, don't set OPENAI_API_BASE. - 1. Note: OpenAI's default API design ends with a v1. An example of the correct configuration is: `OPENAI_API_BASE: "``https://api.openai.com/v1``"` + 1. The official OPENAI_BASE_URL address is `https://api.openai.com/v1` + 1. If the official OPENAI_BASE_URL address is inaccessible in your environment (this can be verified with curl), it's recommended to configure using the reverse proxy OPENAI_BASE_URL provided by libraries such as openai-forward. For instance, `OPENAI_BASE_URL: "``https://api.openai-forward.com/v1``"` + 1. If the official OPENAI_BASE_URL address is inaccessible in your environment (again, verifiable via curl), another option is to configure the OPENAI_PROXY parameter. This way, you can access the official OPENAI_BASE_URL via a local proxy. If you don't need to access via a proxy, please do not enable this configuration; if accessing through a proxy is required, modify it to the correct proxy address. Note that when OPENAI_PROXY is enabled, don't set OPENAI_BASE_URL. + 1. Note: OpenAI's default API design ends with a v1. An example of the correct configuration is: `OPENAI_BASE_URL: "``https://api.openai.com/v1``"` 1. Absolutely! How can I assist you today? @@ -98,7 +98,7 @@ 1. How to change the investment amount? - 1. You can view all commands by typing `python startup.py --help` + 1. You can view all commands by typing `metagpt --help` 1. Which version of Python is more stable? @@ -134,7 +134,7 @@ 1. Configuration instructions for SD Skills: The SD interface is currently deployed based on *https://github.com/AUTOMATIC1111/stable-diffusion-webui* **For environmental configurations and model downloads, please refer to the aforementioned GitHub repository. To initiate the SD service that supports API calls, run the command specified in cmd with the parameter nowebui, i.e., - 1. > python webui.py --enable-insecure-extension-access --port xxx --no-gradio-queue --nowebui + 1. > python3 webui.py --enable-insecure-extension-access --port xxx --no-gradio-queue --nowebui 1. Once it runs without errors, the interface will be accessible after approximately 1 minute when the model finishes loading. 1. Configure SD_URL and SD_T2I_API in the config.yaml/key.yaml files. 1.  diff --git a/docs/README_CN.md b/docs/README_CN.md index 409bdc7af..2855b5500 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -47,9 +47,9 @@ # 第 2 步：克隆最新仓库到您的本地机器，并进行安装。 cd MetaGPT pip3 install -e. # 或者 pip3 install metagpt # 安装稳定版本 -# 第 3 步：执行startup.py +# 第 3 步：执行metagpt # 拷贝config.yaml为key.yaml，并设置你自己的OPENAI_API_KEY -python3 startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # 第 4 步【可选的】：如果你想在执行过程中保存像象限图、系统设计、序列流程等图表这些产物，可以在第3步前执行该步骤。默认的，框架做了兼容，在不执行该步的情况下，也可以完整跑完整个流程。 # 如果执行，确保您的系统上安装了 NPM。并使用npm安装mermaid-js @@ -60,6 +60,7 @@ # 如果执行，确保您的系统上安装了 NPM。并使用npm安装mermaid- 详细的安装请安装 [cli_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-stable-version) ### Docker安装 +> 注意：在Windows中，你需要将 "/opt/metagpt" 替换为Docker具有创建权限的目录，比如"D:\Users\x\metagpt" ```bash # 步骤1: 下载metagpt官方镜像并准备好config.yaml @@ -74,10 +75,10 @@ # 步骤2: 使用容器运行metagpt演示 -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" ``` -详细的安装请安装 [docker_install](https://docs.deepwisdom.ai/zhcn/guide/get_started/installation.html#%E4%BD%BF%E7%94%A8docker%E5%AE%89%E8%A3%85) +详细的安装请安装 [docker_install](https://docs.deepwisdom.ai/main/zh/guide/get_started/installation.html#%E4%BD%BF%E7%94%A8docker%E5%AE%89%E8%A3%85) ### 快速开始的演示视频 - 在 [MetaGPT Huggingface Space](https://huggingface.co/spaces/deepwisdom/MetaGPT) 上进行体验 @@ -87,19 +88,19 @@ ### 快速开始的演示视频 https://github.com/geekan/MetaGPT/assets/34952977/34345016-5d13-489d-b9f9-b82ace413419 ## 教程 -- 🗒 [在线文档](https://docs.deepwisdom.ai/zhcn/) -- 💻 [如何使用](https://docs.deepwisdom.ai/zhcn/guide/get_started/quickstart.html) -- 🔎 [MetaGPT的能力及应用场景](https://docs.deepwisdom.ai/zhcn/guide/get_started/introduction.html) +- 🗒 [在线文档](https://docs.deepwisdom.ai/main/zh/) +- 💻 [如何使用](https://docs.deepwisdom.ai/main/zh/guide/get_started/quickstart.html) +- 🔎 [MetaGPT的能力及应用场景](https://docs.deepwisdom.ai/main/zh/guide/get_started/introduction.html) - 🛠 如何构建你自己的智能体？ - - [MetaGPT的使用和开发教程 | 智能体入门](https://docs.deepwisdom.ai/zhcn/guide/tutorials/agent_101.html) - - [MetaGPT的使用和开发教程 | 多智能体入门](https://docs.deepwisdom.ai/zhcn/guide/tutorials/multi_agent_101.html) + - [MetaGPT的使用和开发教程 | 智能体入门](https://docs.deepwisdom.ai/main/zh/guide/tutorials/agent_101.html) + - [MetaGPT的使用和开发教程 | 多智能体入门](https://docs.deepwisdom.ai/main/zh/guide/tutorials/multi_agent_101.html) - 🧑💻 贡献 - [开发路线图](ROADMAP.md) - 🔖 示例 - - [辩论](https://docs.deepwisdom.ai/zhcn/guide/use_cases/multi_agent/debate.html) - - [调研员](https://docs.deepwisdom.ai/zhcn/guide/use_cases/agent/researcher.html) - - [票据助手](https://docs.deepwisdom.ai/zhcn/guide/use_cases/agent/receipt_assistant.html) -- ❓ [常见问题解答](https://docs.deepwisdom.ai/zhcn/guide/faq.html) + - [辩论](https://docs.deepwisdom.ai/main/zh/guide/use_cases/multi_agent/debate.html) + - [调研员](https://docs.deepwisdom.ai/main/zh/guide/use_cases/agent/researcher.html) + - [票据助手](https://docs.deepwisdom.ai/main/zh/guide/use_cases/agent/receipt_assistant.html) +- ❓ [常见问题解答](https://docs.deepwisdom.ai/main/zh/guide/faq.html) ## 支持 @@ -113,14 +114,14 @@ ### 联系信息 如果您对这个项目有任何问题或反馈，欢迎联系我们。我们非常欢迎您的建议！ -- **邮箱：** alexanderwu@fuzhi.ai +- **邮箱：** alexanderwu@deepwisdom.ai - **GitHub 问题：** 对于更技术性的问题，您也可以在我们的 [GitHub 仓库](https://github.com/geekan/metagpt/issues) 中创建一个新的问题。 我们会在2-3个工作日内回复所有问题。 ## 引用 -引用 [Arxiv paper](https://arxiv.org/abs/2308.00352): +引用 [arXiv paper](https://arxiv.org/abs/2308.00352): ```bibtex @misc{hong2023metagpt, diff --git a/docs/README_JA.md b/docs/README_JA.md index 10cb7ee82..8b2bf1fae 100644 --- a/docs/README_JA.md +++ b/docs/README_JA.md @@ -41,7 +41,7 @@ ## MetaGPT の能力 ## 例（GPT-4 で完全生成） -例えば、`python startup.py "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます +例えば、`metagpt "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます  @@ -60,16 +60,16 @@ ### 伝統的なインストール ```bash # ステップ 1: Python 3.9+ がシステムにインストールされていることを確認してください。これを確認するには: -python --version +python3 --version # ステップ 2: リポジトリをローカルマシンにクローンし、インストールする。 git clone https://github.com/geekan/MetaGPT.git cd MetaGPT pip install -e. -# ステップ 3: startup.py を実行する +# ステップ 3: metagpt を実行する # config.yaml を key.yaml にコピーし、独自の OPENAI_API_KEY を設定します -python3 startup.py "Write a cli snake game" +metagpt "Write a cli snake game" # ステップ 4 [オプション]: 実行中に PRD ファイルなどのアーティファクトを保存する場合は、ステップ 3 の前にこのステップを実行できます。デフォルトでは、フレームワークには互換性があり、この手順を実行しなくてもプロセス全体を完了できます。 # NPM がシステムにインストールされていることを確認してください。次に mermaid-js をインストールします。(お使いのコンピューターに npm がない場合は、Node.js 公式サイトで Node.js https://nodejs.org/ をインストールしてください。） @@ -163,6 +163,7 @@ # NPM がシステムにインストールされていることを確認して 注: この方法は pdf エクスポートに対応していません。 ### Docker によるインストール +> Windowsでは、"/opt/metagpt"をDockerが作成する権限を持つディレクトリに置き換える必要があります。例えば、"D:\Users\x\metagpt"などです。 ```bash # ステップ 1: metagpt 公式イメージをダウンロードし、config.yaml を準備する @@ -177,7 +178,7 @@ # ステップ 2: コンテナで metagpt デモを実行する -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ metagpt/metagpt:latest \ - python startup.py "Write a cli snake game" + metagpt "Write a cli snake game" # コンテナを起動し、その中でコマンドを実行することもできます docker run --name metagpt -d \ @@ -187,7 +188,7 @@ # コンテナを起動し、その中でコマンドを実行することもで metagpt/metagpt:latest docker exec -it metagpt /bin/bash -$ python startup.py "Write a cli snake game" +$ metagpt "Write a cli snake game" ``` コマンド `docker run ...` は以下のことを行います: @@ -195,7 +196,7 @@ # コンテナを起動し、その中でコマンドを実行することもで - 特権モードで実行し、ブラウザの実行権限を得る - ホスト設定ファイル `/opt/metagpt/config/key.yaml` をコンテナ `/app/metagpt/config/key.yaml` にマップします - ホストディレクトリ `/opt/metagpt/workspace` をコンテナディレクトリ `/app/metagpt/workspace` にマップするs -- デモコマンド `python startup.py "Write a cli snake game"` を実行する +- デモコマンド `metagpt "Write a cli snake game"` を実行する ### 自分でイメージをビルドする @@ -218,17 +219,17 @@ # 設定ファイルをコピーし、必要な修正を加える。 | 変数名 | config/key.yaml | env | | --------------------------------------- | ----------------------------------------- | ----------------------------------------------- | | OPENAI_API_KEY # 自分のキーに置き換える | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." | -| OPENAI_API_BASE # オプション | OPENAI_API_BASE: "https://.*)(```.*?)",
- r"(.*?```python.*?\s+)?(?P.*)",
- ):
+ for pattern in (r"(.*?```python.*?\s+)?(?P.*)(```.*?)", r"(.*?```python.*?\s+)?(?P.*)"):
match = re.search(pattern, text, re.DOTALL)
if not match:
continue
@@ -109,18 +131,28 @@ class OutputParser:
try:
content = cls.parse_code(text=content)
except Exception:
- pass
-
- # 尝试解析list
- try:
- content = cls.parse_file_list(text=content)
- except Exception:
- pass
+ # 尝试解析list
+ try:
+ content = cls.parse_file_list(text=content)
+ except Exception:
+ pass
parsed_data[block] = content
return parsed_data
+ @staticmethod
+ def extract_content(text, tag="CONTENT"):
+ # Use regular expression to extract content between [CONTENT] and [/CONTENT]
+ extracted_content = re.search(rf"\[{tag}\](.*?)\[/{tag}\]", text, re.DOTALL)
+
+ if extracted_content:
+ return extracted_content.group(1).strip()
+ else:
+ raise ValueError(f"Could not find content between [{tag}] and [/{tag}]")
+
@classmethod
def parse_data_with_mapping(cls, data, mapping):
+ if "[CONTENT]" in data:
+ data = cls.extract_content(text=data)
block_dict = cls.parse_blocks(data)
parsed_data = {}
for block, content in block_dict.items():
@@ -187,7 +219,7 @@ class OutputParser:
result = ast.literal_eval(structure_text)
# Ensure the result matches the specified data type
- if isinstance(result, list) or isinstance(result, dict):
+ if isinstance(result, (list, dict)):
return result
raise ValueError(f"The extracted structure is not a {data_type}.")
@@ -219,10 +251,15 @@ class CodeParser:
# 遍历所有的block
for block in blocks:
# 如果block不为空，则继续处理
- if block.strip() != "":
+ if block.strip() == "":
+ continue
+ if "\n" not in block:
+ block_title = block
+ block_content = ""
+ else:
# 将block的标题和内容分开，并分别去掉前后的空白字符
block_title, block_content = block.split("\n", 1)
- block_dict[block_title.strip()] = block_content.strip()
+ block_dict[block_title.strip()] = block_content.strip()
return block_dict
@@ -282,9 +319,6 @@ class NoMoneyException(Exception):
def print_members(module, indent=0):
"""
https://stackoverflow.com/questions/1796180/how-can-i-get-a-list-of-all-classes-within-current-module-in-python
- :param module:
- :param indent:
- :return:
"""
prefix = " " * indent
for name, obj in inspect.getmembers(module):
@@ -302,9 +336,16 @@ def print_members(module, indent=0):
def parse_recipient(text):
+ # FIXME: use ActionNode instead.
pattern = r"## Send To:\s*([A-Za-z]+)\s*?" # hard code for now
recipient = re.search(pattern, text)
- return recipient.group(1) if recipient else ""
+ if recipient:
+ return recipient.group(1)
+ pattern = r"Send To:\s*([A-Za-z]+)\s*?"
+ recipient = re.search(pattern, text)
+ if recipient:
+ return recipient.group(1)
+ return ""
def create_func_config(func_schema: dict) -> dict:
@@ -329,3 +370,224 @@ def remove_comments(code_str):
clean_code = re.sub(pattern, replace_func, code_str, flags=re.MULTILINE)
clean_code = os.linesep.join([s.rstrip() for s in clean_code.splitlines() if s.strip()])
return clean_code
+
+
+def get_class_name(cls) -> str:
+ """Return class name"""
+ return f"{cls.__module__}.{cls.__name__}"
+
+
+def any_to_str(val: Any) -> str:
+ """Return the class name or the class name of the object, or 'val' if it's a string type."""
+ if isinstance(val, str):
+ return val
+ elif not callable(val):
+ return get_class_name(type(val))
+ else:
+ return get_class_name(val)
+
+
+def any_to_str_set(val) -> set:
+ """Convert any type to string set."""
+ res = set()
+
+ # Check if the value is iterable, but not a string (since strings are technically iterable)
+ if isinstance(val, (dict, list, set, tuple)):
+ # Special handling for dictionaries to iterate over values
+ if isinstance(val, dict):
+ val = val.values()
+
+ for i in val:
+ res.add(any_to_str(i))
+ else:
+ res.add(any_to_str(val))
+
+ return res
+
+
+def is_subscribed(message: "Message", tags: set):
+ """Return whether it's consumer"""
+ if MESSAGE_ROUTE_TO_ALL in message.send_to:
+ return True
+
+ for i in tags:
+ if i in message.send_to:
+ return True
+ return False
+
+
+def any_to_name(val):
+ """
+ Convert a value to its name by extracting the last part of the dotted path.
+
+ :param val: The value to convert.
+
+ :return: The name of the value.
+ """
+ return any_to_str(val).split(".")[-1]
+
+
+def concat_namespace(*args) -> str:
+ return ":".join(str(value) for value in args)
+
+
+def split_namespace(ns_class_name: str) -> List[str]:
+ return ns_class_name.split(":")
+
+
+def general_after_log(i: "loguru.Logger", sec_format: str = "%0.3f") -> typing.Callable[["RetryCallState"], None]:
+ """
+ Generates a logging function to be used after a call is retried.
+
+ This generated function logs an error message with the outcome of the retried function call. It includes
+ the name of the function, the time taken for the call in seconds (formatted according to `sec_format`),
+ the number of attempts made, and the exception raised, if any.
+
+ :param i: A Logger instance from the loguru library used to log the error message.
+ :param sec_format: A string format specifier for how to format the number of seconds since the start of the call.
+ Defaults to three decimal places.
+ :return: A callable that accepts a RetryCallState object and returns None. This callable logs the details
+ of the retried call.
+ """
+
+ def log_it(retry_state: "RetryCallState") -> None:
+ # If the function name is not known, default to ""
+ if retry_state.fn is None:
+ fn_name = ""
+ else:
+ # Retrieve the callable's name using a utility function
+ fn_name = _utils.get_callback_name(retry_state.fn)
+
+ # Log an error message with the function name, time since start, attempt number, and the exception
+ i.error(
+ f"Finished call to '{fn_name}' after {sec_format % retry_state.seconds_since_start}(s), "
+ f"this was the {_utils.to_ordinal(retry_state.attempt_number)} time calling it. "
+ f"exp: {retry_state.outcome.exception()}"
+ )
+
+ return log_it
+
+
+def read_json_file(json_file: str, encoding="utf-8") -> list[Any]:
+ if not Path(json_file).exists():
+ raise FileNotFoundError(f"json_file: {json_file} not exist, return []")
+
+ with open(json_file, "r", encoding=encoding) as fin:
+ try:
+ data = json.load(fin)
+ except Exception:
+ raise ValueError(f"read json file: {json_file} failed")
+ return data
+
+
+def write_json_file(json_file: str, data: list, encoding=None):
+ folder_path = Path(json_file).parent
+ if not folder_path.exists():
+ folder_path.mkdir(parents=True, exist_ok=True)
+
+ with open(json_file, "w", encoding=encoding) as fout:
+ json.dump(data, fout, ensure_ascii=False, indent=4, default=to_jsonable_python)
+
+
+def import_class(class_name: str, module_name: str) -> type:
+ module = importlib.import_module(module_name)
+ a_class = getattr(module, class_name)
+ return a_class
+
+
+def import_class_inst(class_name: str, module_name: str, *args, **kwargs) -> object:
+ a_class = import_class(class_name, module_name)
+ class_inst = a_class(*args, **kwargs)
+ return class_inst
+
+
+def format_trackback_info(limit: int = 2):
+ return traceback.format_exc(limit=limit)
+
+
+def serialize_decorator(func):
+ async def wrapper(self, *args, **kwargs):
+ try:
+ result = await func(self, *args, **kwargs)
+ return result
+ except KeyboardInterrupt:
+ logger.error(f"KeyboardInterrupt occurs, start to serialize the project, exp:\n{format_trackback_info()}")
+ except Exception:
+ logger.error(f"Exception occurs, start to serialize the project, exp:\n{format_trackback_info()}")
+ self.serialize() # Team.serialize
+
+ return wrapper
+
+
+def role_raise_decorator(func):
+ async def wrapper(self, *args, **kwargs):
+ try:
+ return await func(self, *args, **kwargs)
+ except KeyboardInterrupt as kbi:
+ logger.error(f"KeyboardInterrupt: {kbi} occurs, start to serialize the project")
+ if self.latest_observed_msg:
+ self.rc.memory.delete(self.latest_observed_msg)
+ # raise again to make it captured outside
+ raise Exception(format_trackback_info(limit=None))
+ except Exception:
+ if self.latest_observed_msg:
+ logger.warning(
+ "There is a exception in role's execution, in order to resume, "
+ "we delete the newest role communication message in the role's memory."
+ )
+ # remove role newest observed msg to make it observed again
+ self.rc.memory.delete(self.latest_observed_msg)
+ # raise again to make it captured outside
+ raise Exception(format_trackback_info(limit=None))
+
+ return wrapper
+
+
+@handle_exception
+async def aread(filename: str | Path, encoding=None) -> str:
+ """Read file asynchronously."""
+ async with aiofiles.open(str(filename), mode="r", encoding=encoding) as reader:
+ content = await reader.read()
+ return content
+
+
+async def awrite(filename: str | Path, data: str, encoding=None):
+ """Write file asynchronously."""
+ pathname = Path(filename)
+ pathname.parent.mkdir(parents=True, exist_ok=True)
+ async with aiofiles.open(str(pathname), mode="w", encoding=encoding) as writer:
+ await writer.write(data)
+
+
+async def read_file_block(filename: str | Path, lineno: int, end_lineno: int):
+ if not Path(filename).exists():
+ return ""
+ lines = []
+ async with aiofiles.open(str(filename), mode="r") as reader:
+ ix = 0
+ while ix < end_lineno:
+ ix += 1
+ line = await reader.readline()
+ if ix < lineno:
+ continue
+ if ix > end_lineno:
+ break
+ lines.append(line)
+ return "".join(lines)
+
+
+def list_files(root: str | Path) -> List[Path]:
+ files = []
+ try:
+ directory_path = Path(root)
+ if not directory_path.exists():
+ return []
+ for file_path in directory_path.iterdir():
+ if file_path.is_file():
+ files.append(file_path)
+ else:
+ subfolder_files = list_files(root=file_path)
+ files.extend(subfolder_files)
+ except Exception as e:
+ logger.error(f"Error: {e}")
+ return files
diff --git a/metagpt/utils/cost_manager.py b/metagpt/utils/cost_manager.py
new file mode 100644
index 000000000..ce53f2285
--- /dev/null
+++ b/metagpt/utils/cost_manager.py
@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/8/28
+@Author : mashenquan
+@File : openai.py
+@Desc : mashenquan, 2023/8/28. Separate the `CostManager` class to support user-level cost accounting.
+"""
+
+from typing import NamedTuple
+
+from pydantic import BaseModel
+
+from metagpt.logs import logger
+from metagpt.utils.token_counter import TOKEN_COSTS
+
+
+class Costs(NamedTuple):
+ total_prompt_tokens: int
+ total_completion_tokens: int
+ total_cost: float
+ total_budget: float
+
+
+class CostManager(BaseModel):
+ """Calculate the overhead of using the interface."""
+
+ total_prompt_tokens: int = 0
+ total_completion_tokens: int = 0
+ total_budget: float = 0
+ max_budget: float = 10.0
+ total_cost: float = 0
+
+ def update_cost(self, prompt_tokens, completion_tokens, model):
+ """
+ Update the total cost, prompt tokens, and completion tokens.
+
+ Args:
+ prompt_tokens (int): The number of tokens used in the prompt.
+ completion_tokens (int): The number of tokens used in the completion.
+ model (str): The model used for the API call.
+ """
+ self.total_prompt_tokens += prompt_tokens
+ self.total_completion_tokens += completion_tokens
+ cost = (
+ prompt_tokens * TOKEN_COSTS[model]["prompt"] + completion_tokens * TOKEN_COSTS[model]["completion"]
+ ) / 1000
+ self.total_cost += cost
+ logger.info(
+ f"Total running cost: ${self.total_cost:.3f} | Max budget: ${self.max_budget:.3f} | "
+ f"Current cost: ${cost:.3f}, prompt_tokens: {prompt_tokens}, completion_tokens: {completion_tokens}"
+ )
+
+ def get_total_prompt_tokens(self):
+ """
+ Get the total number of prompt tokens.
+
+ Returns:
+ int: The total number of prompt tokens.
+ """
+ return self.total_prompt_tokens
+
+ def get_total_completion_tokens(self):
+ """
+ Get the total number of completion tokens.
+
+ Returns:
+ int: The total number of completion tokens.
+ """
+ return self.total_completion_tokens
+
+ def get_total_cost(self):
+ """
+ Get the total cost of API calls.
+
+ Returns:
+ float: The total cost of API calls.
+ """
+ return self.total_cost
+
+ def get_costs(self) -> Costs:
+ """Get all costs"""
+ return Costs(self.total_prompt_tokens, self.total_completion_tokens, self.total_cost, self.total_budget)
diff --git a/metagpt/utils/custom_decoder.py b/metagpt/utils/custom_decoder.py
index 373d16356..eb01a1115 100644
--- a/metagpt/utils/custom_decoder.py
+++ b/metagpt/utils/custom_decoder.py
@@ -25,7 +25,7 @@ def py_make_scanner(context):
except IndexError:
raise StopIteration(idx) from None
- if nextchar == '"' or nextchar == "'":
+ if nextchar in ("'", '"'):
if idx + 2 < len(string) and string[idx + 1] == nextchar and string[idx + 2] == nextchar:
# Handle the case where the next two characters are the same as nextchar
return parse_string(string, idx + 3, strict, delimiter=nextchar * 3) # triple quote
diff --git a/metagpt/utils/dependency_file.py b/metagpt/utils/dependency_file.py
new file mode 100644
index 000000000..7cf9a1d49
--- /dev/null
+++ b/metagpt/utils/dependency_file.py
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/22
+@Author : mashenquan
+@File : dependency_file.py
+@Desc: Implementation of the dependency file described in Section 2.2.3.2 of RFC 135.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Set
+
+import aiofiles
+
+from metagpt.utils.common import aread
+from metagpt.utils.exceptions import handle_exception
+
+
+class DependencyFile:
+ """A class representing a DependencyFile for managing dependencies.
+
+ :param workdir: The working directory path for the DependencyFile.
+ """
+
+ def __init__(self, workdir: Path | str):
+ """Initialize a DependencyFile instance.
+
+ :param workdir: The working directory path for the DependencyFile.
+ """
+ self._dependencies = {}
+ self._filename = Path(workdir) / ".dependencies.json"
+
+ async def load(self):
+ """Load dependencies from the file asynchronously."""
+ if not self._filename.exists():
+ return
+ self._dependencies = json.loads(await aread(self._filename))
+
+ @handle_exception
+ async def save(self):
+ """Save dependencies to the file asynchronously."""
+ data = json.dumps(self._dependencies)
+ async with aiofiles.open(str(self._filename), mode="w") as writer:
+ await writer.write(data)
+
+ async def update(self, filename: Path | str, dependencies: Set[Path | str], persist=True):
+ """Update dependencies for a file asynchronously.
+
+ :param filename: The filename or path.
+ :param dependencies: The set of dependencies.
+ :param persist: Whether to persist the changes immediately.
+ """
+ if persist:
+ await self.load()
+
+ root = self._filename.parent
+ try:
+ key = Path(filename).relative_to(root)
+ except ValueError:
+ key = filename
+
+ if dependencies:
+ relative_paths = []
+ for i in dependencies:
+ try:
+ relative_paths.append(str(Path(i).relative_to(root)))
+ except ValueError:
+ relative_paths.append(str(i))
+ self._dependencies[str(key)] = relative_paths
+ elif str(key) in self._dependencies:
+ del self._dependencies[str(key)]
+
+ if persist:
+ await self.save()
+
+ async def get(self, filename: Path | str, persist=True):
+ """Get dependencies for a file asynchronously.
+
+ :param filename: The filename or path.
+ :param persist: Whether to load dependencies from the file immediately.
+ :return: A set of dependencies.
+ """
+ if persist:
+ await self.load()
+
+ root = self._filename.parent
+ try:
+ key = Path(filename).relative_to(root)
+ except ValueError:
+ key = filename
+ return set(self._dependencies.get(str(key), {}))
+
+ def delete_file(self):
+ """Delete the dependency file."""
+ self._filename.unlink(missing_ok=True)
+
+ @property
+ def exists(self):
+ """Check if the dependency file exists."""
+ return self._filename.exists()
diff --git a/metagpt/utils/di_graph_repository.py b/metagpt/utils/di_graph_repository.py
new file mode 100644
index 000000000..8bb5f9bb3
--- /dev/null
+++ b/metagpt/utils/di_graph_repository.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/12/19
+@Author : mashenquan
+@File : di_graph_repository.py
+@Desc : Graph repository based on DiGraph
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import List
+
+import networkx
+
+from metagpt.utils.common import aread, awrite
+from metagpt.utils.graph_repository import SPO, GraphRepository
+
+
+class DiGraphRepository(GraphRepository):
+ def __init__(self, name: str, **kwargs):
+ super().__init__(name=name, **kwargs)
+ self._repo = networkx.DiGraph()
+
+ async def insert(self, subject: str, predicate: str, object_: str):
+ self._repo.add_edge(subject, object_, predicate=predicate)
+
+ async def upsert(self, subject: str, predicate: str, object_: str):
+ pass
+
+ async def update(self, subject: str, predicate: str, object_: str):
+ pass
+
+ async def select(self, subject: str = None, predicate: str = None, object_: str = None) -> List[SPO]:
+ result = []
+ for s, o, p in self._repo.edges(data="predicate"):
+ if subject and subject != s:
+ continue
+ if predicate and predicate != p:
+ continue
+ if object_ and object_ != o:
+ continue
+ result.append(SPO(subject=s, predicate=p, object_=o))
+ return result
+
+ def json(self) -> str:
+ m = networkx.node_link_data(self._repo)
+ data = json.dumps(m)
+ return data
+
+ async def save(self, path: str | Path = None):
+ data = self.json()
+ path = path or self._kwargs.get("root")
+ if not path.exists():
+ path.mkdir(parents=True, exist_ok=True)
+ pathname = Path(path) / self.name
+ await awrite(filename=pathname.with_suffix(".json"), data=data, encoding="utf-8")
+
+ async def load(self, pathname: str | Path):
+ data = await aread(filename=pathname, encoding="utf-8")
+ m = json.loads(data)
+ self._repo = networkx.node_link_graph(m)
+
+ @staticmethod
+ async def load_from(pathname: str | Path) -> GraphRepository:
+ pathname = Path(pathname)
+ name = pathname.with_suffix("").name
+ root = pathname.parent
+ graph = DiGraphRepository(name=name, root=root)
+ if pathname.exists():
+ await graph.load(pathname=pathname)
+ return graph
+
+ @property
+ def root(self) -> str:
+ return self._kwargs.get("root")
+
+ @property
+ def pathname(self) -> Path:
+ p = Path(self.root) / self.name
+ return p.with_suffix(".json")
diff --git a/metagpt/utils/exceptions.py b/metagpt/utils/exceptions.py
new file mode 100644
index 000000000..70ed45910
--- /dev/null
+++ b/metagpt/utils/exceptions.py
@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/12/19 14:46
+@Author : alexanderwu
+@File : exceptions.py
+"""
+
+
+import asyncio
+import functools
+import traceback
+from typing import Any, Callable, Tuple, Type, TypeVar, Union
+
+from metagpt.logs import logger
+
+ReturnType = TypeVar("ReturnType")
+
+
+def handle_exception(
+ _func: Callable[..., ReturnType] = None,
+ *,
+ exception_type: Union[Type[Exception], Tuple[Type[Exception], ...]] = Exception,
+ exception_msg: str = "",
+ default_return: Any = None,
+) -> Callable[..., ReturnType]:
+ """handle exception, return default value"""
+
+ def decorator(func: Callable[..., ReturnType]) -> Callable[..., ReturnType]:
+ @functools.wraps(func)
+ async def async_wrapper(*args: Any, **kwargs: Any) -> ReturnType:
+ try:
+ return await func(*args, **kwargs)
+ except exception_type as e:
+ logger.opt(depth=1).error(
+ f"{e}: {exception_msg}, "
+ f"\nCalling {func.__name__} with args: {args}, kwargs: {kwargs} "
+ f"\nStack: {traceback.format_exc()}"
+ )
+ return default_return
+
+ @functools.wraps(func)
+ def sync_wrapper(*args: Any, **kwargs: Any) -> ReturnType:
+ try:
+ return func(*args, **kwargs)
+ except exception_type as e:
+ logger.opt(depth=1).error(
+ f"Calling {func.__name__} with args: {args}, kwargs: {kwargs} failed: {e}, "
+ f"stack: {traceback.format_exc()}"
+ )
+ return default_return
+
+ if asyncio.iscoroutinefunction(func):
+ return async_wrapper
+ else:
+ return sync_wrapper
+
+ if _func is None:
+ return decorator
+ else:
+ return decorator(_func)
diff --git a/metagpt/utils/file.py b/metagpt/utils/file.py
index f3691549b..f62b44eb8 100644
--- a/metagpt/utils/file.py
+++ b/metagpt/utils/file.py
@@ -6,10 +6,12 @@
@File : file.py
@Describe : General file operations.
"""
-import aiofiles
from pathlib import Path
+import aiofiles
+
from metagpt.logs import logger
+from metagpt.utils.exceptions import handle_exception
class File:
@@ -18,6 +20,7 @@ class File:
CHUNK_SIZE = 64 * 1024
@classmethod
+ @handle_exception
async def write(cls, root_path: Path, filename: str, content: bytes) -> Path:
"""Write the file content to the local specified path.
@@ -32,18 +35,15 @@ class File:
Raises:
Exception: If an unexpected error occurs during the file writing process.
"""
- try:
- root_path.mkdir(parents=True, exist_ok=True)
- full_path = root_path / filename
- async with aiofiles.open(full_path, mode="wb") as writer:
- await writer.write(content)
- logger.debug(f"Successfully write file: {full_path}")
- return full_path
- except Exception as e:
- logger.error(f"Error writing file: {e}")
- raise e
+ root_path.mkdir(parents=True, exist_ok=True)
+ full_path = root_path / filename
+ async with aiofiles.open(full_path, mode="wb") as writer:
+ await writer.write(content)
+ logger.debug(f"Successfully write file: {full_path}")
+ return full_path
@classmethod
+ @handle_exception
async def read(cls, file_path: Path, chunk_size: int = None) -> bytes:
"""Partitioning read the file content from the local specified path.
@@ -57,19 +57,14 @@ class File:
Raises:
Exception: If an unexpected error occurs during the file reading process.
"""
- try:
- chunk_size = chunk_size or cls.CHUNK_SIZE
- async with aiofiles.open(file_path, mode="rb") as reader:
- chunks = list()
- while True:
- chunk = await reader.read(chunk_size)
- if not chunk:
- break
- chunks.append(chunk)
- content = b''.join(chunks)
- logger.debug(f"Successfully read file, the path of file: {file_path}")
- return content
- except Exception as e:
- logger.error(f"Error reading file: {e}")
- raise e
-
+ chunk_size = chunk_size or cls.CHUNK_SIZE
+ async with aiofiles.open(file_path, mode="rb") as reader:
+ chunks = list()
+ while True:
+ chunk = await reader.read(chunk_size)
+ if not chunk:
+ break
+ chunks.append(chunk)
+ content = b"".join(chunks)
+ logger.debug(f"Successfully read file, the path of file: {file_path}")
+ return content
diff --git a/metagpt/utils/file_repository.py b/metagpt/utils/file_repository.py
new file mode 100644
index 000000000..0ddca414d
--- /dev/null
+++ b/metagpt/utils/file_repository.py
@@ -0,0 +1,290 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : git_repository.py
+@Desc: File repository management. RFC 135 2.2.3.2, 2.2.3.4 and 2.2.3.13.
+"""
+from __future__ import annotations
+
+import json
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, List, Set
+
+import aiofiles
+
+from metagpt.config import CONFIG
+from metagpt.logs import logger
+from metagpt.schema import Document
+from metagpt.utils.common import aread
+from metagpt.utils.json_to_markdown import json_to_markdown
+
+
+class FileRepository:
+ """A class representing a FileRepository associated with a Git repository.
+
+ :param git_repo: The associated GitRepository instance.
+ :param relative_path: The relative path within the Git repository.
+
+ Attributes:
+ _relative_path (Path): The relative path within the Git repository.
+ _git_repo (GitRepository): The associated GitRepository instance.
+ """
+
+ def __init__(self, git_repo, relative_path: Path = Path(".")):
+ """Initialize a FileRepository instance.
+
+ :param git_repo: The associated GitRepository instance.
+ :param relative_path: The relative path within the Git repository.
+ """
+ self._relative_path = relative_path
+ self._git_repo = git_repo
+
+ # Initializing
+ self.workdir.mkdir(parents=True, exist_ok=True)
+
+ async def save(self, filename: Path | str, content, dependencies: List[str] = None):
+ """Save content to a file and update its dependencies.
+
+ :param filename: The filename or path within the repository.
+ :param content: The content to be saved.
+ :param dependencies: List of dependency filenames or paths.
+ """
+ pathname = self.workdir / filename
+ pathname.parent.mkdir(parents=True, exist_ok=True)
+ async with aiofiles.open(str(pathname), mode="w") as writer:
+ await writer.write(content)
+ logger.info(f"save to: {str(pathname)}")
+
+ if dependencies is not None:
+ dependency_file = await self._git_repo.get_dependency()
+ await dependency_file.update(pathname, set(dependencies))
+ logger.info(f"update dependency: {str(pathname)}:{dependencies}")
+
+ async def get_dependency(self, filename: Path | str) -> Set[str]:
+ """Get the dependencies of a file.
+
+ :param filename: The filename or path within the repository.
+ :return: Set of dependency filenames or paths.
+ """
+ pathname = self.workdir / filename
+ dependency_file = await self._git_repo.get_dependency()
+ return await dependency_file.get(pathname)
+
+ async def get_changed_dependency(self, filename: Path | str) -> Set[str]:
+ """Get the dependencies of a file that have changed.
+
+ :param filename: The filename or path within the repository.
+ :return: List of changed dependency filenames or paths.
+ """
+ dependencies = await self.get_dependency(filename=filename)
+ changed_files = set(self.changed_files.keys())
+ changed_dependent_files = set()
+ for df in dependencies:
+ rdf = Path(df).relative_to(self._relative_path)
+ if str(rdf) in changed_files:
+ changed_dependent_files.add(df)
+ return changed_dependent_files
+
+ async def get(self, filename: Path | str) -> Document | None:
+ """Read the content of a file.
+
+ :param filename: The filename or path within the repository.
+ :return: The content of the file.
+ """
+ doc = Document(root_path=str(self.root_path), filename=str(filename))
+ path_name = self.workdir / filename
+ if not path_name.exists():
+ return None
+ doc.content = await aread(path_name)
+ return doc
+
+ async def get_all(self) -> List[Document]:
+ """Get the content of all files in the repository.
+
+ :return: List of Document instances representing files.
+ """
+ docs = []
+ for root, dirs, files in os.walk(str(self.workdir)):
+ for file in files:
+ file_path = Path(root) / file
+ relative_path = file_path.relative_to(self.workdir)
+ doc = await self.get(relative_path)
+ docs.append(doc)
+ return docs
+
+ @property
+ def workdir(self):
+ """Return the absolute path to the working directory of the FileRepository.
+
+ :return: The absolute path to the working directory.
+ """
+ return self._git_repo.workdir / self._relative_path
+
+ @property
+ def root_path(self):
+ """Return the relative path from git repository root"""
+ return self._relative_path
+
+ @property
+ def changed_files(self) -> Dict[str, str]:
+ """Return a dictionary of changed files and their change types.
+
+ :return: A dictionary where keys are file paths and values are change types.
+ """
+ files = self._git_repo.changed_files
+ relative_files = {}
+ for p, ct in files.items():
+ if ct.value == "D": # deleted
+ continue
+ try:
+ rf = Path(p).relative_to(self._relative_path)
+ except ValueError:
+ continue
+ relative_files[str(rf)] = ct
+ return relative_files
+
+ @property
+ def all_files(self) -> List:
+ """Get a dictionary of all files in the repository.
+
+ The dictionary includes file paths relative to the current FileRepository.
+
+ :return: A dictionary where keys are file paths and values are file information.
+ :rtype: List
+ """
+ return self._git_repo.get_files(relative_path=self._relative_path)
+
+ def get_change_dir_files(self, dir: Path | str) -> List:
+ """Get the files in a directory that have changed.
+
+ :param dir: The directory path within the repository.
+ :return: List of changed filenames or paths within the directory.
+ """
+ changed_files = self.changed_files
+ children = []
+ for f in changed_files:
+ try:
+ Path(f).relative_to(Path(dir))
+ except ValueError:
+ continue
+ children.append(str(f))
+ return children
+
+ @staticmethod
+ def new_filename():
+ """Generate a new filename based on the current timestamp and a UUID suffix.
+
+ :return: A new filename string.
+ """
+ current_time = datetime.now().strftime("%Y%m%d%H%M%S")
+ return current_time
+ # guid_suffix = str(uuid.uuid4())[:8]
+ # return f"{current_time}x{guid_suffix}"
+
+ async def save_doc(self, doc: Document, with_suffix: str = None, dependencies: List[str] = None):
+ """Save a Document instance as a PDF file.
+
+ This method converts the content of the Document instance to Markdown,
+ saves it to a file with an optional specified suffix, and logs the saved file.
+
+ :param doc: The Document instance to be saved.
+ :type doc: Document
+ :param with_suffix: An optional suffix to append to the saved file's name.
+ :type with_suffix: str, optional
+ :param dependencies: A list of dependencies for the saved file.
+ :type dependencies: List[str], optional
+ """
+ m = json.loads(doc.content)
+ filename = Path(doc.filename).with_suffix(with_suffix) if with_suffix is not None else Path(doc.filename)
+ await self.save(filename=str(filename), content=json_to_markdown(m), dependencies=dependencies)
+ logger.debug(f"File Saved: {str(filename)}")
+
+ @staticmethod
+ async def get_file(filename: Path | str, relative_path: Path | str = ".") -> Document | None:
+ """Retrieve a specific file from the file repository.
+
+ :param filename: The name or path of the file to retrieve.
+ :type filename: Path or str
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ :return: The document representing the file, or None if not found.
+ :rtype: Document or None
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.get(filename=filename)
+
+ @staticmethod
+ async def get_all_files(relative_path: Path | str = ".") -> List[Document]:
+ """Retrieve all files from the file repository.
+
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ :return: A list of documents representing all files in the repository.
+ :rtype: List[Document]
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.get_all()
+
+ @staticmethod
+ async def save_file(filename: Path | str, content, dependencies: List[str] = None, relative_path: Path | str = "."):
+ """Save a file to the file repository.
+
+ :param filename: The name or path of the file to save.
+ :type filename: Path or str
+ :param content: The content of the file.
+ :param dependencies: A list of dependencies for the file.
+ :type dependencies: List[str], optional
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.save(filename=filename, content=content, dependencies=dependencies)
+
+ @staticmethod
+ async def save_as(
+ doc: Document, with_suffix: str = None, dependencies: List[str] = None, relative_path: Path | str = "."
+ ):
+ """Save a Document instance with optional modifications.
+
+ This static method creates a new FileRepository, saves the Document instance
+ with optional modifications (such as a suffix), and logs the saved file.
+
+ :param doc: The Document instance to be saved.
+ :type doc: Document
+ :param with_suffix: An optional suffix to append to the saved file's name.
+ :type with_suffix: str, optional
+ :param dependencies: A list of dependencies for the saved file.
+ :type dependencies: List[str], optional
+ :param relative_path: The relative path within the file repository.
+ :type relative_path: Path or str, optional
+ :return: A boolean indicating whether the save operation was successful.
+ :rtype: bool
+ """
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ return await file_repo.save_doc(doc=doc, with_suffix=with_suffix, dependencies=dependencies)
+
+ async def delete(self, filename: Path | str):
+ """Delete a file from the file repository.
+
+ This method deletes a file from the file repository based on the provided filename.
+
+ :param filename: The name or path of the file to be deleted.
+ :type filename: Path or str
+ """
+ pathname = self.workdir / filename
+ if not pathname.exists():
+ return
+ pathname.unlink(missing_ok=True)
+
+ dependency_file = await self._git_repo.get_dependency()
+ await dependency_file.update(filename=pathname, dependencies=None)
+ logger.info(f"remove dependency key: {str(pathname)}")
+
+ @staticmethod
+ async def delete_file(filename: Path | str, relative_path: Path | str = "."):
+ file_repo = CONFIG.git_repo.new_file_repository(relative_path=relative_path)
+ await file_repo.delete(filename=filename)
diff --git a/metagpt/utils/get_template.py b/metagpt/utils/get_template.py
deleted file mode 100644
index 86c1915f7..000000000
--- a/metagpt/utils/get_template.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-"""
-@Time : 2023/9/19 20:39
-@Author : femto Zheng
-@File : get_template.py
-"""
-from metagpt.config import CONFIG
-
-
-def get_template(templates, format=CONFIG.prompt_format):
- selected_templates = templates.get(format)
- if selected_templates is None:
- raise ValueError(f"Can't find {format} in passed in templates")
-
- # Extract the selected templates
- prompt_template = selected_templates["PROMPT_TEMPLATE"]
- format_example = selected_templates["FORMAT_EXAMPLE"]
-
- return prompt_template, format_example
diff --git a/metagpt/utils/git_repository.py b/metagpt/utils/git_repository.py
new file mode 100644
index 000000000..e9855df05
--- /dev/null
+++ b/metagpt/utils/git_repository.py
@@ -0,0 +1,272 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/11/20
+@Author : mashenquan
+@File : git_repository.py
+@Desc: Git repository management. RFC 135 2.2.3.3.
+"""
+from __future__ import annotations
+
+import shutil
+from enum import Enum
+from pathlib import Path
+from typing import Dict, List
+
+from git.repo import Repo
+from git.repo.fun import is_git_dir
+from gitignore_parser import parse_gitignore
+
+from metagpt.logs import logger
+from metagpt.utils.dependency_file import DependencyFile
+from metagpt.utils.file_repository import FileRepository
+
+
+class ChangeType(Enum):
+ ADDED = "A" # File was added
+ COPIED = "C" # File was copied
+ DELETED = "D" # File was deleted
+ RENAMED = "R" # File was renamed
+ MODIFIED = "M" # File was modified
+ TYPE_CHANGED = "T" # Type of the file was changed
+ UNTRACTED = "U" # File is untracked (not added to version control)
+
+
+class GitRepository:
+ """A class representing a Git repository.
+
+ :param local_path: The local path to the Git repository.
+ :param auto_init: If True, automatically initializes a new Git repository if the provided path is not a Git repository.
+
+ Attributes:
+ _repository (Repo): The GitPython `Repo` object representing the Git repository.
+ """
+
+ def __init__(self, local_path=None, auto_init=True):
+ """Initialize a GitRepository instance.
+
+ :param local_path: The local path to the Git repository.
+ :param auto_init: If True, automatically initializes a new Git repository if the provided path is not a Git repository.
+ """
+ self._repository = None
+ self._dependency = None
+ self._gitignore_rules = None
+ if local_path:
+ self.open(local_path=local_path, auto_init=auto_init)
+
+ def open(self, local_path: Path, auto_init=False):
+ """Open an existing Git repository or initialize a new one if auto_init is True.
+
+ :param local_path: The local path to the Git repository.
+ :param auto_init: If True, automatically initializes a new Git repository if the provided path is not a Git repository.
+ """
+ local_path = Path(local_path)
+ if self.is_git_dir(local_path):
+ self._repository = Repo(local_path)
+ self._gitignore_rules = parse_gitignore(full_path=str(local_path / ".gitignore"))
+ return
+ if not auto_init:
+ return
+ local_path.mkdir(parents=True, exist_ok=True)
+ return self._init(local_path)
+
+ def _init(self, local_path: Path):
+ """Initialize a new Git repository at the specified path.
+
+ :param local_path: The local path where the new Git repository will be initialized.
+ """
+ self._repository = Repo.init(path=Path(local_path))
+
+ gitignore_filename = Path(local_path) / ".gitignore"
+ ignores = ["__pycache__", "*.pyc"]
+ with open(str(gitignore_filename), mode="w") as writer:
+ writer.write("\n".join(ignores))
+ self._repository.index.add([".gitignore"])
+ self._repository.index.commit("Add .gitignore")
+ self._gitignore_rules = parse_gitignore(full_path=gitignore_filename)
+
+ def add_change(self, files: Dict):
+ """Add or remove files from the staging area based on the provided changes.
+
+ :param files: A dictionary where keys are file paths and values are instances of ChangeType.
+ """
+ if not self.is_valid or not files:
+ return
+
+ for k, v in files.items():
+ self._repository.index.remove(k) if v is ChangeType.DELETED else self._repository.index.add([k])
+
+ def commit(self, comments):
+ """Commit the staged changes with the given comments.
+
+ :param comments: Comments for the commit.
+ """
+ if self.is_valid:
+ self._repository.index.commit(comments)
+
+ def delete_repository(self):
+ """Delete the entire repository directory."""
+ if self.is_valid:
+ shutil.rmtree(self._repository.working_dir)
+
+ @property
+ def changed_files(self) -> Dict[str, str]:
+ """Return a dictionary of changed files and their change types.
+
+ :return: A dictionary where keys are file paths and values are change types.
+ """
+ files = {i: ChangeType.UNTRACTED for i in self._repository.untracked_files}
+ changed_files = {f.a_path: ChangeType(f.change_type) for f in self._repository.index.diff(None)}
+ files.update(changed_files)
+ return files
+
+ @staticmethod
+ def is_git_dir(local_path):
+ """Check if the specified directory is a Git repository.
+
+ :param local_path: The local path to check.
+ :return: True if the directory is a Git repository, False otherwise.
+ """
+ git_dir = Path(local_path) / ".git"
+ if git_dir.exists() and is_git_dir(git_dir):
+ return True
+ return False
+
+ @property
+ def is_valid(self):
+ """Check if the Git repository is valid (exists and is initialized).
+
+ :return: True if the repository is valid, False otherwise.
+ """
+ return bool(self._repository)
+
+ @property
+ def status(self) -> str:
+ """Return the Git repository's status as a string."""
+ if not self.is_valid:
+ return ""
+ return self._repository.git.status()
+
+ @property
+ def workdir(self) -> Path | None:
+ """Return the path to the working directory of the Git repository.
+
+ :return: The path to the working directory or None if the repository is not valid.
+ """
+ if not self.is_valid:
+ return None
+ return Path(self._repository.working_dir)
+
+ def archive(self, comments="Archive"):
+ """Archive the current state of the Git repository.
+
+ :param comments: Comments for the archive commit.
+ """
+ logger.info(f"Archive: {list(self.changed_files.keys())}")
+ self.add_change(self.changed_files)
+ self.commit(comments)
+
+ def new_file_repository(self, relative_path: Path | str = ".") -> FileRepository:
+ """Create a new instance of FileRepository associated with this Git repository.
+
+ :param relative_path: The relative path to the file repository within the Git repository.
+ :return: A new instance of FileRepository.
+ """
+ path = Path(relative_path)
+ try:
+ path = path.relative_to(self.workdir)
+ except ValueError:
+ path = relative_path
+ return FileRepository(git_repo=self, relative_path=Path(path))
+
+ async def get_dependency(self) -> DependencyFile:
+ """Get the dependency file associated with the Git repository.
+
+ :return: An instance of DependencyFile.
+ """
+ if not self._dependency:
+ self._dependency = DependencyFile(workdir=self.workdir)
+ return self._dependency
+
+ def rename_root(self, new_dir_name):
+ """Rename the root directory of the Git repository.
+
+ :param new_dir_name: The new name for the root directory.
+ """
+ if self.workdir.name == new_dir_name:
+ return
+ new_path = self.workdir.parent / new_dir_name
+ if new_path.exists():
+ logger.info(f"Delete directory {str(new_path)}")
+ shutil.rmtree(new_path)
+ try:
+ shutil.move(src=str(self.workdir), dst=str(new_path))
+ except Exception as e:
+ logger.warning(f"Move {str(self.workdir)} to {str(new_path)} error: {e}")
+ logger.info(f"Rename directory {str(self.workdir)} to {str(new_path)}")
+ self._repository = Repo(new_path)
+ self._gitignore_rules = parse_gitignore(full_path=str(new_path / ".gitignore"))
+
+ def get_files(self, relative_path: Path | str, root_relative_path: Path | str = None, filter_ignored=True) -> List:
+ """
+ Retrieve a list of files in the specified relative path.
+
+ The method returns a list of file paths relative to the current FileRepository.
+
+ :param relative_path: The relative path within the repository.
+ :type relative_path: Path or str
+ :param root_relative_path: The root relative path within the repository.
+ :type root_relative_path: Path or str
+ :param filter_ignored: Flag to indicate whether to filter files based on .gitignore rules.
+ :type filter_ignored: bool
+ :return: A list of file paths in the specified directory.
+ :rtype: List[str]
+ """
+ try:
+ relative_path = Path(relative_path).relative_to(self.workdir)
+ except ValueError:
+ relative_path = Path(relative_path)
+
+ if not root_relative_path:
+ root_relative_path = Path(self.workdir) / relative_path
+ files = []
+ try:
+ directory_path = Path(self.workdir) / relative_path
+ if not directory_path.exists():
+ return []
+ for file_path in directory_path.iterdir():
+ if file_path.is_file():
+ rpath = file_path.relative_to(root_relative_path)
+ files.append(str(rpath))
+ else:
+ subfolder_files = self.get_files(
+ relative_path=file_path, root_relative_path=root_relative_path, filter_ignored=False
+ )
+ files.extend(subfolder_files)
+ except Exception as e:
+ logger.error(f"Error: {e}")
+ if not filter_ignored:
+ return files
+ filtered_files = self.filter_gitignore(filenames=files, root_relative_path=root_relative_path)
+ return filtered_files
+
+ def filter_gitignore(self, filenames: List[str], root_relative_path: Path | str = None) -> List[str]:
+ """
+ Filter a list of filenames based on .gitignore rules.
+
+ :param filenames: A list of filenames to be filtered.
+ :type filenames: List[str]
+ :param root_relative_path: The root relative path within the repository.
+ :type root_relative_path: Path or str
+ :return: A list of filenames that pass the .gitignore filtering.
+ :rtype: List[str]
+ """
+ if root_relative_path is None:
+ root_relative_path = self.workdir
+ files = []
+ for filename in filenames:
+ pathname = root_relative_path / filename
+ if self._gitignore_rules(str(pathname)):
+ continue
+ files.append(filename)
+ return files
diff --git a/metagpt/utils/graph_repository.py b/metagpt/utils/graph_repository.py
new file mode 100644
index 000000000..1a6f29a6b
--- /dev/null
+++ b/metagpt/utils/graph_repository.py
@@ -0,0 +1,200 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+@Time : 2023/12/19
+@Author : mashenquan
+@File : graph_repository.py
+@Desc : Superclass for graph repository.
+"""
+
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import List
+
+from pydantic import BaseModel
+
+from metagpt.logs import logger
+from metagpt.repo_parser import ClassInfo, ClassRelationship, RepoFileInfo
+from metagpt.utils.common import concat_namespace
+
+
+class GraphKeyword:
+ IS = "is"
+ OF = "Of"
+ ON = "On"
+ CLASS = "class"
+ FUNCTION = "function"
+ HAS_FUNCTION = "has_function"
+ SOURCE_CODE = "source_code"
+ NULL = ""
+ GLOBAL_VARIABLE = "global_variable"
+ CLASS_FUNCTION = "class_function"
+ CLASS_PROPERTY = "class_property"
+ HAS_CLASS_FUNCTION = "has_class_function"
+ HAS_CLASS_PROPERTY = "has_class_property"
+ HAS_CLASS = "has_class"
+ HAS_PAGE_INFO = "has_page_info"
+ HAS_CLASS_VIEW = "has_class_view"
+ HAS_SEQUENCE_VIEW = "has_sequence_view"
+ HAS_ARGS_DESC = "has_args_desc"
+ HAS_TYPE_DESC = "has_type_desc"
+
+
+class SPO(BaseModel):
+ subject: str
+ predicate: str
+ object_: str
+
+
+class GraphRepository(ABC):
+ def __init__(self, name: str, **kwargs):
+ self._repo_name = name
+ self._kwargs = kwargs
+
+ @abstractmethod
+ async def insert(self, subject: str, predicate: str, object_: str):
+ pass
+
+ @abstractmethod
+ async def upsert(self, subject: str, predicate: str, object_: str):
+ pass
+
+ @abstractmethod
+ async def update(self, subject: str, predicate: str, object_: str):
+ pass
+
+ @abstractmethod
+ async def select(self, subject: str = None, predicate: str = None, object_: str = None) -> List[SPO]:
+ pass
+
+ @property
+ def name(self) -> str:
+ return self._repo_name
+
+ @staticmethod
+ async def update_graph_db_with_file_info(graph_db: "GraphRepository", file_info: RepoFileInfo):
+ await graph_db.insert(subject=file_info.file, predicate=GraphKeyword.IS, object_=GraphKeyword.SOURCE_CODE)
+ file_types = {".py": "python", ".js": "javascript"}
+ file_type = file_types.get(Path(file_info.file).suffix, GraphKeyword.NULL)
+ await graph_db.insert(subject=file_info.file, predicate=GraphKeyword.IS, object_=file_type)
+ for c in file_info.classes:
+ class_name = c.get("name", "")
+ # file -> class
+ await graph_db.insert(
+ subject=file_info.file,
+ predicate=GraphKeyword.HAS_CLASS,
+ object_=concat_namespace(file_info.file, class_name),
+ )
+ # class detail
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, class_name),
+ predicate=GraphKeyword.IS,
+ object_=GraphKeyword.CLASS,
+ )
+ methods = c.get("methods", [])
+ for fn in methods:
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, class_name),
+ predicate=GraphKeyword.HAS_CLASS_FUNCTION,
+ object_=concat_namespace(file_info.file, class_name, fn),
+ )
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, class_name, fn),
+ predicate=GraphKeyword.IS,
+ object_=GraphKeyword.CLASS_FUNCTION,
+ )
+ for f in file_info.functions:
+ # file -> function
+ await graph_db.insert(
+ subject=file_info.file, predicate=GraphKeyword.HAS_FUNCTION, object_=concat_namespace(file_info.file, f)
+ )
+ # function detail
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, f), predicate=GraphKeyword.IS, object_=GraphKeyword.FUNCTION
+ )
+ for g in file_info.globals:
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, g),
+ predicate=GraphKeyword.IS,
+ object_=GraphKeyword.GLOBAL_VARIABLE,
+ )
+ for code_block in file_info.page_info:
+ if code_block.tokens:
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, *code_block.tokens),
+ predicate=GraphKeyword.HAS_PAGE_INFO,
+ object_=code_block.model_dump_json(),
+ )
+ for k, v in code_block.properties.items():
+ await graph_db.insert(
+ subject=concat_namespace(file_info.file, k, v),
+ predicate=GraphKeyword.HAS_PAGE_INFO,
+ object_=code_block.model_dump_json(),
+ )
+
+ @staticmethod
+ async def update_graph_db_with_class_views(graph_db: "GraphRepository", class_views: List[ClassInfo]):
+ for c in class_views:
+ filename, _ = c.package.split(":", 1)
+ await graph_db.insert(subject=filename, predicate=GraphKeyword.IS, object_=GraphKeyword.SOURCE_CODE)
+ file_types = {".py": "python", ".js": "javascript"}
+ file_type = file_types.get(Path(filename).suffix, GraphKeyword.NULL)
+ await graph_db.insert(subject=filename, predicate=GraphKeyword.IS, object_=file_type)
+ await graph_db.insert(subject=filename, predicate=GraphKeyword.HAS_CLASS, object_=c.package)
+ await graph_db.insert(
+ subject=c.package,
+ predicate=GraphKeyword.IS,
+ object_=GraphKeyword.CLASS,
+ )
+ for vn, vt in c.attributes.items():
+ # class -> property
+ await graph_db.insert(
+ subject=c.package,
+ predicate=GraphKeyword.HAS_CLASS_PROPERTY,
+ object_=concat_namespace(c.package, vn),
+ )
+ # property detail
+ await graph_db.insert(
+ subject=concat_namespace(c.package, vn),
+ predicate=GraphKeyword.IS,
+ object_=GraphKeyword.CLASS_PROPERTY,
+ )
+ await graph_db.insert(
+ subject=concat_namespace(c.package, vn), predicate=GraphKeyword.HAS_TYPE_DESC, object_=vt
+ )
+ for fn, desc in c.methods.items():
+ if "