diff --git a/README.md b/README.md index ed2c118b..70d03b62 100644 --- a/README.md +++ b/README.md @@ -1,75 +1,57 @@
- Arch Logo + Plano Logo
- -_Arch is a models-native (edge and service) proxy server for agents._

- Arch handles the *pesky plumbing work* in building AI agents — like applying guardrails, routing prompts to the right agent, generating hyper-rich information traces for RL, and unifying access to any LLM. It’s a language and framework friendly infrastructure layer designed to help you build and ship agentic apps faster. + _Plano is a models-native proxy and data plane for agents._

+ Plano pulls out the rote plumbing work and decouples you from brittle framework abstractions, centralizing what shouldn’t be bespoke in every codebase - like agent routing and orchestration, rich agentic signals and traces for continuous improvement, guardrail filters for moderation, and smart LLM routing APIs for UX and DX agility. Use any language or AI framework, and deliver agents faster to production. [Quickstart](#Quickstart) • [Demos](#Demos) • -[Route LLMs](#use-arch-as-a-llm-router) • -[Build agentic apps with Arch](#Build-Agentic-Apps-with-Arch) • -[Documentation](https://docs.archgw.com) • +[Route LLMs](#use-plano-as-a-llm-router) • +[Build Agentic Apps with Plano](#Build-Agentic-Apps-with-Plano) • +[Documentation](https://docs.planoai.dev) • [Contact](#Contact) -[![pre-commit](https://github.com/katanemo/arch/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/katanemo/arch/actions/workflows/pre-commit.yml) -[![rust tests (prompt and llm gateway)](https://github.com/katanemo/arch/actions/workflows/rust_tests.yml/badge.svg)](https://github.com/katanemo/arch/actions/workflows/rust_tests.yml) -[![e2e tests](https://github.com/katanemo/arch/actions/workflows/e2e_tests.yml/badge.svg)](https://github.com/katanemo/arch/actions/workflows/e2e_tests.yml) -[![Build and Deploy Documentation](https://github.com/katanemo/arch/actions/workflows/static.yml/badge.svg)](https://github.com/katanemo/arch/actions/workflows/static.yml) - +[![pre-commit](https://github.com/katanemo/plano/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/katanemo/plano/actions/workflows/pre-commit.yml) +[![rust tests (prompt and llm gateway)](https://github.com/katanemo/plano/actions/workflows/rust_tests.yml/badge.svg)](https://github.com/katanemo/plano/actions/workflows/rust_tests.yml) +[![e2e tests](https://github.com/katanemo/plano/actions/workflows/e2e_tests.yml/badge.svg)](https://github.com/katanemo/plano/actions/workflows/e2e_tests.yml) +[![Build and Deploy Documentation](https://github.com/katanemo/plano/actions/workflows/static.yml/badge.svg)](https://github.com/katanemo/plano/actions/workflows/static.yml)
-# About The Latest Release: -[0.3.20] [Preference-aware multi LLM routing for Claude Code 2.0](demos/use_cases/claude_code_router/README.md)
high-level network architecture for ArchGW - - # Overview -Arch - Build fast, hyper-personalized agents with intelligent infra | Product Hunt +Building agentic demos is easy. Shipping agentic applications safely, reliably, and repeatably to production is hard. After the thrill of a quick hack, you end up building the “hidden middleware” to reach production: routing logic to reach the right agent, guardrail hooks for safety and moderation, evaluation and observability glue for continuous learning, and model/provider quirks scattered across frameworks and application code. -AI demos are easy to hack. But once you move past a prototype, you’re stuck building and maintaining low-level plumbing code that slows down real innovation. For example: +Plano solves this by moving core delivery concerns into a unified, out-of-process dataplane. -- **Routing & orchestration.** Put routing in code and you’ve got two choices: maintain it yourself or live with a framework’s baked-in logic. Either way, keeping routing consistent means pushing code changes across all your agents, slowing iteration and turning every policy tweak into a refactor instead of a config flip. -- **Model integration churn.** Frameworks wire LLM integrations directly into code abstractions, making it hard to add or swap models without touching application code — meaning you’ll have to do codewide search/replace every time you want to experiment with a new model or version. -- **Observability & governance.** Logging, tracing, and guardrails are baked in as tightly coupled features, so bringing in best-of-breed solutions is painful and often requires digging through the guts of a framework. -- **Prompt engineering overhead**. Input validation, clarifying vague user input, and coercing outputs into the right schema all pile up, turning what should be design work into low-level plumbing work. -- **Brittle upgrades**. Every change (new model, new guardrail, new trace format) means patching and redeploying application servers. Contrast that with bouncing a central proxy—one upgrade, instantly consistent everywhere. +- **🚦 Orchestration:** Low-latency orchestration between agents, and add new agents without changing app code +- **🔗 Model Agility:** Route [by model name, alias (semantic names) or automatically via preferences](#use-plano-as-a-llm-router) +- **🕵 Agentic Signals™:** Zero-code capture of [behavior signals](#observability) plus OTEL traces/metrics across every agent. +- **🛡️ Moderation & Memory Hooks:** Build jailbreak protection, add moderation policies and memory consistently via [Filter Chains](https://docs.planoai.dev/concepts/filter_chain.html). -With Arch, you can move faster by focusing on higher-level objectives in a language and framework agnostic way. **Arch** was built by the contributors of [Envoy Proxy](https://www.envoyproxy.io/) with the belief that: +Plano pulls rote plumbing out of your framework so you can stay focused on what matters most: the core product logic of your agentic applications. Plano is backed by [industry-leading LLM research](https://planoai.dev/research) and built on [Envoy](https://envoyproxy.io) by its core contributors, who built critical infrastructure at scale for modern worklaods. ->Prompts are nuanced and opaque user requests, which require the same capabilities as traditional HTTP requests including secure handling, intelligent routing, robust observability, and integration with backend (API) systems to improve speed and accuracy for common agentic scenarios – all outside core application logic.* +**High-Level Network Sequence Diagram**: +![high-level network plano arcitecture for Plano](docs/source/_static/img/plano_network_diagram_high_level.png) -**Core Features**: - - - `🚦 Route to Agents`: Engineered with purpose-built [LLMs](https://huggingface.co/collections/katanemo/arch-function-66f209a693ea8df14317ad68) for fast (<100ms) agent routing and hand-off - - `🔗 Route to LLMs`: Unify access to LLMs with support for [three routing strategies](#use-arch-as-a-llm-router). - - `⛨ Guardrails`: Centrally configure and prevent harmful outcomes and ensure safe user interactions - - `⚡ Tools Use`: For common agentic scenarios let Arch instantly clarify and convert prompts to tools/API calls - - `🕵 Observability`: W3C compatible request tracing and LLM metrics that instantly plugin with popular tools - - `🧱 Built on Envoy`: Arch runs alongside app servers as a containerized process, and builds on top of [Envoy's](https://envoyproxy.io) proven HTTP management and scalability features to handle ingress and egress traffic related to prompts and LLMs. - -**High-Level Sequence Diagram**: -![high-level network architecture for ArchGW](docs/source/_static/img/arch_network_diagram_high_level.png) - -**Jump to our [docs](https://docs.archgw.com)** to learn how you can use Arch to improve the speed, security and personalization of your GenAI apps. +**Jump to our [docs](https://docs.planoai.dev)** to learn how you can use Plano to improve the speed, safety and obervability of your agentic applications. > [!IMPORTANT] -> Today, the function calling LLM (Arch-Function) designed for the agentic and RAG scenarios is hosted free of charge in the US-central region. To offer consistent latencies and throughput, and to manage our expenses, we will enable access to the hosted version via developers keys soon, and give you the option to run that LLM locally. For more details see this issue [#258](https://github.com/katanemo/archgw/issues/258) +> Plano and the Arch family of LLMs (like Plano-Orchestrator-4B, Arch-Router, etc) are hosted free of charge in the US-central region to give you a great first-run developer experience of Plano. To scale and run in production, you can either run these LLMs locally or contact us on [Discord](https://discord.gg/pGZf2gcwEc) for API keys. ## Contact To get in touch with us, please join our [discord server](https://discord.gg/pGZf2gcwEc). We will be monitoring that actively and offering support there. ## Demos -* [Sample App: Weather Forecast Agent](demos/samples_python/weather_forecast/README.md) - A sample agentic weather forecasting app that highlights core function calling capabilities of Arch. +* [Sample App: Weather Forecast Agent](demos/samples_python/weather_forecast/README.md) - A sample agentic weather forecasting app that highlights core function calling capabilities of Plano. * [Sample App: Network Operator Agent](demos/samples_python/network_switch_operator_agent/README.md) - A simple network device switch operator agent that can retrieve device statistics and reboot them. -* [Use Case: Connecting to SaaS APIs](demos/use_cases/spotify_bearer_auth) - Connect 3rd party SaaS APIs to your agentic chat experience. + ## Quickstart -Follow this quickstart guide to use Arch as a router for local or hosted LLMs, including dynamic routing. Later in the section we will see how you can Arch to build highly capable agentic applications, and to provide e2e observability. +Follow this quickstart guide to use Plano as a router for local or hosted LLMs, including dynamic routing. Later in the section we will see how you can Plano to build highly capable agentic applications, and to provide e2e observability. ### Prerequisites @@ -79,98 +61,19 @@ Before you begin, ensure you have the following: 2. [Docker compose](https://docs.docker.com/compose/install/) (v2.29) 3. [Python](https://www.python.org/downloads/) (v3.13) -Arch's CLI allows you to manage and interact with the Arch gateway efficiently. To install the CLI, simply run the following command: +Plano's CLI allows you to manage and interact with the Plano gateway efficiently. To install the CLI, simply run the following command: > [!TIP] -> We recommend that developers create a new Python virtual environment to isolate dependencies before installing Arch. This ensures that archgw and its dependencies do not interfere with other packages on your system. +> We recommend that developers create a new Python virtual environment to isolate dependencies before installing Plano. This ensures that plano and its dependencies do not interfere with other packages on your system. ```console $ python3.12 -m venv venv $ source venv/bin/activate # On Windows, use: venv\Scripts\activate -$ pip install archgw==0.3.22 +$ pip install plano==0.4.0 ``` -### Use Arch as a LLM Router -Arch supports three powerful routing strategies for LLMs: model-based routing, alias-based routing, and preference-based routing. Each strategy offers different levels of abstraction and control for managing your LLM infrastructure. - -#### Model-based Routing -Model-based routing allows you to configure specific models with static routing. This is ideal when you need direct control over which models handle specific requests. Arch supports 11+ LLM providers including OpenAI, Anthropic, DeepSeek, Mistral, Groq, and more. - -```yaml -version: v0.1.0 - -listeners: - egress_traffic: - address: 0.0.0.0 - port: 12000 - message_format: openai - timeout: 30s - -llm_providers: - - model: openai/gpt-4o - access_key: $OPENAI_API_KEY - default: true - - - model: anthropic/claude-3-5-sonnet-20241022 - access_key: $ANTHROPIC_API_KEY - -``` - -You can then route to specific models using any OpenAI-compatible client: - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://127.0.0.1:12000/v1", api_key="test") - -# Route to specific model -response = client.chat.completions.create( - model="anthropic/claude-3-5-sonnet-20241022", - messages=[{"role": "user", "content": "Explain quantum computing"}] -) -``` - -#### Alias-based Routing -Alias-based routing lets you create semantic model names that map to underlying providers. This approach decouples your application code from specific model names, making it easy to experiment with different models or handle provider changes. - -```yaml -version: v0.1.0 - -listeners: - egress_traffic: - address: 0.0.0.0 - port: 12000 - message_format: openai - timeout: 30s - -llm_providers: - - model: openai/gpt-4o - access_key: $OPENAI_API_KEY - - - model: anthropic/claude-3-5-sonnet-20241022 - access_key: $ANTHROPIC_API_KEY - -model_aliases: - # Model aliases - friendly names that map to actual model names - fast-model: - target: gpt-4o-mini - - reasoning-model: - target: gpt-4o - - creative-model: - target: claude-3-5-sonnet-20241022 -``` - -Use semantic aliases in your application code: - -```python -# Your code uses semantic names instead of provider-specific ones -response = client.chat.completions.create( - model="reasoning-model", # Routes to best available reasoning model - messages=[{"role": "user", "content": "Solve this complex problem..."}] -) -``` +### Use Plano as a LLM Router +Plano supports multiple powerful routing strategies for LLMs. [Model-based routing](https://docs.arch.com/guides/llm_router.html#model-based-routing) gives you direct control over specific models and supports 11+ LLM providers including OpenAI, Anthropic, DeepSeek, Mistral, Groq, and more. [Alias-based routing](https://docs.arch.com/guides/llm_router.html#alias-based-routing) lets you create semantic model names that decouple your application code from specific providers, making it easy to experiment with different models or handle provider changes without refactoring. For full configuration examples and code walkthroughs, see our [routing guides](https://docs.arch.com/guides/llm_router.html). #### Preference-aligned Routing Preference-aligned routing provides intelligent, dynamic model selection based on natural language descriptions of tasks and preferences. Instead of hardcoded routing logic, you describe what each model is good at using plain English. @@ -203,23 +106,25 @@ llm_providers: description: analyzing existing code for bugs, improvements, and optimization ``` -Arch uses a lightweight 1.5B autoregressive model to intelligently map user prompts to these preferences, automatically selecting the best model for each request. This approach adapts to intent drift, supports multi-turn conversations, and avoids brittle embedding-based classifiers or manual if/else chains. No retraining required when adding models or updating policies — routing is governed entirely by human-readable rules. -**Learn More**: Check our [documentation](https://docs.archgw.com/concepts/llm_providers/llm_providers.html) for comprehensive provider setup guides and routing strategies. You can learn more about the design, benchmarks, and methodology behind preference-based routing in our paper: + +Plano uses a lightweight 1.5B autoregressive model to intelligently map user prompts to these preferences, automatically selecting the best model for each request. This approach adapts to intent drift, supports multi-turn conversations, and avoids brittle embedding-based classifiers or manual if/else chains. No retraining required when adding models or updating policies — routing is governed entirely by human-readable rules. + +**Learn More**: Check our [documentation](https://docs.plano.com/concepts/llm_providers/llm_providers.html) for comprehensive provider setup guides and routing strategies. You can learn more about the design, benchmarks, and methodology behind preference-based routing in our paper:
- Arch Router Paper Preview + Plano Router Paper Preview
-### Build Agentic Apps with Arch +### Build Agentic Apps with Plano -In following quickstart we will show you how easy it is to build AI agent with Arch gateway. We will build a currency exchange agent using following simple steps. For this demo we will use `https://api.frankfurter.dev/` to fetch latest price for currencies and assume USD as base currency. +In following quickstart we will show you how easy it is to build AI agent with Plano gateway. We will build a currency exchange agent using following simple steps. For this demo we will use `https://api.frankfurter.dev/` to fetch latest price for currencies and assume USD as base currency. -#### Step 1. Create arch config file +#### Step 1. Create plano config file -Create `arch_config.yaml` file with following content, +Create `plano_config.yaml` file with following content, ```yaml version: v0.1.0 @@ -271,14 +176,14 @@ endpoints: protocol: https ``` -#### Step 2. Start arch gateway with currency conversion config +#### Step 2. Start plano gateway with currency conversion config ```sh -$ archgw up arch_config.yaml -2024-12-05 16:56:27,979 - cli.main - INFO - Starting archgw cli version: 0.3.22 +$ plano up plano_config.yaml +2024-12-05 16:56:27,979 - cli.main - INFO - Starting plano cli version: 0.4.0 2024-12-05 16:56:28,485 - cli.utils - INFO - Schema validation successful! -2024-12-05 16:56:28,485 - cli.main - INFO - Starting arch model server and arch gateway +2024-12-05 16:56:28,485 - cli.main - INFO - Starting plano model server and plano gateway 2024-12-05 16:56:51,647 - cli.core - INFO - Container is healthy! ``` @@ -310,39 +215,13 @@ $ curl --header 'Content-Type: application/json' \ ``` -## [Observability](https://docs.archgw.com/guides/observability/observability.html) -Arch is designed to support best-in class observability by supporting open standards. Please read our [docs](https://docs.archgw.com/guides/observability/observability.html) on observability for more details on tracing, metrics, and logs. The screenshot below is from our integration with Signoz (among others) +## [Observability](https://docs.plano.com/guides/observability/observability.html) +Plano is designed to support best-in class observability by supporting open standards. Please read our [docs](https://docs.plano.com/guides/observability/observability.html) on observability for more details on tracing, metrics, and logs. The screenshot below is from our integration with Signoz (among others) ![alt text](docs/source/_static/img/tracing.png) -## Debugging - -When debugging issues / errors application logs and access logs provide key information to give you more context on whats going on with the system. Arch gateway runs in info log level and following is a typical output you could see in a typical interaction between developer and arch gateway, - -``` -$ archgw up --service archgw --foreground -... -[2025-03-26 18:32:01.350][26][info] prompt_gateway: on_http_request_body: sending request to model server -[2025-03-26 18:32:01.851][26][info] prompt_gateway: on_http_call_response: model server response received -[2025-03-26 18:32:01.852][26][info] prompt_gateway: on_http_call_response: dispatching api call to developer endpoint: weather_forecast_service, path: /weather, method: POST -[2025-03-26 18:32:01.882][26][info] prompt_gateway: on_http_call_response: developer api call response received: status code: 200 -[2025-03-26 18:32:01.882][26][info] prompt_gateway: on_http_call_response: sending request to upstream llm -[2025-03-26 18:32:01.883][26][info] llm_gateway: on_http_request_body: provider: gpt-4o-mini, model requested: None, model selected: gpt-4o-mini -[2025-03-26 18:32:02.818][26][info] llm_gateway: on_http_response_body: time to first token: 1468ms -[2025-03-26 18:32:04.532][26][info] llm_gateway: on_http_response_body: request latency: 3183ms -... -``` - -Log level can be changed to debug to get more details. To enable debug logs edit (supervisord.conf)[arch/supervisord.conf], change the log level `--component-log-level wasm:info` to `--component-log-level wasm:debug`. And after that you need to rebuild docker image and restart the arch gateway using following set of commands, - -``` -# make sure you are at the root of the repo -$ archgw build -# go to your service that has arch_config.yaml file and issue following command, -$ archgw up --service archgw --foreground -``` ## Contribution -We would love feedback on our [Roadmap](https://github.com/orgs/katanemo/projects/1) and we welcome contributions to **Arch**! +We would love feedback on our [Roadmap](https://github.com/orgs/katanemo/projects/1) and we welcome contributions to **Plano**! Whether you're fixing bugs, adding new features, improving documentation, or creating tutorials, your help is much appreciated. Please visit our [Contribution Guide](CONTRIBUTING.md) for more details diff --git a/docs/source/_static/favicon.ico b/docs/source/_static/favicon.ico index 29dc5902..a1b75eb4 100644 Binary files a/docs/source/_static/favicon.ico and b/docs/source/_static/favicon.ico differ diff --git a/docs/source/_static/img/PlanoTagline.svg b/docs/source/_static/img/PlanoTagline.svg new file mode 100644 index 00000000..c0c10548 --- /dev/null +++ b/docs/source/_static/img/PlanoTagline.svg @@ -0,0 +1,56 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/source/_static/img/arch-logo.png b/docs/source/_static/img/arch-logo.png deleted file mode 100644 index bbffb318..00000000 Binary files a/docs/source/_static/img/arch-logo.png and /dev/null differ diff --git a/docs/source/_static/img/arch-nav-logo.png b/docs/source/_static/img/arch-nav-logo.png deleted file mode 100644 index 5a1a7776..00000000 Binary files a/docs/source/_static/img/arch-nav-logo.png and /dev/null differ diff --git a/docs/source/_static/img/arch-system-architecture.jpg b/docs/source/_static/img/arch-system-architecture.jpg deleted file mode 100644 index 3c8839a7..00000000 Binary files a/docs/source/_static/img/arch-system-architecture.jpg and /dev/null differ diff --git a/docs/source/_static/img/arch_network_diagram_high_level.png b/docs/source/_static/img/arch_network_diagram_high_level.png deleted file mode 100644 index e83e7165..00000000 Binary files a/docs/source/_static/img/arch_network_diagram_high_level.png and /dev/null differ diff --git a/docs/source/_static/img/network-topology-agent.jpg b/docs/source/_static/img/network-topology-agent.jpg deleted file mode 100644 index 50ba9a64..00000000 Binary files a/docs/source/_static/img/network-topology-agent.jpg and /dev/null differ diff --git a/docs/source/_static/img/network-topology-ingress-egress.jpg b/docs/source/_static/img/network-topology-ingress-egress.jpg deleted file mode 100644 index 03e36e77..00000000 Binary files a/docs/source/_static/img/network-topology-ingress-egress.jpg and /dev/null differ diff --git a/docs/source/_static/img/network-topology-ingress-egress.png b/docs/source/_static/img/network-topology-ingress-egress.png new file mode 100644 index 00000000..c2e55584 Binary files /dev/null and b/docs/source/_static/img/network-topology-ingress-egress.png differ diff --git a/docs/source/_static/img/plano-system-architecture.png b/docs/source/_static/img/plano-system-architecture.png new file mode 100644 index 00000000..792477d5 Binary files /dev/null and b/docs/source/_static/img/plano-system-architecture.png differ diff --git a/docs/source/_static/img/plano_network_diagram_high_level.png b/docs/source/_static/img/plano_network_diagram_high_level.png new file mode 100644 index 00000000..da1c5b92 Binary files /dev/null and b/docs/source/_static/img/plano_network_diagram_high_level.png differ diff --git a/docs/source/build_with_arch/agent.rst b/docs/source/build_with_arch/agent.rst deleted file mode 100644 index cd70d669..00000000 --- a/docs/source/build_with_arch/agent.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. _arch_agent_guide: - -Agentic Apps -============= - -Arch helps you build personalized agentic applications by calling application-specific (API) functions via user prompts. -This involves any predefined functions or APIs you want to expose to users to perform tasks, gather information, -or manipulate data. This capability is generally referred to as :ref:`function calling `, where -you can support “agentic” apps tailored to specific use cases - from updating insurance claims to creating ad campaigns - via prompts. - -Arch analyzes prompts, extracts critical information from prompts, engages in lightweight conversation with the user to -gather any missing parameters and makes API calls so that you can focus on writing business logic. Arch does this via its -purpose-built `Arch-Function `_ - -the fastest (200ms p50 - 12x faser than GPT-4o) and cheapest (44x than GPT-4o) function calling LLM that matches or outperforms -frontier LLMs. - -.. image:: includes/agent/function-calling-flow.jpg - :width: 100% - :align: center - - -Single Function Call --------------------- -In the most common scenario, users will request a single action via prompts, and Arch efficiently processes the -request by extracting relevant parameters, validating the input, and calling the designated function or API. Here -is how you would go about enabling this scenario with Arch: - -Step 1: Define Prompt Targets -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: includes/agent/function-calling-agent.yaml - :language: yaml - :linenos: - :emphasize-lines: 19-49 - :caption: Prompt Target Example Configuration - -Step 2: Process Request Parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once the prompt targets are configured as above, handling those parameters is - -.. literalinclude:: includes/agent/parameter_handling.py - :language: python - :linenos: - :caption: Parameter handling with Flask - -Parallel & Multiple Function Calling ------------------------------------- -In more complex use cases, users may request multiple actions or need multiple APIs/functions to be called -simultaneously or sequentially. With Arch, you can handle these scenarios efficiently using parallel or multiple -function calling. This allows your application to engage in a broader range of interactions, such as updating -different datasets, triggering events across systems, or collecting results from multiple services in one prompt. - -Arch-FC1B is built to manage these parallel tasks efficiently, ensuring low latency and high throughput, even -when multiple functions are invoked. It provides two mechanisms to handle these cases: - -Step 1: Define Prompt Targets -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When enabling multiple function calling, define the prompt targets in a way that supports multiple functions or -API calls based on the user's prompt. These targets can be triggered in parallel or sequentially, depending on -the user's intent. - -Example of Multiple Prompt Targets in YAML: - -.. literalinclude:: includes/agent/function-calling-agent.yaml - :language: yaml - :linenos: - :emphasize-lines: 19-49 - :caption: Prompt Target Example Configuration diff --git a/docs/source/build_with_arch/multi_turn.rst b/docs/source/build_with_arch/multi_turn.rst deleted file mode 100644 index 3bb31002..00000000 --- a/docs/source/build_with_arch/multi_turn.rst +++ /dev/null @@ -1,90 +0,0 @@ -.. _arch_multi_turn_guide: - -Multi-Turn -========== -Developers often `struggle `_ to efficiently handle -``follow-up`` or ``clarification`` questions. Specifically, when users ask for changes or additions to previous responses, it requires developers to -re-write prompts using LLMs with precise prompt engineering techniques. This process is slow, manual, error prone and adds latency and token cost for -common scenarios that can be managed more efficiently. - -Arch is highly capable of accurately detecting and processing prompts in multi-turn scenarios so that you can buil fast and accurate agents in minutes. -Below are some cnversational examples that you can build via Arch. Each example is enriched with annotations (via ** [Arch] ** ) that illustrates how Arch -processess conversational messages on your behalf. - -.. Note:: - The following section assumes that you have some knowledge about the core concepts of Arch, such as :ref:`prompt_targets `. - If you haven't familizaried yourself with Arch's concepts, we recommend you first read the :ref:`tech overview ` section firtst. - Additionally, the conversation examples below assume the usage of the following :ref:`arch_config.yaml ` file. - -Example 1: Adjusting Retrieval ------------------------------- -.. code-block:: text - - User: What are the benefits of renewable energy? - **[Arch]**: Check if there is an available that can handle this user query. - **[Arch]**: Found "get_info_for_energy_source" prompt_target in arch_config.yaml. Forward prompt to the endpoint configured in "get_info_for_energy_source" - ... - Assistant: Renewable energy reduces greenhouse gas emissions, lowers air pollution, and provides sustainable power sources like solar and wind. - - User: Include cost considerations in the response. - **[Arch]**: Follow-up detected. Forward prompt history to the "get_info_for_energy_source" prompt_target and post the following parameters consideration="cost" - ... - Assistant: Renewable energy reduces greenhouse gas emissions, lowers air pollution, and provides sustainable power sources like solar and wind. While the initial setup costs can be high, long-term savings from reduced fuel expenses and government incentives make it cost-effective. - - -Example 2: Switching Intent ---------------------------- -.. code-block:: text - - User: What are the symptoms of diabetes? - **[Arch]**: Check if there is an available that can handle this user query. - **[Arch]**: Found "diseases_symptoms" prompt_target in arch_config.yaml. Forward disease=diabeteres to "diseases_symptoms" prompt target - ... - Assistant: Common symptoms include frequent urination, excessive thirst, fatigue, and blurry vision. - - User: How is it diagnosed? - **[Arch]**: New intent detected. - **[Arch]**: Found "disease_diagnoses" prompt_target in arch_config.yaml. Forward disease=diabeteres to "disease_diagnoses" prompt target - ... - Assistant: Diabetes is diagnosed through blood tests like fasting blood sugar, A1C, or an oral glucose tolerance test. - - -Build Multi-Turn RAG Apps --------------------------- -The following section describes how you can easilly add support for multi-turn scenarios via Arch. You process and manage multi-turn prompts -just like you manage single-turn ones. Arch handles the conpleixity of detecting the correct intent based on the last user prompt and -the covnersational history, extracts relevant parameters needed by downstream APIs, and dipatches calls to any upstream LLMs to summarize the -response from your APIs. - - -.. _multi_turn_subsection_prompt_target: - -Step 1: Define Arch Config -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: includes/multi_turn/prompt_targets_multi_turn.yaml - :language: yaml - :caption: Arch Config - :linenos: - -Step 2: Process Request in Flask -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once the prompt targets are configured as above, handle parameters across multi-turn as if its a single-turn request - -.. literalinclude:: includes/multi_turn/multi_turn_rag.py - :language: python - :caption: Parameter handling with Flask - :linenos: - -Demo App -~~~~~~~~ - -For your convenience, we've built a `demo app `_ -that you can test and modify locally for multi-turn RAG scenarios. - -.. figure:: includes/multi_turn/mutli-turn-example.png - :width: 100% - :align: center - - Example multi-turn user conversation showing adjusting retrieval diff --git a/docs/source/build_with_arch/rag.rst b/docs/source/build_with_arch/rag.rst deleted file mode 100644 index eafd6e2b..00000000 --- a/docs/source/build_with_arch/rag.rst +++ /dev/null @@ -1,52 +0,0 @@ -.. _arch_rag_guide: - -RAG Apps -======== - -The following section describes how Arch can help you build faster, smarter and more accurate -Retrieval-Augmented Generation (RAG) applications, including fast and accurate RAG in multi-turn -converational scenarios. - -What is Retrieval-Augmented Generation (RAG)? ---------------------------------------------- -RAG applications combine retrieval-based methods with generative AI models to provide more accurate, -contextually relevant, and reliable outputs. These applications leverage external data sources to augment -the capabilities of Large Language Models (LLMs), enabling them to retrieve and integrate specific information -rather than relying solely on the LLM's internal knowledge. - -Parameter Extraction for RAG ----------------------------- - -To build RAG (Retrieval Augmented Generation) applications, you can configure prompt targets with parameters, -enabling Arch to retrieve critical information in a structured way for processing. This approach improves the -retrieval quality and speed of your application. By extracting parameters from the conversation, you can pull -the appropriate chunks from a vector database or SQL-like data store to enhance accuracy. With Arch, you can -streamline data retrieval and processing to build more efficient and precise RAG applications. - -Step 1: Define Prompt Targets -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: includes/rag/prompt_targets.yaml - :language: yaml - :caption: Prompt Targets - :linenos: - -Step 2: Process Request Parameters in Flask -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once the prompt targets are configured as above, handling those parameters is - -.. literalinclude:: includes/rag/parameter_handling.py - :language: python - :caption: Parameter handling with Flask - :linenos: - -Multi-Turn RAG (Follow-up Questions) -------------------------------------- -Developers often `struggle `_ to efficiently handle -``follow-up`` or ``clarification`` questions. Specifically, when users ask for changes or additions to previous responses, it requires developers to -re-write prompts using LLMs with precise prompt engineering techniques. This process is slow, manual, error prone and adds signifcant latency to the -user experience. - -Arch is highly capable of accurately detecting and processing prompts in a multi-turn scenarios so that you can buil fast and accurate RAG apps in -minutes. For additional details on how to build multi-turn RAG applications please refer to our :ref:`multi-turn ` docs. diff --git a/docs/source/build_with_arch/includes/agent/function-calling-agent.yaml b/docs/source/build_with_plano/includes/agent/function-calling-agent.yaml similarity index 100% rename from docs/source/build_with_arch/includes/agent/function-calling-agent.yaml rename to docs/source/build_with_plano/includes/agent/function-calling-agent.yaml diff --git a/docs/source/build_with_arch/includes/agent/function-calling-flow.jpg b/docs/source/build_with_plano/includes/agent/function-calling-flow.jpg similarity index 100% rename from docs/source/build_with_arch/includes/agent/function-calling-flow.jpg rename to docs/source/build_with_plano/includes/agent/function-calling-flow.jpg diff --git a/docs/source/build_with_arch/includes/agent/parameter_handling.py b/docs/source/build_with_plano/includes/agent/parameter_handling.py similarity index 100% rename from docs/source/build_with_arch/includes/agent/parameter_handling.py rename to docs/source/build_with_plano/includes/agent/parameter_handling.py diff --git a/docs/source/build_with_arch/includes/multi_turn/multi_turn_rag.py b/docs/source/build_with_plano/includes/multi_turn/multi_turn_rag.py similarity index 100% rename from docs/source/build_with_arch/includes/multi_turn/multi_turn_rag.py rename to docs/source/build_with_plano/includes/multi_turn/multi_turn_rag.py diff --git a/docs/source/build_with_arch/includes/multi_turn/mutli-turn-example.png b/docs/source/build_with_plano/includes/multi_turn/mutli-turn-example.png similarity index 100% rename from docs/source/build_with_arch/includes/multi_turn/mutli-turn-example.png rename to docs/source/build_with_plano/includes/multi_turn/mutli-turn-example.png diff --git a/docs/source/build_with_arch/includes/multi_turn/prompt_targets_multi_turn.yaml b/docs/source/build_with_plano/includes/multi_turn/prompt_targets_multi_turn.yaml similarity index 100% rename from docs/source/build_with_arch/includes/multi_turn/prompt_targets_multi_turn.yaml rename to docs/source/build_with_plano/includes/multi_turn/prompt_targets_multi_turn.yaml diff --git a/docs/source/build_with_arch/includes/rag/parameter_handling.py b/docs/source/build_with_plano/includes/rag/parameter_handling.py similarity index 100% rename from docs/source/build_with_arch/includes/rag/parameter_handling.py rename to docs/source/build_with_plano/includes/rag/parameter_handling.py diff --git a/docs/source/build_with_arch/includes/rag/prompt_targets.yaml b/docs/source/build_with_plano/includes/rag/prompt_targets.yaml similarity index 100% rename from docs/source/build_with_arch/includes/rag/prompt_targets.yaml rename to docs/source/build_with_plano/includes/rag/prompt_targets.yaml diff --git a/docs/source/concepts/agents.rst b/docs/source/concepts/agents.rst new file mode 100644 index 00000000..5a51196c --- /dev/null +++ b/docs/source/concepts/agents.rst @@ -0,0 +1,76 @@ +.. _agents: + +Agents +====== + +Agents are autonomous systems that handle wide-ranging, open-ended tasks by calling models in a loop until the work is complete. Unlike deterministic :ref:`prompt targets `, agents have access to tools, reason about which actions to take, and adapt their behavior based on intermediate results—making them ideal for complex workflows that require multi-step reasoning, external API calls, and dynamic decision-making. + +Plano helps developers build and scale multi-agent systems by managing the orchestration layer—deciding which agent(s) or LLM(s) should handle each request, and in what sequence—while developers focus on implementing agent logic in any language or framework they choose. + +Agent Orchestration +------------------- + +**Plano-Orchestrator** is a family of state-of-the-art routing and orchestration models that decide which agent(s) should handle each request, and in what sequence. Built for real-world multi-agent deployments, it analyzes user intent and conversation context to make precise routing and orchestration decisions while remaining efficient enough for low-latency production use across general chat, coding, and long-context multi-turn conversations. + +This allows development teams to: + +* **Scale multi-agent systems**: Route requests across multiple specialized agents without hardcoding routing logic in application code. +* **Improve performance**: Direct requests to the most appropriate agent based on intent, reducing unnecessary handoffs and improving response quality. +* **Enhance debuggability**: Centralized routing decisions are observable through Plano's tracing and logging, making it easier to understand why a particular agent was selected. + +Inner Loop vs. Outer Loop +-------------------------- + +Plano distinguishes between the **inner loop** (agent implementation logic) and the **outer loop** (orchestration and routing): + +Inner Loop (Agent Logic) +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The inner loop is where your agent lives—the business logic that decides which tools to call, how to interpret results, and when the task is complete. You implement this in any language or framework: + +* **Python agents**: Using frameworks like LangChain, LlamaIndex, CrewAI, or custom Python code. +* **JavaScript/TypeScript agents**: Using frameworks like LangChain.js or custom Node.js implementations. +* **Any other AI famreowkr**: Agents are just HTTP services that Plano can route to. + +Your agent controls: + +* Which tools or APIs to call in response to a prompt. +* How to interpret tool results and decide next steps. +* When to call the LLM for reasoning or summarization. +* When the task is complete and what response to return. + +.. note:: + **Making LLM Calls from Agents** + + When your agent needs to call an LLM for reasoning, summarization, or completion, you should route those calls through Plano's Model Proxy rather than calling LLM providers directly. This gives you: + + * **Consistent responses**: Normalized response formats across all :ref:`LLM providers `, whether you're using OpenAI, Anthropic, Azure OpenAI, or any OpenAI-compatible provider. + * **Rich agentic signals**: Automatic capture of function calls, tool usage, reasoning steps, and model behavior—surfaced through traces and metrics without instrumenting your agent code. + * **Smart model routing**: Leverage :ref:`model-based, alias-based, or preference-aligned routing ` to dynamically select the best model for each task based on cost, performance, or custom policies. + + By routing LLM calls through the Model Proxy, your agents remain decoupled from specific providers and can benefit from centralized policy enforcement, observability, and intelligent routing—all managed in the outer loop. For a step-by-step guide, see :ref:`implementing_routing` in the LLM Router guide. + +Outer Loop (Orchestration) +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The outer loop is Plano's orchestration layer—it manages the lifecycle of requests across agents and LLMs: + +* **Intent analysis**: Plano-Orchestrator analyzes incoming prompts to determine user intent and conversation context. +* **Routing decisions**: Routes requests to the appropriate agent(s) or LLM(s) based on capabilities, context, and availability. +* **Sequencing**: Determines whether multiple agents need to collaborate and in what order. +* **Lifecycle management**: Handles retries, failover, circuit breaking, and load balancing across agent instances. + +By managing the outer loop, Plano allows you to: + +* Add new agents without changing routing logic in existing agents. +* Run multiple versions or variants of agents for A/B testing or canary deployments. +* Apply consistent :ref:`filter chains ` (guardrails, context enrichment) before requests reach agents. +* Monitor and debug multi-agent workflows through centralized observability. + +Key Benefits +------------ + +* **Language and framework agnostic**: Write agents in any language; Plano orchestrates them via HTTP. +* **Reduced complexity**: Agents focus on task logic; Plano handles routing, retries, and cross-cutting concerns. +* **Better observability**: Centralized tracing shows which agents were called, in what sequence, and why. +* **Easier scaling**: Add more agent instances or new agent types without refactoring existing code. diff --git a/docs/source/concepts/filter_chain.rst b/docs/source/concepts/filter_chain.rst new file mode 100644 index 00000000..2b29c13a --- /dev/null +++ b/docs/source/concepts/filter_chain.rst @@ -0,0 +1,73 @@ +.. _filter_chain: + +Filter Chains +============== + +Filter chains are Plano's way of capturing **reusable workflow steps** in the dataplane, without duplication and coupling logic into application code. A filter chain is an ordered list of **mutations** that a request flows through before reaching its final destination —such as an agent, an LLM, or a tool backend. Each filter is a network-addressable service/path that can: + +1. Inspect the incoming prompt, metadata, and conversation state. +2. Mutate or enrich the request (for example, rewrite queries or build context). +3. Short-circuit the flow and return a response early (for example, block a request on a compliance failure). +4. Emit structured logs and traces so you can debug and continuously improve your agents. + +In other words, filter chains provide a lightweight programming model over HTTP for building reusable steps +in your agent architectures. + +Typical Use Cases +----------------- + +Without a dataplane programming model, teams tend to spread logic like query rewriting, compliance checks, +context building, and routing decisions across many agents and frameworks. This quickly becomes hard to reason +about and even harder to evolve. + +Filter chains show up most often in patterns like: + +* **Guardrails and Compliance**: Enforcing content policies, stripping or masking sensitive data, and blocking obviously unsafe or off-topic requests before they reach an agent. +* **Query rewriting, RAG, and Memory**: Rewriting user queries for retrieval, normalizing entities, and assembling RAG context envelopes while pulling in relevant memory (for example, conversation history, user profiles, or prior tool results) before calling a model or tool. +* **Cross-cutting Observability**: Injecting correlation IDs, sampling traces, or logging enriched request metadata at consistent points in the request path. + +Because these behaviors live in the dataplane rather than inside individual agents, you define them once, attach them to many agents and prompt targets, and can add, remove, or reorder them without changing application code. + +Configuration example +--------------------- + +The example below shows a configuration where an agent uses a filter chain with two filters: a query rewriter, +and a context builder that prepares retrieval context before the agent runs. + +.. literalinclude:: ../../source/resources/includes/plano_config_agents_filters.yaml + :language: yaml + :linenos: + :emphasize-lines: 29-39 + :caption: Example Configuration + +In this setup: + +* The ``filters`` section defines the reusable filters, each running as its own HTTP/MCP service. +* The ``listeners`` section wires the ``rag_agent`` behind an ``agent`` listener and attaches a ``filter_chain`` with ``query_rewriter`` followed by ``context_builder``. +* When a request arrives at ``agent_1``, Plano executes the filters in order before handing control to ``rag_agent``. + + +Filter Chain Programming Model (HTTP and MCP) +--------------------------------------------- + +Filters are implemented as simple RESTful endpoints reachable via HTTP. If you want to use the `Model Context Protocol (MCP) `_, you can configure that as well, which makes it easy to write filters in any language. However, you can also write a filter as a plain HTTP service. + +When defining a filter in Plano configuration, the following fields are optional: + +* ``type``: Controls the filter runtime (defaults to ``mcp``). You can omit this if you're using MCP. +* ``transport``: Controls how Plano talks to the filter (defaults to ``streamable-http`` for efficient streaming interactions over HTTP). You can omit this for standard HTTP transport. +* ``tool``: Names the MCP tool Plano will invoke (by default, the filter ``id``). You can omit this if the tool name matches your filter id. + +In practice, you typically only need to specify ``id`` and ``url`` to get started. Plano's sensible defaults mean a filter can be as simple as an HTTP endpoint. If you want to customize the runtime or protocol, those fields are there, but they're optional. + +Filters communicate the outcome of their work via HTTP status codes: + +* **HTTP 200 (Success)**: The filter successfully processed the request. If the filter mutated the request (e.g., rewrote a query or enriched context), those mutations are passed downstream. +* **HTTP 4xx (User Error)**: The request violates a filter's rules or constraints—for example, content moderation policies or compliance checks. The request is terminated, and the error is returned to the caller. This is *not* a fatal error; it represents expected user-facing policy enforcement. +* **HTTP 5xx (Fatal Error)**: An unexpected failure in the filter itself (for example, a crash or misconfiguration). Plano will surface the error back to the caller and record it in logs and traces. + +This semantics allows filters to enforce guardrails and policies (4xx) without blocking the entire system, while still surfacing critical failures (5xx) for investigation. + +If any filter fails or decides to terminate the request early (for example, after a policy violation), Plano will +surface that outcome back to the caller and record it in logs and traces. This makes filter chains a safe and +powerful abstraction for evolving your agent workflows over time. diff --git a/docs/source/concepts/includes/arch_config.yaml b/docs/source/concepts/includes/plano_config.yaml similarity index 79% rename from docs/source/concepts/includes/arch_config.yaml rename to docs/source/concepts/includes/plano_config.yaml index 69b24f41..e1dc0749 100644 --- a/docs/source/concepts/includes/arch_config.yaml +++ b/docs/source/concepts/includes/plano_config.yaml @@ -1,27 +1,16 @@ -version: v0.1.0 +version: v0.2.0 listeners: ingress_traffic: address: 0.0.0.0 port: 10000 - message_format: openai - timeout: 30s # Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way -llm_providers: +model_providers: - access_key: $OPENAI_API_KEY model: openai/gpt-4o default: true -# default system prompt used by all prompt targets -system_prompt: You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions. - -prompt_guards: - input_guards: - jailbreak: - on_exception: - message: Looks like you're curious about my abilities, but I can only provide assistance within my programmed parameters. - prompt_targets: - name: information_extraction default: true diff --git a/docs/source/concepts/listeners.rst b/docs/source/concepts/listeners.rst new file mode 100644 index 00000000..d8b78e11 --- /dev/null +++ b/docs/source/concepts/listeners.rst @@ -0,0 +1,79 @@ +.. _plano_overview_listeners: + +Listeners +--------- +**Listeners** are a top-level primitive in Plano that bind network traffic to the dataplane. They simplify the +configuration required to accept incoming connections from downstream clients (edge) and to expose a unified egress +endpoint for calls from your applications to upstream LLMs. + +Plano builds on Envoy's Listener subsystem to streamline connection management for developers. It hides most of +Envoy's complexity behind sensible defaults and a focused configuration surface, so you can bind listeners without +deep knowledge of Envoy’s configuration model while still getting secure, reliable, and performant connections. + +Listeners are modular building blocks: you can configure only inbound listeners (for edge proxying and guardrails), +only outbound/model-proxy listeners (for LLM routing from your services), or both together. This lets you fit Plano +cleanly into existing architectures, whether you need it at the edge, behind the firewall, or across the full +request path. + + +Network Topology +^^^^^^^^^^^^^^^^ + +The diagram below shows how inbound and outbound traffic flow through Plano and how listeners relate to agents, +prompt targets, and upstream LLMs: + +.. image:: /_static/img/network-topology-ingress-egress.png + :width: 100% + :align: center + + +Inbound (Agent & Prompt Target) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Developers configure **inbound listeners** to accept connections from clients such as web frontends, backend +services, or other gateways. An inbound listener acts as the primary entry point for prompt traffic, handling +initial connection setup, TLS termination, guardrails, and forwarding incoming traffic to the appropriate prompt +targets or agents. + +There are two primary types of inbound connections exposed via listeners: + +* **Agent Inbound (Edge)**: Clients (web/mobile apps or other services) connect to Plano, send prompts, and receive + responses. This is typically your public/edge listener where Plano applies guardrails, routing, and orchestration + before returning results to the caller. + +* **Prompt Target Inbound (Edge)**: Your application server calls Plano's internal listener targeting + :ref:`prompt targets ` that can invoke tools and LLMs directly on its behalf. + +Inbound listeners are where you attach :ref:`Filter Chains ` so that safety and context-building happen +consistently at the edge. + +Outbound (Model Proxy & Egress) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Plano also exposes an **egress listener** that your applications call when sending requests to upstream LLM providers +or self-hosted models. From your application's perspective this looks like a single OpenAI-compatible HTTP endpoint +(for example, ``http://127.0.0.1:12000/v1``), while Plano handles provider selection, retries, and failover behind +the scenes. + +Under the hood, Plano opens outbound HTTP(S) connections to upstream LLM providers using its unified API surface and +smart model routing. For more details on how Plano talks to models and how providers are configured, see +:ref:`LLM providers `. + +Configure Listeners +^^^^^^^^^^^^^^^^^^^ + +Listeners are configured via the ``listeners`` block in your Plano configuration. You can define one or more inbound +listeners (for example, ``type:edge``) or one or more outbound/model listeners (for example, ``type:model``), or both +in the same deployment. + +To configure an inbound (edge) listener, add a ``listeners`` block to your configuration file and define at least one +listener with address, port, and protocol details: + +.. literalinclude:: ./includes/plano_config.yaml + :language: yaml + :linenos: + :lines: 1-13 + :emphasize-lines: 3-7 + :caption: Example Configuration + +When you start Plano, you specify a listener address/port that you want to bind downstream. Plano also exposes a +predefined internal listener (``127.0.0.1:12000``) that you can use to proxy egress calls originating from your +application to LLMs (API-based or hosted) via prompt targets. diff --git a/docs/source/concepts/llm_providers/client_libraries.rst b/docs/source/concepts/llm_providers/client_libraries.rst index 5c3aa0b4..b9316fb6 100644 --- a/docs/source/concepts/llm_providers/client_libraries.rst +++ b/docs/source/concepts/llm_providers/client_libraries.rst @@ -3,7 +3,7 @@ Client Libraries ================ -Arch provides a unified interface that works seamlessly with multiple client libraries and tools. You can use your preferred client library without changing your existing code - just point it to Arch's gateway endpoints. +Plano provides a unified interface that works seamlessly with multiple client libraries and tools. You can use your preferred client library without changing your existing code - just point it to Plano's gateway endpoints. Supported Clients ------------------ @@ -16,7 +16,7 @@ Supported Clients Gateway Endpoints ----------------- -Arch exposes two main endpoints: +Plano exposes three main endpoints: .. list-table:: :header-rows: 1 @@ -26,13 +26,15 @@ Arch exposes two main endpoints: - Purpose * - ``http://127.0.0.1:12000/v1/chat/completions`` - OpenAI-compatible chat completions (LLM Gateway) + * - ``http://127.0.0.1:12000/v1/responses`` + - OpenAI Responses API with :ref:`conversational state management ` (LLM Gateway) * - ``http://127.0.0.1:12000/v1/messages`` - Anthropic-compatible messages (LLM Gateway) OpenAI (Python) SDK ------------------- -The OpenAI SDK works with any provider through Arch's OpenAI-compatible endpoint. +The OpenAI SDK works with any provider through Plano's OpenAI-compatible endpoint. **Installation:** @@ -46,7 +48,7 @@ The OpenAI SDK works with any provider through Arch's OpenAI-compatible endpoint from openai import OpenAI - # Point to Arch's LLM Gateway + # Point to Plano's LLM Gateway client = OpenAI( api_key="test-key", # Can be any value for local testing base_url="http://127.0.0.1:12000/v1" @@ -96,7 +98,7 @@ The OpenAI SDK works with any provider through Arch's OpenAI-compatible endpoint **Using with Non-OpenAI Models:** -The OpenAI SDK can be used with any provider configured in Arch: +The OpenAI SDK can be used with any provider configured in Plano: .. code-block:: python @@ -124,10 +126,92 @@ The OpenAI SDK can be used with any provider configured in Arch: ] ) +OpenAI Responses API (Conversational State) +------------------------------------------- + +The OpenAI Responses API (``v1/responses``) enables multi-turn conversations with automatic state management. Plano handles conversation history for you, so you don't need to manually include previous messages in each request. + +See :ref:`managing_conversational_state` for detailed configuration and storage backend options. + +**Installation:** + +.. code-block:: bash + + pip install openai + +**Basic Multi-Turn Conversation:** + +.. code-block:: python + + from openai import OpenAI + + # Point to Plano's LLM Gateway + client = OpenAI( + api_key="test-key", + base_url="http://127.0.0.1:12000/v1" + ) + + # First turn - creates a new conversation + response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[ + {"role": "user", "content": "My name is Alice"} + ] + ) + + # Extract response_id for conversation continuity + response_id = response.id + print(f"Assistant: {response.choices[0].message.content}") + + # Second turn - continues the conversation + # Plano automatically retrieves and merges previous context + response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[ + {"role": "user", "content": "What's my name?"} + ], + metadata={"response_id": response_id} # Reference previous conversation + ) + + print(f"Assistant: {response.choices[0].message.content}") + # Output: "Your name is Alice" + +**Using with Any Provider:** + +The Responses API works with any LLM provider configured in Plano: + +.. code-block:: python + + # Multi-turn conversation with Claude + response = client.chat.completions.create( + model="claude-3-5-sonnet-20241022", + messages=[ + {"role": "user", "content": "Let's discuss quantum physics"} + ] + ) + + response_id = response.id + + # Continue conversation - Plano manages state regardless of provider + response = client.chat.completions.create( + model="claude-3-5-sonnet-20241022", + messages=[ + {"role": "user", "content": "Tell me more about entanglement"} + ], + metadata={"response_id": response_id} + ) + +**Key Benefits:** + +* **Reduced payload size**: No need to send full conversation history in each request +* **Provider flexibility**: Use any configured LLM provider with state management +* **Automatic context merging**: Plano handles conversation continuity behind the scenes +* **Production-ready storage**: Configure :ref:`PostgreSQL or memory storage ` based on your needs + Anthropic (Python) SDK ---------------------- -The Anthropic SDK works with any provider through Arch's Anthropic-compatible endpoint. +The Anthropic SDK works with any provider through Plano's Anthropic-compatible endpoint. **Installation:** @@ -141,7 +225,7 @@ The Anthropic SDK works with any provider through Arch's Anthropic-compatible en import anthropic - # Point to Arch's LLM Gateway + # Point to Plano's LLM Gateway client = anthropic.Anthropic( api_key="test-key", # Can be any value for local testing base_url="http://127.0.0.1:12000" @@ -192,7 +276,7 @@ The Anthropic SDK works with any provider through Arch's Anthropic-compatible en **Using with Non-Anthropic Models:** -The Anthropic SDK can be used with any provider configured in Arch: +The Anthropic SDK can be used with any provider configured in Plano: .. code-block:: python @@ -284,7 +368,7 @@ For direct HTTP requests or integration with any programming language: Cross-Client Compatibility -------------------------- -One of Arch's key features is cross-client compatibility. You can: +One of Plano's key features is cross-client compatibility. You can: **Use OpenAI SDK with Claude Models:** diff --git a/docs/source/concepts/llm_providers/llm_providers.rst b/docs/source/concepts/llm_providers/llm_providers.rst index 782a7163..85a02dab 100644 --- a/docs/source/concepts/llm_providers/llm_providers.rst +++ b/docs/source/concepts/llm_providers/llm_providers.rst @@ -1,16 +1,16 @@ .. _llm_providers: -LLM Providers -============= -**LLM Providers** are a top-level primitive in Arch, helping developers centrally define, secure, observe, -and manage the usage of their LLMs. Arch builds on Envoy's reliable `cluster subsystem `_ +Model (LLM) Providers +===================== +**LLM Providers** are a top-level primitive in Plano, helping developers centrally define, secure, observe, +and manage the usage of their LLMs. Plano builds on Envoy's reliable `cluster subsystem `_ to manage egress traffic to LLMs, which includes intelligent routing, retry and fail-over mechanisms, ensuring high availability and fault tolerance. This abstraction also enables developers to seamlessly switch between LLM providers or upgrade LLM versions, simplifying the integration and scaling of LLMs across applications. Today, we are enabling you to connect to 11+ different AI providers through a unified interface with advanced routing and management capabilities. -Whether you're using OpenAI, Anthropic, Azure OpenAI, local Ollama models, or any OpenAI-compatible provider, Arch provides seamless integration with enterprise-grade features. +Whether you're using OpenAI, Anthropic, Azure OpenAI, local Ollama models, or any OpenAI-compatible provider, Plano provides seamless integration with enterprise-grade features. Core Capabilities ----------------- @@ -18,29 +18,29 @@ Core Capabilities **Multi-Provider Support** Connect to any combination of providers simultaneously (see :ref:`supported_providers` for full details): -- **First-Class Providers**: Native integrations with OpenAI, Anthropic, DeepSeek, Mistral, Groq, Google Gemini, Together AI, xAI, Azure OpenAI, and Ollama -- **OpenAI-Compatible Providers**: Any provider implementing the OpenAI Chat Completions API standard +- First-Class Providers: Native integrations with OpenAI, Anthropic, DeepSeek, Mistral, Groq, Google Gemini, Together AI, xAI, Azure OpenAI, and Ollama +- OpenAI-Compatible Providers: Any provider implementing the OpenAI Chat Completions API standard **Intelligent Routing** Three powerful routing approaches to optimize model selection: -- **Model-based Routing**: Direct routing to specific models using provider/model names (see :ref:`supported_providers`) -- **Alias-based Routing**: Semantic routing using custom aliases (see :ref:`model_aliases`) -- **Preference-aligned Routing**: Intelligent routing using the Arch-Router model (see :ref:`preference_aligned_routing`) +- Model-based Routing: Direct routing to specific models using provider/model names (see :ref:`supported_providers`) +- Alias-based Routing: Semantic routing using custom aliases (see :ref:`model_aliases`) +- Preference-aligned Routing: Intelligent routing using the Plano-Router model (see :ref:`preference_aligned_routing`) **Unified Client Interface** Use your preferred client library without changing existing code (see :ref:`client_libraries` for details): -- **OpenAI Python SDK**: Full compatibility with all providers -- **Anthropic Python SDK**: Native support with cross-provider capabilities -- **cURL & HTTP Clients**: Direct REST API access for any programming language -- **Custom Integrations**: Standard HTTP interfaces for seamless integration +- OpenAI Python SDK: Full compatibility with all providers +- Anthropic Python SDK: Native support with cross-provider capabilities +- cURL & HTTP Clients: Direct REST API access for any programming language +- Custom Integrations: Standard HTTP interfaces for seamless integration Key Benefits ------------ - **Provider Flexibility**: Switch between providers without changing client code -- **Three Routing Methods**: Choose from model-based, alias-based, or preference-aligned routing (using `Arch-Router-1.5B `_) strategies +- **Three Routing Methods**: Choose from model-based, alias-based, or preference-aligned routing (using `Plano-Router-1.5B `_) strategies - **Cost Optimization**: Route requests to cost-effective models based on complexity - **Performance Optimization**: Use fast models for simple tasks, powerful models for complex reasoning - **Environment Management**: Configure different models for different environments diff --git a/docs/source/concepts/llm_providers/supported_providers.rst b/docs/source/concepts/llm_providers/supported_providers.rst index be063885..a8237ca7 100644 --- a/docs/source/concepts/llm_providers/supported_providers.rst +++ b/docs/source/concepts/llm_providers/supported_providers.rst @@ -3,10 +3,10 @@ Supported Providers & Configuration =================================== -Arch provides first-class support for multiple LLM providers through native integrations and OpenAI-compatible interfaces. This comprehensive guide covers all supported providers, their available chat models, and detailed configuration instructions. +Plano provides first-class support for multiple LLM providers through native integrations and OpenAI-compatible interfaces. This comprehensive guide covers all supported providers, their available chat models, and detailed configuration instructions. .. note:: - **Model Support:** Arch supports all chat models from each provider, not just the examples shown in this guide. The configurations below demonstrate common models for reference, but you can use any chat model available from your chosen provider. + **Model Support:** Plano supports all chat models from each provider, not just the examples shown in this guide. The configurations below demonstrate common models for reference, but you can use any chat model available from your chosen provider. Configuration Structure ----------------------- @@ -50,7 +50,7 @@ Any provider that implements the OpenAI API interface can be configured using cu Supported API Endpoints ------------------------ -Arch supports the following standardized endpoints across providers: +Plano supports the following standardized endpoints across providers: .. list-table:: :header-rows: 1 @@ -524,7 +524,7 @@ Amazon Bedrock **Provider Prefix:** ``amazon_bedrock/`` -**API Endpoint:** Arch automatically constructs the endpoint as: +**API Endpoint:** Plano automatically constructs the endpoint as: - Non-streaming: ``/model/{model-id}/converse`` - Streaming: ``/model/{model-id}/converse-stream`` diff --git a/docs/source/concepts/prompt_target.rst b/docs/source/concepts/prompt_target.rst index c708a9e7..95b5b567 100644 --- a/docs/source/concepts/prompt_target.rst +++ b/docs/source/concepts/prompt_target.rst @@ -1,15 +1,17 @@ .. _prompt_target: Prompt Target -============== +============= +A Prompt Target is a deterministic, task-specific backend function or API endpoint that your application calls via Plano. +Unlike agents (which handle wide-ranging, open-ended tasks), prompt targets are designed for focused, specific workloads where Plano can add value through input clarification and validation. -**Prompt Targets** are a core concept in Arch, empowering developers to clearly define how user prompts are interpreted, processed, and routed within their generative AI applications. Prompts can seamlessly be routed either to specialized AI agents capable of handling sophisticated, context-driven tasks or to targeted tools provided by your application, offering users a fast, precise, and personalized experience. +Plano helps by: -This section covers the essentials of prompt targets—what they are, how to configure them, their practical uses, and recommended best practices—to help you fully utilize this feature in your applications. +* **Clarifying and validating input**: Plano enriches incoming prompts with metadata (e.g., detecting follow-ups or clarifying requests) and can extract structured parameters from natural language before passing them to your backend. +* **Enabling high determinism**: Since the task is specific and well-defined, Plano can reliably extract the information your backend needs without ambiguity. +* **Reducing backend work**: Your backend receives clean, validated, structured inputs—so you can focus on business logic instead of parsing and validation. -What Are Prompt Targets? ------------------------- -Prompt targets are endpoints within Arch that handle specific types of user prompts. They act as the bridge between user inputs and your backend agents or tools (APIs), enabling Arch to route, process, and manage prompts efficiently. Defining prompt targets helps you decouple your application's core logic from processing and handling complexities, leading to clearer code organization, better scalability, and easier maintenance. +For example, a prompt target might be "schedule a meeting" (specific task, deterministic inputs like date, time, attendees) or "retrieve documents" (well-defined RAG query with clear intent). Prompt targets are typically called from your application code via Plano's internal listener. .. table:: @@ -33,16 +35,11 @@ Below are the key features of prompt targets that empower developers to build ef - **Input Management**: Specify required and optional parameters for each target. - **Tools Integration**: Seamlessly connect prompts to backend APIs or functions. - **Error Handling**: Direct errors to designated handlers for streamlined troubleshooting. -- **Metadata Enrichment**: Attach additional context to prompts for enhanced processing. - -Configuring Prompt Targets --------------------------- -Configuring prompt targets involves defining them in Arch's configuration file. Each Prompt target specifies how a particular type of prompt should be handled, including the endpoint to invoke and any parameters required. +- **Multi-Turn Support**: Manage follow-up prompts and clarifications in conversational flows. Basic Configuration ~~~~~~~~~~~~~~~~~~~ - -A prompt target configuration includes the following elements: +Configuring prompt targets involves defining them in Plano's configuration file. Each Prompt target specifies how a particular type of prompt should be handled, including the endpoint to invoke and any parameters required. A prompt target configuration includes the following elements: .. vale Vale.Spelling = NO @@ -55,8 +52,8 @@ A prompt target configuration includes the following elements: Defining Parameters ~~~~~~~~~~~~~~~~~~~ -Parameters are the pieces of information that Arch needs to extract from the user's prompt to perform the desired action. -Each parameter can be marked as required or optional. Here is a full list of parameter attributes that Arch can support: +Parameters are the pieces of information that Plano needs to extract from the user's prompt to perform the desired action. +Each parameter can be marked as required or optional. Here is a full list of parameter attributes that Plano can support: .. table:: :width: 100% @@ -98,50 +95,92 @@ Example Configuration For Tools name: api_server path: /weather -Example Configuration For Agents -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _plano_multi_turn_guide: -.. code-block:: yaml - :caption: Agent Orchestration Configuration Example +Multi-Turn +~~~~~~~~~~ +Developers often `struggle `_ to efficiently handle +``follow-up`` or ``clarification`` questions. Specifically, when users ask for changes or additions to previous responses, it requires developers to +re-write prompts using LLMs with precise prompt engineering techniques. This process is slow, manual, error prone and adds latency and token cost for +common scenarios that can be managed more efficiently. - overrides: - use_agent_orchestrator: true +Plano is highly capable of accurately detecting and processing prompts in multi-turn scenarios so that you can buil fast and accurate agents in minutes. +Below are some cnversational examples that you can build via Plano. Each example is enriched with annotations (via ** [Plano] ** ) that illustrates how Plano +processess conversational messages on your behalf. - prompt_targets: - - name: sales_agent - description: handles queries related to sales and purchases +Example 1: Adjusting Retrieval - - name: issues_and_repairs - description: handles issues, repairs, or refunds +.. code-block:: text - - name: escalate_to_human - description: escalates to human agent + User: What are the benefits of renewable energy? + **[Plano]**: Check if there is an available that can handle this user query. + **[Plano]**: Found "get_info_for_energy_source" prompt_target in arch_config.yaml. Forward prompt to the endpoint configured in "get_info_for_energy_source" + ... + Assistant: Renewable energy reduces greenhouse gas emissions, lowers air pollution, and provides sustainable power sources like solar and wind. -.. note:: - Today, you can use Arch to coordinate more specific agentic scenarios via tools and function calling, or use it for high-level agent routing and hand off scenarios. In the future, we plan to offer you the ability to combine these two approaches for more complex scenarios. Please see `github issues `_ for more details. + User: Include cost considerations in the response. + **[Plano]**: Follow-up detected. Forward prompt history to the "get_info_for_energy_source" prompt_target and post the following parameters consideration="cost" + ... + Assistant: Renewable energy reduces greenhouse gas emissions, lowers air pollution, and provides sustainable power sources like solar and wind. While the initial setup costs can be high, long-term savings from reduced fuel expenses and government incentives make it cost-effective. -Routing Logic -------------- -Prompt targets determine where and how user prompts are processed. Arch uses intelligent routing logic to ensure that prompts are directed to the appropriate targets based on their intent and context. -Default Targets -~~~~~~~~~~~~~~~ -For general-purpose prompts that do not match any specific prompt target, Arch routes them to a designated default target. This is useful for handling open-ended queries like document summarization or information extraction. +Example 2: Switching Intent +--------------------------- +.. code-block:: text -Intent Matching -~~~~~~~~~~~~~~~ -Arch analyzes the user's prompt to determine its intent and matches it with the most suitable prompt target based on the name and description defined in the configuration. + User: What are the symptoms of diabetes? + **[Plano]**: Check if there is an available that can handle this user query. + **[Plano]**: Found "diseases_symptoms" prompt_target in arch_config.yaml. Forward disease=diabeteres to "diseases_symptoms" prompt target + ... + Assistant: Common symptoms include frequent urination, excessive thirst, fatigue, and blurry vision. -For example: + User: How is it diagnosed? + **[Plano]**: New intent detected. + **[Plano]**: Found "disease_diagnoses" prompt_target in arch_config.yaml. Forward disease=diabeteres to "disease_diagnoses" prompt target + ... + Assistant: Diabetes is diagnosed through blood tests like fasting blood sugar, A1C, or an oral glucose tolerance test. -.. code-block:: bash - Prompt: "Can you reboot the router?" - Matching Target: reboot_device (based on description matching "reboot devices") +Build Multi-Turn RAG Apps +------------------------- +The following section describes how you can easilly add support for multi-turn scenarios via Plano. You process and manage multi-turn prompts +just like you manage single-turn ones. Plano handles the conpleixity of detecting the correct intent based on the last user prompt and +the covnersational history, extracts relevant parameters needed by downstream APIs, and dipatches calls to any upstream LLMs to summarize the +response from your APIs. +.. _multi_turn_subsection_prompt_target: + +Step 1: Define Plano Config +--------------------------- + +.. literalinclude:: ../build_with_plano/includes/multi_turn/prompt_targets_multi_turn.yaml + :language: yaml + :caption: Plano Config + :linenos: + +Step 2: Process Request in Flask +-------------------------------- + +Once the prompt targets are configured as above, handle parameters across multi-turn as if its a single-turn request + +.. literalinclude:: ../build_with_plano/includes/multi_turn/multi_turn_rag.py + :language: python + :caption: Parameter handling with Flask + :linenos: + +Demo App +-------- + +For your convenience, we've built a `demo app `_ +that you can test and modify locally for multi-turn RAG scenarios. + +.. figure:: ../build_with_plano/includes/multi_turn/mutli-turn-example.png + :width: 100% + :align: center + + Example multi-turn user conversation showing adjusting retrieval + Summary --------- -Prompt targets are essential for defining how user prompts are handled within your generative AI applications using Arch. - -By carefully configuring prompt targets, you can ensure that prompts are accurately routed, necessary parameters are extracted, and backend services are invoked seamlessly. This modular approach not only simplifies your application's architecture but also enhances scalability, maintainability, and overall user experience. +~~~~~~~ +By carefully designing prompt targets as deterministic, task-specific entry points, you ensure that prompts are routed to the right workload, necessary parameters are cleanly extracted and validated, and backend services are invoked with structured inputs. This clear separation between prompt handling and business logic simplifies your architecture, makes behavior more predictable and testable, and improves the scalability and maintainability of your agentic applications. diff --git a/docs/source/concepts/tech_overview/error_target.rst b/docs/source/concepts/tech_overview/error_target.rst deleted file mode 100644 index e901ae5e..00000000 --- a/docs/source/concepts/tech_overview/error_target.rst +++ /dev/null @@ -1,53 +0,0 @@ -.. _error_target: - -Error Target -============= - -**Error targets** are designed to capture and manage specific issues or exceptions that occur during Arch's function or system's execution. - -These endpoints receive errors forwarded from Arch when issues arise, such as improper function/API calls, guardrail violations, or other processing errors. -The errors are communicated to the application via headers like ``X-Arch-[ERROR-TYPE]``, enabling you to respond appropriately and handle errors gracefully. - - -Key Concepts ------------- - -- **Error Type**: Categorizes the nature of the error, such as "ValidationError" or "RuntimeError." These error types help in identifying what kind of issue occurred and provide context for troubleshooting. - -- **Error Message**: A clear, human-readable message describing the error. This should provide enough detail to inform users or developers of the root cause or required action. - -- **Parameter-Specific Errors**: Errors that arise due to invalid or missing parameters when invoking a function. These errors are critical for ensuring the correctness of inputs. - - -Error Header Example --------------------- - -.. code-block:: bash - :caption: Error Header Example - - HTTP/1.1 400 Bad Request - X-Arch-Error-Type: FunctionValidationError - X-Arch-Error-Message: Tools call parsing failure - X-Arch-Target-Prompt: createUser - Content-Type: application/json - - "messages": [ - { - "role": "user", - "content": "Please create a user with the following ID: 1234" - }, - { - "role": "system", - "content": "Expected a string for 'user_id', but got an integer." - } - ] - - -Best Practices and Tips ------------------------ - -- **Graceful Degradation**: If an error occurs, fail gracefully by providing fallback logic or alternative flows when possible. - -- **Log Errors**: Always log errors on the server side for later analysis. - -- **Client-Side Handling**: Make sure the client can interpret error responses and provide meaningful feedback to the user. Clients should not display raw error codes or stack traces but rather handle them gracefully. diff --git a/docs/source/concepts/tech_overview/listener.rst b/docs/source/concepts/tech_overview/listener.rst deleted file mode 100644 index dd486986..00000000 --- a/docs/source/concepts/tech_overview/listener.rst +++ /dev/null @@ -1,37 +0,0 @@ -.. _arch_overview_listeners: - -Listener ---------- -**Listener** is a top level primitive in Arch, which simplifies the configuration required to bind incoming -connections from downstream clients, and for egress connections to LLMs (hosted or API) - -Arch builds on Envoy's Listener subsystem to streamline connection management for developers. Arch minimizes -the complexity of Envoy's listener setup by using best-practices and exposing only essential settings, -making it easier for developers to bind connections without deep knowledge of Envoy’s configuration model. This -simplification ensures that connections are secure, reliable, and optimized for performance. - -Downstream (Ingress) -^^^^^^^^^^^^^^^^^^^^^^ -Developers can configure Arch to accept connections from downstream clients. A downstream listener acts as the -primary entry point for incoming traffic, handling initial connection setup, including network filtering, guardrails, -and additional network security checks. For more details on prompt security and safety, -see :ref:`here `. - -Upstream (Egress) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Arch automatically configures a listener to route requests from your application to upstream LLM API providers (or hosts). -When you start Arch, it creates a listener for egress traffic based on the presence of the ``listener`` configuration -section in the configuration file. Arch binds itself to a local address such as ``127.0.0.1:12000/v1`` or a DNS-based -address like ``arch.local:12000/v1`` for outgoing traffic. For more details on LLM providers, read :ref:`here `. - -Configure Listener -^^^^^^^^^^^^^^^^^^ - -To configure a Downstream (Ingress) Listener, simply add the ``listener`` directive to your configuration file: - -.. literalinclude:: ../includes/arch_config.yaml - :language: yaml - :linenos: - :lines: 1-18 - :emphasize-lines: 3-7 - :caption: Example Configuration diff --git a/docs/source/concepts/tech_overview/model_serving.rst b/docs/source/concepts/tech_overview/model_serving.rst deleted file mode 100644 index 53a4e377..00000000 --- a/docs/source/concepts/tech_overview/model_serving.rst +++ /dev/null @@ -1,45 +0,0 @@ -.. _model_serving: - -Model Serving -============= - -Arch is a set of `two` self-contained processes that are designed to run alongside your application -servers (or on a separate host connected via a network). The first process is designated to manage low-level -networking and HTTP related concerns, and the other process is for model serving, which helps Arch make -intelligent decisions about the incoming prompts. The model server is designed to call the purpose-built -LLMs in Arch. - -.. image:: /_static/img/arch-system-architecture.jpg - :align: center - :width: 40% - - -Arch' is designed to be deployed in your cloud VPC, on a on-premises host, and can work on devices that don't -have a GPU. Note, GPU devices are need for fast and cost-efficient use, so that Arch (model server, specifically) -can process prompts quickly and forward control back to the application host. There are three modes in which Arch -can be configured to run its **model server** subsystem: - -Local Serving (CPU - Moderate) ------------------------------- -The following bash commands enable you to configure the model server subsystem in Arch to run local on device -and only use CPU devices. This will be the slowest option but can be useful in dev/test scenarios where GPUs -might not be available. - -.. code-block:: console - - $ archgw up --local-cpu - -Cloud Serving (GPU - Blazing Fast) ----------------------------------- -The command below instructs Arch to intelligently use GPUs locally for fast intent detection, but default to -cloud serving for function calling and guardrails scenarios to dramatically improve the speed and overall performance -of your applications. - -.. code-block:: console - - $ archgw up - -.. Note:: - Arch's model serving in the cloud is priced at $0.05M/token (156x cheaper than GPT-4o) with average latency - of 200ms (10x faster than GPT-4o). Please refer to our :ref:`Get Started ` to know - how to generate API keys for model serving diff --git a/docs/source/concepts/tech_overview/prompt.rst b/docs/source/concepts/tech_overview/prompt.rst deleted file mode 100644 index 0ba7b316..00000000 --- a/docs/source/concepts/tech_overview/prompt.rst +++ /dev/null @@ -1,127 +0,0 @@ -.. _arch_overview_prompt_handling: - -Prompts -======= - -Arch's primary design point is to securely accept, process and handle prompts. To do that effectively, -Arch relies on Envoy's HTTP `connection management `_, -subsystem and its **prompt handler** subsystem engineered with purpose-built LLMs to -implement critical functionality on behalf of developers so that you can stay focused on business logic. - -Arch's **prompt handler** subsystem interacts with the **model subsystem** through Envoy's cluster manager system to ensure robust, resilient and fault-tolerant experience in managing incoming prompts. - -.. seealso:: - Read more about the :ref:`model subsystem ` and how the LLMs are hosted in Arch. - -Messages --------- - -Arch accepts messages directly from the body of the HTTP request in a format that follows the `Hugging Face Messages API `_. -This design allows developers to pass a list of messages, where each message is represented as a dictionary -containing two key-value pairs: - - - **Role**: Defines the role of the message sender, such as "user" or "assistant". - - **Content**: Contains the actual text of the message. - - -Prompt Guard ------------------ - -Arch is engineered with `Arch-Guard `_, an industry leading safety layer, powered by a -compact and high-performing LLM that monitors incoming prompts to detect and reject jailbreak attempts - -ensuring that unauthorized or harmful behaviors are intercepted early in the process. - -To add jailbreak guardrails, see example below: - -.. literalinclude:: ../includes/arch_config.yaml - :language: yaml - :linenos: - :lines: 1-25 - :emphasize-lines: 21-25 - :caption: Example Configuration - -.. Note:: - As a roadmap item, Arch will expose the ability for developers to define custom guardrails via Arch-Guard, - and add support for additional safety checks defined by developers and hazardous categories like, violent crimes, privacy, hate, - etc. To offer feedback on our roadmap, please visit our `github page `_ - - -Prompt Targets --------------- - -Once a prompt passes any configured guardrail checks, Arch processes the contents of the incoming conversation -and identifies where to forward the conversation to via its ``prompt target`` primitive. Prompt targets are endpoints -that receive prompts that are processed by Arch. For example, Arch enriches incoming prompts with metadata like knowing -when a user's intent has changed so that you can build faster, more accurate RAG apps. - -Configuring ``prompt_targets`` is simple. See example below: - -.. literalinclude:: ../includes/arch_config.yaml - :language: yaml - :linenos: - :emphasize-lines: 39-53 - :caption: Example Configuration - - -.. seealso:: - - Check :ref:`Prompt Target ` for more details! - -Intent Matching -^^^^^^^^^^^^^^^ - -Arch uses fast text embedding and intent recognition approaches to first detect the intent of each incoming prompt. -This intent matching phase analyzes the prompt's content and matches it against predefined prompt targets, ensuring that each prompt is forwarded to the most appropriate endpoint. -Arch’s intent matching framework considers both the name and description of each prompt target, and uses a composite matching score between embedding similarity and intent classification scores to enhance accuracy in forwarding decisions. - -- **Intent Recognition**: NLI techniques further refine the matching process by evaluating the semantic alignment between the prompt and potential targets. - -- **Text Embedding**: By embedding the prompt and comparing it to known target vectors, Arch effectively identifies the closest match, ensuring that the prompt is handled by the correct downstream service. - -Agentic Apps via Prompt Targets -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To support agentic apps, like scheduling travel plans or sharing comments on a document - via prompts, Arch uses its function calling abilities to extract critical information from the incoming prompt (or a set of prompts) needed by a downstream backend API or function call before calling it directly. -For more details on how you can build agentic applications using Arch, see our full guide :ref:`here `: - -.. Note:: - `Arch-Function `_ is a collection of dedicated agentic models engineered in Arch to extract information from a (set of) prompts and executes necessary backend API calls. - This allows for efficient handling of agentic tasks, such as scheduling data retrieval, by dynamically interacting with backend services. - Arch-Function achieves state-of-the-art performance, comparable with frontier models like Claude Sonnet 3.5 ang GPT-4, while being 44x cheaper ($0.10M/token hosted) and 10x faster (p50 latencies of 200ms). - -Prompting LLMs --------------- -Arch is a single piece of software that is designed to manage both ingress and egress prompt traffic, drawing its distributed proxy nature from the robust `Envoy `_. -This makes it extremely efficient and capable of handling upstream connections to LLMs. -If your application is originating code to an API-based LLM, simply use the OpenAI client and configure it with Arch. -By sending traffic through Arch, you can propagate traces, manage and monitor traffic, apply rate limits, and utilize a large set of traffic management capabilities in a centralized way. - -.. Attention:: - When you start Arch, it automatically creates a listener port for egress calls to upstream LLMs. This is based on the - ``llm_providers`` configuration section in the ``arch_config.yml`` file. Arch binds itself to a local address such as - ``127.0.0.1:12000``. - - -Example: Using OpenAI Client with Arch as an Egress Gateway -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code-block:: python - - import openai - - # Set the OpenAI API base URL to the Arch gateway endpoint - openai.api_base = "http://127.0.0.1:12000" - - # No need to set openai.api_key since it's configured in Arch's gateway - - # Use the OpenAI client as usual - response = openai.Completion.create( - model="text-davinci-003", - prompt="What is the capital of France?" - ) - - print("OpenAI Response:", response.choices[0].text.strip()) - -In these examples, the OpenAI client is used to send traffic directly through the Arch egress proxy to the LLM of your choice, such as OpenAI. -The OpenAI client is configured to route traffic via Arch by setting the proxy to ``127.0.0.1:12000``, assuming Arch is running locally and bound to that address and port. -This setup allows you to take advantage of Arch's advanced traffic management features while interacting with LLM APIs like OpenAI. diff --git a/docs/source/concepts/tech_overview/request_lifecycle.rst b/docs/source/concepts/tech_overview/request_lifecycle.rst deleted file mode 100644 index 160eb85e..00000000 --- a/docs/source/concepts/tech_overview/request_lifecycle.rst +++ /dev/null @@ -1,170 +0,0 @@ -.. _lifecycle_of_a_request: - -Request Lifecycle -================= - -Below we describe the events in the lifecycle of a request passing through an Arch gateway instance. We first -describe how Arch fits into the request path and then the internal events that take place following -the arrival of a request at Arch from downstream clients. We follow the request until the corresponding -dispatch upstream and the response path. - -.. image:: /_static/img/network-topology-ingress-egress.jpg - :width: 100% - :align: center - -Terminology ------------ - -We recommend that you get familiar with some of the :ref:`terminology ` used in Arch -before reading this section. - -Network topology ----------------- - -How a request flows through the components in a network (including Arch) depends on the network’s topology. -Arch can be used in a wide variety of networking topologies. We focus on the inner operation of Arch below, -but briefly we address how Arch relates to the rest of the network in this section. - -- **Downstream(Ingress)** listeners take requests from upstream clients like a web UI or clients that forward - prompts to you local application responses from the application flow back through Arch to the downstream. - -- **Upstream(Egress)** listeners take requests from the application and forward them to LLMs. - -.. image:: /_static/img/network-topology-ingress-egress.jpg - :width: 100% - :align: center - -In practice, Arch can be deployed on the edge and as an internal load balancer between AI agents. A request path may -traverse multiple Arch gateways: - -.. image:: /_static/img/network-topology-agent.jpg - :width: 100% - :align: center - - -High level architecture ------------------------ -Arch is a set of **two** self-contained processes that are designed to run alongside your application servers -(or on a separate server connected to your application servers via a network). The first process is designated -to manage HTTP-level networking and connection management concerns (protocol management, request id generation, -header sanitization, etc.), and the other process is for **model serving**, which helps Arch make intelligent -decisions about the incoming prompts. The model server hosts the purpose-built LLMs to -manage several critical, but undifferentiated, prompt related tasks on behalf of developers. - - -The request processing path in Arch has three main parts: - -* :ref:`Listener subsystem ` which handles **downstream** and **upstream** request - processing. It is responsible for managing the downstream (ingress) and the upstream (egress) request - lifecycle. The downstream and upstream HTTP/2 codec lives here. -* :ref:`Prompt handler subsystem ` which is responsible for selecting and - forwarding prompts ``prompt_targets`` and establishes the lifecycle of any **upstream** connection to a - hosted endpoint that implements domain-specific business logic for incoming prompts. This is where knowledge - of targets and endpoint health, load balancing and connection pooling exists. -* :ref:`Model serving subsystem ` which helps Arch make intelligent decisions about the - incoming prompts. The model server is designed to call the purpose-built LLMs in Arch. - -The three subsystems are bridged with either the HTTP router filter, and the cluster manager subsystems of Envoy. - -Also, Arch utilizes `Envoy event-based thread model `_. -A main thread is responsible for the server lifecycle, configuration processing, stats, etc. and some number of -:ref:`worker threads ` process requests. All threads operate around an event loop (`libevent `_) -and any given downstream TCP connection will be handled by exactly one worker thread for its lifetime. Each worker -thread maintains its own pool of TCP connections to upstream endpoints. - -Worker threads rarely share state and operate in a trivially parallel fashion. This threading model -enables scaling to very high core count CPUs. - -Configuration -------------- - -Today, only support a static bootstrap configuration file for simplicity today: - -.. literalinclude:: ../includes/arch_config.yaml - :language: yaml - - -Request Flow (Ingress) ----------------------- - -A brief outline of the lifecycle of a request and response using the example configuration above: - -1. **TCP Connection Establishment**: - A TCP connection from downstream is accepted by an Arch listener running on a worker thread. - The listener filter chain provides SNI and other pre-TLS information. The transport socket, typically TLS, - decrypts incoming data for processing. - -2. **Prompt Guardrails Check**: - Arch first checks the incoming prompts for guardrails such as jailbreak attempts. This ensures - that harmful or unwanted behaviors are detected early in the request processing pipeline. - -3. **Intent Matching**: - The decrypted data stream is de-framed by the HTTP/2 codec in Arch's HTTP connection manager. Arch performs - intent matching via is **prompt-handler** subsystem using the name and description of the defined prompt targets, - determining which endpoint should handle the prompt. - -4. **Parameter Gathering with Arch-Function**: - If a prompt target requires specific parameters, Arch engages Arch-FC to extract the necessary details - from the incoming prompt(s). This process gathers the critical information needed for downstream API calls. - -5. **API Call Execution**: - Arch routes the prompt to the appropriate backend API or function call. If an endpoint cluster is identified, - load balancing is performed, circuit breakers are checked, and the request is proxied to the upstream endpoint. - -6. **Default Summarization by Upstream LLM**: - By default, if no specific endpoint processing is needed, the prompt is sent to an upstream LLM for summarization. - This ensures that responses are concise and relevant, enhancing user experience in RAG (Retrieval Augmented Generation) - and agentic applications. - -7. **Error Handling and Forwarding**: - Errors encountered during processing, such as failed function calls or guardrail detections, are forwarded to - designated error targets. Error details are communicated through specific headers to the application: - - - ``X-Function-Error-Code``: Code indicating the type of function call error. - - ``X-Prompt-Guard-Error-Code``: Code specifying violations detected by prompt guardrails. - - Additional headers carry messages and timestamps to aid in debugging and logging. - -8. **Response Handling**: - The upstream endpoint’s TLS transport socket encrypts the response, which is then proxied back downstream. - Responses pass through HTTP filters in reverse order, ensuring any necessary processing or modification before final delivery. - - -Request Flow (Egress) ---------------------- - -A brief outline of the lifecycle of a request and response in the context of egress traffic from an application to Large Language Models (LLMs) via Arch: - -1. **HTTP Connection Establishment to LLM**: - Arch initiates an HTTP connection to the upstream LLM service. This connection is handled by Arch’s egress listener - running on a worker thread. The connection typically uses a secure transport protocol such as HTTPS, ensuring the - prompt data is encrypted before being sent to the LLM service. - -2. **Rate Limiting**: - Before sending the request to the LLM, Arch applies rate-limiting policies to ensure that the upstream LLM service - is not overwhelmed by excessive traffic. Rate limits are enforced per client or service, ensuring fair usage and - preventing accidental or malicious overload. If the rate limit is exceeded, Arch may return an appropriate HTTP - error (e.g., 429 Too Many Requests) without sending the prompt to the LLM. - -3. **Load Balancing to (hosted) LLM Endpoints**: - After passing the rate-limiting checks, Arch routes the prompt to the appropriate LLM endpoint. - If multiple LLM providers instances are available, load balancing is performed to distribute traffic evenly - across the instances. Arch checks the health of the LLM endpoints using circuit breakers and health checks, - ensuring that the prompt is only routed to a healthy, responsive instance. - -4. **Response Reception and Forwarding**: - Once the LLM processes the prompt, Arch receives the response from the LLM service. The response is typically a - generated text, completion, or summarization. Upon reception, Arch decrypts (if necessary) and handles the response, - passing it through any egress processing pipeline defined by the application, such as logging or additional response filtering. - - -Post-request processing -^^^^^^^^^^^^^^^^^^^^^^^^ -Once a request completes, the stream is destroyed. The following also takes places: - -* The post-request :ref:`monitoring ` are updated (e.g. timing, active requests, upgrades, health checks). - Some statistics are updated earlier however, during request processing. Stats are batched and written by the main - thread periodically. -* :ref:`Access logs ` are written to the access log -* :ref:`Trace ` spans are finalized. If our example request was traced, a - trace span, describing the duration and details of the request would be created by the HCM when - processing request headers and then finalized by the HCM during post-request processing. diff --git a/docs/source/concepts/tech_overview/terminology.rst b/docs/source/concepts/tech_overview/terminology.rst deleted file mode 100644 index 4257a20f..00000000 --- a/docs/source/concepts/tech_overview/terminology.rst +++ /dev/null @@ -1,50 +0,0 @@ -.. _arch_terminology: - -Terminology -============ - -A few definitions before we dive into the main architecture documentation. Also note, Arch borrows from Envoy's terminology -to keep things consistent in logs and traces, and introduces and clarifies concepts are is relates to LLM applications. - -**Agent**: An application that uses LLMs to handle wide-ranging tasks from users via prompts. This could be as simple -as retrieving or summarizing data from an API, or being able to trigger complex actions like adjusting ad campaigns, or -changing travel plans via prompts. - -**Arch Config**: Arch operates based on a configuration that controls the behavior of a single instance of the Arch gateway. -This where you enable capabilities like LLM routing, fast function calling (via prompt_targets), applying guardrails, and enabling critical -features like metrics and tracing. For the full configuration reference of `arch_config.yaml` see :ref:`here `. - -**Downstream(Ingress)**: An downstream client (web application, etc.) connects to Arch, sends prompts, and receives responses. - -**Upstream(Egress)**: An upstream host that receives connections and prompts from Arch, and returns context or responses for a prompt - -.. image:: /_static/img/network-topology-ingress-egress.jpg - :width: 100% - :align: center - -**Listener**: A :ref:`listener ` is a named network location (e.g., port, address, path etc.) that Arch -listens on to process prompts before forwarding them to your application server endpoints. rch enables you to configure one listener -for downstream connections (like port 80, 443) and creates a separate internal listener for calls that initiate from your application -code to LLMs. - -.. Note:: - - When you start Arch, you specify a listener address/port that you want to bind downstream. But, Arch uses are predefined port - that you can use (``127.0.0.1:12000``) to proxy egress calls originating from your application to LLMs (API-based or hosted). - For more details, check out :ref:`LLM providers `. - -**Prompt Target**: Arch offers a primitive called :ref:`prompt target ` to help separate business logic from -undifferentiated work in building generative AI apps. Prompt targets are endpoints that receive prompts that are processed by Arch. -For example, Arch enriches incoming prompts with metadata like knowing when a request is a follow-up or clarifying prompt so that you -can build faster, more accurate retrieval (RAG) apps. To support agentic apps, like scheduling travel plans or sharing comments on a -document - via prompts, Arch uses its function calling abilities to extract critical information from the incoming prompt (or a set of -prompts) needed by a downstream backend API or function call before calling it directly. - -**Model Serving**: Arch is a set of `two` self-contained processes that are designed to run alongside your application servers -(or on a separate host connected via a network).The :ref:`model serving ` process helps Arch make intelligent decisions -about the incoming prompts. The model server is designed to call the (fast) purpose-built LLMs in Arch. - -**Error Target**: :ref:`Error targets ` are those endpoints that receive forwarded errors from Arch when issues arise, -such as failing to properly call a function/API, detecting violations of guardrails, or encountering other processing errors. -These errors are communicated to the application via headers ``X-Arch-[ERROR-TYPE]``, allowing it to handle the errors gracefully -and take appropriate actions. diff --git a/docs/source/conf.py b/docs/source/conf.py index 11048b61..7d5f22ba 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -12,10 +12,10 @@ from sphinx.util.docfields import Field from sphinxawesome_theme import ThemeOptions from sphinxawesome_theme.postprocess import Icons -project = "Arch Docs" +project = "Plano Docs" copyright = "2025, Katanemo Labs, Inc" author = "Katanemo Labs, Inc" -release = " v0.3.22" +release = " v0.4" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration @@ -72,7 +72,7 @@ theme_options = ThemeOptions( awesome_external_links=True, extra_header_link_icons={ "repository on GitHub": { - "link": "https://github.com/katanemo/arch", + "link": "https://github.com/katanemo/plano", "icon": ( ' None: ) ], ) - - app.add_css_file("_static/custom.css") diff --git a/docs/source/get_started/intro_to_arch.rst b/docs/source/get_started/intro_to_arch.rst deleted file mode 100644 index 27211f67..00000000 --- a/docs/source/get_started/intro_to_arch.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. _intro_to_arch: - -Intro to Arch -============= -AI demos are easy to build. But past the thrill of a quick hack, you are left building, maintaining and scaling low-level plumbing code for agents that slows down AI innovation. -For example: - -- You want to build specialized agents, but get stuck writing **routing and handoff** code. -- You bogged down with prompt engineering work to **clarify user intent and validate inputs**. -- You want to **quickly and safely use new LLMs** but get stuck writing integration code. -- You waste cycles writing and maintaining **observability** code, when it can be transparent. -- You want to **apply guardrails**, but have to write custom code for each prompt and LLM. - -Arch is designed to solve these problems by providing a unified, out-of-process architecture that integrates with your existing application stack, enabling you to focus on building high-level features rather than plumbing — all without locking you into a framework. - -.. figure:: /_static/img/arch_network_diagram_high_level.png - :width: 100% - :align: center - - High-level network flow of where Arch Gateway sits in your agentic stack. Designed for both ingress and egress prompt traffic. - - -`Arch `_ is a smart edge and AI gateway for AI-native apps - built by the contributors of Envoy Proxy with the belief that: - - *Prompts are nuanced and opaque user requests, which require the same capabilities as traditional HTTP requests - including secure handling, intelligent routing, robust observability, and integration with backend (API) - systems for personalization - all outside business logic.* - -In practice, achieving the above goal is incredibly difficult. Arch attempts to do so by providing the following high level features: - -**Out-of-process architecture, built on** `Envoy `_: -Arch takes a dependency on Envoy and is a self-contained process that is designed to run alongside your application servers. -Arch uses Envoy's HTTP connection management subsystem, HTTP L7 filtering and telemetry capabilities to extend the functionality exclusively for prompts and LLMs. -This gives Arch several advantages: - -* Arch builds on Envoy's proven success. Envoy is used at massive scale by the leading technology companies of our time including `AirBnB `_, `Dropbox `_, `Google `_, `Reddit `_, `Stripe `_, etc. Its battle tested and scales linearly with usage and enables developers to focus on what really matters: application features and business logic. - -* Arch works with any application language. A single Arch deployment can act as gateway for AI applications written in Python, Java, C++, Go, Php, etc. - -* Arch can be deployed and upgraded quickly across your infrastructure transparently without the horrid pain of deploying library upgrades in your applications. - -**Engineered with Fast Task-Specific LLMs (TLMs):** Arch is engineered with specialized LLMs that are designed for the fast, cost-effective and accurate handling of prompts. -These LLMs are designed to be best-in-class for critical tasks like: - -* **Function Calling:** Arch helps you easily personalize your applications by enabling calls to application-specific (API) operations via user prompts. - This involves any predefined functions or APIs you want to expose to users to perform tasks, gather information, or manipulate data. - With function calling, you have flexibility to support "agentic" experiences tailored to specific use cases - from updating insurance claims to creating ad campaigns - via prompts. - Arch analyzes prompts, extracts critical information from prompts, engages in lightweight conversation to gather any missing parameters and makes API calls so that you can focus on writing business logic. - For more details, read :ref:`Function Calling `. - -* **Prompt Guard:** Arch helps you improve the safety of your application by applying prompt guardrails in a centralized way for better governance hygiene. - With prompt guardrails you can prevent ``jailbreak attempts`` present in user's prompts without having to write a single line of code. - To learn more about how to configure guardrails available in Arch, read :ref:`Prompt Guard `. - -**Traffic Management:** Arch offers several capabilities for LLM calls originating from your applications, including smart retries on errors from upstream LLMs, and automatic cut-over to other LLMs configured in Arch for continuous availability and disaster recovery scenarios. -Arch extends Envoy's `cluster subsystem `_ to manage upstream connections to LLMs so that you can build resilient AI applications. - -**Front/edge Gateway:** There is substantial benefit in using the same software at the edge (observability, traffic shaping algorithms, applying guardrails, etc.) as for outbound LLM inference use cases. -Arch has the feature set that makes it exceptionally well suited as an edge gateway for AI applications. -This includes TLS termination, applying guardrail early in the process, intelligent parameter gathering from prompts, and prompt-based routing to backend APIs. - -**Best-In Class Monitoring:** Arch offers several monitoring metrics that help you understand three critical aspects of -your application: latency, token usage, and error rates by an upstream LLM provider. Latency measures the speed at which -your application is responding to users, which includes metrics like time to first token (TFT), time per output token (TOT) -metrics, and the total latency as perceived by users. - -**End-to-End Tracing:** Arch propagates trace context using the W3C Trace Context standard, specifically through the ``traceparent`` header. -This allows each component in the system to record its part of the request flow, enabling end-to-end tracing across the entire application. -By using OpenTelemetry, Arch ensures that developers can capture this trace data consistently and in a format compatible with various observability tools. -For more details, read :ref:`Tracing `. diff --git a/docs/source/get_started/intro_to_plano.rst b/docs/source/get_started/intro_to_plano.rst new file mode 100644 index 00000000..43b11f46 --- /dev/null +++ b/docs/source/get_started/intro_to_plano.rst @@ -0,0 +1,56 @@ +.. _intro_to_plano: + +Intro to Plano +============== +Building agentic demos is easy. Delivering agentic applications safely, reliably, and repeatably to production is hard. After a quick hack, you end up building the "hidden AI middleware" to reach production: routing logic to reach the right agent, guardrail hooks for safety and moderation, evaluation and observability glue for continuous learning, and model/provider quirks — scattered across frameworks and application code. + +Plano solves this by moving core delivery concerns into a unified, out-of-process dataplane. Core capabilities: + +- **🚦 Orchestration:** Low-latency orchestration between agents, and add new agents without changing app code. When routing lives inside app code, it becomes hard to evolve and easy to duplicate. Moving orchestration into a centrally managed dataplane lets you change strategies without touching your agents, improving performance and reducing maintenance burden while avoiding tight coupling. + +- **🛡️ Guardrails & Memory Hooks:** Apply jailbreak protection, content policies, and context workflows (e.g., rewriting, retrieval, redaction) once via :ref:`Filter Chains ` at the dataplane. Instead of re-implementing these in every agentic service, you get centralized governance, reduced code duplication, and consistent behavior across your stack. + +- **🔗 Model Agility:** Route by model, alias (semantic names), or automatically via preferences so agents stay decoupled from specific providers. Swap or add models without refactoring prompts, tool-calling, or streaming handlers throughout your codebase by using Plano's smart routing and unified API. + +- **🕵 Agentic Signals™:** Zero-code capture of behavior signals, traces, and metrics consistently across every agent. Rather than stitching together logging and metrics per framework, Plano surfaces traces, token usage, and learning signals in one place so you can iterate safely. + +Built by core contributors to the widely adopted Envoy Proxy _, Plano gives you a production‑grade foundation for agentic applications. It helps **developers** stay focused on the core logic of their agents, helps **product teams** shorten feedback loops for learning, and helps **engineering teams** standardize policy and safety across agents and LLMs. Plano is grounded in open protocols (de facto: OpenAI‑style v1/responses, de jure: MCP) and proven patterns like sidecar deployments, so it plugs in cleanly while remaining robust, scalable, and flexible. + +In practice, achieving the above goal is incredibly difficult. Plano attempts to do so by providing the following high level features: + +.. figure:: /_static/img/plano_network_diagram_high_level.png + :width: 100% + :align: center + + High-level network flow of where Plano sits in your agentic stack. Designed for both ingress and egress prompt traffic. + +**Engineered with Task-Specific LLMs (TLMs):** Plano is engineered with specialized LLMs that are designed for fast, cost-effective and accurate handling of prompts. +These LLMs are designed to be best-in-class for critical tasks like: + +* **Agent Orchestration:** `Plano-Orchestrator `_ is a family of state-of-the-art routing and orchestration models that decide which agent(s) or LLM(s) should handle each request, and in what sequence. Built for real-world multi-agent deployments, it analyzes user intent and conversation context to make precise routing and orchestration decisions while remaining efficient enough for low-latency production use across general chat, coding, and long-context multi-turn conversations. + +* **Function Calling:** Plano lets you expose application-specific (API) operations as tools so that your agents can update records, fetch data, or trigger determininistic workflows via prompts. Under the hood this is backed by Arch-Function-Chat; for more details, read :ref:`Function Calling `. + +* **Guardrails:** Plano helps you improve the safety of your application by applying prompt guardrails in a centralized way for better governance hygiene. + With prompt guardrails you can prevent ``jailbreak attempts`` present in user's prompts without having to write a single line of code. + To learn more about how to configure guardrails available in Plano, read :ref:`Prompt Guard `. + +**Model Proxy:** Plano offers several capabilities for LLM calls originating from your applications, including smart retries on errors from upstream LLMs and automatic cut-over to other LLMs configured in Plano for continuous availability and disaster recovery scenarios. From your application's perspective you keep using an OpenAI-compatible API, while Plano owns resiliency and failover policies in one place. +Plano extends Envoy's `cluster subsystem `_ to manage upstream connections to LLMs so that you can build resilient, provider-agnostic AI applications. + +**Edge Proxy:** There is substantial benefit in using the same software at the edge (observability, traffic shaping algorithms, applying guardrails, etc.) as for outbound LLM inference use cases. Plano has the feature set that makes it exceptionally well suited as an edge gateway for AI applications. +This includes TLS termination, applying guardrails early in the request flow, and intelligently deciding which agent(s) or LLM(s) should handle each request and in what sequence. In practice, you configure listeners and policies once, and every inbound and outbound call flows through the same hardened gateway. + +**Zero-Code Agent Signals™ & Tracing:** Zero-code capture of behavior signals, traces, and metrics consistently across every agent. Plano propagates trace context using the W3C Trace Context standard, specifically through the ``traceparent`` header. This allows each component in the system to record its part of the request flow, enabling end-to-end tracing across the entire application. By using OpenTelemetry, Plano ensures that developers can capture this trace data consistently and in a format compatible with various observability tools. + +**Best-In Class Monitoring:** Plano offers several monitoring metrics that help you understand three critical aspects of your application: latency, token usage, and error rates by an upstream LLM provider. Latency measures the speed at which your application is responding to users, which includes metrics like time to first token (TFT), time per output token (TOT) metrics, and the total latency as perceived by users. + +**Out-of-process architecture, built on** `Envoy `_: +Plano takes a dependency on Envoy and is a self-contained process that is designed to run alongside your application servers. Plano uses Envoy's HTTP connection management subsystem, HTTP L7 filtering and telemetry capabilities to extend the functionality exclusively for prompts and LLMs. +This gives Plano several advantages: + +* Plano builds on Envoy's proven success. Envoy is used at massive scale by the leading technology companies of our time including `AirBnB `_, `Dropbox `_, `Google `_, `Reddit `_, `Stripe `_, etc. Its battle tested and scales linearly with usage and enables developers to focus on what really matters: application features and business logic. + +* Plano works with any application language. A single Plano deployment can act as gateway for AI applications written in Python, Java, C++, Go, Php, etc. + +* Plano can be deployed and upgraded quickly across your infrastructure transparently without the horrid pain of deploying library upgrades in your applications. diff --git a/docs/source/get_started/overview.rst b/docs/source/get_started/overview.rst index 8a9a43a7..56c24f2b 100644 --- a/docs/source/get_started/overview.rst +++ b/docs/source/get_started/overview.rst @@ -1,38 +1,38 @@ .. _overview: - Overview -============ -`Arch `_ is a smart edge and AI gateway for AI agents - one that is natively designed to handle and process prompts, not just network traffic. +======== +`Plano `_ is delivery infrastructure for agentic apps —a models-native proxy and dataplane that helps you ship agents to production faster and operate them reliably. -Built by contributors to the widely adopted `Envoy Proxy `_, Arch handles the *pesky low-level work* in building agentic apps — like applying guardrails, clarifying vague user input, routing prompts to the right agent, and unifying access to any LLM. It’s a protocol-friendly and framework-agnostic infrastructure layer designed to help you build and ship agentic apps faster. +Plano pulls out the rote plumbing work (the “hidden AI middleware”) and decouples you from brittle, ever‑changing framework abstractions. It centralizes what shouldn’t be bespoke in every codebase like agent routing and orchestration, rich agentic signals and traces for continuous improvement, guardrail filters for safety and moderation, and smart LLM routing APIs for UX and DX agility. Use any language or AI framework, and ship agents to production faster with Plano. -In this documentation, you will learn how to quickly set up Arch to trigger API calls via prompts, apply prompt guardrails without writing any application-level logic, -simplify the interaction with upstream LLMs, and improve observability all while simplifying your application development process. +Built by core contributors to the widely adopted Envoy Proxy _, Plano gives you a production‑grade foundation for agentic applications. It helps **developers** stay focused on the core logic of their agents, helps **product teams** shorten feedback loops for learning, and helps **engineering teams** standardize policy and safety across agents and LLMs. Plano is grounded in open protocols (de facto: OpenAI‑style v1/responses, de jure: MCP) and proven patterns like sidecar deployments, so it plugs in cleanly while remaining robust, scalable, and flexible. -.. figure:: /_static/img/arch_network_diagram_high_level.png +In this documentation, you’ll learn how to set up Plano quickly, trigger API calls via prompts, apply guardrails without tight coupling with application code, simplify model and provider integration, and improve observability — so that you can focus on what matters most: the core product logic of your agents. + +.. figure:: /_static/img/plano_network_diagram_high_level.png :width: 100% :align: center - High-level network flow of where Arch Gateway sits in your agentic stack. Designed for both ingress and egress prompt traffic. + High-level network flow of where Plano sits in your agentic stack. Designed for both ingress and egress traffic. Get Started ----------- -This section introduces you to Arch and helps you get set up quickly: +This section introduces you to Plano and helps you get set up quickly: .. grid:: 3 .. grid-item-card:: :octicon:`apps` Overview :link: overview.html - Overview of Arch and Doc navigation + Overview of Plano and Doc navigation - .. grid-item-card:: :octicon:`book` Intro to Arch - :link: intro_to_arch.html + .. grid-item-card:: :octicon:`book` Intro to Plano + :link: intro_to_plano.html - Explore Arch's features and developer workflow + Explore Plano's features and developer workflow .. grid-item-card:: :octicon:`rocket` Quickstart :link: quickstart.html @@ -43,7 +43,7 @@ This section introduces you to Arch and helps you get set up quickly: Concepts -------- -Deep dive into essential ideas and mechanisms behind Arch: +Deep dive into essential ideas and mechanisms behind Plano: .. grid:: 3 @@ -60,12 +60,12 @@ Deep dive into essential ideas and mechanisms behind Arch: .. grid-item-card:: :octicon:`workflow` Prompt Target :link: ../concepts/prompt_target.html - Understand how Arch handles prompts + Understand how Plano handles prompts Guides ------ -Step-by-step tutorials for practical Arch use cases and scenarios: +Step-by-step tutorials for practical Plano use cases and scenarios: .. grid:: 3 @@ -82,22 +82,22 @@ Step-by-step tutorials for practical Arch use cases and scenarios: .. grid-item-card:: :octicon:`issue-opened` Observability :link: ../guides/observability/observability.html - Learn to monitor and troubleshoot Arch + Learn to monitor and troubleshoot Plano -Build with Arch ---------------- +Build with Plano +---------------- -For developers extending and customizing Arch for specialized needs: +For developers extending and customizing Plano for specialized needs: .. grid:: 2 .. grid-item-card:: :octicon:`dependabot` Agentic Workflow - :link: ../build_with_arch/agent.html + :link: ../build_with_plano/agent.html - Discover how to create and manage custom agents within Arch + Discover how to create and manage custom agents within Plano .. grid-item-card:: :octicon:`stack` RAG Application - :link: ../build_with_arch/rag.html + :link: ../build_with_plano/rag.html Integrate RAG for knowledge-driven responses diff --git a/docs/source/get_started/quickstart.rst b/docs/source/get_started/quickstart.rst index 546285c2..4c7d0a7e 100644 --- a/docs/source/get_started/quickstart.rst +++ b/docs/source/get_started/quickstart.rst @@ -1,9 +1,9 @@ .. _quickstart: Quickstart -================ +========== -Follow this guide to learn how to quickly set up Arch and integrate it into your generative AI applications. +Follow this guide to learn how to quickly set up Plano and integrate it into your generative AI applications. Prerequisites @@ -15,32 +15,31 @@ Before you begin, ensure you have the following: 2. `Docker Compose `_ (v2.29) 3. `Python `_ (v3.10+) -Arch's CLI allows you to manage and interact with the Arch gateway efficiently. To install the CLI, simply run the following command: +Plano's CLI allows you to manage and interact with the Plano efficiently. To install the CLI, simply run the following command: .. tip:: - We recommend that developers create a new Python virtual environment to isolate dependencies before installing Arch. This ensures that ``archgw`` and its dependencies do not interfere with other packages on your system. - + We recommend that developers create a new Python virtual environment to isolate dependencies before installing Plano. This ensures that ``plano`` and its dependencies do not interfere with other packages on your system. .. code-block:: console $ python -m venv venv $ source venv/bin/activate # On Windows, use: venv\Scripts\activate - $ pip install archgw==0.3.22 + $ pip install plano==0.4.0 -Build AI Agent with Arch Gateway --------------------------------- +Build Agentic Apps with Plano +----------------------------- -In the following quickstart, we will show you how easy it is to build an AI agent with the Arch gateway. We will build a currency exchange agent using the following simple steps. For this demo, we will use `https://api.frankfurter.dev/` to fetch the latest prices for currencies and assume USD as the base currency. +In the following quickstart, we will show you how easy it is to build an AI agent with the Plano. We will build a currency exchange agent using the following simple steps. For this demo, we will use `https://api.frankfurter.dev/` to fetch the latest prices for currencies and assume USD as the base currency. -Step 1. Create arch config file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 1. Create plano config file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Create ``arch_config.yaml`` file with the following content: +Create ``plano_config.yaml`` file with the following content: .. code-block:: yaml - version: v0.1.0 + version: v0.1.0 listeners: ingress_traffic: @@ -49,19 +48,13 @@ Create ``arch_config.yaml`` file with the following content: message_format: openai timeout: 30s - llm_providers: + model_providers: - access_key: $OPENAI_API_KEY model: openai/gpt-4o system_prompt: | You are a helpful assistant. - prompt_guards: - input_guards: - jailbreak: - on_exception: - message: Looks like you're curious about my abilities, but I can only provide assistance for currency exchange. - prompt_targets: - name: currency_exchange description: Get currency exchange rate from USD to other currencies @@ -88,16 +81,16 @@ Create ``arch_config.yaml`` file with the following content: endpoint: api.frankfurter.dev:443 protocol: https -Step 2. Start arch gateway with currency conversion config -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 2. Start plano with currency conversion config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: sh - $ archgw up arch_config.yaml - 2024-12-05 16:56:27,979 - cli.main - INFO - Starting archgw cli version: 0.1.5 + $ plano up plano_config.yaml + 2024-12-05 16:56:27,979 - cli.main - INFO - Starting plano cli version: 0.1.5 ... 2024-12-05 16:56:28,485 - cli.utils - INFO - Schema validation successful! - 2024-12-05 16:56:28,485 - cli.main - INFO - Starting arch model server and arch gateway + 2024-12-05 16:56:28,485 - cli.main - INFO - Starting plano model server and plano gateway ... 2024-12-05 16:56:51,647 - cli.core - INFO - Container is healthy! @@ -129,15 +122,15 @@ And to get the list of supported currencies: "Here is a list of the currencies that are supported for conversion from USD, along with their symbols:\n\n1. AUD - Australian Dollar\n2. BGN - Bulgarian Lev\n3. BRL - Brazilian Real\n4. CAD - Canadian Dollar\n5. CHF - Swiss Franc\n6. CNY - Chinese Renminbi Yuan\n7. CZK - Czech Koruna\n8. DKK - Danish Krone\n9. EUR - Euro\n10. GBP - British Pound\n11. HKD - Hong Kong Dollar\n12. HUF - Hungarian Forint\n13. IDR - Indonesian Rupiah\n14. ILS - Israeli New Sheqel\n15. INR - Indian Rupee\n16. ISK - Icelandic Króna\n17. JPY - Japanese Yen\n18. KRW - South Korean Won\n19. MXN - Mexican Peso\n20. MYR - Malaysian Ringgit\n21. NOK - Norwegian Krone\n22. NZD - New Zealand Dollar\n23. PHP - Philippine Peso\n24. PLN - Polish Złoty\n25. RON - Romanian Leu\n26. SEK - Swedish Krona\n27. SGD - Singapore Dollar\n28. THB - Thai Baht\n29. TRY - Turkish Lira\n30. USD - United States Dollar\n31. ZAR - South African Rand\n\nIf you want to convert USD to any of these currencies, you can select the one you are interested in." -Use Arch Gateway as LLM Router ------------------------------- +Use Plano as a LLM Router +------------------------- -Step 1. Create arch config file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 1. Create plano config file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Arch operates based on a configuration file where you can define LLM providers, prompt targets, guardrails, etc. Below is an example configuration that defines OpenAI and Mistral LLM providers. +Plano operates based on a configuration file where you can define LLM providers, prompt targets, guardrails, etc. Below is an example configuration that defines OpenAI and Mistral LLM providers. -Create ``arch_config.yaml`` file with the following content: +Create ``plano_config.yaml`` file with the following content: .. code-block:: yaml @@ -150,7 +143,7 @@ Create ``arch_config.yaml`` file with the following content: message_format: openai timeout: 30s - llm_providers: + model_providers: - access_key: $OPENAI_API_KEY model: openai/gpt-4o default: true @@ -158,19 +151,19 @@ Create ``arch_config.yaml`` file with the following content: - access_key: $MISTRAL_API_KEY model: mistralministral-3b-latest -Step 2. Start arch gateway -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 2. Start plano +~~~~~~~~~~~~~~~~~~~ Once the config file is created, ensure that you have environment variables set up for ``MISTRAL_API_KEY`` and ``OPENAI_API_KEY`` (or these are defined in a ``.env`` file). -Start the Arch gateway: +Start the Plano gateway: .. code-block:: console - $ archgw up arch_config.yaml - 2024-12-05 11:24:51,288 - cli.main - INFO - Starting archgw cli version: 0.1.5 + $ plano up plano_config.yaml + 2024-12-05 11:24:51,288 - cli.main - INFO - Starting plano cli version: 0.1.5 2024-12-05 11:24:51,825 - cli.utils - INFO - Schema validation successful! - 2024-12-05 11:24:51,825 - cli.main - INFO - Starting arch model server and arch gateway + 2024-12-05 11:24:51,825 - cli.main - INFO - Starting plano model server and plano gateway ... 2024-12-05 11:25:16,131 - cli.core - INFO - Container is healthy! @@ -180,7 +173,7 @@ Step 3: Interact with LLM Step 3.1: Using OpenAI Python client ++++++++++++++++++++++++++++++++++++ -Make outbound calls via the Arch gateway: +Make outbound calls via the Plano gateway: .. code-block:: python @@ -188,14 +181,14 @@ Make outbound calls via the Arch gateway: # Use the OpenAI client as usual client = OpenAI( - # No need to set a specific openai.api_key since it's configured in Arch's gateway + # No need to set a specific openai.api_key since it's configured in Plano's gateway api_key='--', - # Set the OpenAI API base URL to the Arch gateway endpoint + # Set the OpenAI API base URL to the Plano gateway endpoint base_url="http://127.0.0.1:12000/v1" ) response = client.chat.completions.create( - # we select model from arch_config file + # we select model from plano_config file model="--", messages=[{"role": "user", "content": "What is the capital of France?"}], ) @@ -225,12 +218,12 @@ Step 3.2: Using curl command ], } -You can override model selection using the ``x-arch-llm-provider-hint`` header. For example, to use Mistral, use the following curl command: +You can override model selection using the ``x-plano-llm-provider-hint`` header. For example, to use Mistral, use the following curl command: .. code-block:: bash $ curl --header 'Content-Type: application/json' \ - --header 'x-arch-llm-provider-hint: ministral-3b' \ + --header 'x-plano-llm-provider-hint: ministral-3b' \ --data '{"messages": [{"role": "user","content": "What is the capital of France?"}], "model": "none"}' \ http://localhost:12000/v1/chat/completions @@ -253,10 +246,10 @@ You can override model selection using the ``x-arch-llm-provider-hint`` header. Next Steps ========== -Congratulations! You've successfully set up Arch and made your first prompt-based request. To further enhance your GenAI applications, explore the following resources: +Congratulations! You've successfully set up Plano and made your first prompt-based request. To further enhance your GenAI applications, explore the following resources: - :ref:`Full Documentation `: Comprehensive guides and references. -- `GitHub Repository `_: Access the source code, contribute, and track updates. -- `Support `_: Get help and connect with the Arch community . +- `GitHub Repository `_: Access the source code, contribute, and track updates. +- `Support `_: Get help and connect with the Plano community . -With Arch, building scalable, fast, and personalized GenAI applications has never been easier. Dive deeper into Arch's capabilities and start creating innovative AI-driven experiences today! +With Plano, building scalable, fast, and personalized GenAI applications has never been easier. Dive deeper into Plano's capabilities and start creating innovative AI-driven experiences today! diff --git a/docs/source/guides/agent_routing.rst b/docs/source/guides/agent_routing.rst deleted file mode 100644 index 5effad69..00000000 --- a/docs/source/guides/agent_routing.rst +++ /dev/null @@ -1,105 +0,0 @@ -.. _agent_routing: - -Agent Routing and Hand Off -=========================== - -Agent Routing and Hand Off is a key feature in Arch that enables intelligent routing of user prompts to specialized AI agents or human agents based on the nature and complexity of the user's request. - -This capability significantly enhances the efficiency and personalization of interactions, ensuring each prompt receives the most appropriate and effective handling. The following section describes -the workflow, configuration, and implementation of Agent routing and hand off in Arch. - -#. **Agent Selection** - When a user submits a prompt, Arch analyzes the input to determine the intent and complexity. Based on the analysis, Arch selects the most suitable agent configured within your application to handle the specific category of the user's request—such as sales inquiries, technical issues, or complex scenarios requiring human attention. - -#. **Prompt Routing** - After selecting the appropriate agent, Arch routes the user's prompt to the designated agent's endpoint and waits for the agent to respond back with the processed output or further instructions. - -#. **Hand Off** - Based on follow-up queries from the user, Arch repeats the process of analysis, agent selection, and routing to ensure a seamless hand off between AI agents as needed. - -.. code-block:: yaml - :caption: Agent Routing and Hand Off Configuration Example - - prompt_targets: - - name: sales_agent - description: Handles queries related to sales and purchases - - - name: issues_and_repairs - description: handles issues, repairs, or refunds - - - name: escalate_to_human - description: escalates to human agent - -.. code-block:: python - :caption: Agent Routing and Hand Off Implementation Example via FastAPI - - class Agent: - def __init__(self, role: str, instructions: str): - self.system_prompt = f"You are a {role}.\n{instructions}" - - def handle(self, req: ChatCompletionsRequest): - messages = [{"role": "system", "content": self.get_system_prompt()}] + [ - message.model_dump() for message in req.messages - ] - return call_openai(messages, req.stream) #call_openai is a placeholder for the actual API call - - def get_system_prompt(self) -> str: - return self.system_prompt - - # Define your agents - AGENTS = { - "sales_agent": Agent( - role="sales agent", - instructions=( - "Always answer in a sentence or less.\n" - "Follow the following routine with the user:\n" - "1. Engage\n" - "2. Quote ridiculous price\n" - "3. Reveal caveat if user agrees." - ), - ), - "issues_and_repairs": Agent( - role="issues and repairs agent", - instructions="Propose a solution, offer refund if necessary.", - ), - "escalate_to_human": Agent( - role="human escalation agent", instructions="Escalate issues to a human." - ), - "unknown_agent": Agent( - role="general assistant", instructions="Assist the user in general queries." - ), - } - - #handle the request from arch gateway - @app.post("/v1/chat/completions") - def completion_api(req: ChatCompletionsRequest, request: Request): - - agent_name = req.metadata.get("agent-name", "unknown_agent") - agent = AGENTS.get(agent_name) - logger.info(f"Routing to agent: {agent_name}") - - return agent.handle(req) - -.. note:: - The above example demonstrates a simple implementation of Agent Routing and Hand Off using FastAPI. For the full implementation of this example - please see our `GitHub demo `_. - -Example Use Cases ------------------ -Agent Routing and Hand Off is particularly beneficial in scenarios such as: - -- **Customer Support**: Routing common customer queries to automated support agents, while escalating complex or sensitive issues to human support staff. -- **Sales and Marketing**: Automatically directing potential leads and sales inquiries to specialized sales agents for timely and targeted follow-ups. -- **Technical Assistance**: Managing user-reported issues, repairs, or refunds by assigning them to the correct technical or support agent efficiently. - -Best Practices and Tips ------------------------- -When implementing Agent Routing and Hand Off in your applications, consider these best practices: - -- Clearly define agent responsibilities: Ensure each agent or human endpoint has a clear, specific description of the prompts they handle, reducing mis-routing. -- Monitor and optimize routes: Regularly review how prompts are routed to adjust and optimize agent definitions and configurations. - -.. note:: - To observe traffic to and from agents, please read more about :ref:`observability ` in Arch. - -By carefully configuring and managing your Agent routing and hand off, you can significantly improve your application's responsiveness, performance, and overall user satisfaction. diff --git a/docs/source/guides/function_calling.rst b/docs/source/guides/function_calling.rst index 54113dd0..a6c5ce9f 100644 --- a/docs/source/guides/function_calling.rst +++ b/docs/source/guides/function_calling.rst @@ -126,7 +126,7 @@ Step 3: Arch Takes Over Once you have defined the functions and configured the prompt targets, Arch Gateway takes care of the remaining work. It will automatically validate parameters, and ensure that the required parameters (e.g., location) are present in the prompt, and add validation rules if necessary. -.. figure:: /_static/img/arch_network_diagram_high_level.png +.. figure:: /_static/img/plano_network_diagram_high_level.png :width: 100% :align: center diff --git a/docs/source/guides/llm_router.rst b/docs/source/guides/llm_router.rst index 963df0f0..2eb5a6c1 100644 --- a/docs/source/guides/llm_router.rst +++ b/docs/source/guides/llm_router.rst @@ -5,7 +5,7 @@ LLM Routing With the rapid proliferation of large language models (LLM) — each optimized for different strengths, style, or latency/cost profile — routing has become an essential technique to operationalize the use of different models. -Arch provides three distinct routing approaches to meet different use cases: +Plano provides three distinct routing approaches to meet different use cases: 1. **Model-based Routing**: Direct routing to specific models using provider/model names 2. **Alias-based Routing**: Semantic routing using custom aliases that map to underlying models @@ -42,7 +42,7 @@ Preference-aligned Routing (Arch-Router) Traditional LLM routing approaches face significant limitations: they evaluate performance using benchmarks that often fail to capture human preferences, select from fixed model pools, and operate as "black boxes" without practical mechanisms for encoding user preferences. -Arch's preference-aligned routing addresses these challenges by applying a fundamental engineering principle: decoupling. The framework separates route selection (matching queries to human-readable policies) from model assignment (mapping policies to specific LLMs). This separation allows you to define routing policies using descriptive labels like ``Domain: 'finance', Action: 'analyze_earnings_report'`` rather than cryptic identifiers, while independently configuring which models handle each policy. +Plano's preference-aligned routing addresses these challenges by applying a fundamental engineering principle: decoupling. The framework separates route selection (matching queries to human-readable policies) from model assignment (mapping policies to specific LLMs). This separation allows you to define routing policies using descriptive labels like ``Domain: 'finance', Action: 'analyze_earnings_report'`` rather than cryptic identifiers, while independently configuring which models handle each policy. The `Arch-Router `_ model automatically selects the most appropriate LLM based on: @@ -65,7 +65,7 @@ For direct model routing, the process is straightforward: #. **Provider Validation** - Arch validates that the specified provider and model are configured and available. + Plano validates that the specified provider and model are configured and available. #. **Direct Routing** @@ -87,11 +87,11 @@ For alias-based routing, the process includes name resolution: #. **Alias Resolution** - Arch resolves the alias to the actual provider/model name based on configuration. + Plano resolves the alias to the actual provider/model name based on configuration. #. **Model Selection** - If the alias maps to multiple models, Arch selects one based on availability and load balancing. + If the alias maps to multiple models, Plano selects one based on availability and load balancing. #. **Request Forwarding** @@ -159,6 +159,8 @@ In summary, Arch-Router demonstrates: - **Production-Ready Performance**: Optimized for low-latency, high-throughput applications in multi-model environments. +.. _implementing_routing: + Implementing Routing -------------------- @@ -390,7 +392,7 @@ Best practicesm .. The model is not trained to process raw image or audio inputs. While it can handle textual queries *about* these modalities (e.g., "generate an image of a cat"), it cannot interpret encoded multimedia data directly. .. - **❌ Function Calling:** -.. This model is designed for **semantic preference matching**, not exact intent classification or tool execution. For structured function invocation, use models in the **Arch-Function-Calling** collection. +.. This model is designed for **semantic preference matching**, not exact intent classification or tool execution. For structured function invocation, use models in the **Plano-Function-Calling** collection. .. - **❌ System Prompt Dependency:** .. Arch-Router routes based solely on the user’s conversation history. It does not use or rely on system prompts for routing decisions. diff --git a/docs/source/guides/observability/access_logging.rst b/docs/source/guides/observability/access_logging.rst index ca102df5..4e66f711 100644 --- a/docs/source/guides/observability/access_logging.rst +++ b/docs/source/guides/observability/access_logging.rst @@ -3,14 +3,14 @@ Access Logging ============== -Access logging in Arch refers to the logging of detailed information about each request and response that flows through Arch. -It provides visibility into the traffic passing through Arch, which is crucial for monitoring, debugging, and analyzing the +Access logging in Plano refers to the logging of detailed information about each request and response that flows through Plano. +It provides visibility into the traffic passing through Plano, which is crucial for monitoring, debugging, and analyzing the behavior of AI applications and their interactions. Key Features ^^^^^^^^^^^^ * **Per-Request Logging**: - Each request that passes through Arch is logged. This includes important metadata such as HTTP method, + Each request that passes through Plano is logged. This includes important metadata such as HTTP method, path, response status code, request duration, upstream host, and more. * **Integration with Monitoring Tools**: Access logs can be exported to centralized logging systems (e.g., ELK stack or Fluentd) or used to feed monitoring and alerting systems. @@ -19,7 +19,7 @@ Key Features How It Works ^^^^^^^^^^^^ -Arch gateway exposes access logs for every call it manages on your behalf. By default these access logs can be found under ``~/archgw_logs``. For example: +Plano exposes access logs for every call it manages on your behalf. By default these access logs can be found under ``~/archgw_logs``. For example: .. code-block:: console diff --git a/docs/source/guides/observability/monitoring.rst b/docs/source/guides/observability/monitoring.rst index 26718d07..db206bec 100644 --- a/docs/source/guides/observability/monitoring.rst +++ b/docs/source/guides/observability/monitoring.rst @@ -8,11 +8,11 @@ and instrumentation for generating, collecting, processing, and exporting teleme metrics, and logs. Its flexible design supports a wide range of backends and seamlessly integrates with modern application tools. -Arch acts a *source* for several monitoring metrics related to **prompts** and **LLMs** natively integrated +Plano acts a *source* for several monitoring metrics related to **agents** and **LLMs** natively integrated via `OpenTelemetry `_ to help you understand three critical aspects of your application: latency, token usage, and error rates by an upstream LLM provider. Latency measures the speed at which your application is responding to users, which includes metrics like time to first token (TFT), time per output token (TOT) metrics, and -the total latency as perceived by users. Below are some screenshots how Arch integrates natively with tools like +the total latency as perceived by users. Below are some screenshots how Plano integrates natively with tools like `Grafana `_ via `Promethus `_ @@ -32,7 +32,7 @@ Metrics Dashboard (via Grafana) Configure Monitoring ~~~~~~~~~~~~~~~~~~~~ -Arch gateway publishes stats endpoint at http://localhost:19901/stats. As noted above, Arch is a source for metrics. To view and manipulate dashbaords, you will +Plano publishes stats endpoint at http://localhost:19901/stats. As noted above, Plano is a source for metrics. To view and manipulate dashbaords, you will need to configiure `Promethus `_ (as a metrics store) and `Grafana `_ for dashboards. Below are some sample configuration files for both, respectively. diff --git a/docs/source/guides/observability/tracing.rst b/docs/source/guides/observability/tracing.rst index 8dda14fd..cdfc2de2 100644 --- a/docs/source/guides/observability/tracing.rst +++ b/docs/source/guides/observability/tracing.rst @@ -41,7 +41,7 @@ Benefits of Using ``Traceparent`` Headers How to Initiate A Trace ----------------------- -1. **Enable Tracing Configuration**: Simply add the ``random_sampling`` in ``tracing`` section to 100`` flag to in the :ref:`listener ` config +1. **Enable Tracing Configuration**: Simply add the ``random_sampling`` in ``tracing`` section to 100`` flag to in the :ref:`listener ` config 2. **Trace Context Propagation**: Arch automatically propagates the ``traceparent`` header. When a request is received, Arch will: diff --git a/docs/source/guides/orchestration.rst b/docs/source/guides/orchestration.rst new file mode 100644 index 00000000..42b47759 --- /dev/null +++ b/docs/source/guides/orchestration.rst @@ -0,0 +1,224 @@ +.. _agent_routing: + +Orchestration +============== + +Building multi-agent systems allow you to route requests across multiple specialized agents, each designed to handle specific types of tasks. +Plano makes it easy to build and scale these systems by managing the orchestration layer—deciding which agent(s) should handle each request—while you focus on implementing individual agent logic. + +This guide shows you how to configure and implement multi-agent orchestration in Plano. + +How It Works +------------ + +Plano's orchestration layer analyzes incoming prompts and routes them to the most appropriate agent based on user intent and conversation context. The workflow is: + +1. **User submits a prompt**: The request arrives at Plano's agent listener. +2. **Agent selection**: Plano analyzes the prompt to determine user intent and complexity, and routes the request to the most suitable agent configured in your system—such as a sales agent, technical support agent, or RAG agent. +3. **Agent handles request**: The selected agent processes the prompt using its specialized logic and tools. +4. **Seamless handoffs**: For multi-turn conversations, Plano repeats the intent analysis for each follow-up query, enabling smooth handoffs between agents as the conversation evolves. + +Configuration +------------- + +Configure your agents in the ``listeners`` section of your ``plano_config.yaml``: + +.. code-block:: yaml + :caption: Multi-Agent Configuration Example + + listeners: + - type: agent + name: agent_listener + port: 8001 + agents: + - id: sales_agent + description: Handles sales inquiries, product recommendations, and pricing questions + endpoint: http://sales-service:8000/agent + + - id: support_agent + description: Handles technical issues, troubleshooting, and repairs + endpoint: http://support-service:8000/agent + + - id: rag_agent + description: Answers questions using retrieval augmented generation + endpoint: http://rag-service:8000/agent + filter_chain: + - query_rewriter + - context_builder + +**Key Configuration Elements:** + +* **agent listener**: A listener of ``type: agent`` tells Plano to perform intent analysis and routing for incoming requests. +* **agents list**: Define each agent with an ``id``, ``description`` (used for routing decisions), and ``endpoint`` (where Plano forwards requests). +* **filter_chain**: Optionally attach :ref:`filter chains ` to agents for guardrails, query rewriting, or context enrichment. + +Implementation +-------------- + +Agents are HTTP services that receive routed requests from Plano. Here's how to implement a simple multi-agent system using Python and FastAPI: + +.. code-block:: python + :caption: Multi-Agent Implementation Example + + class Agent: + def __init__(self, role: str, instructions: str): + self.system_prompt = f"You are a {role}.\n{instructions}" + + def handle(self, req: ChatCompletionsRequest): + messages = [{"role": "system", "content": self.get_system_prompt()}] + [ + message.model_dump() for message in req.messages + ] + return call_openai(messages, req.stream) #call_openai is a placeholder for the actual API call + + def get_system_prompt(self) -> str: + return self.system_prompt + + # Define your agents + AGENTS = { + "sales_agent": Agent( + role="sales agent", + instructions=( + "Always answer in a sentence or less.\n" + "Follow the following routine with the user:\n" + "1. Engage\n" + "2. Quote ridiculous price\n" + "3. Reveal caveat if user agrees." + ), + ), + "issues_and_repairs": Agent( + role="issues and repairs agent", + instructions="Propose a solution, offer refund if necessary.", + ), + "escalate_to_human": Agent( + role="human escalation agent", instructions="Escalate issues to a human." + ), + "unknown_agent": Agent( + role="general assistant", instructions="Assist the user in general queries." + ), + } + + #handle the request from arch gateway + @app.post("/v1/chat/completions") + def completion_api(req: ChatCompletionsRequest, request: Request): + + agent_name = req.metadata.get("agent-name", "unknown_agent") + agent = AGENTS.get(agent_name) + logger.info(f"Routing to agent: {agent_name}") + + return agent.handle(req) + +**How Requests Flow:** + +1. User sends a prompt to Plano's agent listener (e.g., "I need help with a billing issue"). +2. Plano-Orchestrator analyzes the intent and routes to the ``support_agent``. +3. Plano forwards the request to ``http://support-service:8000/agent`` with metadata indicating which agent was selected. +4. Your agent service receives the request, processes it using its specialized logic, and returns a response. +5. Plano forwards the agent's response back to the user. + +.. note:: + For a complete working example with multiple agents, see our `multi-agent orchestration demo `_ on GitHub. + +Common Use Cases +---------------- + +Multi-agent orchestration is particularly powerful for: + +**Customer Support** + +Route common queries to automated support agents while escalating complex or sensitive issues to human support staff. + +.. code-block:: yaml + + agents: + - id: tier1_support + description: Handles common FAQs, password resets, and basic troubleshooting + - id: tier2_support + description: Handles complex technical issues requiring deep product knowledge + - id: human_escalation + description: Escalates sensitive issues or unresolved problems to human agents + +**Sales and Marketing** + +Direct potential leads and sales inquiries to specialized sales agents for timely, targeted follow-ups. + +.. code-block:: yaml + + agents: + - id: product_recommendation + description: Recommends products based on user needs and preferences + - id: pricing_agent + description: Provides pricing information and quotes + - id: sales_closer + description: Handles final negotiations and closes deals + +**Technical Documentation and Support** + +Combine RAG agents for documentation lookup with specialized troubleshooting agents. + +.. code-block:: yaml + + agents: + - id: docs_agent + description: Retrieves relevant documentation and guides + filter_chain: + - query_rewriter + - context_builder + - id: troubleshoot_agent + description: Diagnoses and resolves technical issues step by step + +Best Practices +-------------- + +**Write Clear Agent Descriptions** + +Agent descriptions are used by Plano-Orchestrator to make routing decisions. Be specific about what each agent handles: + +.. code-block:: yaml + + # Good - specific and actionable + - id: refund_agent + description: Processes refund requests for orders within 30 days, validates return eligibility + + # Less ideal - too vague + - id: refund_agent + description: Handles refunds + +**Use Filter Chains for Cross-Cutting Concerns** + +Apply :ref:`filter chains ` to agents that need guardrails, context enrichment, or query rewriting: + +.. code-block:: yaml + + agents: + - id: rag_agent + description: Answers questions using company knowledge base + filter_chain: + - compliance_check # Ensure queries comply with policies + - query_rewriter # Optimize query for retrieval + - context_builder # Fetch relevant docs + +**Monitor and Optimize Routing** + +Regularly review which agents handle which requests to identify mis-routing patterns and adjust agent descriptions: + +.. code-block:: yaml + + # Monitor routing decisions through Plano's tracing + # Adjust descriptions if you see unexpected routing behavior + +**Route LLM Calls Through Plano's Model Proxy** + +When your agents need to call LLMs, route those calls through Plano's :ref:`Model Proxy ` for consistent responses, smart routing, and rich observability. See :ref:`Making LLM Calls from Agents ` for details. + +Next Steps +---------- + +* Learn more about :ref:`agents ` and the inner vs. outer loop model +* Explore :ref:`filter chains ` for adding guardrails and context enrichment +* See :ref:`observability ` for monitoring multi-agent workflows +* Review the :ref:`LLM Providers ` guide for model routing within agents + +.. note:: + To observe traffic to and from agents, please read more about :ref:`observability ` in Plano. + +By carefully configuring and managing your Agent routing and hand off, you can significantly improve your application's responsiveness, performance, and overall user satisfaction. diff --git a/docs/source/guides/prompt_guard.rst b/docs/source/guides/prompt_guard.rst index df2f1c0d..9db88ce5 100644 --- a/docs/source/guides/prompt_guard.rst +++ b/docs/source/guides/prompt_guard.rst @@ -1,66 +1,89 @@ .. _prompt_guard: -Prompt Guard -============= +Guardrails +========== -**Prompt guard** is a security and validation feature offered in Arch to protect agents, by filtering and analyzing prompts before they reach your application logic. -In applications where prompts generate responses or execute specific actions based on user inputs, prompt guard minimizes risks like malicious inputs (or misaligned outputs). -By adding a layer of input scrutiny, prompt guards ensures safer, more reliable, and accurate interactions with agents. +**Guardrails** are Plano's way of applying safety and validation checks to prompts before they reach your application logic. They are typically implemented as +filters in a :ref:`Filter Chain ` attached to an agent, so every request passes through a consistent processing layer. + + +Why Guardrails +-------------- +Guardrails are essential for maintaining control over AI-driven applications. They help enforce organizational policies, ensure compliance with regulations +(like GDPR or HIPAA), and protect users from harmful or inappropriate content. In applications where prompts generate responses or trigger actions, guardrails +minimize risks like malicious inputs, off-topic queries, or misaligned outputs—adding a consistent layer of input scrutiny that makes interactions safer, +more reliable, and easier to reason about. -Why Prompt Guard ----------------- .. vale Vale.Spelling = NO -- **Prompt Sanitization via Arch-Guard** - - **Jailbreak Prevention**: Detects and filters inputs that might attempt jailbreak attacks, like alternating LLM intended behavior, exposing the system prompt, or bypassing ethnics safety. +- **Jailbreak Prevention**: Detect and filter inputs that attempt to change LLM behavior, expose system prompts, or bypass safety policies. +- **Domain and Topicality Enforcement**: Ensure that agents only respond to prompts within an approved domain (for example, finance-only or healthcare-only use cases) and reject unrelated queries. +- **Dynamic Error Handling**: Provide clear error messages when requests violate policy, helping users correct their inputs. -- **Dynamic Error Handling** - - **Automatic Correction**: Applies error-handling techniques to suggest corrections for minor input errors, such as typos or misformatted data. - - **Feedback Mechanism**: Provides informative error messages to users, helping them understand how to correct input mistakes or adhere to guidelines. +How Guardrails Work +------------------- -.. Note:: - Today, Arch offers support for jailbreak via Arch-Guard. We will be adding support for additional guards in Q1, 2025 (including response guardrails) +In Plano, guardrails are usually implemented as filters that run as HTTP services. Each filter receives the incoming prompt and related metadata, evaluates it +against policy, and either lets the request continue (HTTP 200) or terminates it early with an appropriate error code (typically HTTP 4xx for policy failures). -What Is Arch-Guard -~~~~~~~~~~~~~~~~~~ -`Arch-Guard `_ is a robust classifier model specifically trained on a diverse corpus of prompt attacks. -It excels at detecting explicitly malicious prompts, providing an essential layer of security for LLM applications. +The example below shows a simple, plain-Python HTTP service that acts as a topicality guardrail: it rejects any prompt that is not related to the +"weather" domain. -By embedding Arch-Guard within the Arch architecture, we empower developers to build robust, LLM-powered applications while prioritizing security and safety. With Arch-Guard, you can navigate the complexities of prompt management with confidence, knowing you have a reliable defense against malicious input. +.. code-block:: python + :caption: Example topicality guard filter in plain Python (FastAPI) + + from fastapi import FastAPI, Request, HTTPException + + app = FastAPI() + + ALLOWED_KEYWORDS = {"weather", "forecast", "temperature", "rain", "snow", "humidity"} + + @app.post("/guardrails/topic") + async def topic_guard(request: Request): + body = await request.json() + # Expecting an OpenAI-style request body with messages + messages = body.get("messages", []) + user_content = " ".join( + m["content"] for m in messages if m.get("role") == "user" + ).lower() + + if not any(keyword in user_content for keyword in ALLOWED_KEYWORDS): + # Return 400 to indicate a policy failure (not a server error) + raise HTTPException( + status_code=400, + detail={ + "error": "off_topic", + "message": "This assistant only answers weather-related questions.", + }, + ) + + # If the prompt is on-topic, just pass the original body through + return body -Example Configuration -~~~~~~~~~~~~~~~~~~~~~ -Here is an example of using Arch-Guard in Arch: +To wire this guardrail into Plano, you define a listener of ``type: agent`` and attach a filter chain with a single filter that points +to the Python service above. -.. literalinclude:: includes/arch_config.yaml - :language: yaml - :linenos: - :lines: 22-26 - :caption: Arch-Guard Example Configuration +.. code-block:: yaml + :caption: Listener (type: agent) with a topicality guard filter -How Arch-Guard Works ----------------------- + filters: + - id: topicality_guard + url: http://topic-guard:8000/guardrails/topic -#. **Pre-Processing Stage** - - As a request or prompt is received, Arch Guard first performs validation. If any violations are detected, the input is flagged, and a tailored error message may be returned. - -#. **Error Handling and Feedback** - - If the prompt contains errors or does not meet certain criteria, the user receives immediate feedback or correction suggestions, enhancing usability and reducing the chance of repeated input mistakes. - -Benefits of Using Arch Guard ------------------------------- - -- **Enhanced Security**: Protects against injection attacks, harmful content, and misuse, securing both system and user data. - -- **Better User Experience**: Clear feedback and error correction improve user interactions by guiding them to correct input formats and constraints. + listeners: + - type: agent + name: agent_listener + port: 8001 + router: arch_agent_router + agents: + - id: rag_agent + description: virtual assistant for retrieval augmented generation tasks + filter_chain: + - topicality_guard -Summary -------- - -Prompt guard is an essential tool for any prompt-based system that values security, accuracy, and compliance. -By implementing Prompt Guard, developers can provide a robust layer of input validation and security, leading to better-performing, reliable, and safer applications. +When a request arrives at ``agent_listener``, Plano will first call the ``topicality_guard`` filter. If the filter returns **HTTP 200**, +the request continues on to the configured agent or prompt target. If the filter returns **HTTP 400**, Plano returns that error back to +the caller and does not forward the request further—enforcing your domain guardrail without changing any application code. diff --git a/docs/source/guides/state.rst b/docs/source/guides/state.rst new file mode 100644 index 00000000..f62169ba --- /dev/null +++ b/docs/source/guides/state.rst @@ -0,0 +1,255 @@ +.. _managing_conversational_state: + +Conversational State +===================== + +The OpenAI Responses API (``v1/responses``) is designed for multi-turn conversations where context needs to persist across requests. Plano provides a unified ``v1/responses`` API that works with **any LLM provider**—OpenAI, Anthropic, Azure OpenAI, DeepSeek, or any OpenAI-compatible provider—while automatically managing conversational state for you. + +Unlike the traditional Chat Completions API where you manually manage conversation history by including all previous messages in each request, Plano handles state management behind the scenes. This means you can use the Responses API with any model provider, and Plano will persist conversation context across requests—making it ideal for building conversational agents that remember context without bloating every request with full message history. + +How It Works +------------ + +When a client calls the Responses API: + +1. **First request**: Plano generates a unique ``resp_id`` and stores the conversation state (messages, model, provider, timestamp). +2. **Subsequent requests**: The client includes the ``previous_resp_id`` from the previous response. Plano retrieves the stored conversation state, merges it with the new input, and sends the combined context to the LLM. +3. **Response**: The LLM sees the full conversation history without the client needing to resend all previous messages. + +This pattern dramatically reduces bandwidth and makes it easier to build multi-turn agents—Plano handles the state plumbing so you can focus on agent logic. + +**Example Using OpenAI Python SDK:** + +.. code-block:: python + + from openai import OpenAI + + # Point to Plano's Model Proxy endpoint + client = OpenAI( + api_key="test-key", + base_url="http://127.0.0.1:12000/v1" + ) + + # First turn - Plano creates a new conversation state + response = client.responses.create( + model="gpt-4o-mini", # Works with any configured provider + input="My name is Alice and I like Python" + ) + + # Save the response_id for conversation continuity + resp_id = response.id + print(f"Assistant: {response.output_text}") + + # Second turn - Plano automatically retrieves previous context + resp2 = client.responses.create( + model="claude-sonnet-4-20250514", # Different model/provider, make sure its configured in plano_config.yaml + input="Please list all the messages you have received in our conversation, numbering each one.", + previous_response_id=resp_id, + ) + + print(f"Assistant: {resp2.output_text}") + # Output: "Your name is Alice and your favorite language is Python" + +Notice how the second request only includes the new user message—Plano automatically merges it with the stored conversation history before sending to the LLM. + +Configuration Overview +---------------------- + +State storage is configured in the ``state_storage`` section of your ``plano_config.yaml``: + +.. literalinclude:: ../resources/includes/arch_config_state_storage_example.yaml + :language: yaml + :lines: 21-30 + :linenos: + :emphasize-lines: 3,6-10 + +Plano supports two storage backends: + +* **Memory**: Fast, ephemeral storage for development and testing. State is lost when Plano restarts. +* **PostgreSQL**: Durable, production-ready storage with support for Supabase and self-hosted PostgreSQL instances. + +.. note:: + If you don't configure ``state_storage``, conversation state management is **disabled**. The Responses API will still work, but clients must manually include full conversation history in each request (similar to the Chat Completions API behavior). + +Memory Storage (Development) +---------------------------- + +Memory storage keeps conversation state in-memory using a thread-safe ``HashMap``. It's perfect for local development, demos, and testing, but all state is lost when Plano restarts. + +**Configuration** + +Add this to your ``plano_config.yaml``: + +.. code-block:: yaml + + state_storage: + type: memory + +That's it. No additional setup required. + +**When to Use Memory Storage** + +* Local development and debugging +* Demos and proof-of-concepts +* Automated testing environments +* Single-instance deployments where persistence isn't critical + +**Limitations** + +* State is lost on restart +* Not suitable for production workloads +* Cannot scale across multiple Plano instances + +PostgreSQL Storage (Production) +-------------------------------- + +PostgreSQL storage provides durable, production-grade conversation state management. It works with both self-hosted PostgreSQL and Supabase (PostgreSQL-as-a-service), making it ideal for scaling multi-agent systems in production. + +Prerequisites +^^^^^^^^^^^^^ + +Before configuring PostgreSQL storage, you need: + +1. A PostgreSQL database (version 12 or later) +2. Database credentials (host, user, password) +3. The ``conversation_states`` table created in your database + +**Setting Up the Database** + +Run the SQL schema to create the required table: + +.. literalinclude:: ../resources/db_setup/conversation_states.sql + :language: sql + :linenos: + +**Using psql:** + +.. code-block:: bash + + psql $DATABASE_URL -f docs/db_setup/conversation_states.sql + +**Using Supabase Dashboard:** + +1. Log in to your Supabase project +2. Navigate to the SQL Editor +3. Copy and paste the SQL from ``docs/db_setup/conversation_states.sql`` +4. Run the query + +Configuration +^^^^^^^^^^^^^ + +Once the database table is created, configure Plano to use PostgreSQL storage: + +.. code-block:: yaml + + state_storage: + type: postgres + connection_string: "postgresql://user:password@host:5432/database" + +**Using Environment Variables** + +You should **never** hardcode credentials. Use environment variables instead: + +.. code-block:: yaml + + state_storage: + type: postgres + connection_string: "postgresql://myuser:$DB_PASSWORD@db.example.com:5432/postgres" + +Then set the environment variable before running Plano: + +.. code-block:: bash + + export DB_PASSWORD="your-secure-password" + # Run Plano or config validation + ./plano + +.. warning:: + **Special Characters in Passwords**: If your password contains special characters like ``#``, ``@``, or ``&``, you must URL-encode them in the connection string. For example, ``MyPass#123`` becomes ``MyPass%23123``. + +Supabase Connection Strings +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Supabase requires different connection strings depending on your network setup. Most users should use the **Session Pooler** connection string. + +**IPv4 Networks (Most Common)** + +Use the Session Pooler connection string (port 5432): + +.. code-block:: text + + postgresql://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler.supabase.com:5432/postgres + +**IPv6 Networks** + +Use the direct connection (port 5432): + +.. code-block:: text + + postgresql://postgres:[PASSWORD]@db.[PROJECT-REF].supabase.co:5432/postgres + +**Finding Your Connection String** + +1. Go to your Supabase project dashboard +2. Navigate to **Settings → Database → Connection Pooling** +3. Copy the **Session mode** connection string +4. Replace ``[YOUR-PASSWORD]`` with your actual database password +5. URL-encode special characters in the password + +**Example Configuration** + +.. code-block:: yaml + + state_storage: + type: postgres + connection_string: "postgresql://postgres.myproject:$DB_PASSWORD@aws-0-us-west-2.pooler.supabase.com:5432/postgres" + +Then set the environment variable: + +.. code-block:: bash + + # If your password is "MyPass#123", encode it as "MyPass%23123" + export DB_PASSWORD="MyPass%23123" + +Troubleshooting +--------------- + +**"Table 'conversation_states' does not exist"** + +Run the SQL schema from ``docs/db_setup/conversation_states.sql`` against your database. + +**Connection errors with Supabase** + +* Verify you're using the correct connection string format (Session Pooler for IPv4) +* Check that your password is URL-encoded if it contains special characters +* Ensure your Supabase project hasn't paused due to inactivity (free tier) + +**Permission errors** + +Ensure your database user has the following permissions: + +.. code-block:: sql + + GRANT SELECT, INSERT, UPDATE, DELETE ON conversation_states TO your_user; + +**State not persisting across requests** + +* Verify ``state_storage`` is configured in your ``plano_config.yaml`` +* Check Plano logs for state storage initialization messages +* Ensure the client is sending the ``prev_response_id={$response_id}`` from previous responses + +Best Practices +-------------- + +1. **Use environment variables for credentials**: Never hardcode database passwords in configuration files. +2. **Start with memory storage for development**: Switch to PostgreSQL when moving to production. +3. **Implement cleanup policies**: Prevent unbounded growth by regularly archiving or deleting old conversations. +4. **Monitor storage usage**: Track conversation state table size and query performance in production. +5. **Test failover scenarios**: Ensure your application handles storage backend failures gracefully. + +Next Steps +---------- + +* Learn more about building :ref:`agents ` that leverage conversational state +* Explore :ref:`filter chains ` for enriching conversation context +* See the :ref:`LLM Providers ` guide for configuring model routing diff --git a/docs/source/index.rst b/docs/source/index.rst index 26bea5dd..32306f63 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,17 +1,15 @@ -Welcome to Arch! -================ +Welcome to Plano! +================= -.. image:: /_static/img/arch-logo.png +.. image:: /_static/img/PlanoTagline.svg :width: 100% :align: center -.. raw:: html +`Plano `_ is delivery infrastructure for agentic apps. A models-native proxy and data plane designed to help you build agents faster, and deliver them reliably to production. - Arch - Build fast, hyper-personalized agents with intelligent infra | Product Hunt +Plano pulls out the rote plumbing work (aka “hidden AI middleware”) and decouples you from brittle, ever‑changing framework abstractions. It centralizes what shouldn’t be bespoke in every codebase like agent routing and orchestration, rich agentic signals and traces for continuous improvement, guardrail filters for safety and moderation, and smart LLM routing APIs for UX and DX agility. Use any language or AI framework, and ship agents to production faster with Plano. -`Arch `_ is a models-native edge and LLM proxy/gateway for AI agents - one that is natively designed to handle and process prompts, not just network traffic. - -Built by contributors to the widely adopted `Envoy Proxy `_, Arch handles the *pesky low-level work* in building agentic apps — like applying guardrails, clarifying vague user input, routing prompts to the right agent, and unifying access to any LLM. It’s a language and framework friendly infrastructure layer designed to help you build and ship agentic apps faster. +Built by contributors to the widely adopted `Envoy Proxy `_, Plano **helps developers** focus more on the core product logic of agents, **product teams** accelerate feedback loops for reinforcement learning, and **engineering teams** standardize policies and access controls across every agent and LLM for safer, more reliable scaling. .. tab-set:: @@ -23,7 +21,7 @@ Built by contributors to the widely adopted `Envoy Proxy ` which handles **downstream** and **upstream** request + processing. It is responsible for managing the inbound(edge) and outbound(egress) request lifecycle. The downstream and upstream HTTP/2 codec lives here. This also includes the lifecycle of any **upstream** connection to an LLM provider or tool backend. The listenser subsystmem manages connection pools, load balancing, retries, and failover. + +* :ref:`Bright Staff controller subsystem ` is Plano's memory-efficient, lightweight controller for agentic traffic. It sits inside the Plano data plane and makes real-time decisions about how prompts are handled, forwarded, and processed. + +These two subsystems are bridged with either the HTTP router filter, and the cluster manager subsystems of Envoy. + +Also, Plano utilizes `Envoy event-based thread model `_. A main thread is responsible for the server lifecycle, configuration processing, stats, etc. and some number of :ref:`worker threads ` process requests. All threads operate around an event loop (`libevent `_) and any given downstream TCP connection will be handled by exactly one worker thread for its lifetime. Each worker thread maintains its own pool of TCP connections to upstream endpoints. + +Worker threads rarely share state and operate in a trivially parallel fashion. This threading model +enables scaling to very high core count CPUs. + +Request Flow (Ingress) +---------------------- + +A brief outline of the lifecycle of a request and response using the example configuration above: + +1. **TCP Connection Establishment**: + A TCP connection from downstream is accepted by an Plano listener running on a worker thread. + The listener filter chain provides SNI and other pre-TLS information. The transport socket, typically TLS, + decrypts incoming data for processing. + +3. **Routing Decision (Agent vs Prompt Target)**: + The decrypted data stream is de-framed by the HTTP/2 codec in Plano's HTTP connection manager. Plano performs + intent matching (via the Bright Staff controller and prompt-handling logic) using the configured agents and + :ref:`prompt targets `, determining whether this request should be handled by an agent workflow + (with optional :ref:`Filter Chains `) or by a deterministic prompt target. + +4a. **Agent Path: Orchestration and Filter Chains** + + If the request is routed to an **agent**, Plano executes any attached :ref:`Filter Chains ` first. These filters can apply guardrails, rewrite prompts, or enrich context (for example, RAG retrieval) before the agent runs. Once filters complete, the Bright Staff controller orchestrates which downstream tools, APIs, or LLMs the agent should call and in what sequence. + + * Plano may call one or more backend APIs or tools on behalf of the agent. + * If an endpoint cluster is identified, load balancing is performed, circuit breakers are checked, and the request is proxied to the appropriate upstream endpoint. + * If no specific endpoint is required, the prompt is sent to an upstream LLM using Plano's model proxy for + completion or summarization. + + For more on agent workflows and orchestration, see :ref:`Prompt Targets and Agents ` and + :ref:`Agent Filter Chains `. + +4b. **Prompt Target Path: Deterministic Tool/API Calls** + + If the request is routed to a **prompt target**, Plano treats it as a deterministic, task-specific call. + Plano engages its function-calling and parameter-gathering capabilities to extract the necessary details + from the incoming prompt(s) and produce the structured inputs your backend expects. + + * **Parameter Gathering**: Plano extracts and validates parameters defined on the prompt target (for example, + currency symbols, dates, or entity identifiers) so your backend does not need to parse natural language. + * **API Call Execution**: Plano then routes the call to the configured backend endpoint. If an endpoint cluster is identified, load balancing and circuit-breaker checks are applied before proxying the request upstream. + + For more on how to design and configure prompt targets, see :ref:`Prompt Target `. + +5. **Error Handling and Forwarding**: + Errors encountered during processing, such as failed function calls or guardrail detections, are forwarded to + designated error targets. Error details are communicated through specific headers to the application: + + - ``X-Function-Error-Code``: Code indicating the type of function call error. + - ``X-Prompt-Guard-Error-Code``: Code specifying violations detected by prompt guardrails. + - Additional headers carry messages and timestamps to aid in debugging and logging. + +6. **Response Handling**: + The upstream endpoint’s TLS transport socket encrypts the response, which is then proxied back downstream. + Responses pass through HTTP filters in reverse order, ensuring any necessary processing or modification before final delivery. + + +Request Flow (Egress) +--------------------- + +A brief outline of the lifecycle of a request and response in the context of egress traffic from an application to Large Language Models (LLMs) via Plano: + +1. **HTTP Connection Establishment to LLM**: + Plano initiates an HTTP connection to the upstream LLM service. This connection is handled by Plano’s egress listener running on a worker thread. The connection typically uses a secure transport protocol such as HTTPS, ensuring the prompt data is encrypted before being sent to the LLM service. + +2. **Rate Limiting**: + Before sending the request to the LLM, Plano applies rate-limiting policies to ensure that the upstream LLM service is not overwhelmed by excessive traffic. Rate limits are enforced per client or service, ensuring fair usage and preventing accidental or malicious overload. If the rate limit is exceeded, Plano may return an appropriate HTTP error (e.g., 429 Too Many Requests) without sending the prompt to the LLM. + +3. **Seamless Request Transformation and Smart Routing**: + After rate limiting, Plano normalizes the outgoing request into a provider-agnostic shape and applies smart routing decisions using the configured :ref:`LLM Providers `. This includes translating client-specific conventions into a unified OpenAI-style contract, enriching or overriding parameters (for example, temperature or max tokens) based on policy, and choosing the best target model or provider using :ref:`model-based, alias-based, or preference-aligned routing `. + +4. **Load Balancing to (hosted) LLM Endpoints**: + After smart routing selects the target provider/model, Plano routes the prompt to the appropriate LLM endpoint. + If multiple LLM provider instances are available, load balancing is performed to distribute traffic evenly + across the instances. Plano checks the health of the LLM endpoints using circuit breakers and health checks, + ensuring that the prompt is only routed to a healthy, responsive instance. + +5. **Response Reception and Forwarding**: + Once the LLM processes the prompt, Plano receives the response from the LLM service. The response is typically a generated text, completion, or summarization. Upon reception, Plano decrypts (if necessary) and handles the response, passing it through any egress processing pipeline defined by the application, such as logging or additional response filtering. + + +Post-request processing +^^^^^^^^^^^^^^^^^^^^^^^^ +Once a request completes, the stream is destroyed. The following also takes places: + +* The post-request :ref:`monitoring ` are updated (e.g. timing, active requests, upgrades, health checks). + Some statistics are updated earlier however, during request processing. Stats are batched and written by the main + thread periodically. +* :ref:`Access logs ` are written to the access log +* :ref:`Trace ` spans are finalized. If our example request was traced, a + trace span, describing the duration and details of the request would be created by the HCM when + processing request headers and then finalized by the HCM during post-request processing. + + +Configuration +------------- + +Today, only support a static bootstrap configuration file for simplicity today: + +.. literalinclude:: ../../concepts/includes/plano_config.yaml + :language: yaml diff --git a/docs/source/concepts/tech_overview/tech_overview.rst b/docs/source/resources/tech_overview/tech_overview.rst similarity index 71% rename from docs/source/concepts/tech_overview/tech_overview.rst rename to docs/source/resources/tech_overview/tech_overview.rst index a6b52c70..23c8febb 100644 --- a/docs/source/concepts/tech_overview/tech_overview.rst +++ b/docs/source/resources/tech_overview/tech_overview.rst @@ -6,10 +6,6 @@ Tech Overview .. toctree:: :maxdepth: 2 - terminology - threading_model - listener - prompt - model_serving request_lifecycle - error_target + model_serving + threading_model diff --git a/docs/source/concepts/tech_overview/threading_model.rst b/docs/source/resources/tech_overview/threading_model.rst similarity index 72% rename from docs/source/concepts/tech_overview/threading_model.rst rename to docs/source/resources/tech_overview/threading_model.rst index 4eae0d10..4f02ba15 100644 --- a/docs/source/concepts/tech_overview/threading_model.rst +++ b/docs/source/resources/tech_overview/threading_model.rst @@ -3,17 +3,17 @@ Threading Model =============== -Arch builds on top of Envoy's single process with multiple threads architecture. +Plano builds on top of Envoy's single process with multiple threads architecture. A single *primary* thread controls various sporadic coordination tasks while some number of *worker* threads perform filtering, and forwarding. Once a connection is accepted, the connection spends the rest of its lifetime bound to a single worker thread. All the functionality around prompt handling from a downstream client is handled in a separate worker thread. -This allows the majority of Arch to be largely single threaded (embarrassingly parallel) with a small amount +This allows the majority of Plano to be largely single threaded (embarrassingly parallel) with a small amount of more complex code handling coordination between the worker threads. -Generally, Arch is written to be 100% non-blocking. +Generally, Plano is written to be 100% non-blocking. .. tip::