diff --git a/docs/source/concepts/prompt_target.rst b/docs/source/concepts/prompt_target.rst index d8f8df91..8e5994c7 100644 --- a/docs/source/concepts/prompt_target.rst +++ b/docs/source/concepts/prompt_target.rst @@ -3,14 +3,13 @@ Prompt Target ============== -**Prompt Targets** are a fundamental component of Arch, enabling developers to define how different types of user prompts are processed and routed within their generative AI applications. -This section provides an in-depth look at prompt targets, including their purpose, configuration, usage, and best practices to help you effectively leverage this feature in your projects. +**Prompt Targets** are a core concept in Arch, empowering developers to clearly define how user prompts are interpreted, processed, and routed within their generative AI applications. Prompts can seamlessly be routed either to specialized AI agents capable of handling sophisticated, context-driven tasks or to targeted tools provided by your application, offering users a fast, precise, and personalized experience. + +This section covers the essentials of prompt targets—what they are, how to configure them, their practical uses, and recommended best practices—to help you fully utilize this feature in your applications. What Are Prompt Targets? ------------------------ -Prompt targets are predefined endpoints within Arch that handle specific types of user prompts. -They act as the bridge between user inputs and your backend services or APIs, enabling Arch to route, process, and manage prompts efficiently. -By defining prompt targets, you can separate your application's business logic from the complexities of prompt processing, ensuring a cleaner and more maintainable codebase. +Prompt targets are endpoints within Arch that handle specific types of user prompts. They act as the bridge between user inputs and your backend agemts or tools (APIs), enabling Arch to route, process, and manage prompts efficiently. Defining prompt targets helps you decouple your application's core logic from processing and handling complexities, leading to clearer code organization, better scalability, and easier maintenance. .. table:: @@ -21,7 +20,7 @@ By defining prompt targets, you can separate your application's business logic f ==================== ============================================ Intent Recognition Identify the purpose of a user prompt. Parameter Extraction Extract necessary data from the prompt. - API Invocation Call relevant backend services or functions. + Invocation Call relevant backend agents or tools (APIs). Response Handling Process and return responses to the user. ==================== ============================================ @@ -30,16 +29,15 @@ Key Features Below are the key features of prompt targets that empower developers to build efficient, scalable, and personalized GenAI solutions: -- **Modular Design**: Define multiple prompt targets to handle diverse functionalities. -- **Parameter Management**: Specify required and optional parameters for each target. -- **Function Integration**: Seamlessly connect prompts to backend APIs or functions. +- **Design Scenarios**: Define prompt targets to effectively handle specific agentic scenarios. +- **Input Management**: Specify required and optional parameters for each target. +- **Tools Integration**: Seamlessly connect prompts to backend APIs or functions. - **Error Handling**: Direct errors to designated handlers for streamlined troubleshooting. - **Metadata Enrichment**: Attach additional context to prompts for enhanced processing. Configuring Prompt Targets -------------------------- -Configuring prompt targets involves defining them in Arch's configuration file. -Each Prompt target specifies how a particular type of prompt should be handled, including the endpoint to invoke and any parameters required. +Configuring prompt targets involves defining them in Arch's configuration file. Each Prompt target specifies how a particular type of prompt should be handled, including the endpoint to invoke and any parameters required. Basic Configuration ~~~~~~~~~~~~~~~~~~~ @@ -50,7 +48,7 @@ A prompt target configuration includes the following elements: - ``name``: A unique identifier for the prompt target. - ``description``: A brief explanation of what the prompt target does. -- ``endpoint``: The API endpoint or function that handles the prompt. +- ``endpoint``: Required if you want to call a tool or specific API. ``name`` and ``path`` ``http_method`` are the three attributes of the endpoint. - ``parameters`` (Optional): A list of parameters to extract from the prompt. .. _defining_prompt_target_parameters: @@ -69,6 +67,7 @@ Each parameter can be marked as required or optional. Here is a full list of par ``name (req.)`` Specifies name of the parameter. ``description (req.)`` Provides a human-readable explanation of the parameter's purpose. ``type (req.)`` Specifies the data type. Supported types include: **int**, **str**, **float**, **bool**, **list**, **set**, **dict**, **tuple** + ``in_path`` Indicates whether the parameter is part of the path in the endpoint url. Valid values: **true** or **false** ``default`` Specifies a default value for the parameter if not provided by the user. ``format`` Specifies a format for the parameter value. For example: `2019-12-31` for a date value. ``enum`` Lists of allowable values for the parameter with data type matching the ``type`` attribute. **Usage Example**: ``enum: ["celsius`", "fahrenheit"]`` @@ -76,10 +75,11 @@ Each parameter can be marked as required or optional. Here is a full list of par ``required`` Indicates whether the parameter is mandatory or optional. Valid values: **true** or **false** ======================== ============================================================================ -Example Configuration -~~~~~~~~~~~~~~~~~~~~~ +Example Configuration For Tools +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: yaml + :caption: Tools and Function Calling Configuration Example prompt_targets: - name: get_weather @@ -98,16 +98,35 @@ Example Configuration name: api_server path: /weather +Example Configuration For Agents +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + :caption: Agent Orchestration Configuration Example + + overrides: + use_agent_orchestrator: true + + prompt_targets: + - name: sales_agent + description: handles queries related to sales and purchases + + - name: issues_and_repairs + description: handles issues, repairs, or refunds + + - name: escalate_to_human + description: escalates to human agent + +.. note:: + Today, you can use Arch to coordinate more specific agentic scenarios via tools and function calling, or use it for high-level agent routing and hand off scenarios. In the future, we plan to offer you the ability to combine these two approaches for more complex scenarios. Please see `github issues `_ for more details. Routing Logic ------------- -Prompt targets determine where and how user prompts are processed. -Arch uses intelligent routing logic to ensure that prompts are directed to the appropriate targets based on their intent and context. +Prompt targets determine where and how user prompts are processed. Arch uses intelligent routing logic to ensure that prompts are directed to the appropriate targets based on their intent and context. Default Targets ~~~~~~~~~~~~~~~ -For general-purpose prompts that do not match any specific prompt target, Arch routes them to a designated default target. -This is useful for handling open-ended queries like document summarization or information extraction. +For general-purpose prompts that do not match any specific prompt target, Arch routes them to a designated default target. This is useful for handling open-ended queries like document summarization or information extraction. Intent Matching ~~~~~~~~~~~~~~~ @@ -124,5 +143,5 @@ For example: Summary -------- Prompt targets are essential for defining how user prompts are handled within your generative AI applications using Arch. -By carefully configuring prompt targets, you can ensure that prompts are accurately routed, necessary parameters are extracted, and backend services are invoked seamlessly. -This modular approach not only simplifies your application's architecture but also enhances scalability, maintainability, and overall user experience. + +By carefully configuring prompt targets, you can ensure that prompts are accurately routed, necessary parameters are extracted, and backend services are invoked seamlessly. This modular approach not only simplifies your application's architecture but also enhances scalability, maintainability, and overall user experience. diff --git a/docs/source/conf.py b/docs/source/conf.py index 399566f0..91a182c1 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -15,7 +15,7 @@ from sphinxawesome_theme.postprocess import Icons project = "Arch Docs" copyright = "2025, Katanemo Labs, Inc" author = "Katanemo Labs, Inc" -release = " v0.2.3" +release = " v0.2.4" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration diff --git a/docs/source/get_started/intro_to_arch.rst b/docs/source/get_started/intro_to_arch.rst index 97188ab1..14a84bab 100644 --- a/docs/source/get_started/intro_to_arch.rst +++ b/docs/source/get_started/intro_to_arch.rst @@ -3,10 +3,17 @@ Intro to Arch ============= -Arch is an intelligent `(Layer 7) `_ gateway designed for generative AI apps, agents, copilots that work with prompts. -Engineered with purpose-built large language models (LLMs), Arch handles all the critical but undifferentiated tasks related to the handling and processing of prompts, including -detecting and rejecting jailbreak attempts, intelligently calling “backend” APIs to fulfill the user's request represented in a prompt, routing to and offering disaster recovery -between upstream LLMs, and managing the observability of prompts and LLM interactions in a centralized way. +Arch is an intelligent proxy server designed agentic applications. **Move faster** by letting Arch handle the **pesky heavy lifting** in building agents: +fast input clarification, agent routing, seamless integration of prompts with tools for common tasks, and unified access and observability of LLMs. + +Past the thrill of an AI demo, have you found yourself hitting these walls? You know, the all too familiar ones: + +- You break a prompt into specialized ones, but **get stuck writing routing** and handoff logic? +- You want use new LLMs, but **struggle to quickly add LLMs** without writing integration logic? +- You're **trapped in tedious prompting work** to clarify inputs and user intents? +- You're **wasting cycles** choosing and integrating **code for observability** instead of it just happening transparently? + +And you think to youself, can't I move faster by focusing on higher-level objectives in a language and framework agnostic way? Well, you can! .. figure:: /_static/img/arch_network_diagram_high_level.png :width: 100% @@ -15,7 +22,7 @@ between upstream LLMs, and managing the observability of prompts and LLM interac High-level network flow of where Arch Gateway sits in your agentic stack. Designed for both ingress and egress prompt traffic. -**The project was born out of the belief that:** +**Arch Gateway was built by the contributors of Envoy Proxy with the belief that:** *Prompts are nuanced and opaque user requests, which require the same capabilities as traditional HTTP requests including secure handling, intelligent routing, robust observability, and integration with backend (API) diff --git a/docs/source/get_started/overview.rst b/docs/source/get_started/overview.rst index 07a347af..0f1930b1 100644 --- a/docs/source/get_started/overview.rst +++ b/docs/source/get_started/overview.rst @@ -3,7 +3,11 @@ Overview ============ -Welcome to Arch, the intelligent prompt gateway designed to help developers build **fast**, **secure**, and **personalized** generative AI apps at ANY scale. +Welcome to Arch, The intelligent (edge and LLM) proxy server for agentic applications. + +Move **faster** by letting Arch handle the pesky heavy lifting in building agents: **fast input clarification**, **agent routing**, +seamless integration of prompts with **tools for common tasks**, and **unified access and observability of LLMs**. + In this documentation, you will learn how to quickly set up Arch to trigger API calls via prompts, apply prompt guardrails without writing any application-level logic, simplify the interaction with upstream LLMs, and improve observability all while simplifying your application development process. diff --git a/docs/source/guides/agent_routing.rst b/docs/source/guides/agent_routing.rst new file mode 100644 index 00000000..418d674d --- /dev/null +++ b/docs/source/guides/agent_routing.rst @@ -0,0 +1,105 @@ +.. _agent_routing: + +Agent Routing and Hand Off +=========================== + +Agent Routing and Hand Off is a key feature in Arch that enables intelligent routing of user prompts to specialized AI agents or human agents based on the nature and complexity of the user's request. + +This capability significantly enhances the efficiency and personalization of interactions, ensuring each prompt receives the most appropriate and effective handling. The following section describes +the workflow, configuration, and implementation of Agent routing and hand off in Arch. + +#. **Agent Selection** + When a user submits a prompt, Arch analyzes the input to determine the intent and complexity. Based on the analysis, Arch selects the most suitable agent configured within your application to handle the specific category of the user's request—such as sales inquiries, technical issues, or complex scenarios requiring human attention. + +#. **Prompt Routing** + After selecting the appropriate agent, Arch routes the user's prompt to the designated agent's endpoint and waits for the agent to respond back with the processed output or further instructions. + +#. **Hand Off** + Based on follow-up queries from the user, Arch repeats the process of analysis, agent selection, and routing to ensure a seamless hand off between AI agents as needed. + +.. code-block:: yaml + :caption: Agent Routing and Hand Off Configuration Example + + prompt_targets: + - name: sales_agent + description: Handles queries related to sales and purchases + + - name: issues_and_repairs + description: handles issues, repairs, or refunds + + - name: escalate_to_human + description: escalates to human agent + +.. code-block:: python + :caption: Agent Routing and Hand Off Implementation Example via FastAPI + + class Agent: + def __init__(self, role: str, instructions: str): + self.system_prompt = f"You are a {role}.\n{instructions}" + + def handle(self, req: ChatCompletionsRequest): + messages = [{"role": "system", "content": self.get_system_prompt()}] + [ + message.model_dump() for message in req.messages + ] + return call_openai(messages, req.stream) #call_openai is a placeholder for the actual API call + + def get_system_prompt(self) -> str: + return self.system_prompt + + # Define your agents + AGENTS = { + "sales_agent": Agent( + role="sales agent", + instructions=( + "Always answer in a sentence or less.\n" + "Follow the following routine with the user:\n" + "1. Engage\n" + "2. Quote ridiculous price\n" + "3. Reveal caveat if user agrees." + ), + ), + "issues_and_repairs": Agent( + role="issues and repairs agent", + instructions="Propose a solution, offer refund if necessary.", + ), + "escalate_to_human": Agent( + role="human escalation agent", instructions="Escalate issues to a human." + ), + "unknown_agent": Agent( + role="general assistant", instructions="Assist the user in general queries." + ), + } + + #handle the request from arch gateway + @app.post("/v1/chat/completions") + def completion_api(req: ChatCompletionsRequest, request: Request): + + agent_name = req.metadata.get("agent-name", "unknown_agent") + agent = AGENTS.get(agent_name) + logger.info(f"Routing to agent: {agent_name}") + + return agent.handle(req) + +.. note:: + The above example demonstrates a simple implementation of Agent Routing and Hand Off using FastAPI. For the full implemenation of this example + please see our `GitHub demo `_. + +Example Use Cases +----------------- +Agent Routing and Hand Off is particularly beneficial in scenarios such as: + +- **Customer Support**: Routing common customer queries to automated support agents, while escalating complex or sensitive issues to human support staff. +- **Sales and Marketing**: Automatically directing potential leads and sales inquiries to specialized sales agents for timely and targeted follow-ups. +- **Technical Assistance**: Managing user-reported issues, repairs, or refunds by assigning them to the correct technical or support agent efficiently. + +Best Practices and Tips +------------------------ +When implementing Agent Routing and Hand Off in your applications, consider these best practices: + +- Clearly Define Agent Responsibilities: Ensure each agent or human endpoint has a clear, specific description of the prompts they handle, reducing misrouting. +- Monitor and Optimize Routes: Regularly review how prompts are routed to adjust and optimize agent definitions and configurations. + +.. note:: + To observe traffic to and from agents, please read more about :ref:`observabiliuty ` in Arch. + +By carefully configuring and managing your Agent routing and hand off, you can significantly improve your application's responsiveness, performance, and overall user satisfaction. diff --git a/docs/source/index.rst b/docs/source/index.rst index 3588382d..aefc63bf 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,11 +12,11 @@ Welcome to Arch!

Build fast, observable, and personalized GenAI apps

- Arch - Build fast, hyper-personalized agents with intelligent infra | Product Hunt + Arch - Build fast, hyper-personalized agents with intelligent infra | Product Hunt -`Arch `_ is an intelligent gateway for agents - an infrastructure primitive for GenAI (built by the contributors of `Envoy `_ ). The project was born out of the belief that: +`Arch `_ is an intelligent (edge and LLM) proxy, exclusively designed for prompts and agents - and built by the contributors of the widely adopted and loved `Envoy `_ ). - *Prompts are nuanced and opaque user requests, which require the same capabilities as traditional HTTP requests including secure handling, intelligent routing, robust observability, and integration with backend (API) systems for personalization - all outside business logic.* +**Move faster** by letting Arch handle the **pesky heavy lifting** in building agents: fast input clarification, agent routing, seamless integration of prompts with tools for common tasks, and unified access and observability of LLMs - all outside business logic. .. tab-set:: @@ -50,6 +50,7 @@ Welcome to Arch! :maxdepth: 2 guides/prompt_guard + guides/agent_routing guides/function_calling guides/observability/observability