mirror of
https://github.com/katanemo/plano.git
synced 2026-06-26 15:39:40 +02:00
updating plano docs, README and CLI
This commit is contained in:
Salman Paracha
parent
2f9121407b
commit
28fd430efd
61 changed files with 1449 additions and 1306 deletions
|
|
@ -3,7 +3,7 @@
|
|||
Client Libraries
|
||||
================
|
||||
|
||||
Arch provides a unified interface that works seamlessly with multiple client libraries and tools. You can use your preferred client library without changing your existing code - just point it to Arch's gateway endpoints.
|
||||
Plano provides a unified interface that works seamlessly with multiple client libraries and tools. You can use your preferred client library without changing your existing code - just point it to Plano's gateway endpoints.
|
||||
|
||||
Supported Clients
|
||||
------------------
|
||||
|
|
@ -16,7 +16,7 @@ Supported Clients
|
|||
Gateway Endpoints
|
||||
-----------------
|
||||
|
||||
Arch exposes two main endpoints:
|
||||
Plano exposes three main endpoints:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
|
@ -26,13 +26,15 @@ Arch exposes two main endpoints:
|
|||
- Purpose
|
||||
* - ``http://127.0.0.1:12000/v1/chat/completions``
|
||||
- OpenAI-compatible chat completions (LLM Gateway)
|
||||
* - ``http://127.0.0.1:12000/v1/responses``
|
||||
- OpenAI Responses API with :ref:`conversational state management <managing_conversational_state>` (LLM Gateway)
|
||||
* - ``http://127.0.0.1:12000/v1/messages``
|
||||
- Anthropic-compatible messages (LLM Gateway)
|
||||
|
||||
OpenAI (Python) SDK
|
||||
-------------------
|
||||
|
||||
The OpenAI SDK works with any provider through Arch's OpenAI-compatible endpoint.
|
||||
The OpenAI SDK works with any provider through Plano's OpenAI-compatible endpoint.
|
||||
|
||||
**Installation:**
|
||||
|
||||
|
|
@ -46,7 +48,7 @@ The OpenAI SDK works with any provider through Arch's OpenAI-compatible endpoint
|
|||
|
||||
from openai import OpenAI
|
||||
|
||||
# Point to Arch's LLM Gateway
|
||||
# Point to Plano's LLM Gateway
|
||||
client = OpenAI(
|
||||
api_key="test-key", # Can be any value for local testing
|
||||
base_url="http://127.0.0.1:12000/v1"
|
||||
|
|
@ -96,7 +98,7 @@ The OpenAI SDK works with any provider through Arch's OpenAI-compatible endpoint
|
|||
|
||||
**Using with Non-OpenAI Models:**
|
||||
|
||||
The OpenAI SDK can be used with any provider configured in Arch:
|
||||
The OpenAI SDK can be used with any provider configured in Plano:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -124,10 +126,92 @@ The OpenAI SDK can be used with any provider configured in Arch:
|
|||
]
|
||||
)
|
||||
|
||||
OpenAI Responses API (Conversational State)
|
||||
-------------------------------------------
|
||||
|
||||
The OpenAI Responses API (``v1/responses``) enables multi-turn conversations with automatic state management. Plano handles conversation history for you, so you don't need to manually include previous messages in each request.
|
||||
|
||||
See :ref:`managing_conversational_state` for detailed configuration and storage backend options.
|
||||
|
||||
**Installation:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install openai
|
||||
|
||||
**Basic Multi-Turn Conversation:**
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
# Point to Plano's LLM Gateway
|
||||
client = OpenAI(
|
||||
api_key="test-key",
|
||||
base_url="http://127.0.0.1:12000/v1"
|
||||
)
|
||||
|
||||
# First turn - creates a new conversation
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-4o-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "My name is Alice"}
|
||||
]
|
||||
)
|
||||
|
||||
# Extract response_id for conversation continuity
|
||||
response_id = response.id
|
||||
print(f"Assistant: {response.choices[0].message.content}")
|
||||
|
||||
# Second turn - continues the conversation
|
||||
# Plano automatically retrieves and merges previous context
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-4o-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's my name?"}
|
||||
],
|
||||
metadata={"response_id": response_id} # Reference previous conversation
|
||||
)
|
||||
|
||||
print(f"Assistant: {response.choices[0].message.content}")
|
||||
# Output: "Your name is Alice"
|
||||
|
||||
**Using with Any Provider:**
|
||||
|
||||
The Responses API works with any LLM provider configured in Plano:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Multi-turn conversation with Claude
|
||||
response = client.chat.completions.create(
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
messages=[
|
||||
{"role": "user", "content": "Let's discuss quantum physics"}
|
||||
]
|
||||
)
|
||||
|
||||
response_id = response.id
|
||||
|
||||
# Continue conversation - Plano manages state regardless of provider
|
||||
response = client.chat.completions.create(
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
messages=[
|
||||
{"role": "user", "content": "Tell me more about entanglement"}
|
||||
],
|
||||
metadata={"response_id": response_id}
|
||||
)
|
||||
|
||||
**Key Benefits:**
|
||||
|
||||
* **Reduced payload size**: No need to send full conversation history in each request
|
||||
* **Provider flexibility**: Use any configured LLM provider with state management
|
||||
* **Automatic context merging**: Plano handles conversation continuity behind the scenes
|
||||
* **Production-ready storage**: Configure :ref:`PostgreSQL or memory storage <managing_conversational_state>` based on your needs
|
||||
|
||||
Anthropic (Python) SDK
|
||||
----------------------
|
||||
|
||||
The Anthropic SDK works with any provider through Arch's Anthropic-compatible endpoint.
|
||||
The Anthropic SDK works with any provider through Plano's Anthropic-compatible endpoint.
|
||||
|
||||
**Installation:**
|
||||
|
||||
|
|
@ -141,7 +225,7 @@ The Anthropic SDK works with any provider through Arch's Anthropic-compatible en
|
|||
|
||||
import anthropic
|
||||
|
||||
# Point to Arch's LLM Gateway
|
||||
# Point to Plano's LLM Gateway
|
||||
client = anthropic.Anthropic(
|
||||
api_key="test-key", # Can be any value for local testing
|
||||
base_url="http://127.0.0.1:12000"
|
||||
|
|
@ -192,7 +276,7 @@ The Anthropic SDK works with any provider through Arch's Anthropic-compatible en
|
|||
|
||||
**Using with Non-Anthropic Models:**
|
||||
|
||||
The Anthropic SDK can be used with any provider configured in Arch:
|
||||
The Anthropic SDK can be used with any provider configured in Plano:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -284,7 +368,7 @@ For direct HTTP requests or integration with any programming language:
|
|||
Cross-Client Compatibility
|
||||
--------------------------
|
||||
|
||||
One of Arch's key features is cross-client compatibility. You can:
|
||||
One of Plano's key features is cross-client compatibility. You can:
|
||||
|
||||
**Use OpenAI SDK with Claude Models:**
|
||||
|
||||
|
|
|
|||
|
|
@ -1,16 +1,16 @@
|
|||
.. _llm_providers:
|
||||
|
||||
LLM Providers
|
||||
=============
|
||||
**LLM Providers** are a top-level primitive in Arch, helping developers centrally define, secure, observe,
|
||||
and manage the usage of their LLMs. Arch builds on Envoy's reliable `cluster subsystem <https://www.envoyproxy.io/docs/envoy/v1.31.2/intro/arch_overview/upstream/cluster_manager>`_
|
||||
Model (LLM) Providers
|
||||
=====================
|
||||
**LLM Providers** are a top-level primitive in Plano, helping developers centrally define, secure, observe,
|
||||
and manage the usage of their LLMs. Plano builds on Envoy's reliable `cluster subsystem <https://www.envoyproxy.io/docs/envoy/v1.31.2/intro/arch_overview/upstream/cluster_manager>`_
|
||||
to manage egress traffic to LLMs, which includes intelligent routing, retry and fail-over mechanisms,
|
||||
ensuring high availability and fault tolerance. This abstraction also enables developers to seamlessly
|
||||
switch between LLM providers or upgrade LLM versions, simplifying the integration and scaling of LLMs
|
||||
across applications.
|
||||
|
||||
Today, we are enabling you to connect to 11+ different AI providers through a unified interface with advanced routing and management capabilities.
|
||||
Whether you're using OpenAI, Anthropic, Azure OpenAI, local Ollama models, or any OpenAI-compatible provider, Arch provides seamless integration with enterprise-grade features.
|
||||
Whether you're using OpenAI, Anthropic, Azure OpenAI, local Ollama models, or any OpenAI-compatible provider, Plano provides seamless integration with enterprise-grade features.
|
||||
|
||||
Core Capabilities
|
||||
-----------------
|
||||
|
|
@ -18,29 +18,29 @@ Core Capabilities
|
|||
**Multi-Provider Support**
|
||||
Connect to any combination of providers simultaneously (see :ref:`supported_providers` for full details):
|
||||
|
||||
- **First-Class Providers**: Native integrations with OpenAI, Anthropic, DeepSeek, Mistral, Groq, Google Gemini, Together AI, xAI, Azure OpenAI, and Ollama
|
||||
- **OpenAI-Compatible Providers**: Any provider implementing the OpenAI Chat Completions API standard
|
||||
- First-Class Providers: Native integrations with OpenAI, Anthropic, DeepSeek, Mistral, Groq, Google Gemini, Together AI, xAI, Azure OpenAI, and Ollama
|
||||
- OpenAI-Compatible Providers: Any provider implementing the OpenAI Chat Completions API standard
|
||||
|
||||
**Intelligent Routing**
|
||||
Three powerful routing approaches to optimize model selection:
|
||||
|
||||
- **Model-based Routing**: Direct routing to specific models using provider/model names (see :ref:`supported_providers`)
|
||||
- **Alias-based Routing**: Semantic routing using custom aliases (see :ref:`model_aliases`)
|
||||
- **Preference-aligned Routing**: Intelligent routing using the Arch-Router model (see :ref:`preference_aligned_routing`)
|
||||
- Model-based Routing: Direct routing to specific models using provider/model names (see :ref:`supported_providers`)
|
||||
- Alias-based Routing: Semantic routing using custom aliases (see :ref:`model_aliases`)
|
||||
- Preference-aligned Routing: Intelligent routing using the Plano-Router model (see :ref:`preference_aligned_routing`)
|
||||
|
||||
**Unified Client Interface**
|
||||
Use your preferred client library without changing existing code (see :ref:`client_libraries` for details):
|
||||
|
||||
- **OpenAI Python SDK**: Full compatibility with all providers
|
||||
- **Anthropic Python SDK**: Native support with cross-provider capabilities
|
||||
- **cURL & HTTP Clients**: Direct REST API access for any programming language
|
||||
- **Custom Integrations**: Standard HTTP interfaces for seamless integration
|
||||
- OpenAI Python SDK: Full compatibility with all providers
|
||||
- Anthropic Python SDK: Native support with cross-provider capabilities
|
||||
- cURL & HTTP Clients: Direct REST API access for any programming language
|
||||
- Custom Integrations: Standard HTTP interfaces for seamless integration
|
||||
|
||||
Key Benefits
|
||||
------------
|
||||
|
||||
- **Provider Flexibility**: Switch between providers without changing client code
|
||||
- **Three Routing Methods**: Choose from model-based, alias-based, or preference-aligned routing (using `Arch-Router-1.5B <https://huggingface.co/katanemo/Arch-Router-1.5B>`_) strategies
|
||||
- **Three Routing Methods**: Choose from model-based, alias-based, or preference-aligned routing (using `Plano-Router-1.5B <https://huggingface.co/katanemo/Plano-Router-1.5B>`_) strategies
|
||||
- **Cost Optimization**: Route requests to cost-effective models based on complexity
|
||||
- **Performance Optimization**: Use fast models for simple tasks, powerful models for complex reasoning
|
||||
- **Environment Management**: Configure different models for different environments
|
||||
|
|
|
|||
|
|
@ -3,10 +3,10 @@
|
|||
Supported Providers & Configuration
|
||||
===================================
|
||||
|
||||
Arch provides first-class support for multiple LLM providers through native integrations and OpenAI-compatible interfaces. This comprehensive guide covers all supported providers, their available chat models, and detailed configuration instructions.
|
||||
Plano provides first-class support for multiple LLM providers through native integrations and OpenAI-compatible interfaces. This comprehensive guide covers all supported providers, their available chat models, and detailed configuration instructions.
|
||||
|
||||
.. note::
|
||||
**Model Support:** Arch supports all chat models from each provider, not just the examples shown in this guide. The configurations below demonstrate common models for reference, but you can use any chat model available from your chosen provider.
|
||||
**Model Support:** Plano supports all chat models from each provider, not just the examples shown in this guide. The configurations below demonstrate common models for reference, but you can use any chat model available from your chosen provider.
|
||||
|
||||
Configuration Structure
|
||||
-----------------------
|
||||
|
|
@ -50,7 +50,7 @@ Any provider that implements the OpenAI API interface can be configured using cu
|
|||
Supported API Endpoints
|
||||
------------------------
|
||||
|
||||
Arch supports the following standardized endpoints across providers:
|
||||
Plano supports the following standardized endpoints across providers:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
|
@ -524,7 +524,7 @@ Amazon Bedrock
|
|||
|
||||
**Provider Prefix:** ``amazon_bedrock/``
|
||||
|
||||
**API Endpoint:** Arch automatically constructs the endpoint as:
|
||||
**API Endpoint:** Plano automatically constructs the endpoint as:
|
||||
- Non-streaming: ``/model/{model-id}/converse``
|
||||
- Streaming: ``/model/{model-id}/converse-stream``
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue