mirror of
https://github.com/katanemo/plano.git
synced 2026-06-17 15:25:17 +02:00
add instructions on how to add custom inference endpoint
This commit is contained in:
parent
12e4d8d8b8
commit
90f77384f5
2 changed files with 12 additions and 6 deletions
|
|
@ -14,12 +14,6 @@ llm_providers:
|
|||
endpoint: host.docker.internal:11434
|
||||
default: true
|
||||
|
||||
- name: arch-fc-cloud
|
||||
provider_interface: openai
|
||||
model: Arch-Function
|
||||
endpoint: api.fc.archgw.com:443
|
||||
protocol: https
|
||||
|
||||
system_prompt: |
|
||||
You are a helpful assistant.
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,18 @@ calls, handling retries, managing rate limits, and ensuring seamless integration
|
|||
LLMs. Simply configure the details of the LLMs your application will use, and Arch offers a unified interface to
|
||||
make outbound LLM calls.
|
||||
|
||||
Adding custom LLM Provider
|
||||
--------------------------
|
||||
Out of the box we provide support for openai and mistral llm providers. But if you want to add your custom provider you can follow the below steps:
|
||||
|
||||
1. Add the provider in the `llm_providers` section of the `arch_config.yaml` file. And supply details of how to reach to the provider using "endpoint" and protocol.
|
||||
|
||||
- name: huggingface-qwen-2.5
|
||||
provider_interface: openai
|
||||
model: qwen-2.5
|
||||
endpoint: endpoint_address:443
|
||||
protocol: https
|
||||
|
||||
Example: Using the OpenAI Python SDK
|
||||
------------------------------------
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue