add instructions on how to add custom inference endpoint

This commit is contained in:
Adil Hafeez 2025-01-24 16:33:15 -08:00
parent 12e4d8d8b8
commit 90f77384f5
No known key found for this signature in database
GPG key ID: 9B18EF7691369645
2 changed files with 12 additions and 6 deletions

View file

@ -14,12 +14,6 @@ llm_providers:
endpoint: host.docker.internal:11434
default: true
- name: arch-fc-cloud
provider_interface: openai
model: Arch-Function
endpoint: api.fc.archgw.com:443
protocol: https
system_prompt: |
You are a helpful assistant.

View file

@ -32,6 +32,18 @@ calls, handling retries, managing rate limits, and ensuring seamless integration
LLMs. Simply configure the details of the LLMs your application will use, and Arch offers a unified interface to
make outbound LLM calls.
Adding custom LLM Provider
--------------------------
Out of the box we provide support for openai and mistral llm providers. But if you want to add your custom provider you can follow the below steps:
1. Add the provider in the `llm_providers` section of the `arch_config.yaml` file. And supply details of how to reach to the provider using "endpoint" and protocol.
- name: huggingface-qwen-2.5
provider_interface: openai
model: qwen-2.5
endpoint: endpoint_address:443
protocol: https
Example: Using the OpenAI Python SDK
------------------------------------