From cc223f52fc1735eb69423092b1df82f933e88b63 Mon Sep 17 00:00:00 2001 From: Adil Hafeez Date: Fri, 24 Jan 2025 16:45:12 -0800 Subject: [PATCH] add more details --- docs/source/concepts/llm_provider.rst | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/source/concepts/llm_provider.rst b/docs/source/concepts/llm_provider.rst index 2e4e8350..24cf2c14 100644 --- a/docs/source/concepts/llm_provider.rst +++ b/docs/source/concepts/llm_provider.rst @@ -34,18 +34,30 @@ make outbound LLM calls. Adding custom LLM Provider -------------------------- -Out of the box we provide support for openai and mistral llm providers. But if you want to add your custom provider you can follow the below steps: -1. Add the provider in the `llm_providers` section of the `arch_config.yaml` file. And supply details of how to reach to the provider using "endpoint" and protocol. +We support any OpenAI compliant LLM provider for example mistral, openai, ollama etc. Out of the box we provide support for openai and mistral. But if you want to add your custom provider you can follow the below steps: + +1. Add the llm provider in the `llm_providers` section of the `arch_config.yaml` file. And supply details of how to reach to the provider using "endpoint" and specify `https` in `protocol`` if your endpoint is hosted on public cloud. .. code-block:: yaml - - name: huggingface-qwen-2.5 + - name: local-llama provider_interface: openai - model: qwen-2.5 - endpoint: endpoint_address:443 + model: llama3.2 + endpoint: host.docker.internal:11434 + + +For example following code block shows you how to add mistral llm provider in the `arch_config.yaml` file. + +.. code-block:: yaml + + - name: mistral-ai + provider_interface: openai + model: llama3.2 + endpoint: api.mistral.ai:443 protocol: https + Example: Using the OpenAI Python SDK ------------------------------------