<spanid="arch-overview-prompt-handling"></span><h1>Prompts<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#prompts"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h1>
<p>Arch’s <strong>prompt handler</strong> subsystem interacts with the <strong>model subsytem</strong> through Envoy’s cluster manager system to ensure robust, resilient and fault-tolerant experience in managing incoming prompts.</p>
<divclass="admonition seealso">
<pclass="admonition-title">See also</p>
<p>Read more about the <aclass="reference internal"href="model_serving.html#model-serving"><spanclass="std std-ref">model subsystem</span></a> and how the LLMs are hosted in Arch.</p>
<h2>Messages<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#messages"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#messages'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h2>
<p>Arch accepts messages directly from the body of the HTTP request in a format that follows the <aclass="reference external"href="https://huggingface.co/docs/text-generation-inference/en/messages_api"rel="nofollow noopener">Hugging Face Messages API<svgfill="currentColor"height="1em"stroke="none"viewbox="0 96 960 960"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M188 868q-11-11-11-28t11-28l436-436H400q-17 0-28.5-11.5T360 336q0-17 11.5-28.5T400 296h320q17 0 28.5 11.5T760 336v320q0 17-11.5 28.5T720 696q-17 0-28.5-11.5T680 656V432L244 868q-11 11-28 11t-28-11Z"></path></svg></a>.
This design allows developers to pass a list of messages, where each message is represented as a dictionary
containing two key-value pairs:</p>
<blockquote>
<div><ulclass="simple">
<li><p><strong>Role</strong>: Defines the role of the message sender, such as “user” or “assistant”.</p></li>
<li><p><strong>Content</strong>: Contains the actual text of the message.</p></li>
<p>Arch is engineered with <aclass="reference external"href="https://huggingface.co/collections/katanemo/arch-guard-6702bdc08b889e4bce8f446d"rel="nofollow noopener">Arch-Guard<svgfill="currentColor"height="1em"stroke="none"viewbox="0 96 960 960"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M188 868q-11-11-11-28t11-28l436-436H400q-17 0-28.5-11.5T360 336q0-17 11.5-28.5T400 296h320q17 0 28.5 11.5T760 336v320q0 17-11.5 28.5T720 696q-17 0-28.5-11.5T680 656V432L244 868q-11 11-28 11t-28-11Z"></path></svg></a>, an industry leading safety layer, powered by a
<divclass="code-block-caption"><spanclass="caption-text">Example Configuration</span><a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#id1"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></div>
</span><spanid="line-6"><spanclass="linenos"> 6</span><spanclass="w"></span><spanclass="c1"># Defines how Arch should parse the content from application/json or text/pain Content-type in the http request</span>
</span><spanid="line-9"><spanclass="linenos"> 9</span><spanclass="c1"># Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way</span>
</span><spanid="line-19"><spanclass="linenos">19</span><spanclass="nt">system_prompt</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.</span>
</mark></span><spanid="line-25"><mark><spanclass="linenos">25</span><spanclass="w"></span><spanclass="nt">message</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Looks like you're curious about my abilities, but I can only provide assistance within my programmed parameters.</span>
and identifies where to forwad the conversation to via its <codeclass="docutils literal notranslate"><spanclass="pre">prompt</span><spanclass="pre">target</span></code> primitve. Prompt targets are endpoints
<divclass="code-block-caption"><spanclass="caption-text">Example Configuration</span><a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#id2"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></div>
</span><spanid="line-6"><spanclass="linenos"> 6</span><spanclass="w"></span><spanclass="c1"># Defines how Arch should parse the content from application/json or text/pain Content-type in the http request</span>
</span><spanid="line-9"><spanclass="linenos"> 9</span><spanclass="c1"># Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way</span>
</span><spanid="line-19"><spanclass="linenos">19</span><spanclass="nt">system_prompt</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.</span>
</span><spanid="line-25"><spanclass="linenos">25</span><spanclass="w"></span><spanclass="nt">message</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Looks like you're curious about my abilities, but I can only provide assistance within my programmed parameters.</span>
</span><spanid="line-30"><spanclass="linenos">30</span><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">handel all scenarios that are question and answer in nature. Like summarization, information extraction, etc.</span>
</span><spanid="line-34"><spanclass="linenos">34</span><spanclass="w"></span><spanclass="c1"># Arch uses the default LLM and treats the response from the endpoint as the prompt to send to the LLM</span>
</span><spanid="line-36"><spanclass="linenos">36</span><spanclass="w"></span><spanclass="c1"># override system prompt for this prompt target</span>
</span><spanid="line-37"><spanclass="linenos">37</span><spanclass="w"></span><spanclass="nt">system_prompt</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">You are a helpful information extraction assistant. Use the information that is provided to you.</span>
</mark></span><spanid="line-40"><mark><spanclass="linenos">40</span><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Reboot a specific network device</span>
</mark></span><spanid="line-47"><mark><spanclass="linenos">47</span><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Identifier of the network device to reboot.</span>
</mark></span><spanid="line-51"><mark><spanclass="linenos">51</span><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Confirmation flag to proceed with reboot.</span>
</span><spanid="line-60"><spanclass="linenos">60</span><spanclass="c1"># Arch creates a round-robin load balancing between different endpoints, managed via the cluster subsystem.</span>
</span><spanid="line-63"><spanclass="linenos">63</span><spanclass="w"></span><spanclass="c1"># value could be ip address or a hostname with port</span>
</span><spanid="line-64"><spanclass="linenos">64</span><spanclass="w"></span><spanclass="c1"># this could also be a list of endpoints for load balancing</span>
</span><spanid="line-65"><spanclass="linenos">65</span><spanclass="w"></span><spanclass="c1"># for example endpoint: [ ip1:port, ip2:port ]</span>
</span><spanid="line-67"><spanclass="linenos">67</span><spanclass="w"></span><spanclass="c1"># max time to wait for a connection to be established</span>
<p>Check <aclass="reference internal"href="../prompt_target.html#prompt-target"><spanclass="std std-ref">Prompt Target</span></a> for more details!</p>
<h3>Intent Matching<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#intent-matching"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#intent-matching'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h3>
<p>Arch uses fast text embedding and intent recognition approaches to first detect the intent of each incoming prompt.
This intent matching phase analyzes the prompt’s content and matches it against predefined prompt targets, ensuring that each prompt is forwarded to the most appropriate endpoint.
Arch’s intent matching framework considers both the name and description of each prompt target, and uses a composite matching score between embedding similarity and intent classification scores to enchance accuracy in forwarding decisions.</p>
<li><p><strong>Intent Recognition</strong>: NLI techniques further refine the matching process by evaluating the semantic alignment between the prompt and potential targets.</p></li>
<li><p><strong>Text Embedding</strong>: By embedding the prompt and comparing it to known target vectors, Arch effectively identifies the closest match, ensuring that the prompt is handled by the correct downstream service.</p></li>
<h3>Agentic Apps via Prompt Targets<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#agentic-apps-via-prompt-targets"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#agentic-apps-via-prompt-targets'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h3>
<p>To support agentic apps, like scheduling travel plans or sharing comments on a document - via prompts, Arch uses its function calling abilities to extract critical information from the incoming prompt (or a set of prompts) needed by a downstream backend API or function call before calling it directly.
For more details on how you can build agentic applications using Arch, see our full guide <aclass="reference internal"href="../../build_with_arch/agent.html#arch-agent-guide"><spanclass="std std-ref">here</span></a>:</p>
<p><aclass="reference external"href="https://huggingface.co/collections/katanemo/arch-function-66f209a693ea8df14317ad68"rel="nofollow noopener">Arch-Function<svgfill="currentColor"height="1em"stroke="none"viewbox="0 96 960 960"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M188 868q-11-11-11-28t11-28l436-436H400q-17 0-28.5-11.5T360 336q0-17 11.5-28.5T400 296h320q17 0 28.5 11.5T760 336v320q0 17-11.5 28.5T720 696q-17 0-28.5-11.5T680 656V432L244 868q-11 11-28 11t-28-11Z"></path></svg></a> is a collection of dedicated agentic models engineered in Arch to extract information from a (set of) prompts and executes necessary backend API calls.
This allows for efficient handling of agentic tasks, such as scheduling data retrieval, by dynamically interacting with backend services.
Arch-Function achieves state-of-the-art performance, comparable with frontier models like Claude Sonnet 3.5 ang GPT-4, while being 44x cheaper ($0.10M/token hosted) and 10x faster (p50 latencies of 200ms).</p>
<p>Arch is a single piece of software that is designed to manage both ingress and egress prompt traffic, drawing its distributed proxy nature from the robust <aclass="reference external"href="https://envoyproxy.io"rel="nofollow noopener">Envoy<svgfill="currentColor"height="1em"stroke="none"viewbox="0 96 960 960"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M188 868q-11-11-11-28t11-28l436-436H400q-17 0-28.5-11.5T360 336q0-17 11.5-28.5T400 296h320q17 0 28.5 11.5T760 336v320q0 17-11.5 28.5T720 696q-17 0-28.5-11.5T680 656V432L244 868q-11 11-28 11t-28-11Z"></path></svg></a>.
This makes it extremely efficient and capable of handling upstream connections to LLMs.
If your application is originating code to an API-based LLM, simply use the OpenAI client and configure it with Arch.
By sending traffic through Arch, you can propagate traces, manage and monitor traffic, apply rate limits, and utilize a large set of traffic management capabilities in a centralized way.</p>
<codeclass="docutils literal notranslate"><spanclass="pre">llm_providers</span></code> configuration section in the <codeclass="docutils literal notranslate"><spanclass="pre">arch_config.yml</span></code> file. Arch binds itself to a local address such as
<h3>Example: Using OpenAI Client with Arch as an Egress Gateway<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#example-using-openai-client-with-arch-as-an-egress-gateway"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#example-using-openai-client-with-arch-as-an-egress-gateway'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h3>
The OpenAI client is configured to route traffic via Arch by setting the proxy to <codeclass="docutils literal notranslate"><spanclass="pre">127.0.0.1:12000</span></code>, assuming Arch is running locally and bound to that address and port.
<li><a:data-current="activeSection === '#example-using-openai-client-with-arch-as-an-egress-gateway'"class="reference internal"href="#example-using-openai-client-with-arch-as-an-egress-gateway">Example: Using OpenAI Client with Arch as an Egress Gateway</a></li>
