<spanid="lifecycle-of-a-request"></span><h1>Request Lifecycle<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#request-lifecycle"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h1>
<p>Below we describe the events in the lifecycle of a request passing through an Arch gateway instance. We first
describe how Arch fits into the request path and then the internal events that take place following
the arrival of a request at Arch from downtream clients. We follow the request until the corresponding
<h2>Terminology<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#terminology"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#terminology'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h2>
<p>We recommend that you get familiar with some of the <aclass="reference internal"href="terminology.html#arch-terminology"><spanclass="std std-ref">terminology</span></a> used in Arch
before reading this section.</p>
</section>
<sectionid="network-topology">
<h2>Network topology<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#network-topology"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#network-topology'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h2>
<p>How a request flows through the components in a network (including Arch) depends on the network’s topology.
Arch can be used in a wide variety of networking topologies. We focus on the inner operation of Arch below,
but briefly we address how Arch relates to the rest of the network in this section.</p>
<ulclass="simple">
<li><p><strong>Downstream(Ingress)</strong> listeners take requests from upstream clients like a web UI or clients that forward
prompts to you local application responses from the application flow back through Arch to the downstream.</p></li>
<li><p><strong>Upstream(Egress)</strong> listeners take requests from the application and forward them to LLMs.</p></li>
<h2>High level architecture<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#high-level-architecture"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#high-level-architecture'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h2>
<p>Arch is a set of <strong>two</strong> self-contained processes that are designed to run alongside your application servers
(or on a separate server connected to your application servers via a network). The first process is designated
to manage HTTP-level networking and connection management concerns (protocol management, request id generation,
header sanitization, etc.), and the other process is for <strong>model serving</strong>, which helps Arch make intelligent
decisions about the incoming prompts. The model server hosts the purpose-built LLMs to
manage several critical, but undifferentiated, prompt related tasks on behalf of developers.</p>
<p>The request processing path in Arch has three main parts:</p>
<ulclass="simple">
<li><p><aclass="reference internal"href="listener.html#arch-overview-listeners"><spanclass="std std-ref">Listener subsystem</span></a> which handles <strong>downstream</strong> and <strong>upstream</strong> request
processing. It is responsible for managing the downstream (ingress) and the upstream (egress) request
lifecycle. The downstream and upstream HTTP/2 codec lives here.</p></li>
<li><p><aclass="reference internal"href="prompt.html#arch-overview-prompt-handling"><spanclass="std std-ref">Prompt handler subsystem</span></a> which is responsible for selecting and
forwarding prompts <codeclass="docutils literal notranslate"><spanclass="pre">prompt_targets</span></code> and establishes the lifecycle of any <strong>upstream</strong> connection to a
hosted endpoint that implements domain-specific business logic for incoming promots. This is where knowledge
of targets and endpoint health, load balancing and connection pooling exists.</p></li>
<li><p><aclass="reference internal"href="model_serving.html#model-serving"><spanclass="std std-ref">Model serving subsystem</span></a> which helps Arch make intelligent decisions about the
A main thread is responsible forthe server lifecycle, configuration processing, stats, etc. and some number of
<aclass="reference internal"href="threading_model.html#arch-overview-threading"><spanclass="std std-ref">worker threads</span></a> process requests. All threads operate around an event loop (<aclass="reference external"href="https://libevent.org/"rel="nofollow noopener">libevent<svgfill="currentColor"height="1em"stroke="none"viewbox="0 96 960 960"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M188 868q-11-11-11-28t11-28l436-436H400q-17 0-28.5-11.5T360 336q0-17 11.5-28.5T400 296h320q17 0 28.5 11.5T760 336v320q0 17-11.5 28.5T720 696q-17 0-28.5-11.5T680 656V432L244 868q-11 11-28 11t-28-11Z"></path></svg></a>)
and any given downstream TCP connection will be handled by exactly one worker thread for its lifetime. Each worker
thread maintains its own pool of TCP connections to upstream endpoints.</p>
<p>Worker threads rarely share state and operate in a trivially parallel fashion. This threading model
enables scaling to very high core count CPUs.</p>
</section>
<sectionid="configuration">
<h2>Configuration<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#configuration"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#configuration'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h2>
<p>Today, only support a static bootstrap configuration file for simplicity today:</p>
</span><spanid="line-4"><spanclass="w"></span><spanclass="nt">address</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">0.0.0.0</span><spanclass="w"></span><spanclass="c1"># or 127.0.0.1</span>
</span><spanid="line-6"><spanclass="w"></span><spanclass="c1"># Defines how Arch should parse the content from application/json or text/pain Content-type in the http request</span>
</span><spanid="line-19"><spanclass="nt">system_prompt</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.</span>
</span><spanid="line-25"><spanclass="w"></span><spanclass="nt">message</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Looks like you're curious about my abilities, but I can only provide assistance within my programmed parameters.</span>
</span><spanid="line-30"><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">handel all scenarios that are question and answer in nature. Like summarization, information extraction, etc.</span>
</span><spanid="line-34"><spanclass="w"></span><spanclass="c1"># Arch uses the default LLM and treats the response from the endpoint as the prompt to send to the LLM</span>
</span><spanid="line-36"><spanclass="w"></span><spanclass="c1"># override system prompt for this prompt target</span>
</span><spanid="line-37"><spanclass="w"></span><spanclass="nt">system_prompt</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">You are a helpful information extraction assistant. Use the information that is provided to you.</span>
</span><spanid="line-40"><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Reboot a specific network device</span>
</span><spanid="line-47"><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Identifier of the network device to reboot.</span>
</span><spanid="line-51"><spanclass="w"></span><spanclass="nt">description</span><spanclass="p">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">Confirmation flag to proceed with reboot.</span>
</span><spanid="line-60"><spanclass="c1"># Arch creates a round-robin load balancing between different endpoints, managed via the cluster subsystem.</span>
<li><p><strong>Error Handling and Forwarding</strong>:
Errors encountered during processing, such as failed function calls or guardrail detections, are forwarded to
designated error targets. Error details are communicated through specific headers to the application:</p>
<ulclass="simple">
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">X-Function-Error-Code</span></code>: Code indicating the type of function call error.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">X-Prompt-Guard-Error-Code</span></code>: Code specifying violations detected by prompt guardrails.</p></li>
<li><p>Additional headers carry messages and timestamps to aid in debugging and logging.</p></li>
</ul>
</li>
<li><p><strong>Response Handling</strong>:
The upstream endpoint’s TLS transport socket encrypts the response, which is then proxied back downstream.
Responses pass through HTTP filters in reverse order, ensuring any necessary processing or modification before final delivery.</p></li>
</ol>
</section>
<sectionid="request-flow-egress">
<h2>Request Flow (Egress)<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#request-flow-egress"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#request-flow-egress'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h2>
<p>A brief outline of the lifecycle of a request and response in the context of egress traffic from an application to Large Language Models (LLMs) via Arch:</p>
<li><p><strong>HTTP Connection Establishment to LLM</strong>:
Arch initiates an HTTP connection to the upstream LLM service. This connection is handled by Arch’s egress listener
running on a worker thread. The connection typically uses a secure transport protocol such as HTTPS, ensuring the
prompt data is encrypted before being sent to the LLM service.</p></li>
<li><p><strong>Rate Limiting</strong>:
Before sending the request to the LLM, Arch applies rate-limiting policies to ensure that the upstream LLM service
is not overwhelmed by excessive traffic. Rate limits are enforced per client or service, ensuring fair usage and
preventing accidental or malicious overload. If the rate limit is exceeded, Arch may return an appropriate HTTP
error (e.g., 429 Too Many Requests) without sending the prompt to the LLM.</p></li>
<li><p><strong>Load Balancing to (hosted) LLM Endpoints</strong>:
After passing the rate-limiting checks, Arch routes the prompt to the appropriate LLM endpoint.
If multiple LLM providers instances are available, load balancing is performed to distribute traffic evenly
across the instances. Arch checks the health of the LLM endpoints using circuit breakers and health checks,
ensuring that the prompt is only routed to a healthy, responsive instance.</p></li>
<li><p><strong>Response Reception and Forwarding</strong>:
Once the LLM processes the prompt, Arch receives the response from the LLM service. The response is typically a
generated text, completion, or summarization. Upon reception, Arch decrypts (if necessary) and handles the response,
passing it through any egress processing pipeline defined by the application, such as logging or additional response filtering.</p></li>
</ol>
<sectionid="post-request-processing">
<h3>Post-request processing<a@click.prevent="window.navigator.clipboard.writeText($el.href); $el.setAttribute('data-tooltip', 'Copied!'); setTimeout(() => $el.setAttribute('data-tooltip', 'Copy link to this element'), 2000)"aria-label="Copy link to this element"class="headerlink"data-tooltip="Copy link to this element"href="#post-request-processing"x-intersect.margin.0%.0%.-70%.0%="activeSection ='#post-request-processing'"><svgheight="1em"viewbox="0 0 24 24"width="1em"xmlns="http://www.w3.org/2000/svg"><pathd="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76 0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71 0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71 0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76 0 5-2.24 5-5s-2.24-5-5-5z"></path></svg></a></h3>
<p>Once a request completes, the stream is destroyed. The following also takes places:</p>
<ulclass="simple">
<li><p>The post-request <aclass="reference internal"href="../../guides/observability/monitoring.html#monitoring"><spanclass="std std-ref">monitoring</span></a> are updated (e.g. timing, active requests, upgrades, health checks).
Some statistics are updated earlier however, during request processing. Stats are batchedand written by the main
thread periodically.</p></li>
<li><p><aclass="reference internal"href="../../guides/observability/access_logging.html#arch-access-logging"><spanclass="std std-ref">Access logs</span></a> are written to the access log</p></li>
<li><p><aclass="reference internal"href="../../guides/observability/tracing.html#arch-overview-tracing"><spanclass="std std-ref">Trace</span></a> spans are finalized. If our example request was traced, a
trace span, describing the duration and details of the request would be created by the HCM when
processing request headers and then finalized by the HCM during post-request processing.</p></li>
