update getting started guide and add llm gateway and prompt gateway samples (#330)

README.md

@@ -39,88 +39,178 @@ To get in touch with us, please join our [discord server](https://discord.gg/pGZ
 
 ## Quickstart
 
-Follow this guide to learn how to quickly set up Arch and integrate it into your generative AI applications.
+Follow this quickstart guide to use arch gateway to build a simple AI agent. Laster in the section we will see how you can Arch Gateway to manage access keys, provide unified access to upstream LLMs and to provide e2e observability.
 
 ### Prerequisites
 
 Before you begin, ensure you have the following:
 
-- `Docker` & `Python` installed on your system
-- `API Keys` for LLM providers (if using external LLMs)
-
-
-### Step 1: Install Arch
-
-Arch's CLI allows you to manage and interact with the Arch gateway efficiently. To install the CLI, simply run the following command:
-Tip: We recommend that developers create a new Python virtual environment to isolate dependencies before installing Arch. This ensures that archgw and its dependencies do not interfere with other packages on your system.
-
-Make sure you have following utilities installed before proceeding further,
-
 1. [Docker System](https://docs.docker.com/get-started/get-docker/) (v24)
 2. [Docker compose](https://docs.docker.com/compose/install/) (v2.29)
 3. [Python](https://www.python.org/downloads/) (v3.12)
-4. [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer) (v1.8.3. *Note: only needed for local development*)
 
+Arch's CLI allows you to manage and interact with the Arch gateway efficiently. To install the CLI, simply run the following command:
+
+> [!TIP]
+> We recommend that developers create a new Python virtual environment to isolate dependencies before installing Arch. This ensures that archgw and its dependencies do not interfere with other packages on your system.
 
 ```console
 $ python -m venv venv
 $ source venv/bin/activate   # On Windows, use: venv\Scripts\activate
-$ pip install archgw
+$ pip install archgw==0.1.5
 ```
 
-### Step 2: Configure Arch with your application
+### Build AI Agent with Arch Gateway
 
-Arch operates based on a configuration file where you can define LLM providers, prompt targets, guardrails, etc.
-Below is an example configuration to get you started:
+In following quickstart we will show you how easy it is to build AI agent with Arch gateway. We will build a currency exchange agent using following simple steps. For this demo we will use `https://api.frankfurter.dev/` to fetch latest price for currencies and assume USD as base currency.
+
+#### Step 1. Create arch config file
+
+Create `arch_config.yaml` file with following content,
 
 ```yaml
 version: v0.1
+
 listener:
-  address: 127.0.0.1
-  port: 8080 #If you configure port 443, you'll need to update the listener with tls_certificates
+  address: 0.0.0.0
+  port: 10000
   message_format: huggingface
+  connect_timeout: 0.005s
 
-# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
 llm_providers:
-  - name: OpenAI
-    provider: openai
+  - name: gpt-4o
     access_key: $OPENAI_API_KEY
-    model: gpt-3.5-turbo
-    default: true
+    provider: openai
+    model: gpt-4o
 
-# default system prompt used by all prompt targets
 system_prompt: |
-  You are a network assistant that helps operators with a better understanding of network traffic flow and perform actions on networking operations. No advice on manufacturers or purchasing decisions.
+  You are a helpful assistant.
+
+prompt_guards:
+  input_guards:
+    jailbreak:
+      on_exception:
+        message: Looks like you're curious about my abilities, but I can only provide assistance for currency exchange.
 
 prompt_targets:
-    - name: device_summary
-      description: Retrieve network statistics for specific devices within a time range
-      endpoint:
-        name: app_server
-        path: /agent/device_summary
-      parameters:
-        - name: device_ids
-          type: list
-          description: A list of device identifiers (IDs) to retrieve statistics for.
-          required: true  # device_ids are required to get device statistics
-        - name: days
-          type: int
-          description: The number of days for which to gather device statistics.
-          default: "7"
+  - name: currency_exchange
+    description: Get currency exchange rate from USD to other currencies
+    parameters:
+      - name: currency_symbol
+        description: the currency that needs conversion
+        required: true
+        type: str
+        in_path: true
+    endpoint:
+      name: frankfurther_api
+      path: /v1/latest?base=USD&symbols={currency_symbol}
+    system_prompt: |
+      You are a helpful assistant. Show me the currency symbol you want to convert from USD.
+
+  - name: get_supported_currencies
+    description: Get list of supported currencies for conversion
+    endpoint:
+      name: frankfurther_api
+      path: /v1/currencies
 
-# Arch creates a round-robin load balancing between different endpoints, managed via the cluster subsystem.
 endpoints:
-  app_server:
-    # value could be ip address or a hostname with port
-    # this could also be a list of endpoints for load balancing
-    # for example endpoint: [ ip1:port, ip2:port ]
-    endpoint: host.docker.internal:18083
-    # max time to wait for a connection to be established
-    connect_timeout: 0.005s
+  frankfurther_api:
+    endpoint: api.frankfurter.dev:443
+    protocol: https
 ```
-### Step 3: Using OpenAI Client with Arch as an Egress Gateway
 
-Make outbound calls via Arch
+#### Step 2. Start arch gateway with currency conversion config
+
+```sh
+
+$ archgw up arch_config.yaml
+2024-12-05 16:56:27,979 - cli.main - INFO - Starting archgw cli version: 0.1.5
+...
+2024-12-05 16:56:28,485 - cli.utils - INFO - Schema validation successful!
+2024-12-05 16:56:28,485 - cli.main - INFO - Starging arch model server and arch gateway
+...
+2024-12-05 16:56:51,647 - cli.core - INFO - Container is healthy!
+
+```
+
+Once the gateway is up you can start interacting with at port 10000 using openai chat completion API.
+
+Some of the sample queries you can ask could be `what is currency rate for gbp?` or `show me list of currencies for conversion`.
+
+#### Step 3. Interacting with gateway using curl command
+
+Here is a sample curl command you can use to interact,
+
+```bash
+$ curl --header 'Content-Type: application/json' \
+  --data '{"messages": [{"role": "user","content": "what is exchange rate for gbp"}]}' \
+  http://localhost:10000/v1/chat/completions | jq ".choices[0].message.content"
+
+"As of the date provided in your context, December 5, 2024, the exchange rate for GBP (British Pound) from USD (United States Dollar) is 0.78558. This means that 1 USD is equivalent to 0.78558 GBP."
+
+```
+
+And to get list of supported currencies,
+
+```bash
+$ curl --header 'Content-Type: application/json' \
+  --data '{"messages": [{"role": "user","content": "show me list of currencies that are supported for conversion"}]}' \
+  http://localhost:10000/v1/chat/completions | jq ".choices[0].message.content"
+
+"Here is a list of the currencies that are supported for conversion from USD, along with their symbols:\n\n1. AUD - Australian Dollar\n2. BGN - Bulgarian Lev\n3. BRL - Brazilian Real\n4. CAD - Canadian Dollar\n5. CHF - Swiss Franc\n6. CNY - Chinese Renminbi Yuan\n7. CZK - Czech Koruna\n8. DKK - Danish Krone\n9. EUR - Euro\n10. GBP - British Pound\n11. HKD - Hong Kong Dollar\n12. HUF - Hungarian Forint\n13. IDR - Indonesian Rupiah\n14. ILS - Israeli New Sheqel\n15. INR - Indian Rupee\n16. ISK - Icelandic Króna\n17. JPY - Japanese Yen\n18. KRW - South Korean Won\n19. MXN - Mexican Peso\n20. MYR - Malaysian Ringgit\n21. NOK - Norwegian Krone\n22. NZD - New Zealand Dollar\n23. PHP - Philippine Peso\n24. PLN - Polish Złoty\n25. RON - Romanian Leu\n26. SEK - Swedish Krona\n27. SGD - Singapore Dollar\n28. THB - Thai Baht\n29. TRY - Turkish Lira\n30. USD - United States Dollar\n31. ZAR - South African Rand\n\nIf you want to convert USD to any of these currencies, you can select the one you are interested in."
+
+```
+
+### Use Arch Gateway as LLM Router
+
+#### Step 1. Create arch config file
+
+Arch operates based on a configuration file where you can define LLM providers, prompt targets, guardrails, etc. Below is an example configuration that defines openai and mistral LLM providers.
+
+Create `arch_config.yaml` file with following content:
+
+```yaml
+version: v0.1
+
+listener:
+  address: 0.0.0.0
+  port: 10000
+  message_format: huggingface
+  connect_timeout: 0.005s
+
+llm_providers:
+  - name: gpt-4o
+    access_key: $OPENAI_API_KEY
+    provider: openai
+    model: gpt-4o
+    default: true
+
+  - name: ministral-3b
+    access_key: $MISTRAL_API_KEY
+    provider: mistral
+    model: ministral-3b-latest
+```
+
+#### Step 2. Start arch gateway
+
+Once the config file is created ensure that you have env vars setup for `MISTRAL_API_KEY` and `OPENAI_API_KEY` (or these are defined in `.env` file).
+
+Start arch gateway,
+
+```
+$ archgw up arch_config.yaml
+2024-12-05 11:24:51,288 - cli.main - INFO - Starting archgw cli version: 0.1.5
+2024-12-05 11:24:51,825 - cli.utils - INFO - Schema validation successful!
+2024-12-05 11:24:51,825 - cli.main - INFO - Starting arch model server and arch gateway
+...
+2024-12-05 11:25:16,131 - cli.core - INFO - Container is healthy!
+```
+
+### Step 3: Interact with LLM
+
+#### Step 3.1: Using OpenAI python client
+
+Make outbound calls via Arch gateway
 
 ```python
 from openai import OpenAI
@@ -143,12 +233,59 @@ print("OpenAI Response:", response.choices[0].message.content)
 
 ```
 
-### [Observability](https://docs.archgw.com/guides/observability/observability.html)
+#### Step 3.2: Using curl command
+```
+$ curl --header 'Content-Type: application/json' \
+  --data '{"messages": [{"role": "user","content": "What is the capital of France?"}]}' \
+  http://localhost:12000/v1/chat/completions
+
+{
+  ...
+  "model": "gpt-4o-2024-08-06",
+  "choices": [
+    {
+      ...
+      "message": {
+        "role": "assistant",
+        "content": "The capital of France is Paris.",
+      },
+    }
+  ],
+...
+}
+
+```
+
+You can override model selection using `x-arch-llm-provider-hint` header. For example if you want to use mistral using following curl command,
+
+```
+$ curl --header 'Content-Type: application/json' \
+  --header 'x-arch-llm-provider-hint: ministral-3b' \
+  --data '{"messages": [{"role": "user","content": "What is the capital of France?"}]}' \
+  http://localhost:12000/v1/chat/completions
+{
+  ...
+  "model": "ministral-3b-latest",
+  "choices": [
+    {
+      "message": {
+        "role": "assistant",
+        "content": "The capital of France is Paris. It is the most populous city in France and is known for its iconic landmarks such as the Eiffel Tower, the Louvre Museum, and Notre-Dame Cathedral. Paris is also a major global center for art, fashion, gastronomy, and culture.",
+      },
+      ...
+    }
+  ],
+  ...
+}
+
+```
+
+## [Observability](https://docs.archgw.com/guides/observability/observability.html)
 Arch is designed to support best-in class observability by supporting open standards. Please read our [docs](https://docs.archgw.com/guides/observability/observability.html) on observability for more details on tracing, metrics, and logs. The screenshot below is from our integration with Signoz (among others)
 
 ![alt text](docs/source/_static/img/tracing.png)
 
-### Contribution
+## Contribution
 We would love feedback on our [Roadmap](https://github.com/orgs/katanemo/projects/1) and we welcome contributions to **Arch**!
 Whether you're fixing bugs, adding new features, improving documentation, or creating tutorials, your help is much appreciated.
 Please visit our [Contribution Guide](CONTRIBUTING.md) for more details

arch/arch_config_schema.yaml

@@ -28,6 +28,11 @@ properties:
             type: string
           connect_timeout:
             type: string
+          protocol:
+            type: string
+            enum:
+              - http
+              - https
         additionalProperties: false
         required:
           - endpoint
@@ -92,6 +97,8 @@ properties:
                 type: array
                 items:
                   type: string
+              in_path:
+                type: boolean
             additionalProperties: false
             required:
               - name
@@ -104,6 +111,11 @@ properties:
               type: string
             path:
               type: string
+            http_method:
+              type: string
+              enum:
+                - GET
+                - POST
           additionalProperties: false
           required:
             - name

arch/envoy.template.yaml

@@ -500,7 +500,18 @@ static_resources:
                     socket_address:
                       address: {{ cluster.endpoint }}
                       port_value: {{ cluster.port }}
-                  hostname: {{ cluster.name }}
+                  hostname: {{ cluster.endpoint }}
+      {% if cluster.protocol == "https" %}
+      transport_socket:
+        name: envoy.transport_sockets.tls
+        typed_config:
+          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
+          sni: {{ cluster.endpoint }}
+          common_tls_context:
+            tls_params:
+              tls_minimum_protocol_version: TLSv1_2
+              tls_maximum_protocol_version: TLSv1_3
+      {% endif %}
 {% endfor %}
     - name: arch_internal
       connect_timeout: 5s

arch/tools/cli/config_generator.py

@@ -49,7 +49,9 @@ def validate_and_render_schema():
 
     if "prompt_targets" in config_yaml:
         for prompt_target in config_yaml["prompt_targets"]:
-            name = prompt_target.get("endpoint", {}).get("name", "")
+            name = prompt_target.get("endpoint", {}).get("name", None)
+            if not name:
+                continue
             if name not in inferred_clusters:
                 inferred_clusters[name] = {
                     "name": name,

arch/tools/cli/core.py

@@ -196,7 +196,7 @@ def start_arch_modelserver():
         subprocess.run(
             ["archgw_modelserver", "restart"], check=True, start_new_session=True
         )
-        log.info("Successfull ran model_server")
+        log.info("Successfully ran model_server")
     except subprocess.CalledProcessError as e:
         log.info(f"Failed to start model_server. Please check archgw_modelserver logs")
         sys.exit(1)
@@ -212,7 +212,7 @@ def stop_arch_modelserver():
             ["archgw_modelserver", "stop"],
             check=True,
         )
-        log.info("Successfull stopped the archgw model_server")
+        log.info("Successfully stopped the archgw model_server")
     except subprocess.CalledProcessError as e:
         log.info(f"Failed to start model_server. Please check archgw_modelserver logs")
         sys.exit(1)

arch/tools/cli/main.py

@@ -171,7 +171,7 @@ def up(file, path, service):
         log.info(f"Error: {str(e)}")
         sys.exit(1)
 
-    log.info("Starging arch model server and arch gateway")
+    log.info("Starting arch model server and arch gateway")
 
     # Set the ARCH_CONFIG_FILE environment variable
     env_stage = {}

crates/common/src/configuration.rs

@@ -179,8 +179,6 @@ impl Display for LlmProvider {
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct Endpoint {
     pub endpoint: Option<String>,
-    // pub connect_timeout: Option<DurationString>,
-    // pub timeout: Option<DurationString>,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -193,12 +191,33 @@ pub struct Parameter {
     #[serde(rename = "enum")]
     pub enum_values: Option<Vec<String>>,
     pub default: Option<String>,
+    pub in_path: Option<bool>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Hash, Default)]
+pub enum HttpMethod {
+    #[default]
+    #[serde(rename = "GET")]
+    Get,
+    #[serde(rename = "POST")]
+    Post,
+}
+
+impl Display for HttpMethod {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            HttpMethod::Get => write!(f, "GET"),
+            HttpMethod::Post => write!(f, "POST"),
+        }
+    }
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct EndpointDetails {
     pub name: String,
     pub path: Option<String>,
+    #[serde(rename = "http_method")]
+    pub method: Option<HttpMethod>,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]

crates/common/src/lib.rs

@@ -11,3 +11,4 @@ pub mod routing;
 pub mod stats;
 pub mod tokenizer;
 pub mod tracing;
+pub mod path;

crates/common/src/path.rs

@@ -0,0 +1,82 @@
+use std::collections::HashMap;
+
+pub fn replace_params_in_path(path: &str, params: &HashMap<String, String>) -> Result<String, String> {
+    let mut result = String::new();
+    let mut in_param = false;
+    let mut current_param = String::new();
+
+    for c in path.chars() {
+        if c == '{' {
+            in_param = true;
+        } else if c == '}' {
+            in_param = false;
+            let param_name = current_param.clone();
+            if let Some(value) = params.get(&param_name) {
+                result.push_str(value);
+            } else {
+                return Err(format!("Missing value for parameter `{}`", param_name));
+            }
+            current_param.clear();
+        } else {
+            if in_param {
+                current_param.push(c);
+            } else {
+                result.push(c);
+            }
+        }
+    }
+
+    Ok(result)
+}
+
+#[cfg(test)]
+mod test {
+    #[test]
+    fn test_replace_path() {
+        let path = "/cluster.open-cluster-management.io/v1/managedclusters/{cluster_name}";
+        let params = vec![("cluster_name".to_string(), "test1".to_string())]
+            .into_iter()
+            .collect();
+        assert_eq!(
+            super::replace_params_in_path(path, &params),
+            Ok("/cluster.open-cluster-management.io/v1/managedclusters/test1".to_string())
+        );
+
+        let path = "/cluster.open-cluster-management.io/v1/managedclusters";
+        let params = vec![].into_iter().collect();
+        assert_eq!(
+            super::replace_params_in_path(path, &params),
+            Ok("/cluster.open-cluster-management.io/v1/managedclusters".to_string())
+        );
+
+        let path = "/foo/{bar}/baz";
+        let params = vec![("bar".to_string(), "qux".to_string())]
+            .into_iter()
+            .collect();
+        assert_eq!(
+            super::replace_params_in_path(path, &params),
+            Ok("/foo/qux/baz".to_string())
+        );
+
+        let path = "/foo/{bar}/baz/{qux}";
+        let params = vec![
+            ("bar".to_string(), "qux".to_string()),
+            ("qux".to_string(), "quux".to_string()),
+        ]
+        .into_iter()
+        .collect();
+        assert_eq!(
+            super::replace_params_in_path(path, &params),
+            Ok("/foo/qux/baz/quux".to_string())
+        );
+
+        let path = "/foo/{bar}/baz/{qux}";
+        let params = vec![("bar".to_string(), "qux".to_string())]
+            .into_iter()
+            .collect();
+        assert_eq!(
+            super::replace_params_in_path(path, &params),
+            Err("Missing value for parameter `qux`".to_string())
+        );
+    }
+}

crates/prompt_gateway/src/stream_context.rs

@@ -29,6 +29,7 @@ use derivative::Derivative;
 use http::StatusCode;
 use log::{debug, info, trace, warn};
 use proxy_wasm::traits::*;
+use serde_yaml::Value;
 use std::cell::RefCell;
 use std::collections::HashMap;
 use std::rc::Rc;
@@ -892,9 +893,37 @@ impl StreamContext {
         let endpoint = prompt_target.endpoint.unwrap();
         let path: String = endpoint.path.unwrap_or(String::from("/"));
 
+        // only add params that are of string, number and bool type
+        let url_params = tool_params
+            .iter()
+            .filter(|(_, value)| value.is_number() || value.is_string() || value.is_bool())
+            .map(|(key, value)| match value {
+                Value::Number(n) => (key.clone(), n.to_string()),
+                Value::String(s) => (key.clone(), s.clone()),
+                Value::Bool(b) => (key.clone(), b.to_string()),
+                Value::Null => todo!(),
+                Value::Sequence(_) => todo!(),
+                Value::Mapping(_) => todo!(),
+                Value::Tagged(_) => todo!(),
+            })
+            .collect::<HashMap<String, String>>();
+
+        let path = match common::path::replace_params_in_path(&path, &url_params) {
+            Ok(path) => path,
+            Err(e) => {
+                return self.send_server_error(
+                    ServerError::BadRequest {
+                        why: format!("error replacing params in path: {}", e),
+                    },
+                    Some(StatusCode::BAD_REQUEST),
+                );
+            }
+        };
+
+        let http_method = endpoint.method.unwrap_or_default().to_string();
         let mut headers = vec![
             (ARCH_UPSTREAM_HOST_HEADER, endpoint.name.as_str()),
-            (":method", "POST"),
+            (":method", &http_method),
             (":path", &path),
             (":authority", endpoint.name.as_str()),
             ("content-type", "application/json"),

crates/prompt_gateway/tests/integration.rs

@@ -360,6 +360,7 @@ prompt_targets:
     endpoint:
       name: api_server
       path: /weather
+      http_method: POST
     system_prompt: |
       You are a helpful weather forecaster. Use weater data that is provided to you. Please following following guidelines when responding to user queries:
       - Use farenheight for temperature

demos/currency_exchange/README.md

@@ -0,0 +1 @@
+This demo shows how you can use a publicly hosted rest api and interact it using arch gateway.

demos/currency_exchange/arch_config.yaml

@@ -0,0 +1,52 @@
+version: v0.1
+
+listener:
+  address: 0.0.0.0
+  port: 10000
+  message_format: huggingface
+  connect_timeout: 0.005s
+
+llm_providers:
+  - name: gpt-4o
+    access_key: $OPENAI_API_KEY
+    provider: openai
+    model: gpt-4o
+
+system_prompt: |
+  You are a helpful assistant.
+
+prompt_guards:
+  input_guards:
+    jailbreak:
+      on_exception:
+        message: Looks like you're curious about my abilities, but I can only provide assistance for currency exchange.
+
+prompt_targets:
+  - name: currency_exchange
+    description: Get currency exchange rate from USD to other currencies
+    parameters:
+      - name: currency_symbol
+        description: the currency that needs conversion
+        required: true
+        type: str
+        in_path: true
+    endpoint:
+      name: frankfurther_api
+      path: /v1/latest?base=USD&symbols={currency_symbol}
+    system_prompt: |
+      You are a helpful assistant. Show me the currency symbol you want to convert from USD.
+
+  - name: get_supported_currencies
+    description: Get list of supported currencies for conversion
+    endpoint:
+      name: frankfurther_api
+      path: /v1/currencies
+
+endpoints:
+  frankfurther_api:
+    endpoint: api.frankfurter.dev:443
+    protocol: https
+
+tracing:
+  random_sampling: 100
+  trace_arch_internal: true

demos/currency_exchange/docker-compose.yaml

@@ -0,0 +1,21 @@
+services:
+  chatbot_ui:
+    build:
+      context: ../shared/chatbot_ui
+    ports:
+      - "18080:8080"
+    environment:
+      # this is only because we are running the sample app in the same docker container environemtn as archgw
+      - CHAT_COMPLETION_ENDPOINT=http://host.docker.internal:10000/v1
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - ./arch_config.yaml:/app/arch_config.yaml
+
+  jaeger:
+    build:
+      context: ../shared/jaeger
+    ports:
+      - "16686:16686"
+      - "4317:4317"
+      - "4318:4318"

demos/currency_exchange/run_demo.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+set -e
+
+# Function to start the demo
+start_demo() {
+  # Step 1: Check if .env file exists
+  if [ -f ".env" ]; then
+    echo ".env file already exists. Skipping creation."
+  else
+    # Step 2: Create `.env` file and set OpenAI key
+    if [ -z "$OPENAI_API_KEY" ]; then
+      echo "Error: OPENAI_API_KEY environment variable is not set for the demo."
+      exit 1
+    fi
+
+    echo "Creating .env file..."
+    echo "OPENAI_API_KEY=$OPENAI_API_KEY" > .env
+    echo ".env file created with OPENAI_API_KEY."
+  fi
+
+  # Step 3: Start Arch
+  echo "Starting Arch with arch_config.yaml..."
+  archgw up arch_config.yaml
+
+  # Step 4: Start Network Agent
+  echo "Starting Network Agent using Docker Compose..."
+  docker compose up -d  # Run in detached mode
+}
+
+# Function to stop the demo
+stop_demo() {
+  # Step 1: Stop Docker Compose services
+  echo "Stopping Network Agent using Docker Compose..."
+  docker compose down
+
+  # Step 2: Stop Arch
+  echo "Stopping Arch..."
+  archgw down
+}
+
+# Main script logic
+if [ "$1" == "down" ]; then
+  stop_demo
+else
+  # Default action is to bring the demo up
+  start_demo
+fi

demos/hr_agent/arch_config.yaml

@@ -34,6 +34,7 @@ prompt_targets:
       endpoint:
         name: app_server
         path: /agent/workforce
+        http_method: POST
       parameters:
         - name: staffing_type
           type: str
@@ -52,6 +53,7 @@ prompt_targets:
       endpoint:
         name: app_server
         path: /agent/slack_message
+        http_method: POST
       description: sends a slack message on a channel
       parameters:
         - name: slack_message

demos/insurance_agent/arch_config.yaml

@@ -29,6 +29,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /policy/qa
+      http_method: POST
     description: Handle general Q/A related to insurance.
     default: true
 
@@ -37,6 +38,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /policy/coverage
+      http_method: POST
     parameters:
     - name: policy_type
       type: str
@@ -48,6 +50,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /policy/initiate
+      http_method: POST
     description: Start a policy coverage for car, boat, motorcycle or house.
     parameters:
     - name: policy_type
@@ -64,6 +67,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /policy/claim
+      http_method: POST
     description: Update the notes on the claim
     parameters:
     - name: claim_id
@@ -79,6 +83,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /policy/deductible
+      http_method: POST
     description: Update the deductible amount for a specific policy coverage.
     parameters:
     - name: policy_id

demos/network_agent/arch_config.yaml

@@ -22,6 +22,7 @@ prompt_targets:
       endpoint:
         name: app_server
         path: /agent/device_summary
+        http_method: POST
       parameters:
         - name: device_ids
           type: list
@@ -36,6 +37,7 @@ prompt_targets:
       endpoint:
         name: app_server
         path: /agent/device_reboot
+        http_method: POST
       parameters:
         - name: device_ids
           type: list

demos/weather_forecast/arch_config.yaml

@@ -60,6 +60,7 @@ prompt_targets:
     endpoint:
       name: weather_forecast_service
       path: /weather
+      http_method: POST
 
   - name: default_target
     default: true
@@ -67,6 +68,7 @@ prompt_targets:
     endpoint:
       name: weather_forecast_service
       path: /default_target
+      http_method: POST
     system_prompt: |
       You are a helpful assistant! Summarize the user's request and provide a helpful response.
     # if it is set to false arch will send response that it received from this prompt target to the user

docs/source/guides/includes/arch_config.yaml

@@ -31,6 +31,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /agent/summary
+      http_method: POST
     # Arch uses the default LLM and treats the response from the endpoint as the prompt to send to the LLM
     auto_llm_dispatch_on_response: true
     # override system prompt for this prompt target
@@ -41,6 +42,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /agent/action
+      http_method: POST
     parameters:
       - name: device_id
         type: str

docs/source/resources/includes/arch_config_full_reference.yaml

@@ -77,6 +77,7 @@ prompt_targets:
     endpoint:
       name: app_server
       path: /agent/summary
+      http_method: POST
     # Arch uses the default LLM and treats the response from the endpoint as the prompt to send to the LLM
     auto_llm_dispatch_on_response: true
     # override system prompt for this prompt target

model_server/app/main.py

@@ -220,6 +220,12 @@ async def hallucination(req: HallucinationRequest, res: Response):
     if "messages" in req.parameters:
         req.parameters.pop("messages")
 
+    if not req.parameters or len(req.parameters) == 0:
+        return {
+            "params_scores": {},
+            "model": req.model,
+        }
+
     candidate_labels = {f"{k} is {v}": k for k, v in req.parameters.items()}
 
     predictions = classifier(

model_server/pyproject.toml

@@ -29,7 +29,7 @@ openai = "1.50.2"
 tf-keras = "*"
 onnx = "1.17.0"
 onnxruntime = "1.19.2"
-httpx = "*"
+httpx = "0.27.2" # https://community.openai.com/t/typeerror-asyncclient-init-got-an-unexpected-keyword-argument-proxies/1040287
 pytest-asyncio = "*"
 pytest = "*"
 opentelemetry-api = "^1.28.0"