Tech Overview

• Terminology
• Threading Model
• Listener
  • Downstream (Ingress)
  • Upstream (Egress)
  • Configure Listener
• Model Serving
  • Local Serving (CPU - Moderate)
  • Local Serving (GPU- Fast)
  • Cloud Serving (GPU - Blazing Fast)
• Prompt
  • Messages
  • Prompt Guardrails
  • Prompt Targets
  • Prompting LLMs
• Request Lifecycle
  • Terminology
  • Network topology
  • High level architecture
  • Configuration
  • Request Flow (Ingress)
  • Request Flow (Egress)
  • Overview