Thread existing JSON schemas from prompt definitions through the
text-completion service to LLM backends' native structured output
APIs. When a prompt has response-type "json" and a strict-mode
compatible schema, the LLM constrains token selection at the logit
level to guarantee schema-valid output.
Wire-level changes:
- Add response_format and schema fields to TextCompletionRequest
- Update translator to encode/decode new fields
- Pass new fields through LlmService, TextCompletionClient, and
PromptManager
Runtime schema compatibility checker:
- New is_strict_mode_compatible() utility validates schemas against
LLM provider constraints (additionalProperties, required fields,
no unsupported constraints, no open-ended objects)
- Per-prompt eligibility decision: compliant schemas use structured
output, non-compliant schemas fall back to free-text + post-hoc
validation
LLM backend implementations:
- OpenAI: response_format with json_schema, variant-aware top-level
array rejection (openai variant blocks, llama/vllm variants allow)
- New vllm variant for the OpenAI backend
- vLLM (dedicated): response_format in raw HTTP body
- Ollama: format=<schema> parameter
- Claude: tool-use trick (forced tool call with schema as input_schema)
- Mistral: native json_schema response_format
- Llamafile, LM Studio: OpenAI SDK response_format
- Azure OpenAI: AzureOpenAI SDK response_format
- Azure serverless: response_format in raw HTTP body
- TGI: response_format in raw HTTP body
- VertexAI Gemini: response_mime_type + response_schema
- VertexAI Claude: tool-use trick
- Google AI Studio: response_mime_type + response_schema
- Bedrock, Cohere: signature-only (no structured output yet)
Post-hoc jsonschema.validate() retained as defence-in-depth.
Tech spec added: docs/tech-specs/structured-output.md
Update tests
A previous commit moved SDK imports into __init__/methods and
stashed them on self, which broke @patch targets in 24 unit tests.
This fixes the approach: chunker and pdf_decoder use module-level
sentinels with global/if-None guards so imports are still deferred but
patchable. Google AI Studio reverts to standard module-level imports
since the module is only loaded when communicating with Gemini.
Keeps lazy loading on other imports.
* Changed schema for Value -> Term, majorly breaking change
* Following the schema change, Value -> Term into all processing
* Updated Cassandra for g, p, s, o index patterns (7 indexes)
* Reviewed and updated all tests
* Neo4j, Memgraph and FalkorDB remain broken, will look at once settled down
* Tidy up duplicate tech specs in doc directory
* Streaming LLM text-completion service tech spec.
* text-completion and prompt interfaces
* streaming change applied to all LLMs, so far tested with VertexAI
* Skip Pinecone unit tests, upstream module issue is affecting things, tests are passing again
* Added agent streaming, not working and has broken tests
* Added Anthropic support for VertexAI
* Update tests to match code
* Fixed private.json usage with Anthropic (I think).
* Fixed test
---------
Co-authored-by: Cyber MacGeddon <cybermaggedon@gmail.com>
- Changed GoogleAIStudio LLM code to match latest documentation
- Very minor tweak to vertexai LLM code - just matching what's in SDK docs
no actual change to implementation.
- Tweaked VertexAI container build to speed up in dev
- Comments in LLM code to mention which docs it was built from. Google
SDKs are confusing ATM.
- Keeps processing in different flows separate so that data can go to different stores / collections etc.
- Potentially supports different processing flows
- Tidies the processing API with common base-classes for e.g. LLMs, and automatic configuration of 'clients' to use the right queue names in a flow
* - Refactored retry for rate limits into the base class
- ConsumerProducer is derived from Consumer to simplify code
- Added rate_limit_count metrics for rate limit events
* Add rate limit events to VertexAI and Google AI Studio
* Added Grafana rate limit dashboard
* Add rate limit handling to all LLMs
