mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-14 07:42:11 +02:00
feat: LLM-native structured output via JSON schema enforcement (#1037)
Thread existing JSON schemas from prompt definitions through the text-completion service to LLM backends' native structured output APIs. When a prompt has response-type "json" and a strict-mode compatible schema, the LLM constrains token selection at the logit level to guarantee schema-valid output. Wire-level changes: - Add response_format and schema fields to TextCompletionRequest - Update translator to encode/decode new fields - Pass new fields through LlmService, TextCompletionClient, and PromptManager Runtime schema compatibility checker: - New is_strict_mode_compatible() utility validates schemas against LLM provider constraints (additionalProperties, required fields, no unsupported constraints, no open-ended objects) - Per-prompt eligibility decision: compliant schemas use structured output, non-compliant schemas fall back to free-text + post-hoc validation LLM backend implementations: - OpenAI: response_format with json_schema, variant-aware top-level array rejection (openai variant blocks, llama/vllm variants allow) - New vllm variant for the OpenAI backend - vLLM (dedicated): response_format in raw HTTP body - Ollama: format=<schema> parameter - Claude: tool-use trick (forced tool call with schema as input_schema) - Mistral: native json_schema response_format - Llamafile, LM Studio: OpenAI SDK response_format - Azure OpenAI: AzureOpenAI SDK response_format - Azure serverless: response_format in raw HTTP body - TGI: response_format in raw HTTP body - VertexAI Gemini: response_mime_type + response_schema - VertexAI Claude: tool-use trick - Google AI Studio: response_mime_type + response_schema - Bedrock, Cohere: signature-only (no structured output yet) Post-hoc jsonschema.validate() retained as defence-in-depth. Tech spec added: docs/tech-specs/structured-output.md Update tests
This commit is contained in:
parent
f106ae2103
commit
9136526863
27 changed files with 1089 additions and 71 deletions
|
|
@ -247,7 +247,10 @@ class Processor(LlmService):
|
|||
|
||||
return self.model_variants[cache_key]
|
||||
|
||||
async def generate_content(self, system, prompt, model=None, temperature=None):
|
||||
async def generate_content(
|
||||
self, system, prompt, model=None, temperature=None,
|
||||
response_format=None, schema=None,
|
||||
):
|
||||
|
||||
# Use provided model or fall back to default
|
||||
model_name = model or self.default_model
|
||||
|
|
@ -311,7 +314,10 @@ class Processor(LlmService):
|
|||
"""Bedrock supports streaming"""
|
||||
return True
|
||||
|
||||
async def generate_content_stream(self, system, prompt, model=None, temperature=None):
|
||||
async def generate_content_stream(
|
||||
self, system, prompt, model=None, temperature=None,
|
||||
response_format=None, schema=None,
|
||||
):
|
||||
"""Stream content generation from Bedrock"""
|
||||
model_name = model or self.default_model
|
||||
effective_temperature = temperature if temperature is not None else self.temperature
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue