mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-14 15:52:11 +02:00
feat: LLM-native structured output via JSON schema enforcement (#1037)
Thread existing JSON schemas from prompt definitions through the text-completion service to LLM backends' native structured output APIs. When a prompt has response-type "json" and a strict-mode compatible schema, the LLM constrains token selection at the logit level to guarantee schema-valid output. Wire-level changes: - Add response_format and schema fields to TextCompletionRequest - Update translator to encode/decode new fields - Pass new fields through LlmService, TextCompletionClient, and PromptManager Runtime schema compatibility checker: - New is_strict_mode_compatible() utility validates schemas against LLM provider constraints (additionalProperties, required fields, no unsupported constraints, no open-ended objects) - Per-prompt eligibility decision: compliant schemas use structured output, non-compliant schemas fall back to free-text + post-hoc validation LLM backend implementations: - OpenAI: response_format with json_schema, variant-aware top-level array rejection (openai variant blocks, llama/vllm variants allow) - New vllm variant for the OpenAI backend - vLLM (dedicated): response_format in raw HTTP body - Ollama: format=<schema> parameter - Claude: tool-use trick (forced tool call with schema as input_schema) - Mistral: native json_schema response_format - Llamafile, LM Studio: OpenAI SDK response_format - Azure OpenAI: AzureOpenAI SDK response_format - Azure serverless: response_format in raw HTTP body - TGI: response_format in raw HTTP body - VertexAI Gemini: response_mime_type + response_schema - VertexAI Claude: tool-use trick - Google AI Studio: response_mime_type + response_schema - Bedrock, Cohere: signature-only (no structured output yet) Post-hoc jsonschema.validate() retained as defence-in-depth. Tech spec added: docs/tech-specs/structured-output.md Update tests
This commit is contained in:
cybermaggedon
GitHub
parent
f106ae2103
commit
9136526863
27 changed files with 1089 additions and 71 deletions
263
docs/tech-specs/structured-output.md
Normal file
263
docs/tech-specs/structured-output.md
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
---
|
||||
layout: default
|
||||
title: "Structured Output: LLM-Native JSON Schema Enforcement"
|
||||
parent: "Tech Specs"
|
||||
---
|
||||
|
||||
# Structured Output: LLM-Native JSON Schema Enforcement
|
||||
|
||||
## Problem / Opportunity
|
||||
|
||||
TrustGraph's knowledge-graph pipeline relies on LLMs to produce
|
||||
structured JSON output — entity extractions, relationship triples,
|
||||
topic classifications, and other schema-governed artefacts. Today,
|
||||
the correct structure is requested via natural-language instructions
|
||||
embedded in the prompt template: the prompt describes the expected
|
||||
JSON shape, and the system parses the LLM's free-text response,
|
||||
hoping it conforms.
|
||||
|
||||
This approach has several weaknesses:
|
||||
|
||||
1. **Fragile parsing.** LLM responses may include markdown fencing,
|
||||
preamble text, trailing commentary, or minor schema violations
|
||||
(missing fields, wrong types, extra keys). Every consumer must
|
||||
tolerate or work around these deviations, adding defensive code
|
||||
and retry logic.
|
||||
|
||||
2. **Wasted tokens and latency.** A significant portion of each
|
||||
prompt is spent describing the output format in prose. When the
|
||||
model deviates, retries consume additional tokens and add
|
||||
end-to-end latency.
|
||||
|
||||
3. **Silent data-quality issues.** Malformed responses that pass
|
||||
lenient parsing can inject bad data into the knowledge graph —
|
||||
wrong types, truncated lists, invented field names — without
|
||||
raising errors.
|
||||
|
||||
4. **Untapped LLM capability.** Most modern LLMs (OpenAI, Google
|
||||
Gemini, Anthropic Claude, Ollama-hosted models via llama.cpp)
|
||||
support *structured output* or *guided decoding*: the caller
|
||||
supplies a JSON schema and the model constrains token selection at
|
||||
the logit level to guarantee schema-valid output. TrustGraph
|
||||
already defines the required JSON schemas inside its prompt
|
||||
definitions but does not pass them through to the LLM backend.
|
||||
|
||||
### Opportunity
|
||||
|
||||
By threading the existing JSON schemas from prompt definitions
|
||||
through the text-completion service to each LLM backend's native
|
||||
structured-output API, TrustGraph can:
|
||||
|
||||
- **Guarantee valid output** on every call — no parsing heuristics,
|
||||
no retries for format errors.
|
||||
- **Reduce prompt size** by removing prose format instructions that
|
||||
the schema makes redundant.
|
||||
- **Improve data quality** in the knowledge graph by eliminating an
|
||||
entire class of silent ingestion errors.
|
||||
- **Simplify service code** by removing defensive JSON extraction and
|
||||
validation logic from every consumer.
|
||||
|
||||
## Scope
|
||||
|
||||
Prompt definitions declare a `response-type` of `"text"`, `"json"`,
|
||||
or `"jsonl"`. Structured output applies only to prompts that produce
|
||||
machine-readable output (`"json"` and `"jsonl"`).
|
||||
|
||||
JSONL presents a compatibility challenge: LLM structured-output APIs
|
||||
enforce a single top-level JSON schema, but JSONL prompts ask the
|
||||
model to emit one JSON object per line — a format that is not itself
|
||||
valid JSON. Converting JSONL prompts to request a JSON array would
|
||||
conflict with the prompt text and sacrifice truncation resilience
|
||||
(partial JSONL is recoverable line-by-line; a truncated array is
|
||||
broken JSON).
|
||||
|
||||
This spec takes a three-phase approach:
|
||||
|
||||
- **Phase 1** — plumb schemas through to LLM backends with automatic
|
||||
compatibility detection; non-compliant schemas fall back to the
|
||||
current free-text path.
|
||||
- **Phase 2** — fix up non-compliant schemas so more prompts benefit.
|
||||
- **Phase 3** — address JSONL prompts.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Structured Output with Automatic Fallback
|
||||
|
||||
### Design
|
||||
|
||||
Phase 1 threads the JSON schema from the prompt definition through
|
||||
the text-completion service to the LLM backend's native
|
||||
structured-output API. Only prompts with `response-type: "json"` are
|
||||
candidates.
|
||||
|
||||
Not all existing schemas are compatible with LLM structured-output
|
||||
APIs. Rather than require schema changes up front, Phase 1 includes
|
||||
a **runtime compatibility check**: if a schema passes, structured
|
||||
output is used; if not, the prompt falls back to the current
|
||||
free-text path with post-hoc validation. This makes the feature
|
||||
safe to deploy immediately.
|
||||
|
||||
### Strict-Mode Schema Requirements
|
||||
|
||||
LLM providers impose constraints beyond standard JSON Schema
|
||||
validation. A schema is considered strict-mode compatible when:
|
||||
|
||||
- Every `object` has `additionalProperties: false`.
|
||||
- Every property defined in `properties` appears in `required`.
|
||||
Optional fields use a nullable type (e.g. `"type": ["string", "null"]`)
|
||||
instead of omitting the key from `required`.
|
||||
- No `minimum`, `maximum`, `minLength`, `maxLength`, or `pattern`
|
||||
constraints (unsupported by most providers' constrained decoders).
|
||||
- No open-ended objects (`{"type": "object"}` without `properties`).
|
||||
- A schema is present and non-null.
|
||||
|
||||
### Runtime Compatibility Check
|
||||
|
||||
`PromptManager` (or a shared utility) inspects each schema at load
|
||||
time against the strict-mode rules above. The result is a boolean
|
||||
flag per prompt: `structured_output_eligible`.
|
||||
|
||||
- **Eligible** — `response_format` and `schema` are set on the
|
||||
`TextCompletionRequest`; the LLM enforces the schema at generation
|
||||
time.
|
||||
- **Not eligible** — request is sent without schema fields; the
|
||||
current free-text parsing and `jsonschema.validate()` path is used.
|
||||
|
||||
This is a per-prompt decision, not a global switch.
|
||||
|
||||
### Text-Completion Request Changes
|
||||
|
||||
`TextCompletionRequest` gains two optional fields:
|
||||
|
||||
```
|
||||
TextCompletionRequest:
|
||||
system: str
|
||||
prompt: str
|
||||
streaming: bool
|
||||
response_format: str | None # "json" or None (default)
|
||||
schema: dict | None # JSON Schema object or None
|
||||
```
|
||||
|
||||
When `response_format` is `"json"` and `schema` is provided, the LLM
|
||||
backend MUST use its native structured-output mechanism. When either
|
||||
field is absent or null, behaviour is unchanged.
|
||||
|
||||
### LLM Backend Mapping
|
||||
|
||||
Each backend maps `response_format` + `schema` to its provider's
|
||||
native API:
|
||||
|
||||
| Backend | API mechanism |
|
||||
|------------|-------------------------------------------------------|
|
||||
| OpenAI | `response_format={"type": "json_schema", "json_schema": {"name": "...", "schema": ...}}` |
|
||||
| Claude | `tool_use` with a single tool whose `input_schema` is the target schema, or the `response_format` parameter when available |
|
||||
| Gemini | `response_mime_type="application/json"` + `response_schema=...` |
|
||||
| Ollama | `format="json"` + schema in the `format` field (llama.cpp guided decoding) |
|
||||
| Llamafile | `response_format={"type": "json_object"}` + schema |
|
||||
|
||||
Backends that do not support schema-level enforcement (e.g. older
|
||||
Ollama versions) fall back to `response_format=json` without a schema
|
||||
and rely on post-hoc validation.
|
||||
|
||||
### Current Prompt Compatibility
|
||||
|
||||
Of the nine `response-type: "json"` prompts, two are strict-mode
|
||||
compatible today:
|
||||
|
||||
| Prompt | Status | Issue |
|
||||
|--------------------------|-----------|------------------------------------|
|
||||
| `schema-selection` | Ready | — |
|
||||
| `supervisor-decompose` | Ready | — |
|
||||
| `plan-create` | Fixable | Optional fields not in `required` |
|
||||
| `graphql-generation` | Blocked | Open-ended `variables` object; `minimum`/`maximum` on `confidence` |
|
||||
| `plan-step-execute` | Blocked | Open-ended `arguments` object |
|
||||
| `diagnose-structured-data` | No schema | — |
|
||||
| `diagnose-xml` | No schema | — |
|
||||
| `diagnose-json` | No schema | — |
|
||||
| `diagnose-csv` | No schema | — |
|
||||
|
||||
### What Does Not Change
|
||||
|
||||
- Prompt templates and their text content.
|
||||
- The `"text"` and `"jsonl"` response-type paths.
|
||||
- The `TextCompletionResponse` schema.
|
||||
- Post-hoc validation (retained as a defence-in-depth measure).
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Schema Remediation
|
||||
|
||||
Phase 2 expands structured-output coverage by fixing schemas that
|
||||
failed the Phase 1 compatibility check.
|
||||
|
||||
### Fixable Schemas
|
||||
|
||||
**`plan-create`** — `tool_hint` and `depends_on` are optional
|
||||
(present in `properties` but absent from `required`). Fix: add both
|
||||
to `required` and change their types to nullable:
|
||||
|
||||
```json
|
||||
"tool_hint": {"type": ["string", "null"]},
|
||||
"depends_on": {
|
||||
"type": ["array", "null"],
|
||||
"items": {"type": "integer"}
|
||||
}
|
||||
```
|
||||
|
||||
### Schemas Requiring Design Decisions
|
||||
|
||||
**`graphql-generation`** — Two issues:
|
||||
|
||||
- `variables` is an open-ended object (`"additionalProperties": true`)
|
||||
with no fixed properties. Constrained decoding cannot handle
|
||||
arbitrary keys. Options: remove `variables` from the schema and
|
||||
accept it as free-form text within a wrapper, or restructure as a
|
||||
JSON-encoded string field.
|
||||
- `confidence` uses `"minimum": 0.0, "maximum": 1.0`. Fix: remove
|
||||
the numeric bounds; accept any number and clamp in application code
|
||||
if needed.
|
||||
|
||||
**`plan-step-execute`** — `arguments` is an open-ended object with no
|
||||
fixed properties. Same constraint as `graphql-generation.variables`.
|
||||
|
||||
### Missing Schemas
|
||||
|
||||
The four `diagnose-*` prompts have `response-type: "json"` but no
|
||||
schema. Adding schemas for these prompts would bring them into
|
||||
structured-output scope. This requires defining the expected
|
||||
response shape for each diagnostic prompt.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 (Future) — Structured Output for JSONL Prompts
|
||||
|
||||
JSONL prompts ask the LLM to emit multiple JSON objects, one per
|
||||
line. Each object is validated individually against the prompt's
|
||||
schema. The current approach is tolerant of truncation and
|
||||
malformed lines — useful properties for large extractions.
|
||||
|
||||
### Options
|
||||
|
||||
**Option A — Array wrapper.** Change the prompt text to request a
|
||||
JSON array instead of JSONL. Wrap the schema as
|
||||
`{"type": "array", "items": <existing-schema>}` and use structured
|
||||
output. Trade-off: loses line-by-line truncation resilience; requires
|
||||
updating every JSONL prompt template.
|
||||
|
||||
**Option B — Structured output per chunk.** Split the input so each
|
||||
text-completion call produces a single JSON object, then aggregate.
|
||||
Trade-off: more LLM calls; higher latency and cost; may not suit
|
||||
prompts that extract variable-length lists from a single chunk.
|
||||
|
||||
**Option C — Hybrid.** Use structured output with the array-wrapped
|
||||
schema but retain the post-hoc JSONL parser as a fallback for
|
||||
backends that do not support structured output or when the response
|
||||
is truncated. Trade-off: two code paths to maintain.
|
||||
|
||||
**Option D — Status quo.** Leave JSONL prompts on the free-text path
|
||||
with post-hoc validation. Structured output for `"json"` prompts
|
||||
already covers the most schema-sensitive cases; JSONL extraction is
|
||||
inherently more tolerant of partial results.
|
||||
|
||||
Phase 3 design will be selected after earlier phases are deployed and
|
||||
real-world structured-output behaviour is observed across backends.
|
||||
Loading…
Add table
Add a link
Reference in a new issue