From f64b5a92c85ebfba7d337150e412d84f60c90832 Mon Sep 17 00:00:00 2001 From: Andrey Avtomonov Date: Fri, 15 May 2026 12:22:58 +0200 Subject: [PATCH] Refine claude-code backend brainstorm after adversarial review iteration 3 --- .../2026-05-15-claude-code-backend-design.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-05-15-claude-code-backend-design.md b/docs/superpowers/specs/2026-05-15-claude-code-backend-design.md index cdf174ee..ee7006ff 100644 --- a/docs/superpowers/specs/2026-05-15-claude-code-backend-design.md +++ b/docs/superpowers/specs/2026-05-15-claude-code-backend-design.md @@ -45,6 +45,15 @@ ktx ingest // are loaded and re-prove that filesystem MCP/hook/skill // discovery still cannot add reachable tools. settingSources: [], + // Skills are independently controlled by the SDK's `skills` + // option. Per the SDK types, omitting `skills` leaves CLI + // defaults in effect (NOT "skills off"); passing `string[]` + // enables only the listed skills. To prove the "exactly + // curated KTX tools" boundary, the backend MUST explicitly + // disable SDK skills here (e.g. `skills: []` or the current + // SDK-equivalent against the pinned version) in addition to + // `settingSources: []` and built-in tool disabling. + skills: [], mcpServers: { ktx: createSdkMcpServer({ tools: }) }, // Make ONLY KTX MCP tools reachable. Two independent layers: // 1. Disable built-in tools entirely (the current SDK @@ -80,7 +89,7 @@ The Claude Agent SDK is documented to reuse local Claude Code authentication aut | Q3 | KTX MCP tools only; Claude Code built-ins (`Bash`/`Read`/`Edit`/`Write`/`Grep`/`Glob`/`WebFetch`/`Task`) must be unreachable to the model. The exact SDK mechanism (e.g. `disallowedTools`, `tools` configuration, `canUseTool` / permission mode) is an open item for the plan — `allowedTools` alone is auto-approval, not a restriction | Preserves current ingest determinism and blast-radius limits; tool set continues to come from each stage's curated set | | Q4 | New `ClaudeAgentSdkRunnerService` class alongside existing `AgentRunnerService`; both implement the same `runLoop` shape | Avoids polluting the AI-SDK runner with conditional dead deps; clean per-runner deps shape; both call sites in `stage-3-work-units.ts:91` etc. are untouched | | `cwd` | Explicit `cwd: project.projectDir` (resolved at startup via `resolveKtxProjectDir`, not `process.cwd()`) | SDK's `cwd` is semantic (skills, `CLAUDE.md`, file checkpointing); KTX's existing convention is to anchor on `projectDir` regardless of invocation directory | -| Tool adapter | A backend-neutral tool boundary that preserves enough KTX tool definition data to build an SDK `tool(name, description, zodSchema, handler)` for each entry. Today `RunLoopParams.toolSet` is `Record` (AI SDK type) and source-specific tools (e.g. `emit_historic_sql_evidence`) are already raw AI SDK `Tool` objects, not `BaseTool` instances (`packages/context/src/ingest/local-bundle-runtime.ts:543-556`); the runner alone cannot recover the original Zod input schema and KTX handler from those. The plan must either (a) extend the toolset port (`createIngestWuToolset`, equivalents in memory/scan toolsets) so it returns a per-tool descriptor with `name`, `description`, `inputSchema` (a Zod **object** — see schema-shape note), and the KTX handler — convertible to **either** AI SDK or Claude Agent SDK tools at the boundary — and adapt source-specific raw tools to that descriptor, or (b) require both shapes to be produced upstream. `toClaudeAgentSdkTool()` on `BaseTool` is fine for the BaseTool subset but is not sufficient on its own. **Schema-shape note:** KTX's `BaseTool.inputSchema` is typed as `ZodType` (`packages/context/src/tools/base-tool.ts:74-85`) and AI SDK accepts that directly, but the Claude Agent SDK `tool()` helper requires a raw Zod object shape (`AnyZodRawShape`), not an arbitrary `ZodType`. The descriptor contract must therefore (i) constrain `inputSchema` to a `ZodObject` (in practice all KTX tool inputs already are `z.object({...})`) and (ii) extract `.shape` at the Claude-SDK boundary. Non-object tool schemas are unsupported on this backend and must be rejected at startup with a clear error rather than silently mis-adapted. | KTX tools return `{ markdown, structured }`; Claude Agent SDK's `tool()` expects `{ content: [{ type: 'text', text }] }` — flattening `markdown` is straightforward once the underlying schema + handler are reachable | +| Tool adapter | A backend-neutral tool boundary that preserves enough KTX tool definition data to build an SDK `tool(name, description, zodSchema, handler)` for **every entry in the final `RunLoopParams.toolSet`**, not just the toolset-factory outputs. Today `RunLoopParams.toolSet` is `Record` (AI SDK type) and is composed in stage code from multiple sources: factory toolsets (`createIngestWuToolset`, memory toolsets), per-source raw AI SDK tools (e.g. `emit_historic_sql_evidence` at `packages/context/src/ingest/local-bundle-runtime.ts:543-556`), and **stage-local raw AI SDK tools added after the factories** — `load_skill` (`packages/context/src/ingest/ingest-bundle.runner.ts:697-720` for WU runs, `:924-941` for reconcile, `packages/context/src/memory/memory-agent.service.ts:128-156` for the memory agent), `read_raw_file` / `read_raw_span`, `stage_list` / `stage_diff`, `eviction_list`, `emit_*` reconciliation tools (see `packages/context/src/ingest/stages/build-wu-context.ts:87-123` and `packages/context/src/ingest/stages/build-reconcile-context.ts:184-212`), plus the `withVerificationLedger` wrapping layer. The runner alone cannot recover the original Zod input schema and KTX handler from any of those. The plan must either (a) define the backend-neutral descriptor as the contract of the **final** `RunLoopParams` tool map — i.e. every entry, including `buildWuToolSet` / `buildReconcileToolSet` outputs, source-specific raw tools, stage-local `load_skill` / read / stage / eviction / emit tools, and verification-ledger wrappers, must expose `name`, `description`, `inputSchema` (a Zod **object** — see schema-shape note), and the KTX handler so it can be converted to **either** AI SDK or Claude Agent SDK tools at the boundary, or (b) require both AI-SDK and Claude-SDK shapes to be produced in paired form at every composition site after stage-local additions and logging wrappers. `toClaudeAgentSdkTool()` on `BaseTool` is fine for the BaseTool subset but is not sufficient on its own. **Schema-shape note:** KTX's `BaseTool.inputSchema` is typed as `ZodType` (`packages/context/src/tools/base-tool.ts:74-85`) and AI SDK accepts that directly, but the Claude Agent SDK `tool()` helper requires a raw Zod object shape (`AnyZodRawShape`), not an arbitrary `ZodType`. The descriptor contract must therefore (i) constrain `inputSchema` to a `ZodObject` (in practice all KTX tool inputs already are `z.object({...})`) and (ii) extract `.shape` at the Claude-SDK boundary. Non-object tool schemas are unsupported on this backend and must be rejected at startup with a clear error rather than silently mis-adapted. | KTX tools return `{ markdown, structured }`; Claude Agent SDK's `tool()` expects `{ content: [{ type: 'text', text }] }` — flattening `markdown` is straightforward once the underlying schema + handler are reachable | | Q5 | MVP: degraded repair + no telemetry; both documented as known gaps | Fastest path to a working backend; correctness preserved (model self-corrects); follow-up wires both through SDK hooks if/when needed | | Naming | Config value: `'claude-code'` | Names the user-facing thing (the Claude Code session they already authenticated); fits enum semantics (each value names an auth/API surface); avoids productizing the Max subscription | @@ -89,9 +98,9 @@ The Claude Agent SDK is documented to reuse local Claude Code authentication aut The plan should touch (at minimum) these areas. This is a sketch, not the plan. - **Config schema** — depending on the open scope decision (see top of doc), either (a) extend `KtxLlmBackend` in `packages/llm/src/types.ts` and explicitly handle the new value in `createKtxLlmProvider` / `createModelFactory` (`packages/llm/src/model-provider.ts:155-186`) so it does not silently fall through to gateway, **and** at every non-agent LLM consumer (page triage, scan enrichment, scan description generation, relationship LLM proposals); or (b) leave `KtxLlmBackend` alone and add a separate agent-runner backend field to `KtxProjectLlmConfig` whose `'claude-code'` value is consumed only at the agent-runner DI boundary. -- **Tool boundary** — make the per-stage toolset port return descriptors that preserve `name`, `description`, Zod input schema, and the KTX handler so either an AI SDK tool or a Claude Agent SDK `tool()` can be built at the consumer. Touch `LocalIngestToolsetFactory.createIngestWuToolset` and the memory/scan toolset equivalents (`packages/context/src/ingest/local-bundle-runtime.ts:543-556`, `packages/context/src/memory/types.ts:120-126`). Source-specific raw AI SDK tools must be wrapped to the same descriptor shape. `BaseTool.toAiSdkTool()` (`:117-165`) stays; a parallel `toClaudeAgentSdkTool()` may live alongside it but is not the whole solution. +- **Tool boundary** — make the contract of the **final** `RunLoopParams.toolSet` (the map actually handed to `runLoop`, after all stage-local composition and logging wrappers) the descriptor boundary: every entry must preserve `name`, `description`, Zod input schema (constrained to `ZodObject`), and the KTX handler so either an AI SDK tool or a Claude Agent SDK `tool()` can be built at the consumer. This is wider than the factory-level boundary alone. Touch (at minimum): the per-stage toolset ports `LocalIngestToolsetFactory.createIngestWuToolset` and memory/scan equivalents (`packages/context/src/ingest/local-bundle-runtime.ts:543-556`, `packages/context/src/memory/types.ts:120-126`); the stage-local composition in `buildWuToolSet` (`packages/context/src/ingest/stages/build-wu-context.ts:87-123`) and `buildReconcileToolSet` (`packages/context/src/ingest/stages/build-reconcile-context.ts:184-212`); the stage-local raw AI SDK tools constructed inline in `IngestBundleRunner` (`load_skill`, `emit_unmapped_fallback`, and read/raw/stage/eviction/emit tools at `packages/context/src/ingest/ingest-bundle.runner.ts:697-720` and `:924-941`); the memory-agent `load_skill` (`packages/context/src/memory/memory-agent.service.ts:128-156`); and the `withVerificationLedger` wrapper. Source-specific raw AI SDK tools must be wrapped to the same descriptor shape. `BaseTool.toAiSdkTool()` (`:117-165`) stays; a parallel `toClaudeAgentSdkTool()` may live alongside it but is not the whole solution. - **Runner port** — introduce an `AgentRunnerPort` interface (e.g. in `packages/context/src/agent/`) with `runLoop(params: RunLoopParams): Promise` and retype `IngestBundleRunnerDeps.agentRunner` (`packages/context/src/ingest/ports.ts:353`) and `MemoryAgentDeps.agentRunner` (`packages/context/src/memory/types.ts:153`) — plus any other field annotated as the concrete `AgentRunnerService` — to the port. `AgentRunnerService` and the new `ClaudeAgentSdkRunnerService` both implement the port. -- **`packages/context/src/agent/`** — add `claude-agent-sdk-runner.service.ts` exposing `ClaudeAgentSdkRunnerService` with the same `runLoop(params: RunLoopParams)` shape as `AgentRunnerService`. Internals: register the curated KTX tools via `createSdkMcpServer` and `mcpServers`, set `cwd`, `systemPrompt`, `maxTurns: stepBudget`, **set `settingSources: []`** (or whichever value the plan settles on after re-reading the SDK types — the requirement is that no user/project filesystem settings, MCP servers, hooks, skills, plugins, or slash commands are loaded), disable built-in tools, restrict reachable tools to `mcp__ktx__*` (mechanism per the open Q3 item — not `allowedTools` alone), and consume the async iterator to detect stop conditions and map onto `RunLoopResult`. +- **`packages/context/src/agent/`** — add `claude-agent-sdk-runner.service.ts` exposing `ClaudeAgentSdkRunnerService` with the same `runLoop(params: RunLoopParams)` shape as `AgentRunnerService`. Internals: register the curated KTX tools via `createSdkMcpServer` and `mcpServers`, set `cwd`, `systemPrompt`, `maxTurns: stepBudget`, **set `settingSources: []`** (or whichever value the plan settles on after re-reading the SDK types — the requirement is that no user/project filesystem settings, MCP servers, hooks, skills, plugins, or slash commands are loaded), **explicitly disable SDK skills** (e.g. `skills: []` against the pinned SDK version — omitting the option leaves CLI defaults in effect, not "off"), disable built-in tools, restrict reachable tools to `mcp__ktx__*` (mechanism per the open Q3 item — not `allowedTools` alone), and consume the async iterator to detect stop conditions and map onto `RunLoopResult` (using `SDKResultMessage` as the authoritative stop-reason source — see open item 4). - **DI wiring** — modify `resolveAgentRunner` in `packages/context/src/ingest/local-bundle-runtime.ts:580-604` and the agent-runner construction path in `packages/context/src/memory/local-memory.ts:92-110` to branch on the resolved agent-runner backend and construct `ClaudeAgentSdkRunnerService` instead of `AgentRunnerService` when applicable. All call sites (`stage-3-work-units.ts:91`, memory agent, etc.) receive the chosen runner via DI (typed as `AgentRunnerPort`) and do not change behavior. - **Setup / config validation** — when the user selects `claude-code` in `ktx setup`, verify that the local Claude Code SDK auth is **usable**, not just that `~/.claude/` exists. SDK docs establish that the SDK reuses authentication automatically when the user has run `claude` to authenticate; they do not establish directory probing as a sufficient liveness test. The plan must define a usability check (e.g. a minimal SDK probe call that exercises auth, an SDK-provided auth-status helper if one exists, or a documented file-presence check that the SDK docs explicitly endorse). Pure existence-of-`~/.claude/` is not sufficient on its own. - **Docs** — `docs-site/content/docs/concepts/` and `docs-site/content/docs/getting-started/` need a section on the `claude-code` backend, framed as "use your own local Claude Code session." Avoid productizing-Max-sub language. @@ -121,5 +130,5 @@ Real questions the plan will need to answer that we did not lock during brainsto 1. **Model selection per role.** Today KTX has `KtxModelRole = 'default' | 'triage' | 'candidateExtraction' | 'curator' | 'reconcile' | 'repair'` with per-role model IDs. Claude Agent SDK's `query()` accepts a single `model` string per call. The plan needs to decide whether the `claude-code` backend (a) maps each role to a specific Claude model ID per call, (b) uses a single configured model for all roles, or (c) reads role-to-model mapping from the same `ktx.yaml` shape used by other backends. The `'repair'` role specifically is degraded under Q5=A, but the rest still need a binding strategy. 2. **Auth presence check.** Before the first `query()` call, KTX should fail fast with a clear message if the local Claude Code SDK auth is not usable. The check must be a usability test, not just `~/.claude/` directory probing — directory presence does not prove the SDK can actually authenticate (see implementation surface). The detection mechanism (SDK probe call, SDK-provided helper, or a docs-endorsed file-presence test) is open. 3. **`ktx.yaml` schema migration & non-agent LLM consumers.** Decide between the two scope options at the top of this document (global `KtxLlmBackend` extension vs. separate agent-runner backend field). If extending `KtxLlmBackend`: update zod schemas under `packages/context/src/project/`, the setup wizard (`packages/cli/src/setup-models.ts`, `packages/cli/src/commands/setup-commands.ts`), `createKtxLlmProvider` / `createModelFactory` (`packages/llm/src/model-provider.ts:155-186`), and define behavior at every non-agent LLM consumer (`packages/context/src/ingest/page-triage/page-triage.service.ts`, `packages/context/src/scan/local-scan.ts`, `packages/context/src/scan/description-generation.ts`, `packages/context/src/scan/relationship-discovery.ts`). If adding a separate field: only the setup wizard, the project zod schema, and the agent-runner DI factories need to change. -4. **Stop-reason mapping.** The Agent SDK exposes session lifecycle via the async iterator and the `Stop` hook event. The plan needs to define how a Claude Agent SDK session maps to KTX's `RunLoopStopReason = 'budget' | 'natural' | 'error'` (`agent-runner.service.ts:6`). In particular: how to detect that `maxTurns` was hit vs natural completion vs error. +4. **Stop-reason mapping.** The authoritative stop-reason source is the SDK's typed result message (`SDKResultMessage`) emitted by the `query()` async iterator, **not** the `Stop` hook event. The `Stop` hook input only carries `stop_hook_active` and the last assistant text — it does not carry a terminal reason, so it is insufficient for mapping. The `SDKResultMessage` subtypes include `error_max_turns` and a `terminal_reason` field with values like `'max_turns' | 'completed' | …`. Default mapping onto KTX's `RunLoopStopReason = 'budget' | 'natural' | 'error'` (`agent-runner.service.ts:6`): `error_max_turns` or `terminal_reason === 'max_turns'` → `'budget'`; successful completed result → `'natural'`; any other error subtype or non-completed terminal reason → `'error'`. The plan must confirm the exact field names and enum values against the pinned SDK version and lock the mapping table. `Stop` hooks remain a lifecycle side-channel (useful for, e.g., tool-failure tallying via `PostToolUseFailure`), not a stop-reason source. 5. **Tool failure counting.** `stage-3-work-units.ts:132` reads `toolFailureCount?(wu.unitKey)` to fail a WU when any tool call failed. The new runner needs to surface tool failures via the same counting mechanism. The `PostToolUseFailure` hook is the natural integration point.