trustgraph/ts/EFFECT_NATIVE_REWRITE_PLAN.md

# TrustGraph TS Effect-Native Rewrite Plan

## Summary

Bring the TypeScript port to a genuinely Effect-native shape while keeping existing TrustGraph capabilities working. The current native MCP rewrite should be preserved. The remaining work is to remove Promise-first TrustGraph APIs and replace manual host/framework plumbing with Effect v4-native backend, messaging, processor, HTTP/RPC, and CLI layers.

The implementation should proceed in green checkpoints rather than one broad sweep. Each stage should leave the repo closer to Effect-native and should run the smallest relevant gate before moving on.

## Non-Negotiable Architecture Decisions

- TrustGraph-authored production APIs should be Effect-first, not Promise-first.
- Remove exported Promise APIs from backend, messaging, processor, flow service, gateway, CLI, and service runner surfaces.
- Do not keep dual Promise and Effect methods on the same TrustGraph objects.
- Do not rely on Fastify, `@fastify/websocket`, Commander, or production `Effect.runPromise` / `ManagedRuntime` compatibility bridges.
- External JavaScript SDKs that are inherently Promise-based may be wrapped with `Effect.tryPromise`, but that Promise shape must stay behind Effect APIs.
- Gateway HTTP and WebSocket behavior should move to Effect v4 native modules: `effect/unstable/httpapi`, `effect/unstable/http`, `effect/unstable/rpc`, `effect/unstable/socket`, and platform Bun/Node HTTP/socket layers.
- CLI behavior should move to `effect/unstable/cli`.
- The MCP package should stay on the native `effect/unstable/ai/McpServer` implementation and must not reintroduce server-side `@modelcontextprotocol/sdk` or `zod`.

## Current Grounding

- `bun run check:tsgo` is green.
- `bun run build` is green.
- `bun run --cwd packages/mcp test` is green.
- `bun run test` currently fails only in `@trustgraph/base`, in `packages/base/src/__tests__/nats-backend.test.ts`.
- The NATS failures are a symptom of the backend still sitting between old Promise-shaped abstractions and newer JetStream calls. The rewrite should fix the contract, not only update the mock.
- The Effect v4 source of truth for APIs and examples is available at `/home/elpresidank/YeeBois/projects/beep-effect/.repos/effect-v4`.

## Stage 1: Backend And NATS

- Redefine `Message`, `BackendProducer`, `BackendConsumer`, and `PubSubBackend` around `Effect.Effect` return values.
- Make NATS connection, JetStream manager/client, stream initialization, and durable consumer handles scoped resources.
- Replace deprecated receive plumbing with modern JetStream consumer APIs.
- Preserve behavior for stream creation, durable consumer creation, headers, JSON/schema encode/decode, receive timeout returning `null`, ack/nak, close/drain, and tagged `PubSubError` mapping.
- Update NATS tests and in-memory fake backends to the new Effect-native interface.

Verification:

- `bun run --cwd packages/base test src/__tests__/nats-backend.test.ts`
- `bun run --cwd packages/base test`
- `bun run check:tsgo`

## Stage 2: Messaging And Processor Runtime

- Remove Promise facades from `makeProducer`, `makeConsumer`, `makeRequestResponse`, processor `start` / `stop` / `run`, and flow compatibility wrappers.
- Expose Effect values, scoped constructors, Context services, Layers, and finalizers as the only TrustGraph API shape.
- Convert message handlers, config handlers, shutdown callbacks, producer send/flush/close, consumer receive/ack/nak/close, and request/response operations to Effect functions.
- Remove internal `ManagedRuntime.make(Layer.empty)` compatibility runtimes from production source.

Verification:

- `bun run --cwd packages/base test`
- `bun run check:tsgo`

## Stage 3: Flow Service Call Sites

- Convert flow service state methods that currently call `Effect.runPromise` into Effect-valued service methods.
- Keep public feature behavior unchanged for config, flow manager, librarian, knowledge cores, retrieval, embeddings, triples, agent, MCP tool, and text-completion services.
- Convert test fakes to the same Effect-native backend/messaging contracts.

Verification:

- `bun run --cwd packages/flow test`
- `bun run test`
- `bun run check:tsgo`

## Stage 4: Gateway HTTP And RPC

- Remove Fastify and `@fastify/websocket` from `packages/flow`.
- Rebuild the gateway with Effect v4 `HttpApi` groups/endpoints and native RPC/socket layers.
- Preserve existing behavior:
  - `POST /api/v1/workbench/dispatch`
  - `POST /api/v1/:kind`
  - `POST /api/v1/flow/:flow/service/:kind`
  - `POST /api/v1/flow/:flow/load`
  - `GET /api/v1/rpc`
  - `GET /api/v1/metrics`
  - bearer auth behavior and RPC token behavior
  - existing response/error shapes expected by clients and workbench
- Use native Effect HTTP test utilities or route-level protocol tests instead of Fastify injection.

Verification:

- `bun run --cwd packages/flow test`
- `bun run --cwd packages/client test`
- `bun run test`
- `bun run build`

## Stage 5: CLI And Runner Scripts

- Remove Commander from `packages/cli`.
- Rebuild CLI commands with `effect/unstable/cli` and Effect-native socket/API clients.
- Convert service runner scripts to launch Effect programs directly with platform runtime `runMain` style execution.
- Remove `run(): Promise<void>` exports from flow services; export Effect programs/layers and Effect-native `runMain` helpers instead.
- Leave script-only demo/seed tools as follow-up only if they are outside production package source, but do not let production packages depend on Promise facades.

Verification:

- `bun run --cwd packages/cli test`
- `bun run check:tsgo`
- `bun run build`
- `bun run test`

## Stage 6: Cleanup And Acceptance

- Remove obsolete dependencies from package manifests and lockfile:
  - `fastify`
  - `@fastify/websocket`
  - `commander`
  - any legacy dependency made unnecessary by the rewrite
- Preserve unrelated dirty work. Do not revert user changes.
- Use parallel agents for bounded audits or disjoint rewrite slices when useful.
- Use Graphiti memory if available; if unavailable, continue safely and report it skipped.

Final verification:

- `bun run check:tsgo`
- `bun run build`
- `bun run lint`
- `bun run test`
- `bun run --cwd packages/mcp test`
- `bun run workbench:qa` after installing the matching Playwright browser if needed
- `git diff --check`
- `rg -n "fastify|@fastify/websocket|commander" packages package.json bun.lock` has no production dependency/use hits
- `rg -n "Effect\\.runPromise|ManagedRuntime\\.make|Promise<|async function main" packages/base/src packages/flow/src packages/cli/src scripts -g "*.ts"` has no production-source hits except tests or unavoidable external type declarations that are not TrustGraph APIs

Do not mark the goal complete until all required gates are green, or until a real external blocker is reported with the exact failing command, error, and smallest next action.