apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-07-01 01:19:38 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
trustgraph/ts/EFFECT_NATIVE_REWRITE_AUDIT.md
elpresidank 3809a38c46 Migrate flow manager to ref-backed Effect state
2026-06-02 01:26:40 -05:00

19 KiB
Raw Blame History

TrustGraph Effect-Native Rewrite Opportunity Audit

This is the current backlog snapshot for the playbook in ts/EFFECT_NATIVE_REWRITE_PLAYBOOK.md. The branch is ts-port-effect-v4. The unrelated local file .idea/effect.intellij.xml must stay uncommitted.

Inputs

Verified source roots:

  • TrustGraph TS port: /home/elpresidank/YeeBois/dev/trustgraph/ts
  • Effect v4 subtree: /home/elpresidank/YeeBois/projects/beep-effect2/.repos/effect-v4
  • Installed Effect beta used by this workspace: ts/node_modules/effect

Current signal counts from ts/packages after the 2026-06-02 FlowManager ref-backed state slice:

Signal Count
Effect.runPromise 204
Map< 75
WebSocket 47
new Map 56
toPromiseRequestor 0
makeAsyncProcessor 19
receive( 18
while ( 10
new Error 14
new Promise 10
JSON.parse 7
localStorage 9
JSON.stringify 6
setTimeout 4
process.env 3

Notes:

  • The remaining process.env hits are in packages/workbench/playwright.config.ts.
  • In production packages/base, packages/cli, and packages/mcp sources, the strict scans for new Error, new Promise, setTimeout, JSON.parse, JSON.stringify, and direct process.env reads are clean.
  • Effect.runPromise is expected at external Promise compatibility boundaries, but each match should still be audited for avoidable internal runtime ownership.
  • The Effect.runPromise, Map<, and new Map counts increased in this snapshot because the FlowManager slice added focused service tests and Promise compatibility facades while removing the service's internal mutable object state.

Loop Passes

2026-06-02: Base Request/Response Facade

  • Status: migrated and verified.
  • Completed:
    • Request/response startup now owns a scoped Effect runtime handle and maps failures to TrustGraph tagged messaging errors.
    • Runtime shutdown is idempotent and uses scoped fibers.
    • Tests cover Promise compatibility, tagged timeout errors, and tagged lifecycle errors.
  • Verification:
    • bun run --cwd ts/packages/base test
    • bun run --cwd ts/packages/base build
    • bun run --cwd ts check:tsgo
    • bun run --cwd ts build
    • bun run --cwd ts test

2026-06-02: Gateway Dispatcher Requestor Cache

  • Status: migrated and package-verified.
  • Completed:
    • Gateway dispatcher caches scoped EffectRequestResponse handles instead of Promise<RequestResponse> values.
    • Lazy requestor creation is serialized with SynchronizedRef.modifyEffect.
    • Streaming final-marker detection is centralized.
    • Dispatcher cleanup uses Effect scope/error handling instead of manual try/catch.
  • Verification:
    • bun run --cwd ts/packages/flow test
    • bun run --cwd ts/packages/flow build
    • bun run --cwd ts check:tsgo

2026-06-02: Strict Base, CLI, MCP, And tsgo Slice

  • Status: migrated, root-verified, committed, and pushed.
  • Completed:
    • Base messaging, NATS backend, producer, consumer, subscriber, request/response, runtime factories, processor programs, flow specs, and LLM service now use Effect-native boundaries, schema codecs, scoped cleanup, and S.TaggedErrorClass.make(...) errors.
    • CLI commands now run Effect programs at the command boundary, wrap socket lifecycle with Effect.acquireUseRelease, encode JSON through Effect Schema, and write output without console.log.
    • MCP Effect server now loads env/config through Config, wraps gateway calls with Effect.tryPromise, constructs schema classes with .make, and uses tagged errors.
    • MCP stdio compatibility server keeps createMcpServer and run, but uses Effect callbacks/tryPromise/schema encoding internally. run() uses ManagedRuntime; runMain() uses NodeRuntime.runMain.
    • Flow stateful service launch sites now pass an explicit Context.Context into the base processor runtime instead of hiding requirements behind assertions.
  • Verification:
    • cd ts && bun run check
    • cd ts && bun run test
    • cd ts && bun run build
    • git diff --check

2026-06-02: ConfigService Ref-Backed State Slice

  • Status: migrated and root-verified.
  • Completed:
    • ts/packages/flow/src/config/service.ts now models runtime state as a SynchronizedRef<ConfigServiceState> instead of adding mutable store, version, consumer, and producer fields onto the processor object.
    • Config operations have Effect-returning handlers with Promise facades only on the exported compatibility methods.
    • Request narrowing now uses effect/Predicate rather than request-record type assertions.
    • Persistence remains schema-backed and now reads/writes snapshots from the ref-backed state.
    • The consume loop now uses Effect.whileLoop; the remaining consumer.receive(2000) call is a pubsub boundary for this service.
    • Service startup now exposes runMain() through NodeRuntime.runMain. The legacy run() Promise facade uses ManagedRuntime, and ts/scripts/run-config.ts delegates directly to runMain() instead of owning its own catch/process-exit wrapper.
    • Config-service tests cover tagged invalid mutation errors, workspace persistence, legacy load, concurrent ref-backed mutations, and push publishing from the stored producer handle.
  • Verification:
    • bun run --cwd ts/packages/flow build
    • bun run --cwd ts/packages/flow test
    • cd ts && bun run check
    • cd ts && bun run build
    • cd ts && bun run test
    • git diff --check

2026-06-02: RAG And Agent Requestor Bridge Slice

  • Status: migrated, root-verified, committed, and pushed.
  • Completed:
    • ts/packages/flow/src/retrieval/graph-rag.ts and ts/packages/flow/src/retrieval/document-rag.ts now accept EffectRequestResponse clients directly. The engines no longer adapt Effect requestors back to Promise requestors and then wrap those calls in Effect.tryPromise.
    • ts/packages/flow/src/retrieval/graph-rag-service.ts and ts/packages/flow/src/retrieval/document-rag-service.ts now pass native flow requestors directly into the engines.
    • ts/packages/flow/src/agent/react/tools.ts now accepts EffectRequestResponse clients directly for graph RAG, document RAG, triples, and MCP tool calls. Tool input narrowing uses Schema and effect/Predicate rather than local request/response type assertions.
    • ts/packages/flow/src/agent/react/service.ts wires default and configured tools with native Effect requestors instead of toPromiseRequestor.
    • Graph RAG, document RAG, and agent service startup now expose runMain() through NodeRuntime.runMain; their legacy run() Promise facades use ManagedRuntime.
    • ts/scripts/run-graph-rag.ts, ts/scripts/run-document-rag.ts, and ts/scripts/run-agent.ts now delegate to runMain().
  • Verification:
    • bun run --cwd ts/packages/flow build
    • bun run --cwd ts/packages/flow test
    • cd ts && bun run check
    • cd ts && bun run build
    • cd ts && bun run test

2026-06-02: KnowledgeCore Ref-Backed State Slice

  • Status: migrated and root-verified.
  • Completed:
    • ts/packages/flow/src/cores/service.ts now exposes a typed KnowledgeCoreService instead of AsyncProcessorRuntime & Record<string, any>.
    • Runtime state now lives in SynchronizedRef<KnowledgeCoreServiceState> with kgCores, deCores, the request consumer, and response producer.
    • Knowledge operations now have Effect-returning handlers with Promise facades only on exported compatibility methods.
    • Persistence now decodes legacy and current snapshot shapes with Effect Schema and encodes JSON through Schema rather than raw JSON.parse/JSON.stringify plus assertions.
    • The consume loop now uses Effect.whileLoop; the remaining consumer.receive(2000) call is a pubsub boundary for this service.
    • The service exposes runMain() through NodeRuntime.runMain; legacy run() uses ManagedRuntime, and ts/scripts/run-knowledge.ts delegates to runMain().
    • ts/packages/base/src/schema/messages.ts now models legacy hyphenated knowledge request/response aliases so the service can preserve the wire shape without response type assertions.
    • New knowledge-core tests cover ref-backed mutation, graph embedding alias responses, concurrent state updates, and legacy persistence loading.
  • Verification:
    • bun run --cwd ts/packages/base build
    • bun run --cwd ts/packages/flow build
    • bun run --cwd ts/packages/flow test
    • cd ts && bun run check
    • cd ts && bun run build
    • cd ts && bun run test

2026-06-02: Flow Manager And Librarian Runtime Normalization

  • Status: migrated and root-verified.
  • Completed:
    • ts/packages/flow/src/flow-manager/service.ts and ts/packages/flow/src/librarian/service.ts now expose runMain() through NodeRuntime.runMain.
    • Their legacy run() Promise facades now use ManagedRuntime instead of directly owning Effect.runPromise.
    • ts/scripts/run-flow-manager.ts and ts/scripts/run-librarian.ts now delegate to runMain() instead of wrapping startup with local .catch(console.error/process.exit) handlers.
  • Verification:
    • bun run --cwd ts/packages/flow build
    • cd ts && bun run check
    • cd ts && bun run build
    • cd ts && bun run test
    • git diff --check

2026-06-02: FlowManager Ref-Backed State Slice

  • Status: migrated and root-verified.
  • Completed:
    • ts/packages/flow/src/flow-manager/service.ts now exposes a typed FlowManagerService instead of AsyncProcessorRuntime & Record<string, any>.
    • Runtime state now lives in SynchronizedRef<FlowManagerServiceState> with flows, blueprints, the request consumer, response producer, and config request client.
    • Flow operations now have Effect-returning handlers with Promise facades only on exported compatibility methods.
    • Blueprint config loading now narrows runtime values before constructing Blueprint records, replacing the prior parsed as Blueprint shortcut.
    • start-flow and stop-flow mutate the flow map through SynchronizedRef.modifyEffect, making duplicate checks and map updates atomic.
    • The consume loop now uses Effect.whileLoop; the remaining consumer.receive(2000) call is a pubsub boundary for this service.
    • New flow-manager tests cover tagged errors, ref-backed flow mutation, config push/delete requests, blueprint narrowing, duplicate concurrent starts, and message-level flow-error responses.
  • Verification:
    • bun run --cwd ts/packages/flow test -- src/__tests__/flow-manager-service.test.ts
    • bun run --cwd ts/packages/flow build
    • bun run --cwd ts/packages/flow test
    • cd ts && bun run check
    • cd ts && bun run build
    • cd ts && bun run test
    • git diff --check

Subagent Findings To Preserve

  • MCP/workbench:
    • Make the Effect MCP server the canonical implementation. The old stdio server should remain only as compatibility while parity is needed.
    • Workbench BaseApi atoms can move toward AtomRpc or AtomHttpApi after the client API is less Promise-first.
    • MCP env is now Config-backed; continue that policy for future MCP settings.
  • Flow stateful services:
    • Config service, KnowledgeCore service, and FlowManager ref-backed state are complete. Librarian now has native Effect module startup (NodeRuntime.runMain with a ManagedRuntime compatibility facade), but it still has a mutable poller service object. It remains a good candidate for Context services, scoped layers, Ref/SynchronizedRef, Schedule, and managed persistence.
    • Persistence IO should move toward FileSystem or KeyValueStore where the installed beta has the needed provider surface.
  • Base messaging/processors:
    • Subscriber queues/maps, processor/flow Promise compatibility, and dynamic flow state should continue moving toward Queue, Deferred, SynchronizedRef, Schedule, and scoped layers.
    • Existing constructor shims and typed registries in base processors still use type assertions; they need a typed factory/registry redesign rather than more assertions.
  • Gateway/client:
    • Knowledge streams still duplicate legacy end-of-stream handling.
    • Effect RPC client remains Promise-first internally in places and should be turned into a managed runtime or scoped layer.
    • WebSocket adapter shims still contain host-boundary try/catch and normal Error construction.
  • RAG/providers/storage:
    • RAG and agent requestor bridges are complete: toPromiseRequestor has no remaining ts/packages matches.
    • Provider SDKs and storage clients should become managed resources where they have meaningful lifecycle.
    • FalkorDB/Qdrant/Ollama/OpenAI-compatible surfaces still need config, schema, and scope audits.

Ranked Findings

P0: Migrate Librarian Stateful Service To Scoped Effect Service

  • TrustGraph evidence:
    • ts/packages/flow/src/librarian/service.ts
  • Effect primitives:
    • Context, Layer.scoped, Ref, SynchronizedRef, Schedule, Effect.addFinalizer, Config, Schema, FileSystem, KeyValueStore.
  • Rewrite shape:
    • Model one remaining service at a time as a Context service plus scoped layer or ref-backed state slice.
    • Store mutable service state in Ref or SynchronizedRef.
    • Run service main programs with platform runtime entrypoints such as NodeRuntime.runMain; keep ManagedRuntime only for compatibility Promise facades.
    • Replace polling sleep loops with schedules where behavior allows.
    • Decode persisted payloads and config with schemas at boundaries.
  • Tests:
    • Service-specific tests plus cd ts && bun run --cwd packages/flow test.

P1: Finish Client RPC Boundary Modernization

  • TrustGraph evidence:
    • ts/packages/client/src/socket/effect-rpc-client.ts
    • ts/packages/client/src/socket/trustgraph-socket.ts
    • ts/packages/client/src/socket/websocket-adapter.ts
  • Effect primitives:
    • effect/unstable/socket Socket.makeWebSocket, fromWebSocket, toChannel, layerWebSocket.
    • effect/unstable/rpc/RpcClient.layerProtocolSocket.
    • effect/unstable/rpc/RpcSerialization.layerNdjson or layerNdJsonRpc.
    • ManagedRuntime for compatibility facades when a Promise API must remain.
  • Rewrite shape:
    • Treat EffectRpcClient as an internal managed runtime or scoped layer.
    • Expose Promise-returning methods through a thin adapter.
    • Replace normal client Error constructors with tagged errors before they cross into shared Effect code.
  • Tests:
    • cd ts && bun run --cwd packages/client test

P1: Base Processor Registry And Constructor Shims

  • TrustGraph evidence:
    • ts/packages/base/src/processor/async-processor.ts
    • ts/packages/base/src/processor/flow.ts
    • ts/packages/base/src/processor/flow-processor.ts
  • Effect primitives:
    • Schema-backed registries, Context, Layer, Effect.fn, Option, Predicate.
  • Rewrite shape:
    • Replace constructor as unknown as shims with typed factory exports.
    • Replace resource lookup casts with schema-backed typed registry helpers.
    • Do not add assertions to quiet Effect channel inference problems.
  • Tests:
    • cd ts && bun run --cwd packages/base test
    • Root cd ts && bun run check because this surface easily pollutes Effect error and requirement channels.

P1: Make SDK, Storage, And Provider Layers Managed Resources

  • TrustGraph evidence:
    • ts/packages/flow/src/storage/triples/falkordb.ts
    • ts/packages/flow/src/storage/embeddings/qdrant-graph.ts
    • ts/packages/flow/src/storage/embeddings/qdrant-doc.ts
    • ts/packages/flow/src/model/text-completion/*.ts
    • ts/packages/flow/src/embeddings/ollama.ts
  • Effect primitives:
    • Effect.acquireRelease, Layer.scoped, Config, ConfigProvider, Metric, Logger, Effect AI provider layers.
  • Rewrite shape:
    • Move env/config reading into Config loaders and provider-specific layers.
    • Scope SDK clients that need explicit close/disconnect.
    • Remove Effect.void as Effect.Effect<undefined> stream assertions by letting branch return types infer or by restructuring the stream parser.
  • Tests:
    • Provider config tests with ConfigProvider.fromMap.
    • Storage tests with fake clients before changing real resource lifetimes.

P2: Canonicalize MCP Around The Effect Server

  • Status:
    • First blocker slice complete: MCP now builds under strict tsgo and the stdio server has an Effect-backed compatibility implementation.
  • Remaining shape:
    • Decide whether the old SDK/Zod stdio compatibility surface should stay as a wrapper or be removed.
    • Add parity tests before deleting any public entry point.
  • Tests:
    • cd ts && bun run --cwd packages/mcp build
    • Root cd ts && bun run check

P2: Tighten Workbench Platform And Reactivity Usage

  • TrustGraph evidence:
    • ts/packages/workbench/src/atoms/workbench.ts
    • Remaining direct browser state includes localStorage and DOM theme inspection.
  • Effect primitives:
    • BrowserKeyValueStore.layerLocalStorage, BrowserKeyValueStore.layerSessionStorage, BrowserHttpClient, Clipboard, AtomRpc, AtomHttpApi, AtomRegistry, AsyncResult, Reactivity.
  • Rewrite shape:
    • Leave workbench out of the next backend/runtime rewrite wave.
    • Move persistent UI state through browser platform services later.
  • Tests:
    • cd ts && bun run workbench:qa

Recommended PR Order

  1. Librarian or flow-manager scoped state migration.
  2. Client RPC managed runtime/scoped layer cleanup.
  3. Base processor registry and constructor shim redesign.
  4. Gateway RPC callback and client streaming completion cleanup.
  5. Storage/provider managed resource cleanup.
  6. MCP parity/deletion decision and workbench platform polish.

No-Op Rules

Do not flag these as rewrite blockers without additional proof:

  • Promise-returning CLI actions, MCP SDK callbacks, client compatibility methods, and Fastify route handlers at true external boundaries. Boundary code still must map failures into typed errors or wire errors.
  • try/catch blocks at host/tool boundaries only when the catch maps into a typed error or a wire-contract error. Internal exception capture should use Effect.try, Effect.tryPromise, or Result.try.
  • S.Class, S.TaggedErrorClass, Context.Service, Rpc.make, and HttpApi.make when they are required or idiomatic for the Effect API.
  • Plain Map usage for local pure transformations, such as graph utility construction, unless the state is long-lived mutable service state.
  • JSON stringification in tests or wire-contract fixtures. Production JSON encode/decode should prefer schema codecs when the encoded form can be preserved.

Acceptance For Final Loop Completion

The overall playbook loop is complete only when:

  • All remaining playbook signal matches are migrated or documented as no-op external-boundary cases with concrete evidence.
  • No P0/P1/P2 migration item remains in this audit.
  • cd ts && bun run check, cd ts && bun run build, cd ts && bun run test, and git diff --check pass after the final migration slice.