mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-24 04:31:02 +02:00
Migrate strict Effect runtime surfaces
This commit is contained in:
parent
f6878d4dd7
commit
b4ee2b691f
35 changed files with 1717 additions and 1410 deletions
|
|
@ -1,9 +1,8 @@
|
||||||
# TrustGraph Effect-Native Rewrite Opportunity Audit
|
# TrustGraph Effect-Native Rewrite Opportunity Audit
|
||||||
|
|
||||||
This is the first ranked audit produced from the playbook in
|
This is the current backlog snapshot for the playbook in
|
||||||
`ts/EFFECT_NATIVE_REWRITE_PLAYBOOK.md`. It is an opportunity map, not a code
|
`ts/EFFECT_NATIVE_REWRITE_PLAYBOOK.md`. The branch is `ts-port-effect-v4`.
|
||||||
rewrite. The branch was `ts-port-effect-v4`; the only unrelated local file seen
|
The unrelated local file `.idea/effect.intellij.xml` must stay uncommitted.
|
||||||
during the audit was `.idea/effect.intellij.xml`.
|
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
|
|
@ -11,27 +10,38 @@ Verified source roots:
|
||||||
|
|
||||||
- TrustGraph TS port: `/home/elpresidank/YeeBois/dev/trustgraph/ts`
|
- TrustGraph TS port: `/home/elpresidank/YeeBois/dev/trustgraph/ts`
|
||||||
- Effect v4 subtree: `/home/elpresidank/YeeBois/projects/beep-effect2/.repos/effect-v4`
|
- Effect v4 subtree: `/home/elpresidank/YeeBois/projects/beep-effect2/.repos/effect-v4`
|
||||||
- Reactivity fallback: `ts/node_modules/effect/src/unstable/reactivity`
|
- Installed Effect beta used by this workspace: `ts/node_modules/effect`
|
||||||
- Atom React fallback: `ts/packages/workbench/node_modules/@effect/atom-react`
|
|
||||||
|
|
||||||
Signal counts from `ts/packages`:
|
Current signal counts from `ts/packages` after the 2026-06-02 strict tsgo
|
||||||
|
slice:
|
||||||
|
|
||||||
| Signal | Count |
|
| Signal | Count |
|
||||||
| --- | ---: |
|
| --- | ---: |
|
||||||
| `Effect.runPromise` | 71 |
|
| `Effect.runPromise` | 208 |
|
||||||
| `Map<` | 54 |
|
| `Map<` | 58 |
|
||||||
| `JSON.stringify` | 50 |
|
|
||||||
| `WebSocket` | 45 |
|
| `WebSocket` | 45 |
|
||||||
| `process.env` | 44 |
|
| `new Map` | 45 |
|
||||||
| `new Map` | 42 |
|
|
||||||
| `toPromiseRequestor` | 19 |
|
| `toPromiseRequestor` | 19 |
|
||||||
| `makeAsyncProcessor` | 19 |
|
| `makeAsyncProcessor` | 19 |
|
||||||
| `new Promise` | 18 |
|
| `receive(` | 18 |
|
||||||
| `JSON.parse` | 16 |
|
| `while (` | 13 |
|
||||||
| `receive(` | 16 |
|
| `new Error` | 14 |
|
||||||
| `setTimeout` | 13 |
|
| `new Promise` | 10 |
|
||||||
| `while (` | 10 |
|
| `JSON.parse` | 8 |
|
||||||
| `localStorage` | 8 |
|
| `localStorage` | 8 |
|
||||||
|
| `JSON.stringify` | 6 |
|
||||||
|
| `setTimeout` | 4 |
|
||||||
|
| `process.env` | 3 |
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- The remaining `process.env` hits are in `packages/workbench/playwright.config.ts`.
|
||||||
|
- In production `packages/base`, `packages/cli`, and `packages/mcp` sources,
|
||||||
|
the strict scans for `new Error`, `new Promise`, `setTimeout`,
|
||||||
|
`JSON.parse`, `JSON.stringify`, and direct `process.env` reads are clean.
|
||||||
|
- `Effect.runPromise` is expected at external Promise compatibility
|
||||||
|
boundaries, but each match should still be audited for avoidable internal
|
||||||
|
runtime ownership.
|
||||||
|
|
||||||
## Loop Passes
|
## Loop Passes
|
||||||
|
|
||||||
|
|
@ -39,356 +49,256 @@ Signal counts from `ts/packages`:
|
||||||
|
|
||||||
- Status: migrated and verified.
|
- Status: migrated and verified.
|
||||||
- Completed:
|
- Completed:
|
||||||
- `ts/packages/base/src/messaging/request-response.ts:50` now creates an
|
- Request/response startup now owns a scoped Effect runtime handle and maps
|
||||||
explicit `Scope.Closeable` and `:55` builds the existing
|
failures to TrustGraph tagged messaging errors.
|
||||||
`EffectRequestResponse` runtime.
|
- Runtime shutdown is idempotent and uses scoped fibers.
|
||||||
- `ts/packages/base/src/messaging/request-response.ts:91` rejects
|
- Tests cover Promise compatibility, tagged timeout errors, and tagged
|
||||||
not-started calls with `MessagingLifecycleError`, and `:108` maps
|
lifecycle errors.
|
||||||
recipient callback failures into `MessagingDeliveryError`. It no longer
|
|
||||||
constructs normal `Error` values.
|
|
||||||
- `ts/packages/base/src/messaging/runtime.ts:427` now lets
|
|
||||||
request/response own its producer directly, `:442` runs the response
|
|
||||||
dispatcher with `Effect.forkScoped`, and `:445` makes shutdown idempotent.
|
|
||||||
- `ts/packages/base/src/__tests__/request-response.test.ts:115` covers the
|
|
||||||
Promise facade over the Effect runtime, `:143` asserts tagged timeout
|
|
||||||
errors, and `:164` asserts tagged lifecycle errors.
|
|
||||||
- Verification:
|
- Verification:
|
||||||
- `bun run --cwd ts/packages/base test`
|
- `bun run --cwd ts/packages/base test`
|
||||||
- `bun run --cwd ts/packages/base build`
|
- `bun run --cwd ts/packages/base build`
|
||||||
- `bun run --cwd ts check:tsgo`
|
- `bun run --cwd ts check:tsgo`
|
||||||
- `bun run --cwd ts build`
|
- `bun run --cwd ts build`
|
||||||
- `bun run --cwd ts test`
|
- `bun run --cwd ts test`
|
||||||
- Remaining base evidence:
|
|
||||||
- `makeSubscriber(` has no current `ts/packages` call sites after this slice,
|
|
||||||
but `ts/packages/base/src/messaging/index.ts` still exports
|
|
||||||
`makeAsyncQueue`, `makeSubscriber`, and related types.
|
|
||||||
- `ts/packages/base/src/messaging/consumer.ts` still has a Promise polling
|
|
||||||
loop and a normal `Error` constructor.
|
|
||||||
- `ts/packages/base/src/messaging/producer.ts` still throws a normal
|
|
||||||
not-started `Error`.
|
|
||||||
- Decision:
|
|
||||||
- Normal `Error` construction in library internals is migration evidence.
|
|
||||||
Prefer existing `S.TaggedErrorClass` errors from
|
|
||||||
`ts/packages/base/src/errors.ts`, adding new tagged errors when needed.
|
|
||||||
- `try`/`catch` blocks are also migration evidence. Prefer `Effect.try`,
|
|
||||||
`Effect.tryPromise`, or `Result.try` unless the block is a host/tool
|
|
||||||
boundary or test-only helper.
|
|
||||||
|
|
||||||
### 2026-06-02: Gateway Dispatcher Requestor Cache
|
### 2026-06-02: Gateway Dispatcher Requestor Cache
|
||||||
|
|
||||||
- Status: migrated and package-verified.
|
- Status: migrated and package-verified.
|
||||||
- Completed:
|
- Completed:
|
||||||
- `ts/packages/flow/src/gateway/dispatch/manager.ts:121` centralizes
|
- Gateway dispatcher caches scoped `EffectRequestResponse` handles instead
|
||||||
streaming completion detection as `dispatcherManagerIsCompleteResponse`.
|
of `Promise<RequestResponse>` values.
|
||||||
- `ts/packages/flow/src/gateway/dispatch/manager.ts:137` stores requestors
|
- Lazy requestor creation is serialized with `SynchronizedRef.modifyEffect`.
|
||||||
as `EffectRequestResponse` handles, not `Promise<RequestResponse>` values.
|
- Streaming final-marker detection is centralized.
|
||||||
- `ts/packages/flow/src/gateway/dispatch/manager.ts:152` starts the manager
|
- Dispatcher cleanup uses Effect scope/error handling instead of manual
|
||||||
through an Effect program, `:157` creates a `SynchronizedRef` cache, and
|
`try`/`catch`.
|
||||||
`:164` uses `Effect.onError` for scope cleanup instead of a `try`/`catch`
|
|
||||||
block.
|
|
||||||
- `ts/packages/flow/src/gateway/dispatch/manager.ts:200` uses
|
|
||||||
`SynchronizedRef.modifyEffect` so lazy requestor creation and caching are
|
|
||||||
serialized under the manager scope.
|
|
||||||
- `ts/packages/flow/src/gateway/dispatch/manager.ts:267` and `:312` keep
|
|
||||||
Fastify/RPC as Promise boundaries via `Effect.runPromise`; streaming
|
|
||||||
responder failures are mapped with `MessagingDeliveryError` at `:290` and
|
|
||||||
`:340`.
|
|
||||||
- `ts/packages/flow/src/gateway/server.ts:25` accepts an optional injected
|
|
||||||
`PubSubBackend` for tests without changing production NATS defaults.
|
|
||||||
- `ts/packages/flow/src/__tests__/gateway-dispatcher.test.ts:150` verifies
|
|
||||||
scoped requestor reuse and shutdown, `:172` verifies streaming through the
|
|
||||||
centralized completion predicate, and `:192` table-tests all final markers.
|
|
||||||
- Verification:
|
- Verification:
|
||||||
- `bun run --cwd ts/packages/flow test`
|
- `bun run --cwd ts/packages/flow test`
|
||||||
- `bun run --cwd ts/packages/flow build`
|
- `bun run --cwd ts/packages/flow build`
|
||||||
- `bun run --cwd ts check:tsgo`
|
- `bun run --cwd ts check:tsgo`
|
||||||
- Remaining gateway evidence:
|
|
||||||
- `ts/packages/flow/src/gateway/rpc-server.ts` still has Promise callbacks
|
### 2026-06-02: Strict Base, CLI, MCP, And tsgo Slice
|
||||||
around Effect RPC queues.
|
|
||||||
- `ts/packages/flow/src/gateway/server.ts` still has Fastify route
|
- Status: migrated, root-verified, ready to commit.
|
||||||
`try`/`catch` blocks. These are boundary code, but should still be audited
|
- Completed:
|
||||||
for `Effect.tryPromise` wrapping where it improves consistency.
|
- Base messaging, NATS backend, producer, consumer, subscriber,
|
||||||
- `ts/packages/client/src/socket/trustgraph-socket.ts` still duplicates some
|
request/response, runtime factories, processor programs, flow specs, and
|
||||||
streaming final-marker detection on the client side.
|
LLM service now use Effect-native boundaries, schema codecs, scoped
|
||||||
|
cleanup, and `S.TaggedErrorClass.make(...)` errors.
|
||||||
|
- CLI commands now run Effect programs at the command boundary, wrap socket
|
||||||
|
lifecycle with `Effect.acquireUseRelease`, encode JSON through Effect
|
||||||
|
Schema, and write output without `console.log`.
|
||||||
|
- MCP Effect server now loads env/config through `Config`, wraps gateway
|
||||||
|
calls with `Effect.tryPromise`, constructs schema classes with `.make`, and
|
||||||
|
uses tagged errors.
|
||||||
|
- MCP stdio compatibility server keeps `createMcpServer` and `run`, but uses
|
||||||
|
Effect callbacks/tryPromise/schema encoding internally. `run()` uses
|
||||||
|
`ManagedRuntime`; `runMain()` uses `NodeRuntime.runMain`.
|
||||||
|
- Flow stateful service launch sites now pass an explicit `Context.Context`
|
||||||
|
into the base processor runtime instead of hiding requirements behind
|
||||||
|
assertions.
|
||||||
|
- Verification:
|
||||||
|
- `cd ts && bun run check`
|
||||||
|
- `cd ts && bun run test`
|
||||||
|
- `cd ts && bun run build`
|
||||||
|
- `git diff --check`
|
||||||
|
|
||||||
|
## Subagent Findings To Preserve
|
||||||
|
|
||||||
|
- MCP/workbench:
|
||||||
|
- Make the Effect MCP server the canonical implementation. The old stdio
|
||||||
|
server should remain only as compatibility while parity is needed.
|
||||||
|
- Workbench BaseApi atoms can move toward `AtomRpc` or `AtomHttpApi` after
|
||||||
|
the client API is less Promise-first.
|
||||||
|
- MCP env is now Config-backed; continue that policy for future MCP settings.
|
||||||
|
- Flow stateful services:
|
||||||
|
- Config, librarian, cores, and flow-manager still have mutable poller
|
||||||
|
service objects. These remain good candidates for `Context` services,
|
||||||
|
scoped layers, `Ref`/`SynchronizedRef`, `Schedule`, and managed
|
||||||
|
persistence.
|
||||||
|
- Persistence IO should move toward `FileSystem` or `KeyValueStore` where
|
||||||
|
the installed beta has the needed provider surface.
|
||||||
|
- Base messaging/processors:
|
||||||
|
- Subscriber queues/maps, processor/flow Promise compatibility, and dynamic
|
||||||
|
flow state should continue moving toward `Queue`, `Deferred`,
|
||||||
|
`SynchronizedRef`, `Schedule`, and scoped layers.
|
||||||
|
- Existing constructor shims and typed registries in base processors still
|
||||||
|
use type assertions; they need a typed factory/registry redesign rather
|
||||||
|
than more assertions.
|
||||||
|
- Gateway/client:
|
||||||
|
- Knowledge streams still duplicate legacy end-of-stream handling.
|
||||||
|
- Effect RPC client remains Promise-first internally in places and should be
|
||||||
|
turned into a managed runtime or scoped layer.
|
||||||
|
- WebSocket adapter shims still contain host-boundary `try`/`catch` and
|
||||||
|
normal `Error` construction.
|
||||||
|
- RAG/providers/storage:
|
||||||
|
- RAG and agent helpers still adapt Effect requestors back to Promise
|
||||||
|
requestors.
|
||||||
|
- Provider SDKs and storage clients should become managed resources where
|
||||||
|
they have meaningful lifecycle.
|
||||||
|
- FalkorDB/Qdrant/Ollama/OpenAI-compatible surfaces still need config,
|
||||||
|
schema, and scope audits.
|
||||||
|
|
||||||
## Ranked Findings
|
## Ranked Findings
|
||||||
|
|
||||||
### P0: Collapse Base Messaging Promise Facades
|
|
||||||
|
|
||||||
- Impact: 5
|
|
||||||
- Risk: 4
|
|
||||||
- Confidence: 4
|
|
||||||
- TrustGraph evidence:
|
|
||||||
- `ts/packages/base/src/messaging/runtime.ts` already defines Effect
|
|
||||||
producer, consumer, request/response factories, queues, fibers, and scopes.
|
|
||||||
- `ts/packages/base/src/messaging/consumer.ts` still has a manual
|
|
||||||
`while (running)` receive loop, `sleep`, and Promise delay helpers.
|
|
||||||
- `ts/packages/base/src/messaging/subscriber.ts` still manages resolver maps
|
|
||||||
and timeout promises.
|
|
||||||
- `ts/packages/base/src/processor/flow.ts` exposes compatibility scope
|
|
||||||
helpers and converts Effect handles back into Promise-style handles.
|
|
||||||
- Effect evidence:
|
|
||||||
- `effect/Queue`, `effect/PubSub`, `effect/Stream`, `effect/Scope`,
|
|
||||||
`effect/Layer`, `effect/Schedule`, `effect/Ref`.
|
|
||||||
- Sources: `packages/effect/src/Queue.ts`, `PubSub.ts`, `Stream.ts`,
|
|
||||||
`Scope.ts`, `Layer.ts`, `Schedule.ts`, `Ref.ts`.
|
|
||||||
- Rewrite shape:
|
|
||||||
- Make the Effect runtime factories the canonical internal surface.
|
|
||||||
- Keep Promise adapters only at external compatibility boundaries. Rejected
|
|
||||||
values at those boundaries should still be tagged TrustGraph errors.
|
|
||||||
- Replace polling sleep loops with scheduled scoped consumers where possible.
|
|
||||||
- Replace resolver maps with `Queue`, `Deferred`, or `PubSub`-backed routing.
|
|
||||||
- Tests:
|
|
||||||
- `cd ts && bun run --cwd packages/base test`
|
|
||||||
- Existing runtime tests around request/response, flow specs, and consumers
|
|
||||||
should be expanded before removing compatibility behavior.
|
|
||||||
- Blockers:
|
|
||||||
- Public package exports may still expect Promise-shaped producer, consumer,
|
|
||||||
and request/response handles. Inventory callers before changing exports.
|
|
||||||
- First slice completed request/response facade migration. Next base follow-up
|
|
||||||
is either an Effect-backed consumer facade or a public export decision for
|
|
||||||
`subscriber.ts`.
|
|
||||||
|
|
||||||
### P0: Convert Stateful Flow Services To Scoped Effect Services
|
### P0: Convert Stateful Flow Services To Scoped Effect Services
|
||||||
|
|
||||||
- Impact: 5
|
|
||||||
- Risk: 4
|
|
||||||
- Confidence: 4
|
|
||||||
- TrustGraph evidence:
|
- TrustGraph evidence:
|
||||||
- `ts/packages/flow/src/config/service.ts` uses `makeAsyncProcessor`,
|
- `ts/packages/flow/src/config/service.ts`
|
||||||
mutable nested `Map` state, `while (this.running)`, `receive(2000)`,
|
- `ts/packages/flow/src/librarian/service.ts`
|
||||||
`sleep`, JSON persistence, and direct `process.env`.
|
- `ts/packages/flow/src/cores/service.ts`
|
||||||
- `ts/packages/flow/src/librarian/service.ts`, `cores/service.ts`, and
|
- `ts/packages/flow/src/flow-manager/service.ts`
|
||||||
`flow-manager/service.ts` repeat the same service-object pattern.
|
- Effect primitives:
|
||||||
- Effect evidence:
|
|
||||||
- `Context`, `Layer.scoped`, `Ref`, `SynchronizedRef`, `Schedule`,
|
- `Context`, `Layer.scoped`, `Ref`, `SynchronizedRef`, `Schedule`,
|
||||||
`Effect.addFinalizer`, `Config`, `Schema`, `effect/FileSystem`,
|
`Effect.addFinalizer`, `Config`, `Schema`, `FileSystem`,
|
||||||
`effect/unstable/persistence/KeyValueStore`.
|
`KeyValueStore`.
|
||||||
- Sources: `packages/effect/src/Context.ts`, `Layer.ts`, `Ref.ts`,
|
|
||||||
`SynchronizedRef.ts`, `Schedule.ts`, `Config.ts`, `Schema.ts`,
|
|
||||||
`ts/node_modules/effect/src/FileSystem.ts`,
|
|
||||||
`ts/node_modules/effect/src/unstable/persistence/KeyValueStore.ts`.
|
|
||||||
- Rewrite shape:
|
- Rewrite shape:
|
||||||
- Model each service as a `Context` service plus a scoped layer.
|
- Model one service at a time as a `Context` service plus scoped layer.
|
||||||
- Store service state in `Ref` or `SynchronizedRef`, not mutable object fields.
|
- Store mutable service state in `Ref` or `SynchronizedRef`.
|
||||||
- Express persistence with `effect/FileSystem` or
|
- Replace polling sleep loops with schedules where behavior allows.
|
||||||
`KeyValueStore.layerFileSystem` when the installed beta exposes the needed
|
|
||||||
provider.
|
|
||||||
- Decode persisted payloads and config with schemas at boundaries.
|
- Decode persisted payloads and config with schemas at boundaries.
|
||||||
- Tests:
|
- Tests:
|
||||||
- Service-specific tests plus `cd ts && bun run --cwd packages/flow test`.
|
- Service-specific tests plus `cd ts && bun run --cwd packages/flow test`.
|
||||||
- Add persistence round-trip tests before replacing file IO.
|
|
||||||
- Blockers:
|
|
||||||
- These services are behavior-heavy. Do one service per PR after the shared
|
|
||||||
runtime surface is stable.
|
|
||||||
|
|
||||||
### P0: Make Gateway Dispatcher Effect-Native
|
### P0: Remove RAG And Agent `toPromiseRequestor` Bridges
|
||||||
|
|
||||||
- Impact: 5
|
|
||||||
- Risk: 3
|
|
||||||
- Confidence: 4
|
|
||||||
- TrustGraph evidence:
|
|
||||||
- `ts/packages/flow/src/gateway/server.ts` already builds RPC/WebSocket
|
|
||||||
pieces with Effect.
|
|
||||||
- `ts/packages/flow/src/gateway/rpc-server.ts` uses `Queue` and RPC layers.
|
|
||||||
- `ts/packages/flow/src/gateway/dispatch/manager.ts` still keeps
|
|
||||||
`Map<string, Promise<RequestResponse<unknown, unknown>>>`, manual
|
|
||||||
streaming completion checks, and per-publish producer construction.
|
|
||||||
- Effect evidence:
|
|
||||||
- `effect/unstable/rpc` `RpcClient`, `RpcServer`, `RpcSerialization`.
|
|
||||||
- `effect/unstable/socket` `Socket`.
|
|
||||||
- `effect/Queue`, `Stream`, `Scope`, `Layer`.
|
|
||||||
- Sources: `ts/node_modules/effect/src/unstable/rpc/RpcClient.ts`,
|
|
||||||
`RpcServer.ts`, `RpcSerialization.ts`, and
|
|
||||||
`ts/node_modules/effect/src/unstable/socket/Socket.ts`.
|
|
||||||
- Rewrite shape:
|
|
||||||
- Convert dispatcher manager methods to Effect-returning functions internally.
|
|
||||||
- Cache requestors as scoped resources instead of Promise values.
|
|
||||||
- Represent streaming dispatch as `Stream` or `Queue` instead of callback
|
|
||||||
completion detection where the wire protocol allows it.
|
|
||||||
- Keep Fastify route handlers as Promise boundaries.
|
|
||||||
- Tests:
|
|
||||||
- Gateway dispatch tests with fake pubsub.
|
|
||||||
- `cd ts && SKIP_LLM=1 bun run test:pipeline` after implementation.
|
|
||||||
- Blockers:
|
|
||||||
- The gateway is an integration boundary. Preserve current HTTP and WebSocket
|
|
||||||
wire behavior during the first rewrite.
|
|
||||||
- First dispatcher-cache slice is complete. Follow-up gateway work should
|
|
||||||
target RPC server Promise callbacks and client-side streaming completion
|
|
||||||
duplication, not recreate the requestor cache migration.
|
|
||||||
|
|
||||||
### P1: Remove RAG And Agent `toPromiseRequestor` Bridges
|
|
||||||
|
|
||||||
- Impact: 4
|
|
||||||
- Risk: 3
|
|
||||||
- Confidence: 5
|
|
||||||
- TrustGraph evidence:
|
- TrustGraph evidence:
|
||||||
- `ts/packages/flow/src/retrieval/document-rag-service.ts`
|
- `ts/packages/flow/src/retrieval/document-rag-service.ts`
|
||||||
- `ts/packages/flow/src/retrieval/graph-rag-service.ts`
|
- `ts/packages/flow/src/retrieval/graph-rag-service.ts`
|
||||||
- `ts/packages/flow/src/agent/react/service.ts`
|
- `ts/packages/flow/src/agent/react/service.ts`
|
||||||
- All define `toPromiseRequestor` and then immediately adapt Effect
|
- Effect primitives:
|
||||||
requestors back to Promise-style clients.
|
- Existing `EffectRequestResponse`, `Effect.fn`, `Stream`,
|
||||||
- Effect evidence:
|
`Effect.runPromiseWith` at true external boundaries only.
|
||||||
- Existing TrustGraph `EffectRequestResponse` in
|
|
||||||
`ts/packages/base/src/messaging/runtime.ts`.
|
|
||||||
- `effect/Stream`, `Effect.fn`, `Effect.runPromiseWith` for boundary-only
|
|
||||||
execution.
|
|
||||||
- Rewrite shape:
|
- Rewrite shape:
|
||||||
- Update RAG engines and agent helpers to accept Effect requestors or
|
- Update engines and agent helpers to accept Effect requestors or
|
||||||
functions returning `Effect`.
|
Effect-returning functions directly.
|
||||||
- Keep Promise wrappers only for old public APIs or tests that explicitly
|
- Preserve Promise wrappers only for old public APIs or tests that explicitly
|
||||||
verify compatibility.
|
verify compatibility.
|
||||||
- Convert streaming agent flows to `Stream` where possible.
|
|
||||||
- Tests:
|
- Tests:
|
||||||
- Existing RAG and agent service tests.
|
- Existing RAG and agent service tests.
|
||||||
- Add tests that assert requestor errors stay typed through the Effect path.
|
- Add tests that assert requestor errors stay typed through the Effect path.
|
||||||
- Blockers:
|
|
||||||
- Engine call signatures need a small design pass so RAG and agent rewrite in
|
|
||||||
the same direction.
|
|
||||||
|
|
||||||
### P1: Finish Client RPC Boundary Modernization
|
### P1: Finish Client RPC Boundary Modernization
|
||||||
|
|
||||||
- Impact: 4
|
|
||||||
- Risk: 3
|
|
||||||
- Confidence: 4
|
|
||||||
- TrustGraph evidence:
|
- TrustGraph evidence:
|
||||||
- `ts/packages/client/src/socket/effect-rpc-client.ts` already uses
|
- `ts/packages/client/src/socket/effect-rpc-client.ts`
|
||||||
`Socket.makeWebSocket`, `RpcClient.layerProtocolSocket`, and
|
- `ts/packages/client/src/socket/trustgraph-socket.ts`
|
||||||
`RpcSerialization.layerNdjson`.
|
- `ts/packages/client/src/socket/websocket-adapter.ts`
|
||||||
- The same file still owns `scopePromise`, `clientPromise`, repeated
|
- Effect primitives:
|
||||||
`Effect.runPromise`, listener sets, a WebSocket constructor shim, and a
|
- `effect/unstable/socket` `Socket.makeWebSocket`, `fromWebSocket`,
|
||||||
Promise facade.
|
|
||||||
- `ts/packages/client/src/socket/trustgraph-socket.ts` is mostly a
|
|
||||||
compatibility API over the Effect RPC client.
|
|
||||||
- Effect evidence:
|
|
||||||
- `effect/unstable/socket/Socket`: `makeWebSocket`, `fromWebSocket`,
|
|
||||||
`toChannel`, `layerWebSocket`.
|
`toChannel`, `layerWebSocket`.
|
||||||
- `effect/unstable/rpc/RpcClient`: `layerProtocolSocket`.
|
- `effect/unstable/rpc/RpcClient.layerProtocolSocket`.
|
||||||
- `effect/unstable/rpc/RpcSerialization`: `layerNdjson`, `layerNdJsonRpc`.
|
- `effect/unstable/rpc/RpcSerialization.layerNdjson` or `layerNdJsonRpc`.
|
||||||
|
- `ManagedRuntime` for compatibility facades when a Promise API must remain.
|
||||||
- Rewrite shape:
|
- Rewrite shape:
|
||||||
- Treat `EffectRpcClient` as an internal managed runtime or scoped layer.
|
- Treat `EffectRpcClient` as an internal managed runtime or scoped layer.
|
||||||
- Expose Promise-returning methods only through a thin compatibility adapter.
|
- Expose Promise-returning methods through a thin adapter.
|
||||||
- Move browser vs Node WebSocket constructor selection into platform layers.
|
- Replace normal client `Error` constructors with tagged errors before they
|
||||||
|
cross into shared Effect code.
|
||||||
- Tests:
|
- Tests:
|
||||||
- `cd ts && bun run --cwd packages/client test`
|
- `cd ts && bun run --cwd packages/client test`
|
||||||
- Keep timeout/retry tests around `withDispatchRequestPolicy`.
|
|
||||||
- Blockers:
|
### P1: Base Processor Registry And Constructor Shims
|
||||||
- Workbench and CLI still consume Promise-shaped client APIs.
|
|
||||||
|
- TrustGraph evidence:
|
||||||
|
- `ts/packages/base/src/processor/async-processor.ts`
|
||||||
|
- `ts/packages/base/src/processor/flow.ts`
|
||||||
|
- `ts/packages/base/src/processor/flow-processor.ts`
|
||||||
|
- Effect primitives:
|
||||||
|
- Schema-backed registries, `Context`, `Layer`, `Effect.fn`, `Option`,
|
||||||
|
`Predicate`.
|
||||||
|
- Rewrite shape:
|
||||||
|
- Replace constructor `as unknown as` shims with typed factory exports.
|
||||||
|
- Replace resource lookup casts with schema-backed typed registry helpers.
|
||||||
|
- Do not add assertions to quiet Effect channel inference problems.
|
||||||
|
- Tests:
|
||||||
|
- `cd ts && bun run --cwd packages/base test`
|
||||||
|
- Root `cd ts && bun run check` because this surface easily pollutes Effect
|
||||||
|
error and requirement channels.
|
||||||
|
|
||||||
### P1: Make SDK, Storage, And Provider Layers Managed Resources
|
### P1: Make SDK, Storage, And Provider Layers Managed Resources
|
||||||
|
|
||||||
- Impact: 4
|
|
||||||
- Risk: 3
|
|
||||||
- Confidence: 3
|
|
||||||
- TrustGraph evidence:
|
- TrustGraph evidence:
|
||||||
- `ts/packages/flow/src/storage/triples/falkordb.ts`
|
- `ts/packages/flow/src/storage/triples/falkordb.ts`
|
||||||
- `ts/packages/flow/src/storage/embeddings/qdrant-graph.ts`
|
- `ts/packages/flow/src/storage/embeddings/qdrant-graph.ts`
|
||||||
- `ts/packages/flow/src/storage/embeddings/qdrant-doc.ts`
|
- `ts/packages/flow/src/storage/embeddings/qdrant-doc.ts`
|
||||||
- `ts/packages/flow/src/model/text-completion/*.ts`
|
- `ts/packages/flow/src/model/text-completion/*.ts`
|
||||||
- These files create direct SDK clients and read `process.env` in live
|
- `ts/packages/flow/src/embeddings/ollama.ts`
|
||||||
constructors.
|
- Effect primitives:
|
||||||
- Effect evidence:
|
|
||||||
- `Effect.acquireRelease`, `Layer.scoped`, `Config`, `ConfigProvider`,
|
- `Effect.acquireRelease`, `Layer.scoped`, `Config`, `ConfigProvider`,
|
||||||
`effect/FileSystem`, `effect/unstable/persistence/KeyValueStore`,
|
`Metric`, `Logger`, Effect AI provider layers.
|
||||||
`Metric`, `Logger`.
|
|
||||||
- AI provider modules from installed provider packages, with subtree source
|
|
||||||
proof under `packages/ai/*/src`, including `OpenAiLanguageModel.ts`,
|
|
||||||
`AnthropicLanguageModel.ts`, and `OpenRouterLanguageModel.ts`.
|
|
||||||
- Rewrite shape:
|
- Rewrite shape:
|
||||||
- Move env reading into `Config` loaders and provider-specific layers.
|
- Move env/config reading into `Config` loaders and provider-specific layers.
|
||||||
- Scope SDK clients that need explicit close/disconnect.
|
- Scope SDK clients that need explicit close/disconnect.
|
||||||
- Replace `console` or ad hoc logging with `Effect.log*` and metrics where
|
- Remove `Effect.void as Effect.Effect<undefined>` stream assertions by
|
||||||
useful.
|
letting branch return types infer or by restructuring the stream parser.
|
||||||
- Tests:
|
- Tests:
|
||||||
- Provider config tests with `ConfigProvider.fromMap`.
|
- Provider config tests with `ConfigProvider.fromMap`.
|
||||||
- Storage tests with fake clients before changing real resource lifetimes.
|
- Storage tests with fake clients before changing real resource lifetimes.
|
||||||
- Blockers:
|
|
||||||
- Some third-party SDK clients may not have meaningful finalizers. Mark those
|
|
||||||
no-op after proof instead of forcing fake lifecycle code.
|
|
||||||
|
|
||||||
### P2: Canonicalize MCP Around The Effect Server
|
### P2: Canonicalize MCP Around The Effect Server
|
||||||
|
|
||||||
- Impact: 3
|
- Status:
|
||||||
- Risk: 2
|
- First blocker slice complete: MCP now builds under strict tsgo and the
|
||||||
- Confidence: 5
|
stdio server has an Effect-backed compatibility implementation.
|
||||||
- TrustGraph evidence:
|
- Remaining shape:
|
||||||
- `ts/packages/mcp/src/server.ts` is the old SDK/Zod server.
|
- Decide whether the old SDK/Zod stdio compatibility surface should stay as
|
||||||
- `ts/packages/mcp/src/server-effect.ts` has Effect AI tools, schemas,
|
a wrapper or be removed.
|
||||||
`McpServer`, HTTP API integration, and provider layers.
|
- Add parity tests before deleting any public entry point.
|
||||||
- Effect evidence:
|
|
||||||
- `effect/unstable/ai` `Tool`, `Toolkit`, `McpServer`, `McpSchema`,
|
|
||||||
`LanguageModel`.
|
|
||||||
- Sources: `ts/node_modules/effect/src/unstable/ai/Tool.ts`,
|
|
||||||
`Toolkit.ts`, `McpServer.ts`, `McpSchema.ts`, `LanguageModel.ts`.
|
|
||||||
- Rewrite shape:
|
|
||||||
- Do not rewrite the Effect server from scratch.
|
|
||||||
- Make the Effect server canonical after parity checks.
|
|
||||||
- Keep the old server only as compatibility or delete it once entrypoints and
|
|
||||||
tests prove the Effect path is complete.
|
|
||||||
- Tests:
|
- Tests:
|
||||||
- MCP package build/test.
|
- `cd ts && bun run --cwd packages/mcp build`
|
||||||
- Tool parity diff against `server.ts` before removal.
|
- Root `cd ts && bun run check`
|
||||||
- Blockers:
|
|
||||||
- Needs a policy decision about old SDK server lifetime.
|
|
||||||
|
|
||||||
### P2: Tighten Workbench Platform And Reactivity Usage
|
### P2: Tighten Workbench Platform And Reactivity Usage
|
||||||
|
|
||||||
- Impact: 3
|
|
||||||
- Risk: 2
|
|
||||||
- Confidence: 4
|
|
||||||
- TrustGraph evidence:
|
- TrustGraph evidence:
|
||||||
- `ts/packages/workbench/src/atoms/workbench.ts` already uses Atom,
|
- `ts/packages/workbench/src/atoms/workbench.ts`
|
||||||
AsyncResult, Reactivity, browser layers, and metrics.
|
- Remaining direct browser state includes `localStorage` and DOM theme
|
||||||
- Remaining direct browser state includes `localStorage` reads/writes and DOM
|
inspection.
|
||||||
theme inspection.
|
- Effect primitives:
|
||||||
- Effect evidence:
|
|
||||||
- `BrowserKeyValueStore.layerLocalStorage`,
|
- `BrowserKeyValueStore.layerLocalStorage`,
|
||||||
`BrowserKeyValueStore.layerSessionStorage`, `BrowserHttpClient`,
|
`BrowserKeyValueStore.layerSessionStorage`, `BrowserHttpClient`,
|
||||||
`Clipboard`.
|
`Clipboard`, `AtomRpc`, `AtomHttpApi`, `AtomRegistry`, `AsyncResult`,
|
||||||
- `AtomRpc`, `AtomHttpApi`, `AtomRegistry`, `AsyncResult`, `Reactivity`.
|
`Reactivity`.
|
||||||
- Rewrite shape:
|
- Rewrite shape:
|
||||||
- Leave the workbench out of the first rewrite wave.
|
- Leave workbench out of the next backend/runtime rewrite wave.
|
||||||
- Later, move persistent UI state through `BrowserKeyValueStore` and keep
|
- Move persistent UI state through browser platform services later.
|
||||||
remote state in Atom RPC/HTTP API families if the client API becomes fully
|
|
||||||
typed Effect RPC.
|
|
||||||
- Tests:
|
- Tests:
|
||||||
- `cd ts && bun run workbench:qa`.
|
- `cd ts && bun run workbench:qa`
|
||||||
- Blockers:
|
|
||||||
- Workbench is already the most modern surface. Backend/runtime wins should
|
|
||||||
happen first.
|
|
||||||
|
|
||||||
## Recommended PR Order
|
## Recommended PR Order
|
||||||
|
|
||||||
1. Config service scoped state migration.
|
1. Config service scoped state migration.
|
||||||
2. RAG and agent requestor bridge removal.
|
2. RAG and agent requestor bridge removal.
|
||||||
3. Base consumer facade and subscriber export cleanup.
|
3. Client RPC managed runtime/scoped layer cleanup.
|
||||||
4. Client compatibility facade tightening.
|
4. Base processor registry and constructor shim redesign.
|
||||||
5. Gateway RPC callback and client streaming completion cleanup.
|
5. Gateway RPC callback and client streaming completion cleanup.
|
||||||
6. Storage/provider managed resource cleanup.
|
6. Storage/provider managed resource cleanup.
|
||||||
7. MCP canonicalization and Workbench polish.
|
7. MCP parity/deletion decision and workbench platform polish.
|
||||||
|
|
||||||
## No-Op Rules
|
## No-Op Rules
|
||||||
|
|
||||||
Do not flag these as rewrite blockers without additional proof:
|
Do not flag these as rewrite blockers without additional proof:
|
||||||
|
|
||||||
- Promise-returning CLI actions and Fastify route handlers at external
|
- Promise-returning CLI actions, MCP SDK callbacks, client compatibility
|
||||||
boundaries. This does not exempt normal `Error` construction inside shared
|
methods, and Fastify route handlers at true external boundaries. Boundary
|
||||||
library code.
|
code still must map failures into typed errors or wire errors.
|
||||||
- `try`/`catch` blocks at host/tool boundaries only when the catch maps into a
|
- `try`/`catch` blocks at host/tool boundaries only when the catch maps into a
|
||||||
typed error or wire error. Internal exception capture should use `Effect.try`,
|
typed error or a wire-contract error. Internal exception capture should use
|
||||||
`Effect.tryPromise`, or `Result.try`.
|
`Effect.try`, `Effect.tryPromise`, or `Result.try`.
|
||||||
- `S.Class`, `S.TaggedErrorClass`, `Context.Service`, `Rpc.make`, and
|
- `S.Class`, `S.TaggedErrorClass`, `Context.Service`, `Rpc.make`, and
|
||||||
`HttpApi.make` when they are required or idiomatic for the Effect API.
|
`HttpApi.make` when they are required or idiomatic for the Effect API.
|
||||||
- Plain `Map` usage for local pure transformations, such as graph utility
|
- Plain `Map` usage for local pure transformations, such as graph utility
|
||||||
construction, unless the state is long-lived, mutable service state.
|
construction, unless the state is long-lived mutable service state.
|
||||||
- JSON stringification that is part of the TrustGraph wire contract, unless a
|
- JSON stringification in tests or wire-contract fixtures. Production JSON
|
||||||
schema codec can preserve the exact encoded form.
|
encode/decode should prefer schema codecs when the encoded form can be
|
||||||
|
preserved.
|
||||||
|
|
||||||
## Acceptance
|
## Acceptance For Final Loop Completion
|
||||||
|
|
||||||
This audit is complete when:
|
The overall playbook loop is complete only when:
|
||||||
|
|
||||||
- `ts/EFFECT_NATIVE_REWRITE_PLAYBOOK.md` exists.
|
- All remaining playbook signal matches are migrated or documented as no-op
|
||||||
- This ranked audit exists and cites concrete TrustGraph and Effect surfaces.
|
external-boundary cases with concrete evidence.
|
||||||
- `git diff --check` passes for both files.
|
- No P0/P1/P2 migration item remains in this audit.
|
||||||
- No code rewrite is mixed into this audit.
|
- `cd ts && bun run check`, `cd ts && bun run build`, `cd ts && bun run test`,
|
||||||
|
and `git diff --check` pass after the final migration slice.
|
||||||
|
|
|
||||||
|
|
@ -115,11 +115,10 @@ const makeNativeRecordingProcessor = (
|
||||||
events.push(`pubsub:${pubsub.backend.constructor.name}`);
|
events.push(`pubsub:${pubsub.backend.constructor.name}`);
|
||||||
}),
|
}),
|
||||||
});
|
});
|
||||||
const stopEffect = processor.stopEffect;
|
processor.onShutdown(() => {
|
||||||
processor.stopEffect = () => {
|
|
||||||
events.push("native-stop");
|
events.push("native-stop");
|
||||||
return stopEffect();
|
return Promise.resolve();
|
||||||
};
|
});
|
||||||
return processor;
|
return processor;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,8 @@ import {
|
||||||
AckPolicy,
|
AckPolicy,
|
||||||
DeliverPolicy,
|
DeliverPolicy,
|
||||||
} from "nats";
|
} from "nats";
|
||||||
|
import { Effect } from "effect";
|
||||||
|
import * as Predicate from "effect/Predicate";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
|
|
||||||
import type {
|
import type {
|
||||||
|
|
@ -29,6 +31,7 @@ import type {
|
||||||
CreateConsumerOptions,
|
CreateConsumerOptions,
|
||||||
Message,
|
Message,
|
||||||
} from "./types.js";
|
} from "./types.js";
|
||||||
|
import { pubSubError } from "../errors.js";
|
||||||
|
|
||||||
const sc = StringCodec();
|
const sc = StringCodec();
|
||||||
|
|
||||||
|
|
@ -57,36 +60,61 @@ function makeNatsMessage<T>(msg: JsMsg, decoded: T): NatsMessage<T> {
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const hasJsMsg = Predicate.hasProperty("_jsMsg");
|
||||||
|
|
||||||
|
function isAckableJsMsg(value: unknown): value is Pick<JsMsg, "ack" | "nak"> {
|
||||||
|
if (!Predicate.isObject(value)) return false;
|
||||||
|
if (!Predicate.hasProperty(value, "ack")) return false;
|
||||||
|
if (!Predicate.hasProperty(value, "nak")) return false;
|
||||||
|
return typeof value.ack === "function" && typeof value.nak === "function";
|
||||||
|
}
|
||||||
|
|
||||||
|
function isNatsMessage<T>(message: Message<T>): message is NatsMessage<T> {
|
||||||
|
return hasJsMsg(message) && isAckableJsMsg(message._jsMsg);
|
||||||
|
}
|
||||||
|
|
||||||
function makeNatsProducer<T>(
|
function makeNatsProducer<T>(
|
||||||
js: JetStreamClient,
|
js: JetStreamClient,
|
||||||
subject: string,
|
subject: string,
|
||||||
schema?: S.Top,
|
schema?: S.Codec<T, unknown>,
|
||||||
): BackendProducer<T> {
|
): BackendProducer<T> {
|
||||||
return {
|
return {
|
||||||
send: async (message, properties) => {
|
send: (message, properties) =>
|
||||||
const encoded = schema !== undefined
|
Effect.runPromise(
|
||||||
? S.encodeUnknownSync(schema as S.Codec<unknown, unknown>)(message)
|
Effect.gen(function* () {
|
||||||
: message;
|
const encoded = schema !== undefined
|
||||||
const data = sc.encode(JSON.stringify(encoded));
|
? yield* S.encodeUnknownEffect(schema)(message).pipe(
|
||||||
const opts: Record<string, unknown> = {};
|
Effect.mapError((error) => pubSubError(`encode:${subject}`, error)),
|
||||||
|
)
|
||||||
|
: message;
|
||||||
|
const json = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(encoded).pipe(
|
||||||
|
Effect.mapError((error) => pubSubError(`encode-json:${subject}`, error)),
|
||||||
|
);
|
||||||
|
const data = sc.encode(json);
|
||||||
|
const opts: Record<string, unknown> = {};
|
||||||
|
|
||||||
if (properties !== undefined && Object.keys(properties).length > 0) {
|
if (properties !== undefined && Object.keys(properties).length > 0) {
|
||||||
const { headers } = await import("nats");
|
const { headers } = yield* Effect.tryPromise({
|
||||||
const hdrs = headers();
|
try: () => import("nats"),
|
||||||
for (const [key, val] of Object.entries(properties)) {
|
catch: (error) => pubSubError("import:nats-headers", error),
|
||||||
hdrs.append(key, val);
|
});
|
||||||
}
|
const hdrs = headers();
|
||||||
opts.headers = hdrs;
|
for (const [key, val] of Object.entries(properties)) {
|
||||||
}
|
hdrs.append(key, val);
|
||||||
|
}
|
||||||
|
opts.headers = hdrs;
|
||||||
|
}
|
||||||
|
|
||||||
await js.publish(subject, data, opts);
|
yield* Effect.tryPromise({
|
||||||
},
|
try: () => js.publish(subject, data, opts),
|
||||||
flush: async () => {
|
catch: (error) => pubSubError(`publish:${subject}`, error),
|
||||||
// NATS publishes are flushed on the connection level.
|
});
|
||||||
},
|
}),
|
||||||
close: async () => {
|
),
|
||||||
// No per-producer cleanup needed for NATS.
|
// NATS publishes are flushed on the connection level.
|
||||||
},
|
flush: () => Promise.resolve(),
|
||||||
|
// No per-producer cleanup needed for NATS.
|
||||||
|
close: () => Promise.resolve(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -101,60 +129,109 @@ function makeNatsConsumer<T>(
|
||||||
subscription: string,
|
subscription: string,
|
||||||
initialPosition: "latest" | "earliest",
|
initialPosition: "latest" | "earliest",
|
||||||
streamName: string,
|
streamName: string,
|
||||||
schema?: S.Top,
|
schema?: S.Codec<T, unknown>,
|
||||||
): InitializableBackendConsumer<T> {
|
): InitializableBackendConsumer<T> {
|
||||||
let consumer: NatsJsConsumer | null = null;
|
let consumer: NatsJsConsumer | null = null;
|
||||||
|
|
||||||
return {
|
return {
|
||||||
init: async () => {
|
init: () =>
|
||||||
// Stream is already ensured by makeNatsBackend(). Create or bind to a durable consumer.
|
Effect.runPromise(
|
||||||
try {
|
Effect.gen(function* () {
|
||||||
consumer = await js.consumers.get(streamName, subscription);
|
const existing = yield* Effect.tryPromise({
|
||||||
} catch {
|
try: () => js.consumers.get(streamName, subscription),
|
||||||
const deliverPolicy =
|
catch: (error) => pubSubError(`get-consumer:${streamName}:${subscription}`, error),
|
||||||
initialPosition === "earliest"
|
}).pipe(
|
||||||
? DeliverPolicy.All
|
Effect.catch(() =>
|
||||||
: DeliverPolicy.New;
|
Effect.gen(function* () {
|
||||||
|
const deliverPolicy =
|
||||||
|
initialPosition === "earliest"
|
||||||
|
? DeliverPolicy.All
|
||||||
|
: DeliverPolicy.New;
|
||||||
|
|
||||||
await jsm.consumers.add(streamName, {
|
yield* Effect.tryPromise({
|
||||||
durable_name: subscription,
|
try: () =>
|
||||||
ack_policy: AckPolicy.Explicit,
|
jsm.consumers.add(streamName, {
|
||||||
deliver_policy: deliverPolicy,
|
durable_name: subscription,
|
||||||
filter_subject: subject,
|
ack_policy: AckPolicy.Explicit,
|
||||||
});
|
deliver_policy: deliverPolicy,
|
||||||
|
filter_subject: subject,
|
||||||
|
}),
|
||||||
|
catch: (error) => pubSubError(`add-consumer:${streamName}:${subscription}`, error),
|
||||||
|
});
|
||||||
|
|
||||||
consumer = await js.consumers.get(streamName, subscription);
|
return yield* Effect.tryPromise({
|
||||||
}
|
try: () => js.consumers.get(streamName, subscription),
|
||||||
},
|
catch: (error) => pubSubError(`get-consumer:${streamName}:${subscription}`, error),
|
||||||
receive: async (timeoutMs = 2000) => {
|
});
|
||||||
if (consumer === null) throw new Error("Consumer not initialized");
|
}),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
consumer = existing;
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
receive: (timeoutMs = 2000) =>
|
||||||
|
Effect.runPromise(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
const current = consumer;
|
||||||
|
if (current === null) {
|
||||||
|
return yield* pubSubError("receive", "Consumer not initialized");
|
||||||
|
}
|
||||||
|
|
||||||
// Pull a single message with a timeout using the pull-based API.
|
// Pull a single message with a timeout using the pull-based API.
|
||||||
// consumer.next() returns a JsMsg or null when the timeout expires.
|
// consumer.next() returns a JsMsg or null when the timeout expires.
|
||||||
const msg = await consumer.next({ expires: timeoutMs });
|
const msg = yield* Effect.tryPromise({
|
||||||
if (msg === null) return null;
|
try: () => current.next({ expires: timeoutMs }),
|
||||||
|
catch: (error) => pubSubError(`receive:${subject}`, error),
|
||||||
|
});
|
||||||
|
if (msg === null) return null;
|
||||||
|
|
||||||
const parsed = JSON.parse(sc.decode(msg.data));
|
const parsed = yield* S.decodeUnknownEffect(S.UnknownFromJsonString)(sc.decode(msg.data)).pipe(
|
||||||
const decoded = schema !== undefined
|
Effect.mapError((error) => pubSubError(`decode-json:${subject}`, error)),
|
||||||
? S.decodeUnknownSync(schema as S.Codec<unknown, unknown>)(parsed) as T
|
);
|
||||||
: parsed as T;
|
const decoded = schema !== undefined
|
||||||
return makeNatsMessage(msg, decoded);
|
? yield* S.decodeUnknownEffect(schema)(parsed).pipe(
|
||||||
},
|
Effect.mapError((error) => pubSubError(`decode-schema:${subject}`, error)),
|
||||||
acknowledge: async (message) => {
|
)
|
||||||
const natsMsg = message as NatsMessage<T>;
|
: yield* S.decodeUnknownEffect(S.Any)(parsed).pipe(
|
||||||
natsMsg._jsMsg.ack();
|
Effect.mapError((error) => pubSubError(`decode-any:${subject}`, error)),
|
||||||
},
|
);
|
||||||
negativeAcknowledge: async (message) => {
|
return makeNatsMessage(msg, decoded);
|
||||||
const natsMsg = message as NatsMessage<T>;
|
}),
|
||||||
natsMsg._jsMsg.nak();
|
),
|
||||||
},
|
acknowledge: (message) =>
|
||||||
unsubscribe: async () => {
|
Effect.runPromise(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
if (!isNatsMessage(message)) {
|
||||||
|
return yield* pubSubError(`acknowledge:${subject}`, "Message was not produced by NATS backend");
|
||||||
|
}
|
||||||
|
yield* Effect.sync(() => {
|
||||||
|
message._jsMsg.ack();
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
negativeAcknowledge: (message) =>
|
||||||
|
Effect.runPromise(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
if (!isNatsMessage(message)) {
|
||||||
|
return yield* pubSubError(
|
||||||
|
`negative-acknowledge:${subject}`,
|
||||||
|
"Message was not produced by NATS backend",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
yield* Effect.sync(() => {
|
||||||
|
message._jsMsg.nak();
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
unsubscribe: () => {
|
||||||
// The pull-based consumer does not have a persistent subscription to drain.
|
// The pull-based consumer does not have a persistent subscription to drain.
|
||||||
// Clearing the reference is sufficient; the durable consumer persists server-side.
|
// Clearing the reference is sufficient; the durable consumer persists server-side.
|
||||||
consumer = null;
|
consumer = null;
|
||||||
|
return Promise.resolve();
|
||||||
},
|
},
|
||||||
close: async () => {
|
close: () => {
|
||||||
consumer = null;
|
consumer = null;
|
||||||
|
return Promise.resolve();
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
@ -165,19 +242,26 @@ export function makeNatsBackend(url = "nats://localhost:4222"): PubSubBackend {
|
||||||
let jsm: JetStreamManager | null = null;
|
let jsm: JetStreamManager | null = null;
|
||||||
const initializedStreams = new Set<string>();
|
const initializedStreams = new Set<string>();
|
||||||
|
|
||||||
const ensureConnected = async (): Promise<void> => {
|
const ensureConnected = Effect.fn("NatsBackend.ensureConnected")(function* () {
|
||||||
if (connection === null) {
|
if (connection === null) {
|
||||||
connection = await connect({ servers: url });
|
const conn = yield* Effect.tryPromise({
|
||||||
js = connection.jetstream();
|
try: () => connect({ servers: url }),
|
||||||
jsm = await connection.jetstreamManager();
|
catch: (error) => pubSubError("connect", error),
|
||||||
|
});
|
||||||
|
connection = conn;
|
||||||
|
js = conn.jetstream();
|
||||||
|
jsm = yield* Effect.tryPromise({
|
||||||
|
try: () => conn.jetstreamManager(),
|
||||||
|
catch: (error) => pubSubError("jetstream-manager", error),
|
||||||
|
});
|
||||||
}
|
}
|
||||||
};
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Ensure the stream for a given subject exists with a wildcard filter.
|
* Ensure the stream for a given subject exists with a wildcard filter.
|
||||||
* E.g. subject "tg.flow.config-request" → stream "tg_flow" with subjects ["tg.flow.>"]
|
* E.g. subject "tg.flow.config-request" → stream "tg_flow" with subjects ["tg.flow.>"]
|
||||||
*/
|
*/
|
||||||
const ensureStream = async (subject: string): Promise<string> => {
|
const ensureStream = Effect.fn("NatsBackend.ensureStream")(function* (subject: string) {
|
||||||
const parts = subject.split(".");
|
const parts = subject.split(".");
|
||||||
const streamName = parts.slice(0, 2).join("_");
|
const streamName = parts.slice(0, 2).join("_");
|
||||||
|
|
||||||
|
|
@ -186,53 +270,78 @@ export function makeNatsBackend(url = "nats://localhost:4222"): PubSubBackend {
|
||||||
const wildcardSubject = `${parts.slice(0, 2).join(".")}.>`;
|
const wildcardSubject = `${parts.slice(0, 2).join(".")}.>`;
|
||||||
|
|
||||||
const manager = jsm;
|
const manager = jsm;
|
||||||
if (manager === null) throw new Error("NATS backend not connected");
|
if (manager === null) return yield* pubSubError("ensure-stream", "NATS backend not connected");
|
||||||
|
|
||||||
try {
|
yield* Effect.tryPromise({
|
||||||
await manager.streams.info(streamName);
|
try: () => manager.streams.info(streamName),
|
||||||
} catch {
|
catch: (error) => pubSubError(`stream-info:${streamName}`, error),
|
||||||
await manager.streams.add({
|
}).pipe(
|
||||||
name: streamName,
|
Effect.catch(() =>
|
||||||
subjects: [wildcardSubject],
|
Effect.tryPromise({
|
||||||
});
|
try: () =>
|
||||||
}
|
manager.streams.add({
|
||||||
|
name: streamName,
|
||||||
|
subjects: [wildcardSubject],
|
||||||
|
}),
|
||||||
|
catch: (error) => pubSubError(`stream-add:${streamName}`, error),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
);
|
||||||
initializedStreams.add(streamName);
|
initializedStreams.add(streamName);
|
||||||
return streamName;
|
return streamName;
|
||||||
};
|
});
|
||||||
|
|
||||||
return {
|
return {
|
||||||
createProducer: async <T>(options: CreateProducerOptions) => {
|
createProducer: <T>(options: CreateProducerOptions<T>) =>
|
||||||
await ensureConnected();
|
Effect.runPromise(
|
||||||
await ensureStream(options.topic);
|
Effect.gen(function* () {
|
||||||
const client = js;
|
yield* ensureConnected();
|
||||||
if (client === null) throw new Error("NATS backend not connected");
|
yield* ensureStream(options.topic);
|
||||||
return makeNatsProducer<T>(client, options.topic, options.schema);
|
const client = js;
|
||||||
},
|
if (client === null) return yield* pubSubError("create-producer", "NATS backend not connected");
|
||||||
createConsumer: async <T>(options: CreateConsumerOptions) => {
|
return makeNatsProducer<T>(client, options.topic, options.schema);
|
||||||
await ensureConnected();
|
}),
|
||||||
const streamName = await ensureStream(options.topic);
|
),
|
||||||
const client = js;
|
createConsumer: <T>(options: CreateConsumerOptions<T>) =>
|
||||||
const manager = jsm;
|
Effect.runPromise(
|
||||||
if (client === null || manager === null) throw new Error("NATS backend not connected");
|
Effect.gen(function* () {
|
||||||
const consumer = makeNatsConsumer<T>(
|
yield* ensureConnected();
|
||||||
client,
|
const streamName = yield* ensureStream(options.topic);
|
||||||
manager,
|
const client = js;
|
||||||
options.topic,
|
const manager = jsm;
|
||||||
options.subscription,
|
if (client === null || manager === null) {
|
||||||
options.initialPosition ?? "latest",
|
return yield* pubSubError("create-consumer", "NATS backend not connected");
|
||||||
streamName,
|
}
|
||||||
options.schema,
|
const consumer = makeNatsConsumer<T>(
|
||||||
);
|
client,
|
||||||
await consumer.init();
|
manager,
|
||||||
return consumer;
|
options.topic,
|
||||||
},
|
options.subscription,
|
||||||
close: async () => {
|
options.initialPosition ?? "latest",
|
||||||
if (connection !== null) {
|
streamName,
|
||||||
await connection.drain();
|
options.schema,
|
||||||
connection = null;
|
);
|
||||||
js = null;
|
yield* Effect.tryPromise({
|
||||||
jsm = null;
|
try: () => consumer.init(),
|
||||||
}
|
catch: (error) => pubSubError(`init-consumer:${options.topic}`, error),
|
||||||
},
|
});
|
||||||
|
return consumer;
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
close: () =>
|
||||||
|
Effect.runPromise(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
const conn = connection;
|
||||||
|
if (conn !== null) {
|
||||||
|
yield* Effect.tryPromise({
|
||||||
|
try: () => conn.drain(),
|
||||||
|
catch: (error) => pubSubError("close", error),
|
||||||
|
});
|
||||||
|
connection = null;
|
||||||
|
js = null;
|
||||||
|
jsm = null;
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -20,10 +20,10 @@ import { pubSubError } from "../errors.js";
|
||||||
export interface PubSubService {
|
export interface PubSubService {
|
||||||
readonly backend: PubSubBackend;
|
readonly backend: PubSubBackend;
|
||||||
readonly createProducer: <T>(
|
readonly createProducer: <T>(
|
||||||
options: CreateProducerOptions,
|
options: CreateProducerOptions<T>,
|
||||||
) => Effect.Effect<BackendProducer<T>, ReturnType<typeof pubSubError>>;
|
) => Effect.Effect<BackendProducer<T>, ReturnType<typeof pubSubError>>;
|
||||||
readonly createConsumer: <T>(
|
readonly createConsumer: <T>(
|
||||||
options: CreateConsumerOptions,
|
options: CreateConsumerOptions<T>,
|
||||||
) => Effect.Effect<BackendConsumer<T>, ReturnType<typeof pubSubError>>;
|
) => Effect.Effect<BackendConsumer<T>, ReturnType<typeof pubSubError>>;
|
||||||
readonly close: Effect.Effect<void, ReturnType<typeof pubSubError>>;
|
readonly close: Effect.Effect<void, ReturnType<typeof pubSubError>>;
|
||||||
}
|
}
|
||||||
|
|
@ -41,12 +41,12 @@ export class PubSub extends Context.Service<PubSub, PubSubService>()("@trustgrap
|
||||||
export function makePubSubService(backend: PubSubBackend): PubSubService {
|
export function makePubSubService(backend: PubSubBackend): PubSubService {
|
||||||
return {
|
return {
|
||||||
backend,
|
backend,
|
||||||
createProducer: <T>(options: CreateProducerOptions) =>
|
createProducer: <T>(options: CreateProducerOptions<T>) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => backend.createProducer<T>(options),
|
try: () => backend.createProducer<T>(options),
|
||||||
catch: (error) => pubSubError(`createProducer:${options.topic}`, error),
|
catch: (error) => pubSubError(`createProducer:${options.topic}`, error),
|
||||||
}),
|
}),
|
||||||
createConsumer: <T>(options: CreateConsumerOptions) =>
|
createConsumer: <T>(options: CreateConsumerOptions<T>) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => backend.createConsumer<T>(options),
|
try: () => backend.createConsumer<T>(options),
|
||||||
catch: (error) => pubSubError(`createConsumer:${options.topic}`, error),
|
catch: (error) => pubSubError(`createConsumer:${options.topic}`, error),
|
||||||
|
|
|
||||||
|
|
@ -29,21 +29,21 @@ export interface BackendConsumer<T = unknown> {
|
||||||
export type ConsumerType = "shared" | "exclusive" | "failover";
|
export type ConsumerType = "shared" | "exclusive" | "failover";
|
||||||
export type InitialPosition = "latest" | "earliest";
|
export type InitialPosition = "latest" | "earliest";
|
||||||
|
|
||||||
export interface CreateProducerOptions {
|
export interface CreateProducerOptions<T = unknown> {
|
||||||
topic: string;
|
topic: string;
|
||||||
schema?: S.Top;
|
schema?: S.Codec<T, unknown>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CreateConsumerOptions {
|
export interface CreateConsumerOptions<T = unknown> {
|
||||||
topic: string;
|
topic: string;
|
||||||
subscription: string;
|
subscription: string;
|
||||||
initialPosition?: InitialPosition;
|
initialPosition?: InitialPosition;
|
||||||
consumerType?: ConsumerType;
|
consumerType?: ConsumerType;
|
||||||
schema?: S.Top;
|
schema?: S.Codec<T, unknown>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface PubSubBackend {
|
export interface PubSubBackend {
|
||||||
createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>>;
|
createProducer<T>(options: CreateProducerOptions<T>): Promise<BackendProducer<T>>;
|
||||||
createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>>;
|
createConsumer<T>(options: CreateConsumerOptions<T>): Promise<BackendConsumer<T>>;
|
||||||
close(): Promise<void>;
|
close(): Promise<void>;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -168,11 +168,11 @@ export type MessagingRuntimeError =
|
||||||
| FlowResourceNotFoundError;
|
| FlowResourceNotFoundError;
|
||||||
|
|
||||||
export function tooManyRequestsError(message = "Rate limit exceeded"): TooManyRequestsError {
|
export function tooManyRequestsError(message = "Rate limit exceeded"): TooManyRequestsError {
|
||||||
return new TooManyRequestsError({ message });
|
return TooManyRequestsError.make({ message });
|
||||||
}
|
}
|
||||||
|
|
||||||
export function llmError(message: string, errorType = "llm-error"): LlmError {
|
export function llmError(message: string, errorType = "llm-error"): LlmError {
|
||||||
return new LlmError({ message, errorType });
|
return LlmError.make({ message, errorType });
|
||||||
}
|
}
|
||||||
|
|
||||||
export function embeddingsError(
|
export function embeddingsError(
|
||||||
|
|
@ -180,7 +180,7 @@ export function embeddingsError(
|
||||||
error: unknown,
|
error: unknown,
|
||||||
provider?: string,
|
provider?: string,
|
||||||
): EmbeddingsError {
|
): EmbeddingsError {
|
||||||
return new EmbeddingsError({
|
return EmbeddingsError.make({
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
...(provider === undefined ? {} : { provider }),
|
...(provider === undefined ? {} : { provider }),
|
||||||
|
|
@ -188,11 +188,11 @@ export function embeddingsError(
|
||||||
}
|
}
|
||||||
|
|
||||||
export function parseError(message: string): ParseError {
|
export function parseError(message: string): ParseError {
|
||||||
return new ParseError({ message });
|
return ParseError.make({ message });
|
||||||
}
|
}
|
||||||
|
|
||||||
export function pubSubError(operation: string, error: unknown): PubSubError {
|
export function pubSubError(operation: string, error: unknown): PubSubError {
|
||||||
return new PubSubError({ operation, message: errorMessage(error) });
|
return PubSubError.make({ operation, message: errorMessage(error) });
|
||||||
}
|
}
|
||||||
|
|
||||||
export function processorLifecycleError(
|
export function processorLifecycleError(
|
||||||
|
|
@ -200,7 +200,7 @@ export function processorLifecycleError(
|
||||||
operation: string,
|
operation: string,
|
||||||
error: unknown,
|
error: unknown,
|
||||||
): ProcessorLifecycleError {
|
): ProcessorLifecycleError {
|
||||||
return new ProcessorLifecycleError({
|
return ProcessorLifecycleError.make({
|
||||||
processorId,
|
processorId,
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
|
|
@ -212,7 +212,7 @@ export function messagingLifecycleError(
|
||||||
operation: string,
|
operation: string,
|
||||||
error: unknown,
|
error: unknown,
|
||||||
): MessagingLifecycleError {
|
): MessagingLifecycleError {
|
||||||
return new MessagingLifecycleError({
|
return MessagingLifecycleError.make({
|
||||||
resource,
|
resource,
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
|
|
@ -224,7 +224,7 @@ export function messagingDeliveryError(
|
||||||
operation: string,
|
operation: string,
|
||||||
error: unknown,
|
error: unknown,
|
||||||
): MessagingDeliveryError {
|
): MessagingDeliveryError {
|
||||||
return new MessagingDeliveryError({
|
return MessagingDeliveryError.make({
|
||||||
topic,
|
topic,
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
|
|
@ -236,7 +236,7 @@ export function messagingDecodeError(
|
||||||
error: unknown,
|
error: unknown,
|
||||||
topic?: string,
|
topic?: string,
|
||||||
): MessagingDecodeError {
|
): MessagingDecodeError {
|
||||||
return new MessagingDecodeError({
|
return MessagingDecodeError.make({
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
...(topic === undefined ? {} : { topic }),
|
...(topic === undefined ? {} : { topic }),
|
||||||
|
|
@ -247,7 +247,7 @@ export function messagingTimeoutError(
|
||||||
operation: string,
|
operation: string,
|
||||||
timeoutMs: number,
|
timeoutMs: number,
|
||||||
): MessagingTimeoutError {
|
): MessagingTimeoutError {
|
||||||
return new MessagingTimeoutError({
|
return MessagingTimeoutError.make({
|
||||||
operation,
|
operation,
|
||||||
timeoutMs,
|
timeoutMs,
|
||||||
message: `${operation} timed out after ${timeoutMs}ms`,
|
message: `${operation} timed out after ${timeoutMs}ms`,
|
||||||
|
|
@ -259,7 +259,7 @@ export function messagingHandlerError(
|
||||||
subscription: string,
|
subscription: string,
|
||||||
error: unknown,
|
error: unknown,
|
||||||
): MessagingHandlerError {
|
): MessagingHandlerError {
|
||||||
return new MessagingHandlerError({
|
return MessagingHandlerError.make({
|
||||||
topic,
|
topic,
|
||||||
subscription,
|
subscription,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
|
|
@ -271,7 +271,7 @@ export function flowRuntimeError(
|
||||||
operation: string,
|
operation: string,
|
||||||
error: unknown,
|
error: unknown,
|
||||||
): FlowRuntimeError {
|
): FlowRuntimeError {
|
||||||
return new FlowRuntimeError({
|
return FlowRuntimeError.make({
|
||||||
flowName,
|
flowName,
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(error),
|
message: errorMessage(error),
|
||||||
|
|
@ -283,7 +283,7 @@ export function flowResourceNotFoundError(
|
||||||
resourceType: FlowResourceNotFoundError["resourceType"],
|
resourceType: FlowResourceNotFoundError["resourceType"],
|
||||||
resourceName: string,
|
resourceName: string,
|
||||||
): FlowResourceNotFoundError {
|
): FlowResourceNotFoundError {
|
||||||
return new FlowResourceNotFoundError({
|
return FlowResourceNotFoundError.make({
|
||||||
flowName,
|
flowName,
|
||||||
resourceType,
|
resourceType,
|
||||||
resourceName,
|
resourceName,
|
||||||
|
|
|
||||||
|
|
@ -4,9 +4,16 @@
|
||||||
* Python reference: trustgraph-base/trustgraph/base/consumer.py
|
* Python reference: trustgraph-base/trustgraph/base/consumer.py
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { PubSubBackend, BackendConsumer, Message } from "../backend/types.js";
|
import type { BackendConsumer, Message, PubSubBackend } from "../backend/types.js";
|
||||||
import type { Flow } from "../processor/flow.js";
|
import type { Flow } from "../processor/flow.js";
|
||||||
import { TooManyRequestsError } from "../errors.js";
|
import {
|
||||||
|
MessagingHandlerError,
|
||||||
|
TooManyRequestsError,
|
||||||
|
messagingDeliveryError,
|
||||||
|
messagingHandlerError,
|
||||||
|
messagingLifecycleError,
|
||||||
|
} from "../errors.js";
|
||||||
|
import { Duration, Effect } from "effect";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
|
|
||||||
export type MessageHandler<T> = (
|
export type MessageHandler<T> = (
|
||||||
|
|
@ -44,83 +51,140 @@ export interface Consumer<T> {
|
||||||
export function makeConsumer<T>(options: ConsumerOptions<T>): Consumer<T> {
|
export function makeConsumer<T>(options: ConsumerOptions<T>): Consumer<T> {
|
||||||
let backend: BackendConsumer<T> | null = null;
|
let backend: BackendConsumer<T> | null = null;
|
||||||
let running = false;
|
let running = false;
|
||||||
let abortController = new AbortController();
|
const isTooManyRequestsError = S.is(TooManyRequestsError);
|
||||||
const concurrency = options.concurrency ?? 1;
|
const concurrency = options.concurrency ?? 1;
|
||||||
const rateLimitRetryMs = options.rateLimitRetryMs ?? 10_000;
|
const rateLimitRetryMs = options.rateLimitRetryMs ?? 10_000;
|
||||||
|
|
||||||
const handleWithRetry = async (msg: Message<T>, flow: FlowContext): Promise<void> => {
|
const runHandler = (
|
||||||
try {
|
message: T,
|
||||||
await options.handler(msg.value(), msg.properties(), flow);
|
properties: Record<string, string>,
|
||||||
} catch (err) {
|
flow: FlowContext,
|
||||||
if (S.is(TooManyRequestsError)(err)) {
|
): Effect.Effect<void, TooManyRequestsError | MessagingHandlerError> =>
|
||||||
console.warn(`[Consumer] Rate limited, retrying in ${rateLimitRetryMs}ms`);
|
Effect.tryPromise({
|
||||||
await sleep(rateLimitRetryMs);
|
try: () => options.handler(message, properties, flow),
|
||||||
await options.handler(msg.value(), msg.properties(), flow);
|
catch: (error) =>
|
||||||
} else {
|
isTooManyRequestsError(error)
|
||||||
throw err;
|
? error
|
||||||
}
|
: messagingHandlerError(options.topic, options.subscription, error),
|
||||||
|
});
|
||||||
|
|
||||||
|
const handleWithRetry = Effect.fn("Consumer.handleWithRetry")(function* (
|
||||||
|
message: Message<T>,
|
||||||
|
flow: FlowContext,
|
||||||
|
) {
|
||||||
|
const callHandler = runHandler(message.value(), message.properties(), flow);
|
||||||
|
yield* callHandler.pipe(
|
||||||
|
Effect.catchTag("TooManyRequestsError", () =>
|
||||||
|
Effect.logWarning("[Consumer] Rate limited, retrying", {
|
||||||
|
topic: options.topic,
|
||||||
|
subscription: options.subscription,
|
||||||
|
retryMs: rateLimitRetryMs,
|
||||||
|
}).pipe(
|
||||||
|
Effect.flatMap(() => Effect.sleep(Duration.millis(rateLimitRetryMs))),
|
||||||
|
Effect.flatMap(() => callHandler),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
const consumeOnce = Effect.fn("Consumer.consumeOnce")(function* (flow: FlowContext) {
|
||||||
|
const currentBackend = backend;
|
||||||
|
if (currentBackend === null) {
|
||||||
|
return yield* messagingLifecycleError(
|
||||||
|
`${options.topic}:${options.subscription}`,
|
||||||
|
"receive",
|
||||||
|
"Consumer backend not started",
|
||||||
|
);
|
||||||
}
|
}
|
||||||
};
|
|
||||||
|
|
||||||
const consumeLoop = async (flow: FlowContext): Promise<void> => {
|
const message = yield* Effect.tryPromise({
|
||||||
while (running) {
|
try: () => currentBackend.receive(2000),
|
||||||
let msg: Message<T> | null = null;
|
catch: (error) => messagingDeliveryError(options.topic, "receive", error),
|
||||||
try {
|
});
|
||||||
const currentBackend = backend;
|
if (message === null) return;
|
||||||
if (currentBackend === null) throw new Error("Consumer backend not started");
|
|
||||||
|
|
||||||
msg = await currentBackend.receive(2000);
|
yield* handleWithRetry(message, flow).pipe(
|
||||||
if (msg === null) continue;
|
Effect.flatMap(() =>
|
||||||
|
Effect.tryPromise({
|
||||||
|
try: () => currentBackend.acknowledge(message),
|
||||||
|
catch: (error) => messagingDeliveryError(options.topic, "acknowledge", error),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
Effect.catch((error) =>
|
||||||
|
Effect.tryPromise({
|
||||||
|
try: () => currentBackend.negativeAcknowledge(message),
|
||||||
|
catch: (nakError) => messagingDeliveryError(options.topic, "negative-acknowledge", nakError),
|
||||||
|
}).pipe(
|
||||||
|
Effect.catch((nakError) =>
|
||||||
|
Effect.logError("[Consumer] Failed to negative-acknowledge message", {
|
||||||
|
error: nakError.message,
|
||||||
|
topic: nakError.topic,
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
Effect.flatMap(() => Effect.fail(error)),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
await handleWithRetry(msg, flow);
|
const consumeLoop = Effect.fn("Consumer.consumeLoop")(function* (flow: FlowContext) {
|
||||||
await currentBackend.acknowledge(msg);
|
yield* Effect.whileLoop({
|
||||||
} catch (err) {
|
while: () => running,
|
||||||
if (!running) break;
|
body: () =>
|
||||||
console.error("[Consumer] Error in consume loop:", err);
|
consumeOnce(flow).pipe(
|
||||||
if (msg !== null) {
|
Effect.catch((error) => {
|
||||||
try {
|
if (!running) return Effect.void;
|
||||||
const currentBackend = backend;
|
return Effect.logError("[Consumer] Error in consume loop", {
|
||||||
if (currentBackend !== null) {
|
error: error.message,
|
||||||
await currentBackend.negativeAcknowledge(msg);
|
topic: options.topic,
|
||||||
}
|
subscription: options.subscription,
|
||||||
} catch (nakErr) {
|
}).pipe(
|
||||||
console.error("[Consumer] Failed to nak message:", nakErr);
|
Effect.flatMap(() => Effect.sleep(Duration.millis(1000))),
|
||||||
}
|
);
|
||||||
}
|
}),
|
||||||
await sleep(1000);
|
),
|
||||||
}
|
step: () => undefined,
|
||||||
}
|
});
|
||||||
};
|
});
|
||||||
|
|
||||||
return {
|
return {
|
||||||
start: async (flow) => {
|
start: (flow) =>
|
||||||
backend = await options.pubsub.createConsumer<T>({
|
Effect.runPromise(
|
||||||
topic: options.topic,
|
Effect.gen(function* () {
|
||||||
subscription: options.subscription,
|
backend = yield* Effect.tryPromise({
|
||||||
initialPosition: options.initialPosition ?? "latest",
|
try: () =>
|
||||||
});
|
options.pubsub.createConsumer<T>({
|
||||||
|
topic: options.topic,
|
||||||
|
subscription: options.subscription,
|
||||||
|
initialPosition: options.initialPosition ?? "latest",
|
||||||
|
}),
|
||||||
|
catch: (error) =>
|
||||||
|
messagingLifecycleError(`${options.topic}:${options.subscription}`, "create-consumer", error),
|
||||||
|
});
|
||||||
|
|
||||||
running = true;
|
running = true;
|
||||||
|
|
||||||
// Spawn concurrent consumer tasks.
|
const workerIndexes = Array.from({ length: concurrency }, (_value, index) => index);
|
||||||
const tasks = Array.from({ length: concurrency }, () =>
|
yield* Effect.forEach(workerIndexes, () => consumeLoop(flow), {
|
||||||
consumeLoop(flow),
|
concurrency: "unbounded",
|
||||||
);
|
discard: true,
|
||||||
// Run all concurrently: first rejection stops all.
|
});
|
||||||
await Promise.all(tasks);
|
}),
|
||||||
},
|
),
|
||||||
stop: async () => {
|
stop: () =>
|
||||||
running = false;
|
Effect.runPromise(
|
||||||
abortController.abort();
|
Effect.gen(function* () {
|
||||||
abortController = new AbortController();
|
running = false;
|
||||||
if (backend !== null) {
|
const currentBackend = backend;
|
||||||
await backend.close();
|
backend = null;
|
||||||
backend = null;
|
if (currentBackend !== null) {
|
||||||
}
|
yield* Effect.tryPromise({
|
||||||
},
|
try: () => currentBackend.close(),
|
||||||
|
catch: (error) =>
|
||||||
|
messagingLifecycleError(`${options.topic}:${options.subscription}`, "close-consumer", error),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function sleep(ms: number): Promise<void> {
|
|
||||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,7 @@ import type { PubSubBackend } from "../backend/types.js";
|
||||||
import type { ProducerMetrics } from "../metrics/prometheus.js";
|
import type { ProducerMetrics } from "../metrics/prometheus.js";
|
||||||
import { Effect } from "effect";
|
import { Effect } from "effect";
|
||||||
import { makeEffectProducerHandle, type EffectProducer } from "./runtime.js";
|
import { makeEffectProducerHandle, type EffectProducer } from "./runtime.js";
|
||||||
|
import { messagingLifecycleError } from "../errors.js";
|
||||||
|
|
||||||
export interface Producer<T> {
|
export interface Producer<T> {
|
||||||
readonly start: () => Promise<void>;
|
readonly start: () => Promise<void>;
|
||||||
|
|
@ -23,28 +24,38 @@ export function makeProducer<T>(
|
||||||
let effectProducer: EffectProducer<T> | null = null;
|
let effectProducer: EffectProducer<T> | null = null;
|
||||||
|
|
||||||
return {
|
return {
|
||||||
start: async () => {
|
start: () =>
|
||||||
const backend = await pubsub.createProducer<T>({ topic });
|
Effect.runPromise(
|
||||||
effectProducer = makeEffectProducerHandle(backend, {
|
Effect.gen(function* () {
|
||||||
topic,
|
const backend = yield* Effect.tryPromise({
|
||||||
...(metrics === undefined ? {} : { metrics }),
|
try: () => pubsub.createProducer<T>({ topic }),
|
||||||
});
|
catch: (error) => messagingLifecycleError(topic, "create-producer", error),
|
||||||
},
|
});
|
||||||
send: async (id, message) => {
|
effectProducer = makeEffectProducerHandle(backend, {
|
||||||
if (effectProducer === null) throw new Error("Producer not started");
|
topic,
|
||||||
|
...(metrics === undefined ? {} : { metrics }),
|
||||||
await Effect.runPromise(effectProducer.send(id, message));
|
});
|
||||||
},
|
}),
|
||||||
stop: async () => {
|
),
|
||||||
if (effectProducer !== null) {
|
send: (id, message) =>
|
||||||
const producer = effectProducer;
|
effectProducer === null
|
||||||
await Effect.runPromise(
|
? Effect.runPromise(Effect.fail(messagingLifecycleError(topic, "send", "Producer not started")))
|
||||||
producer.flush.pipe(
|
: Effect.runPromise(effectProducer.send(id, message)),
|
||||||
Effect.flatMap(() => producer.close),
|
stop: () =>
|
||||||
),
|
Effect.runPromise(
|
||||||
);
|
Effect.gen(function* () {
|
||||||
effectProducer = null;
|
if (effectProducer !== null) {
|
||||||
}
|
const producer = effectProducer;
|
||||||
},
|
yield* producer.flush.pipe(
|
||||||
|
Effect.flatMap(() => producer.close),
|
||||||
|
Effect.ensuring(
|
||||||
|
Effect.sync(() => {
|
||||||
|
effectProducer = null;
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -44,37 +44,42 @@ export function makeRequestResponse<TReq, TRes>(
|
||||||
let runtime: RequestResponseRuntime<TReq, TRes> | null = null;
|
let runtime: RequestResponseRuntime<TReq, TRes> | null = null;
|
||||||
|
|
||||||
return {
|
return {
|
||||||
start: async () => {
|
start: () =>
|
||||||
if (runtime !== null) return;
|
runtime !== null
|
||||||
|
? Promise.resolve()
|
||||||
|
: Effect.runPromise(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
const scope = yield* Scope.make();
|
||||||
|
const startRuntime = Effect.gen(function* () {
|
||||||
|
const config = yield* loadMessagingRuntimeConfig();
|
||||||
|
const requestor = yield* makeEffectRequestResponseFromPubSub<TReq, TRes>(
|
||||||
|
PubSub.fromBackend(options.pubsub),
|
||||||
|
config,
|
||||||
|
{
|
||||||
|
requestTopic: options.requestTopic,
|
||||||
|
responseTopic: options.responseTopic,
|
||||||
|
subscription: options.subscription,
|
||||||
|
},
|
||||||
|
).pipe(Scope.provide(scope));
|
||||||
|
|
||||||
const scope = await Effect.runPromise(Scope.make());
|
runtime = { scope, requestor };
|
||||||
|
});
|
||||||
|
|
||||||
try {
|
yield* startRuntime.pipe(
|
||||||
const config = await Effect.runPromise(loadMessagingRuntimeConfig());
|
Effect.catch((error) =>
|
||||||
const requestor = await Effect.runPromise(
|
Scope.close(scope, Exit.fail(error)).pipe(
|
||||||
makeEffectRequestResponseFromPubSub<TReq, TRes>(
|
Effect.flatMap(() => Effect.fail(error)),
|
||||||
PubSub.fromBackend(options.pubsub),
|
),
|
||||||
config,
|
),
|
||||||
{
|
);
|
||||||
requestTopic: options.requestTopic,
|
}),
|
||||||
responseTopic: options.responseTopic,
|
),
|
||||||
subscription: options.subscription,
|
stop: () => {
|
||||||
},
|
|
||||||
).pipe(Scope.provide(scope)),
|
|
||||||
);
|
|
||||||
|
|
||||||
runtime = { scope, requestor };
|
|
||||||
} catch (error) {
|
|
||||||
await Effect.runPromise(Scope.close(scope, Exit.fail(error))).catch(() => undefined);
|
|
||||||
throw error;
|
|
||||||
}
|
|
||||||
},
|
|
||||||
stop: async () => {
|
|
||||||
const current = runtime;
|
const current = runtime;
|
||||||
runtime = null;
|
runtime = null;
|
||||||
if (current === null) return;
|
return current === null
|
||||||
|
? Promise.resolve()
|
||||||
await Effect.runPromise(Scope.close(current.scope, Exit.void));
|
: Effect.runPromise(Scope.close(current.scope, Exit.void));
|
||||||
},
|
},
|
||||||
/**
|
/**
|
||||||
* Send a request and wait for responses.
|
* Send a request and wait for responses.
|
||||||
|
|
@ -85,20 +90,24 @@ export function makeRequestResponse<TReq, TRes>(
|
||||||
* Return `true` to indicate the final response has been received.
|
* Return `true` to indicate the final response has been received.
|
||||||
* If omitted, returns the first response.
|
* If omitted, returns the first response.
|
||||||
*/
|
*/
|
||||||
request: async (request, requestOptions) => {
|
request: (request, requestOptions) => {
|
||||||
const current = runtime;
|
const current = runtime;
|
||||||
if (current === null) {
|
if (current === null) {
|
||||||
throw messagingLifecycleError(
|
return Effect.runPromise(
|
||||||
`${options.requestTopic}:${options.responseTopic}`,
|
Effect.fail(
|
||||||
"request",
|
messagingLifecycleError(
|
||||||
"RequestResponse not started",
|
`${options.requestTopic}:${options.responseTopic}`,
|
||||||
|
"request",
|
||||||
|
"RequestResponse not started",
|
||||||
|
),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const timeoutMs = requestOptions?.timeoutMs ?? 300_000;
|
const timeoutMs = requestOptions?.timeoutMs ?? 300_000;
|
||||||
const recipient = requestOptions?.recipient;
|
const recipient = requestOptions?.recipient;
|
||||||
|
|
||||||
return await Effect.runPromise(
|
return Effect.runPromise(
|
||||||
current.requestor.request(request, {
|
current.requestor.request(request, {
|
||||||
timeoutMs,
|
timeoutMs,
|
||||||
...(recipient === undefined
|
...(recipient === undefined
|
||||||
|
|
|
||||||
|
|
@ -44,9 +44,9 @@ export type EffectMessageHandler<T, E = never, R = never> = (
|
||||||
flow: FlowContext<R>,
|
flow: FlowContext<R>,
|
||||||
) => Effect.Effect<void, E, R>;
|
) => Effect.Effect<void, E, R>;
|
||||||
|
|
||||||
export interface EffectProducerOptions {
|
export interface EffectProducerOptions<T = unknown> {
|
||||||
readonly topic: string;
|
readonly topic: string;
|
||||||
readonly schema?: S.Top;
|
readonly schema?: S.Codec<T, unknown>;
|
||||||
readonly metrics?: ProducerMetrics;
|
readonly metrics?: ProducerMetrics;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -62,7 +62,7 @@ export interface EffectConsumerOptions<T, E = never, R = never> {
|
||||||
readonly handler: EffectMessageHandler<T, E, R>;
|
readonly handler: EffectMessageHandler<T, E, R>;
|
||||||
readonly concurrency?: number;
|
readonly concurrency?: number;
|
||||||
readonly initialPosition?: "latest" | "earliest";
|
readonly initialPosition?: "latest" | "earliest";
|
||||||
readonly schema?: S.Top;
|
readonly schema?: S.Codec<T, unknown>;
|
||||||
readonly receiveTimeoutMs?: number;
|
readonly receiveTimeoutMs?: number;
|
||||||
readonly errorBackoffMs?: number;
|
readonly errorBackoffMs?: number;
|
||||||
readonly rateLimitRetryMs?: number;
|
readonly rateLimitRetryMs?: number;
|
||||||
|
|
@ -73,12 +73,12 @@ export interface EffectConsumer {
|
||||||
readonly fibers: ReadonlyArray<Fiber.Fiber<void, never>>;
|
readonly fibers: ReadonlyArray<Fiber.Fiber<void, never>>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface EffectRequestResponseOptions {
|
export interface EffectRequestResponseOptions<TReq = unknown, TRes = unknown> {
|
||||||
readonly requestTopic: string;
|
readonly requestTopic: string;
|
||||||
readonly responseTopic: string;
|
readonly responseTopic: string;
|
||||||
readonly subscription: string;
|
readonly subscription: string;
|
||||||
readonly requestSchema?: S.Top;
|
readonly requestSchema?: S.Codec<TReq, unknown>;
|
||||||
readonly responseSchema?: S.Top;
|
readonly responseSchema?: S.Codec<TRes, unknown>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface EffectRequestOptions<TRes, E = never, R = never> {
|
export interface EffectRequestOptions<TRes, E = never, R = never> {
|
||||||
|
|
@ -96,7 +96,7 @@ export interface EffectRequestResponse<TReq, TRes> {
|
||||||
|
|
||||||
export interface ProducerFactoryService {
|
export interface ProducerFactoryService {
|
||||||
readonly make: <T>(
|
readonly make: <T>(
|
||||||
options: EffectProducerOptions,
|
options: EffectProducerOptions<T>,
|
||||||
) => Effect.Effect<EffectProducer<T>, PubSubError, Scope.Scope>;
|
) => Effect.Effect<EffectProducer<T>, PubSubError, Scope.Scope>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -109,7 +109,7 @@ export interface ConsumerFactoryService {
|
||||||
|
|
||||||
export interface RequestResponseFactoryService {
|
export interface RequestResponseFactoryService {
|
||||||
readonly make: <TReq, TRes>(
|
readonly make: <TReq, TRes>(
|
||||||
options: EffectRequestResponseOptions,
|
options: EffectRequestResponseOptions<TReq, TRes>,
|
||||||
) => Effect.Effect<EffectRequestResponse<TReq, TRes>, PubSubError, Scope.Scope>;
|
) => Effect.Effect<EffectRequestResponse<TReq, TRes>, PubSubError, Scope.Scope>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -138,7 +138,7 @@ export class FlowRuntime extends Context.Service<FlowRuntime, FlowRuntimeService
|
||||||
|
|
||||||
export function makeEffectProducerHandle<T>(
|
export function makeEffectProducerHandle<T>(
|
||||||
backend: BackendProducer<T>,
|
backend: BackendProducer<T>,
|
||||||
options: EffectProducerOptions,
|
options: EffectProducerOptions<T>,
|
||||||
): EffectProducer<T> {
|
): EffectProducer<T> {
|
||||||
return {
|
return {
|
||||||
send: Effect.fn(`Producer.send:${options.topic}`)((id: string, message: T) =>
|
send: Effect.fn(`Producer.send:${options.topic}`)((id: string, message: T) =>
|
||||||
|
|
@ -168,9 +168,9 @@ export function makeEffectProducerHandle<T>(
|
||||||
|
|
||||||
export const makeEffectProducerFromPubSub = Effect.fn("makeEffectProducerFromPubSub")(function* <T>(
|
export const makeEffectProducerFromPubSub = Effect.fn("makeEffectProducerFromPubSub")(function* <T>(
|
||||||
pubsub: PubSubService,
|
pubsub: PubSubService,
|
||||||
options: EffectProducerOptions,
|
options: EffectProducerOptions<T>,
|
||||||
) {
|
) {
|
||||||
const createOptions: CreateProducerOptions = options.schema === undefined
|
const createOptions: CreateProducerOptions<T> = options.schema === undefined
|
||||||
? { topic: options.topic }
|
? { topic: options.topic }
|
||||||
: { topic: options.topic, schema: options.schema };
|
: { topic: options.topic, schema: options.schema };
|
||||||
const backend = yield* pubsub.createProducer<T>(createOptions);
|
const backend = yield* pubsub.createProducer<T>(createOptions);
|
||||||
|
|
@ -326,7 +326,7 @@ export const makeEffectConsumerFromPubSub = Effect.fn("makeEffectConsumerFromPub
|
||||||
options: EffectConsumerOptions<T, E, R>,
|
options: EffectConsumerOptions<T, E, R>,
|
||||||
flow: FlowContext<R>,
|
flow: FlowContext<R>,
|
||||||
) {
|
) {
|
||||||
const createOptions: CreateConsumerOptions = {
|
const createOptions: CreateConsumerOptions<T> = {
|
||||||
topic: options.topic,
|
topic: options.topic,
|
||||||
subscription: options.subscription,
|
subscription: options.subscription,
|
||||||
...(options.initialPosition === undefined ? {} : { initialPosition: options.initialPosition }),
|
...(options.initialPosition === undefined ? {} : { initialPosition: options.initialPosition }),
|
||||||
|
|
@ -422,9 +422,9 @@ export const makeEffectRequestResponseFromPubSub = Effect.fn("makeEffectRequestR
|
||||||
>(
|
>(
|
||||||
pubsub: PubSubService,
|
pubsub: PubSubService,
|
||||||
config: MessagingRuntimeConfig,
|
config: MessagingRuntimeConfig,
|
||||||
options: EffectRequestResponseOptions,
|
options: EffectRequestResponseOptions<TReq, TRes>,
|
||||||
) {
|
) {
|
||||||
const producerOptions: CreateProducerOptions = options.requestSchema === undefined
|
const producerOptions: CreateProducerOptions<TReq> = options.requestSchema === undefined
|
||||||
? { topic: options.requestTopic }
|
? { topic: options.requestTopic }
|
||||||
: { topic: options.requestTopic, schema: options.requestSchema };
|
: { topic: options.requestTopic, schema: options.requestSchema };
|
||||||
const producerBackend = yield* pubsub.createProducer<TReq>(producerOptions);
|
const producerBackend = yield* pubsub.createProducer<TReq>(producerOptions);
|
||||||
|
|
@ -432,7 +432,7 @@ export const makeEffectRequestResponseFromPubSub = Effect.fn("makeEffectRequestR
|
||||||
topic: options.requestTopic,
|
topic: options.requestTopic,
|
||||||
...(options.requestSchema === undefined ? {} : { schema: options.requestSchema }),
|
...(options.requestSchema === undefined ? {} : { schema: options.requestSchema }),
|
||||||
});
|
});
|
||||||
const createOptions: CreateConsumerOptions = {
|
const createOptions: CreateConsumerOptions<TRes> = {
|
||||||
topic: options.responseTopic,
|
topic: options.responseTopic,
|
||||||
subscription: options.subscription,
|
subscription: options.subscription,
|
||||||
...(options.responseSchema === undefined ? {} : { schema: options.responseSchema }),
|
...(options.responseSchema === undefined ? {} : { schema: options.responseSchema }),
|
||||||
|
|
@ -502,7 +502,7 @@ export const makeEffectRequestResponseFromPubSub = Effect.fn("makeEffectRequestR
|
||||||
|
|
||||||
export function makeProducerFactoryService(pubsub: PubSubService): ProducerFactoryService {
|
export function makeProducerFactoryService(pubsub: PubSubService): ProducerFactoryService {
|
||||||
return {
|
return {
|
||||||
make: Effect.fn("ProducerFactory.make")(<T>(options: EffectProducerOptions) =>
|
make: Effect.fn("ProducerFactory.make")(<T>(options: EffectProducerOptions<T>) =>
|
||||||
makeEffectProducerFromPubSub<T>(pubsub, options),
|
makeEffectProducerFromPubSub<T>(pubsub, options),
|
||||||
),
|
),
|
||||||
};
|
};
|
||||||
|
|
@ -526,13 +526,11 @@ export function makeRequestResponseFactoryService(
|
||||||
pubsub: PubSubService,
|
pubsub: PubSubService,
|
||||||
config: MessagingRuntimeConfig,
|
config: MessagingRuntimeConfig,
|
||||||
): RequestResponseFactoryService {
|
): RequestResponseFactoryService {
|
||||||
const make = Effect.fn("RequestResponseFactory.make")(function* <TReq, TRes>(
|
return {
|
||||||
options: EffectRequestResponseOptions,
|
make: Effect.fn("RequestResponseFactory.make")(<TReq, TRes>(
|
||||||
) {
|
options: EffectRequestResponseOptions<TReq, TRes>,
|
||||||
return yield* makeEffectRequestResponseFromPubSub<TReq, TRes>(pubsub, config, options);
|
) => makeEffectRequestResponseFromPubSub<TReq, TRes>(pubsub, config, options)),
|
||||||
}) as RequestResponseFactoryService["make"];
|
};
|
||||||
|
|
||||||
return { make };
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export const ProducerFactoryLive = Layer.effect(
|
export const ProducerFactoryLive = Layer.effect(
|
||||||
|
|
@ -589,7 +587,7 @@ export const MessagingRuntimeLive = Layer.mergeAll(
|
||||||
);
|
);
|
||||||
|
|
||||||
export const runEffectProducerScoped = Effect.fn("runEffectProducerScoped")(function* <T>(
|
export const runEffectProducerScoped = Effect.fn("runEffectProducerScoped")(function* <T>(
|
||||||
options: EffectProducerOptions,
|
options: EffectProducerOptions<T>,
|
||||||
) {
|
) {
|
||||||
const pubsub = yield* PubSub;
|
const pubsub = yield* PubSub;
|
||||||
return yield* makeEffectProducerFromPubSub<T>(pubsub, options);
|
return yield* makeEffectProducerFromPubSub<T>(pubsub, options);
|
||||||
|
|
@ -605,7 +603,7 @@ export const runEffectConsumerScoped = Effect.fn("runEffectConsumerScoped")(func
|
||||||
});
|
});
|
||||||
|
|
||||||
export const runEffectRequestResponseScoped = Effect.fn("runEffectRequestResponseScoped")(function* <TReq, TRes>(
|
export const runEffectRequestResponseScoped = Effect.fn("runEffectRequestResponseScoped")(function* <TReq, TRes>(
|
||||||
options: EffectRequestResponseOptions,
|
options: EffectRequestResponseOptions<TReq, TRes>,
|
||||||
) {
|
) {
|
||||||
const pubsub = yield* PubSub;
|
const pubsub = yield* PubSub;
|
||||||
const config = yield* loadMessagingRuntimeConfig();
|
const config = yield* loadMessagingRuntimeConfig();
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,8 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { PubSubBackend, BackendConsumer } from "../backend/types.js";
|
import type { PubSubBackend, BackendConsumer } from "../backend/types.js";
|
||||||
|
import { Duration, Effect, Fiber } from "effect";
|
||||||
|
import { messagingDeliveryError, messagingLifecycleError, messagingTimeoutError } from "../errors.js";
|
||||||
|
|
||||||
type Resolver<T> = {
|
type Resolver<T> = {
|
||||||
queue: AsyncQueue<T>;
|
queue: AsyncQueue<T>;
|
||||||
|
|
@ -32,28 +34,33 @@ export function makeAsyncQueue<T>(): AsyncQueue<T> {
|
||||||
buffer.push(item);
|
buffer.push(item);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
pop: async (timeoutMs) => {
|
pop: (timeoutMs) => {
|
||||||
const buffered = buffer.shift();
|
const buffered = buffer.shift();
|
||||||
if (buffered !== undefined) return buffered;
|
if (buffered !== undefined) return Promise.resolve(buffered);
|
||||||
|
|
||||||
return new Promise<T>((resolve, reject) => {
|
|
||||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
||||||
|
|
||||||
|
const take = Effect.callback<T>((resume) => {
|
||||||
const waiter = (value: T) => {
|
const waiter = (value: T) => {
|
||||||
if (timer !== undefined) clearTimeout(timer);
|
resume(Effect.succeed(value));
|
||||||
resolve(value);
|
|
||||||
};
|
};
|
||||||
|
|
||||||
waiters.push(waiter);
|
waiters.push(waiter);
|
||||||
|
|
||||||
if (timeoutMs !== undefined) {
|
return Effect.sync(() => {
|
||||||
timer = setTimeout(() => {
|
const idx = waiters.indexOf(waiter);
|
||||||
const idx = waiters.indexOf(waiter);
|
if (idx !== -1) waiters.splice(idx, 1);
|
||||||
if (idx !== -1) waiters.splice(idx, 1);
|
});
|
||||||
reject(new Error(`Queue.pop timed out after ${timeoutMs}ms`));
|
|
||||||
}, timeoutMs);
|
|
||||||
}
|
|
||||||
});
|
});
|
||||||
|
|
||||||
|
return Effect.runPromise(
|
||||||
|
timeoutMs === undefined
|
||||||
|
? take
|
||||||
|
: take.pipe(
|
||||||
|
Effect.timeout(Duration.millis(timeoutMs)),
|
||||||
|
Effect.catchTag("TimeoutError", () =>
|
||||||
|
Effect.fail(messagingTimeoutError("queue.pop", timeoutMs)),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
},
|
},
|
||||||
get length() {
|
get length() {
|
||||||
return buffer.length;
|
return buffer.length;
|
||||||
|
|
@ -77,76 +84,113 @@ export function makeSubscriber<T>(
|
||||||
): Subscriber<T> {
|
): Subscriber<T> {
|
||||||
let backend: BackendConsumer<T> | null = null;
|
let backend: BackendConsumer<T> | null = null;
|
||||||
let running = false;
|
let running = false;
|
||||||
|
let fiber: Fiber.Fiber<void, never> | null = null;
|
||||||
|
|
||||||
// ID-specific subscriptions (request/response correlation)
|
// ID-specific subscriptions (request/response correlation)
|
||||||
const idSubscribers = new Map<string, Resolver<T>>();
|
const idSubscribers = new Map<string, Resolver<T>>();
|
||||||
// Wildcard subscribers (receive all messages)
|
// Wildcard subscribers (receive all messages)
|
||||||
const allSubscribers = new Map<string, Resolver<T>>();
|
const allSubscribers = new Map<string, Resolver<T>>();
|
||||||
|
|
||||||
const dispatchLoop = async (): Promise<void> => {
|
const dispatchLoop = Effect.fn("Subscriber.dispatchLoop")(function* () {
|
||||||
let consecutiveErrors = 0;
|
let consecutiveErrors = 0;
|
||||||
while (running) {
|
const dispatchOnce = Effect.fn("Subscriber.dispatchOnce")(function* () {
|
||||||
try {
|
const currentBackend = backend;
|
||||||
const currentBackend = backend;
|
if (currentBackend === null) {
|
||||||
if (currentBackend === null) throw new Error("Subscriber backend not started");
|
return yield* messagingLifecycleError(
|
||||||
|
`${topic}:${subscription}`,
|
||||||
|
"dispatch",
|
||||||
|
"Subscriber backend not started",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
const msg = await currentBackend.receive(2000);
|
const msg = yield* Effect.tryPromise({
|
||||||
if (msg === null) continue;
|
try: () => currentBackend.receive(2000),
|
||||||
|
catch: (error) => messagingDeliveryError(topic, "receive", error),
|
||||||
|
});
|
||||||
|
if (msg === null) return;
|
||||||
|
|
||||||
consecutiveErrors = 0;
|
consecutiveErrors = 0;
|
||||||
|
|
||||||
const props = msg.properties();
|
const props = msg.properties();
|
||||||
const id = props.id;
|
const id = props.id;
|
||||||
const value = msg.value();
|
const value = msg.value();
|
||||||
|
|
||||||
// Route to ID-specific subscriber
|
// Route to ID-specific subscriber
|
||||||
if (id !== undefined && id.length > 0) {
|
if (id !== undefined && id.length > 0) {
|
||||||
const sub = idSubscribers.get(id);
|
const sub = idSubscribers.get(id);
|
||||||
if (sub !== undefined) {
|
if (sub !== undefined) {
|
||||||
|
sub.queue.push(value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Broadcast to all-subscribers
|
||||||
|
for (const sub of allSubscribers.values()) {
|
||||||
sub.queue.push(value);
|
sub.queue.push(value);
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// Broadcast to all-subscribers
|
yield* Effect.tryPromise({
|
||||||
for (const sub of allSubscribers.values()) {
|
try: () => currentBackend.acknowledge(msg),
|
||||||
sub.queue.push(value);
|
catch: (error) => messagingDeliveryError(topic, "acknowledge", error),
|
||||||
}
|
});
|
||||||
|
});
|
||||||
|
|
||||||
await currentBackend.acknowledge(msg);
|
yield* Effect.whileLoop({
|
||||||
} catch (err) {
|
while: () => running,
|
||||||
if (!running) break;
|
body: () =>
|
||||||
consecutiveErrors++;
|
dispatchOnce().pipe(
|
||||||
if (consecutiveErrors <= 3) {
|
Effect.catch((error) => {
|
||||||
console.error("[Subscriber] Error:", err);
|
if (!running) return Effect.void;
|
||||||
} else if (consecutiveErrors === 4) {
|
consecutiveErrors++;
|
||||||
console.error("[Subscriber] Suppressing further errors (will retry with backoff)");
|
const logEffect = consecutiveErrors <= 3
|
||||||
}
|
? Effect.logError("[Subscriber] Error", { error })
|
||||||
// Exponential backoff: 1s, 2s, 4s, max 10s
|
: consecutiveErrors === 4
|
||||||
const delay = Math.min(1000 * Math.pow(2, consecutiveErrors - 1), 10_000);
|
? Effect.logError("[Subscriber] Suppressing further errors (will retry with backoff)", { error })
|
||||||
await new Promise((r) => setTimeout(r, delay));
|
: Effect.void;
|
||||||
}
|
const delay = Math.min(1000 * 2 ** (consecutiveErrors - 1), 10_000);
|
||||||
}
|
return logEffect.pipe(Effect.flatMap(() => Effect.sleep(Duration.millis(delay))));
|
||||||
};
|
}),
|
||||||
|
),
|
||||||
|
step: () => undefined,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
return {
|
return {
|
||||||
start: async () => {
|
start: () =>
|
||||||
backend = await pubsub.createConsumer<T>({
|
Effect.runPromise(
|
||||||
topic,
|
Effect.gen(function* () {
|
||||||
subscription,
|
backend = yield* Effect.tryPromise({
|
||||||
});
|
try: () =>
|
||||||
running = true;
|
pubsub.createConsumer<T>({
|
||||||
// Start the dispatch loop (fire and forget; runs until stop).
|
topic,
|
||||||
dispatchLoop().catch((err) => {
|
subscription,
|
||||||
if (running === true) console.error("[Subscriber] dispatch loop error:", err);
|
}),
|
||||||
});
|
catch: (error) =>
|
||||||
},
|
messagingLifecycleError(`${topic}:${subscription}`, "create-consumer", error),
|
||||||
stop: async () => {
|
});
|
||||||
running = false;
|
running = true;
|
||||||
if (backend !== null) {
|
fiber = yield* dispatchLoop().pipe(Effect.forkDetach);
|
||||||
await backend.close();
|
}),
|
||||||
backend = null;
|
),
|
||||||
}
|
stop: () =>
|
||||||
},
|
Effect.runPromise(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
running = false;
|
||||||
|
const activeFiber = fiber;
|
||||||
|
fiber = null;
|
||||||
|
if (activeFiber !== null) {
|
||||||
|
yield* Fiber.interrupt(activeFiber);
|
||||||
|
}
|
||||||
|
const currentBackend = backend;
|
||||||
|
if (currentBackend !== null) {
|
||||||
|
backend = null;
|
||||||
|
yield* Effect.tryPromise({
|
||||||
|
try: () => currentBackend.close(),
|
||||||
|
catch: (error) =>
|
||||||
|
messagingLifecycleError(`${topic}:${subscription}`, "close-consumer", error),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
),
|
||||||
subscribe: (id) => {
|
subscribe: (id) => {
|
||||||
const queue = makeAsyncQueue<T>();
|
const queue = makeAsyncQueue<T>();
|
||||||
idSubscribers.set(id, { queue });
|
idSubscribers.set(id, { queue });
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@
|
||||||
|
|
||||||
import type { PubSubBackend } from "../backend/types.js";
|
import type { PubSubBackend } from "../backend/types.js";
|
||||||
import { makeNatsBackend } from "../backend/nats.js";
|
import { makeNatsBackend } from "../backend/nats.js";
|
||||||
import { Effect } from "effect";
|
import { Context, Effect } from "effect";
|
||||||
import { processorLifecycleError, type ProcessorLifecycleError } from "../errors.js";
|
import { processorLifecycleError, type ProcessorLifecycleError } from "../errors.js";
|
||||||
import { loadProcessorRuntimeConfig } from "../runtime/config.js";
|
import { loadProcessorRuntimeConfig } from "../runtime/config.js";
|
||||||
|
|
||||||
|
|
@ -36,10 +36,10 @@ declare const processorRunRequirementsType: unique symbol;
|
||||||
export interface ProcessorRuntime<RunError = ProcessorLifecycleError, RunRequirements = never> {
|
export interface ProcessorRuntime<RunError = ProcessorLifecycleError, RunRequirements = never> {
|
||||||
readonly [processorRunErrorType]?: RunError;
|
readonly [processorRunErrorType]?: RunError;
|
||||||
readonly [processorRunRequirementsType]?: RunRequirements;
|
readonly [processorRunRequirementsType]?: RunRequirements;
|
||||||
readonly start: () => Promise<void>;
|
readonly start: (context: Context.Context<RunRequirements>) => Promise<void>;
|
||||||
readonly stop: () => Promise<void>;
|
readonly stop: () => Promise<void>;
|
||||||
startEffect(): unknown;
|
startEffect: Effect.Effect<void, RunError | ProcessorLifecycleError, RunRequirements>;
|
||||||
stopEffect(): unknown;
|
stopEffect: Effect.Effect<void, ProcessorLifecycleError>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface AsyncProcessorRuntime<
|
export interface AsyncProcessorRuntime<
|
||||||
|
|
@ -53,8 +53,8 @@ export interface AsyncProcessorRuntime<
|
||||||
readonly isRunning: () => boolean;
|
readonly isRunning: () => boolean;
|
||||||
readonly registerConfigHandler: (handler: ConfigHandler) => void;
|
readonly registerConfigHandler: (handler: ConfigHandler) => void;
|
||||||
readonly onShutdown: (callback: () => Promise<void>) => void;
|
readonly onShutdown: (callback: () => Promise<void>) => void;
|
||||||
readonly run: () => Promise<void>;
|
readonly run: (context: Context.Context<RunRequirements>) => Promise<void>;
|
||||||
runEffect(): unknown;
|
runEffect: Effect.Effect<void, RunError | ProcessorLifecycleError, RunRequirements>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface AsyncProcessorRuntimeOptions<
|
export interface AsyncProcessorRuntimeOptions<
|
||||||
|
|
@ -94,8 +94,16 @@ export function makeAsyncProcessor<
|
||||||
}
|
}
|
||||||
|
|
||||||
const shutdown = () => {
|
const shutdown = () => {
|
||||||
console.log(`[${config.id}] Shutting down...`);
|
void Effect.runPromise(
|
||||||
void processor.stop().then(() => process.exit(0));
|
Effect.log(`[${config.id}] Shutting down...`).pipe(
|
||||||
|
Effect.flatMap(() =>
|
||||||
|
Effect.tryPromise({
|
||||||
|
try: () => processor.stop(),
|
||||||
|
catch: (error) => processorLifecycleError(config.id, "signal-shutdown", error),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
).then(() => process.exit(0), () => process.exit(1));
|
||||||
};
|
};
|
||||||
const handlers: RegisteredSignalHandler[] = [
|
const handlers: RegisteredSignalHandler[] = [
|
||||||
{ signal: "SIGINT", handler: shutdown },
|
{ signal: "SIGINT", handler: shutdown },
|
||||||
|
|
@ -125,29 +133,19 @@ export function makeAsyncProcessor<
|
||||||
registerConfigHandler: (handler) => {
|
registerConfigHandler: (handler) => {
|
||||||
configHandlers.push(handler);
|
configHandlers.push(handler);
|
||||||
},
|
},
|
||||||
start: async () => {
|
start: (context) => Effect.runPromiseWith(context)(processor.startEffect),
|
||||||
await Effect.runPromise(
|
stop: () => Effect.runPromise(processor.stopEffect),
|
||||||
processor.startEffect() as Effect.Effect<void, RunError | ProcessorLifecycleError>,
|
|
||||||
);
|
|
||||||
},
|
|
||||||
stop: async () => {
|
|
||||||
await Effect.runPromise(
|
|
||||||
processor.stopEffect() as Effect.Effect<void, ProcessorLifecycleError>,
|
|
||||||
);
|
|
||||||
},
|
|
||||||
onShutdown: (callback) => {
|
onShutdown: (callback) => {
|
||||||
shutdownCallbacks.push(callback);
|
shutdownCallbacks.push(callback);
|
||||||
},
|
},
|
||||||
startEffect() {
|
get startEffect() {
|
||||||
const startProcessor = Effect.fn("trustgraph.processor.start")(function* () {
|
const startProcessor = Effect.fn("trustgraph.processor.start")(function* () {
|
||||||
yield* Effect.sync(() => {
|
yield* Effect.sync(() => {
|
||||||
running = true;
|
running = true;
|
||||||
registerProcessSignalHandlers();
|
registerProcessSignalHandlers();
|
||||||
});
|
});
|
||||||
|
|
||||||
yield* (
|
yield* processor.runEffect;
|
||||||
processor.runEffect() as Effect.Effect<void, RunError, RunRequirements>
|
|
||||||
);
|
|
||||||
});
|
});
|
||||||
return startProcessor().pipe(
|
return startProcessor().pipe(
|
||||||
Effect.withSpan("trustgraph.processor.start", {
|
Effect.withSpan("trustgraph.processor.start", {
|
||||||
|
|
@ -157,7 +155,7 @@ export function makeAsyncProcessor<
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
stopEffect() {
|
get stopEffect() {
|
||||||
const stopProcessor = Effect.fn("trustgraph.processor.stop")(function* () {
|
const stopProcessor = Effect.fn("trustgraph.processor.stop")(function* () {
|
||||||
yield* Effect.sync(() => {
|
yield* Effect.sync(() => {
|
||||||
running = false;
|
running = false;
|
||||||
|
|
@ -180,18 +178,15 @@ export function makeAsyncProcessor<
|
||||||
});
|
});
|
||||||
return stopProcessor();
|
return stopProcessor();
|
||||||
},
|
},
|
||||||
run: () =>
|
run: (context) => Effect.runPromiseWith(context)(processor.runEffect),
|
||||||
Effect.runPromise(
|
get runEffect() {
|
||||||
processor.runEffect() as unknown as Effect.Effect<void, RunError>,
|
|
||||||
),
|
|
||||||
runEffect: () => {
|
|
||||||
if (options.runEffect !== undefined) {
|
if (options.runEffect !== undefined) {
|
||||||
return options.runEffect(processor);
|
return options.runEffect(processor);
|
||||||
}
|
}
|
||||||
return Effect.tryPromise({
|
return Effect.tryPromise({
|
||||||
try: () => options.run?.(processor) ?? Promise.resolve(),
|
try: () => options.run?.(processor) ?? Promise.resolve(),
|
||||||
catch: (error) => processorLifecycleError(config.id, "start", error),
|
catch: (error) => processorLifecycleError(config.id, "start", error),
|
||||||
}) as unknown as Effect.Effect<void, RunError, RunRequirements>;
|
});
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -208,13 +203,21 @@ export const AsyncProcessor = Object.assign(
|
||||||
return makeAsyncProcessor(config);
|
return makeAsyncProcessor(config);
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
async launch<T extends ProcessorRuntime<unknown, unknown>>(
|
launch<T extends ProcessorRuntime<unknown, never>>(
|
||||||
this: new (config: ProcessorConfig) => T,
|
this: new (config: ProcessorConfig) => T,
|
||||||
id: string,
|
id: string,
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
const config = await Effect.runPromise(loadProcessorRuntimeConfig(id));
|
const ProcessorCtor = this;
|
||||||
const processor = new this(config);
|
return Effect.runPromise(
|
||||||
await processor.start();
|
Effect.gen(function* () {
|
||||||
|
const config = yield* loadProcessorRuntimeConfig(id);
|
||||||
|
const processor = new ProcessorCtor(config);
|
||||||
|
yield* Effect.tryPromise({
|
||||||
|
try: () => processor.start(Context.empty()),
|
||||||
|
catch: (error) => processorLifecycleError(id, "launch", error),
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
) as unknown as {
|
) as unknown as {
|
||||||
|
|
@ -224,7 +227,7 @@ export const AsyncProcessor = Object.assign(
|
||||||
<RunError = ProcessorLifecycleError, RunRequirements = never>(
|
<RunError = ProcessorLifecycleError, RunRequirements = never>(
|
||||||
config: ProcessorConfig,
|
config: ProcessorConfig,
|
||||||
): AsyncProcessor<RunError, RunRequirements>;
|
): AsyncProcessor<RunError, RunRequirements>;
|
||||||
launch<T extends ProcessorRuntime<unknown, unknown>>(
|
launch<T extends ProcessorRuntime<unknown, never>>(
|
||||||
this: new (config: ProcessorConfig) => T,
|
this: new (config: ProcessorConfig) => T,
|
||||||
id: string,
|
id: string,
|
||||||
): Promise<void>;
|
): Promise<void>;
|
||||||
|
|
|
||||||
|
|
@ -38,6 +38,7 @@ import {
|
||||||
import { makePubSubService, PubSub } from "../backend/pubsub.js";
|
import { makePubSubService, PubSub } from "../backend/pubsub.js";
|
||||||
import { loadMessagingRuntimeConfig } from "../runtime/messaging-config.js";
|
import { loadMessagingRuntimeConfig } from "../runtime/messaging-config.js";
|
||||||
import { Duration, Effect, Exit, Scope } from "effect";
|
import { Duration, Effect, Exit, Scope } from "effect";
|
||||||
|
import * as Predicate from "effect/Predicate";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
|
|
||||||
interface ConfigPush {
|
interface ConfigPush {
|
||||||
|
|
@ -88,9 +89,7 @@ export interface FlowProcessorRuntime<FlowRequirements = never>
|
||||||
readonly configHandlers: ConfigHandler[];
|
readonly configHandlers: ConfigHandler[];
|
||||||
readonly isRunning: () => boolean;
|
readonly isRunning: () => boolean;
|
||||||
readonly registerConfigHandler: (handler: ConfigHandler) => void;
|
readonly registerConfigHandler: (handler: ConfigHandler) => void;
|
||||||
readonly registerSpecification: <Requirements extends FlowRequirements>(
|
readonly registerSpecification: (spec: Spec<FlowRequirements>) => void;
|
||||||
spec: Spec<Requirements>,
|
|
||||||
) => void;
|
|
||||||
readonly specifications: ReadonlyArray<Spec<FlowRequirements>>;
|
readonly specifications: ReadonlyArray<Spec<FlowRequirements>>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -106,6 +105,20 @@ const ConfigPushSchema = S.Struct({
|
||||||
config: S.Record(S.String, S.Unknown),
|
config: S.Record(S.String, S.Unknown),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const isStringRecord = (value: unknown): value is Record<string, unknown> =>
|
||||||
|
Predicate.isObject(value) && !Array.isArray(value);
|
||||||
|
|
||||||
|
const isTopicsRecord = (value: unknown): value is Record<string, string> =>
|
||||||
|
isStringRecord(value) && Object.values(value).every((item) => typeof item === "string");
|
||||||
|
|
||||||
|
const isFlowDefinition = (value: unknown): value is FlowDefinition => {
|
||||||
|
if (!isStringRecord(value)) return false;
|
||||||
|
const topics = value.topics;
|
||||||
|
const parameters = value.parameters;
|
||||||
|
return (topics === undefined || isTopicsRecord(topics)) &&
|
||||||
|
(parameters === undefined || isStringRecord(parameters));
|
||||||
|
};
|
||||||
|
|
||||||
export function runFlowProcessorDefinitionScoped<
|
export function runFlowProcessorDefinitionScoped<
|
||||||
FlowRequirements = never,
|
FlowRequirements = never,
|
||||||
ConfigHandlerError = never,
|
ConfigHandlerError = never,
|
||||||
|
|
@ -202,11 +215,15 @@ export function runFlowProcessorDefinitionScoped<
|
||||||
FlowRuntime | ProducerFactory | ConsumerFactory | RequestResponseFactory | FlowRequirements
|
FlowRuntime | ProducerFactory | ConsumerFactory | RequestResponseFactory | FlowRequirements
|
||||||
> =>
|
> =>
|
||||||
Effect.gen(function* () {
|
Effect.gen(function* () {
|
||||||
const flowDefs = config.flows as Record<string, FlowDefinition> | undefined;
|
const flowDefs = config.flows;
|
||||||
if (flowDefs === undefined) {
|
if (flowDefs === undefined) {
|
||||||
yield* Effect.log(`[${options.id}] No flows in config push, skipping`);
|
yield* Effect.log(`[${options.id}] No flows in config push, skipping`);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if (!isStringRecord(flowDefs)) {
|
||||||
|
yield* Effect.logWarning(`[${options.id}] Skipping config push: flows is not an object`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
const flowsJson = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(flowDefs).pipe(
|
const flowsJson = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(flowDefs).pipe(
|
||||||
Effect.catch((error) => Effect.succeed(String(error))),
|
Effect.catch((error) => Effect.succeed(String(error))),
|
||||||
|
|
@ -226,7 +243,7 @@ export function runFlowProcessorDefinitionScoped<
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const [name, defn] of Object.entries(flowDefs)) {
|
for (const [name, defn] of Object.entries(flowDefs)) {
|
||||||
if (typeof defn !== "object" || defn === null) {
|
if (!isFlowDefinition(defn)) {
|
||||||
yield* Effect.logWarning(`[${options.id}] Skipping flow "${name}": definition is not an object`);
|
yield* Effect.logWarning(`[${options.id}] Skipping flow "${name}": definition is not an object`);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
@ -353,8 +370,8 @@ export function makeFlowProcessor<FlowRequirements = never>(
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
const startEffect = (): FlowProcessorStartEffect<FlowRequirements> => {
|
const makeStartEffect = (): FlowProcessorStartEffect<FlowRequirements> => {
|
||||||
const effect = base.startEffect() as FlowProcessorStartEffect<FlowRequirements>;
|
const effect = base.startEffect;
|
||||||
return options.provide?.(effect) ?? effect;
|
return options.provide?.(effect) ?? effect;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -362,24 +379,29 @@ export function makeFlowProcessor<FlowRequirements = never>(
|
||||||
...base,
|
...base,
|
||||||
specifications,
|
specifications,
|
||||||
registerSpecification: (spec) => {
|
registerSpecification: (spec) => {
|
||||||
specifications.push(spec as Spec<FlowRequirements>);
|
specifications.push(spec);
|
||||||
},
|
},
|
||||||
startEffect,
|
get startEffect() {
|
||||||
start: async () => {
|
return makeStartEffect();
|
||||||
const pubsub = makePubSubService(base.pubsub);
|
|
||||||
const messagingConfig = await Effect.runPromise(loadMessagingRuntimeConfig());
|
|
||||||
const start = startEffect().pipe(
|
|
||||||
Effect.provideService(PubSub, pubsub),
|
|
||||||
Effect.provideService(ProducerFactory, ProducerFactory.of(makeProducerFactoryService(pubsub))),
|
|
||||||
Effect.provideService(ConsumerFactory, ConsumerFactory.of(makeConsumerFactoryService(pubsub, messagingConfig))),
|
|
||||||
Effect.provideService(
|
|
||||||
RequestResponseFactory,
|
|
||||||
RequestResponseFactory.of(makeRequestResponseFactoryService(pubsub, messagingConfig)),
|
|
||||||
),
|
|
||||||
Effect.provideService(FlowRuntime, FlowRuntime.of({ run: runFlowRuntimeScoped })),
|
|
||||||
) as Effect.Effect<void, PubSubError | FlowRuntimeError | ProcessorLifecycleError>;
|
|
||||||
await Effect.runPromise(Effect.scoped(start));
|
|
||||||
},
|
},
|
||||||
|
start: (context) =>
|
||||||
|
Effect.runPromiseWith(context)(
|
||||||
|
Effect.gen(function* () {
|
||||||
|
const pubsub = makePubSubService(base.pubsub);
|
||||||
|
const messagingConfig = yield* loadMessagingRuntimeConfig();
|
||||||
|
const start = processor.startEffect.pipe(
|
||||||
|
Effect.provideService(PubSub, pubsub),
|
||||||
|
Effect.provideService(ProducerFactory, ProducerFactory.of(makeProducerFactoryService(pubsub))),
|
||||||
|
Effect.provideService(ConsumerFactory, ConsumerFactory.of(makeConsumerFactoryService(pubsub, messagingConfig))),
|
||||||
|
Effect.provideService(
|
||||||
|
RequestResponseFactory,
|
||||||
|
RequestResponseFactory.of(makeRequestResponseFactoryService(pubsub, messagingConfig)),
|
||||||
|
),
|
||||||
|
Effect.provideService(FlowRuntime, FlowRuntime.of({ run: runFlowRuntimeScoped })),
|
||||||
|
);
|
||||||
|
yield* Effect.scoped(start);
|
||||||
|
}),
|
||||||
|
),
|
||||||
};
|
};
|
||||||
|
|
||||||
return processor;
|
return processor;
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@
|
||||||
* Python reference: trustgraph-base/trustgraph/base/flow.py
|
* Python reference: trustgraph-base/trustgraph/base/flow.py
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { Effect, Exit, Scope } from "effect";
|
import { Context, Effect, Exit, Scope } from "effect";
|
||||||
import type { PubSubBackend } from "../backend/types.js";
|
import type { PubSubBackend } from "../backend/types.js";
|
||||||
import { makePubSubService } from "../backend/pubsub.js";
|
import { makePubSubService } from "../backend/pubsub.js";
|
||||||
import {
|
import {
|
||||||
|
|
@ -64,19 +64,20 @@ export function makeFlow<Requirements = never>(
|
||||||
definition: FlowDefinition,
|
definition: FlowDefinition,
|
||||||
specifications: ReadonlyArray<Spec<Requirements>>,
|
specifications: ReadonlyArray<Spec<Requirements>>,
|
||||||
) {
|
) {
|
||||||
const producers = new Map<string, EffectProducer<unknown>>();
|
const producers = new Map<string, EffectProducer<never>>();
|
||||||
const consumers = new Map<string, EffectConsumer>();
|
const consumers = new Map<string, EffectConsumer>();
|
||||||
const requestors = new Map<string, EffectRequestResponse<unknown, unknown>>();
|
const requestors = new Map<string, EffectRequestResponse<never, unknown>>();
|
||||||
const parameters = new Map<string, unknown>();
|
const parameters = new Map<string, unknown>();
|
||||||
let compatibilityScope: Scope.Closeable | null = null;
|
let compatibilityScope: Scope.Closeable | null = null;
|
||||||
|
|
||||||
const ensureCompatibilityScope = async (): Promise<Scope.Closeable> => {
|
const ensureCompatibilityScopeEffect = Effect.fn("Flow.ensureCompatibilityScope")(function* () {
|
||||||
if (compatibilityScope !== null) {
|
if (compatibilityScope !== null) {
|
||||||
return compatibilityScope;
|
return compatibilityScope;
|
||||||
}
|
}
|
||||||
compatibilityScope = await Effect.runPromise(Scope.make());
|
const scope = yield* Scope.make();
|
||||||
return compatibilityScope;
|
compatibilityScope = scope;
|
||||||
};
|
return scope;
|
||||||
|
});
|
||||||
|
|
||||||
const toEffectRequestOptions = <TRes>(
|
const toEffectRequestOptions = <TRes>(
|
||||||
options: FlowRequestOptions<TRes> | undefined,
|
options: FlowRequestOptions<TRes> | undefined,
|
||||||
|
|
@ -105,41 +106,58 @@ export function makeFlow<Requirements = never>(
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
async start(): Promise<void> {
|
start(context: Context.Context<Requirements>): Promise<void> {
|
||||||
if (compatibilityScope !== null) {
|
return Effect.runPromise(
|
||||||
await flow.stop();
|
Effect.gen(function* () {
|
||||||
}
|
if (compatibilityScope !== null) {
|
||||||
await flow.runInCompatibilityScope(
|
yield* flow.stopEffect();
|
||||||
flow.startEffect() as Effect.Effect<void, PubSubError, SpecRuntimeRequirements>,
|
}
|
||||||
pubsub,
|
yield* flow.runInCompatibilityScopeEffect(flow.startEffect(), pubsub, context);
|
||||||
|
}),
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
async stop(): Promise<void> {
|
stop(): Promise<void> {
|
||||||
const scope = compatibilityScope;
|
return Effect.runPromise(flow.stopEffect());
|
||||||
compatibilityScope = null;
|
|
||||||
if (scope !== null) {
|
|
||||||
await Effect.runPromise(Scope.close(scope, Exit.void));
|
|
||||||
}
|
|
||||||
flow.clearResources();
|
|
||||||
},
|
},
|
||||||
async runInCompatibilityScope<A, E>(
|
stopEffect(): Effect.Effect<void> {
|
||||||
effect: Effect.Effect<A, E, SpecRuntimeRequirements>,
|
return Effect.gen(function* () {
|
||||||
|
const scope = compatibilityScope;
|
||||||
|
compatibilityScope = null;
|
||||||
|
if (scope !== null) {
|
||||||
|
yield* Scope.close(scope, Exit.void);
|
||||||
|
}
|
||||||
|
flow.clearResources();
|
||||||
|
});
|
||||||
|
},
|
||||||
|
runInCompatibilityScopeEffect<A, E>(
|
||||||
|
effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>,
|
||||||
runtimePubsub: PubSubBackend,
|
runtimePubsub: PubSubBackend,
|
||||||
): Promise<A> {
|
context: Context.Context<Requirements>,
|
||||||
const scope = await ensureCompatibilityScope();
|
) {
|
||||||
const pubsubService = makePubSubService(runtimePubsub);
|
return Effect.gen(function* () {
|
||||||
const messagingConfig = await Effect.runPromise(loadMessagingRuntimeConfig());
|
const scope = yield* ensureCompatibilityScopeEffect();
|
||||||
return await Effect.runPromise(
|
const pubsubService = makePubSubService(runtimePubsub);
|
||||||
effect.pipe(
|
const messagingConfig = yield* loadMessagingRuntimeConfig();
|
||||||
Effect.provideService(ProducerFactory, ProducerFactory.of(makeProducerFactoryService(pubsubService))),
|
return yield* Effect.provide(
|
||||||
Effect.provideService(ConsumerFactory, ConsumerFactory.of(makeConsumerFactoryService(pubsubService, messagingConfig))),
|
effect.pipe(
|
||||||
Effect.provideService(
|
Effect.provideService(ProducerFactory, ProducerFactory.of(makeProducerFactoryService(pubsubService))),
|
||||||
RequestResponseFactory,
|
Effect.provideService(ConsumerFactory, ConsumerFactory.of(makeConsumerFactoryService(pubsubService, messagingConfig))),
|
||||||
RequestResponseFactory.of(makeRequestResponseFactoryService(pubsubService, messagingConfig)),
|
Effect.provideService(
|
||||||
|
RequestResponseFactory,
|
||||||
|
RequestResponseFactory.of(makeRequestResponseFactoryService(pubsubService, messagingConfig)),
|
||||||
|
),
|
||||||
|
Scope.provide(scope),
|
||||||
),
|
),
|
||||||
Scope.provide(scope),
|
context,
|
||||||
),
|
);
|
||||||
);
|
});
|
||||||
|
},
|
||||||
|
runInCompatibilityScope<A, E>(
|
||||||
|
effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>,
|
||||||
|
runtimePubsub: PubSubBackend,
|
||||||
|
context: Context.Context<Requirements>,
|
||||||
|
): Promise<A> {
|
||||||
|
return Effect.runPromise(flow.runInCompatibilityScopeEffect(effect, runtimePubsub, context));
|
||||||
},
|
},
|
||||||
clearResources(): void {
|
clearResources(): void {
|
||||||
producers.clear();
|
producers.clear();
|
||||||
|
|
@ -147,13 +165,13 @@ export function makeFlow<Requirements = never>(
|
||||||
requestors.clear();
|
requestors.clear();
|
||||||
parameters.clear();
|
parameters.clear();
|
||||||
},
|
},
|
||||||
registerProducer(registerName: string, producer: EffectProducer<unknown>): void {
|
registerProducer<T>(registerName: string, producer: EffectProducer<T>): void {
|
||||||
producers.set(registerName, producer);
|
producers.set(registerName, producer);
|
||||||
},
|
},
|
||||||
registerConsumer(registerName: string, consumer: EffectConsumer): void {
|
registerConsumer(registerName: string, consumer: EffectConsumer): void {
|
||||||
consumers.set(registerName, consumer);
|
consumers.set(registerName, consumer);
|
||||||
},
|
},
|
||||||
registerRequestor(registerName: string, rr: EffectRequestResponse<unknown, unknown>): void {
|
registerRequestor<TReq, TRes>(registerName: string, rr: EffectRequestResponse<TReq, TRes>): void {
|
||||||
requestors.set(registerName, rr);
|
requestors.set(registerName, rr);
|
||||||
},
|
},
|
||||||
setParameter(parameterName: string, value: unknown): void {
|
setParameter(parameterName: string, value: unknown): void {
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@
|
||||||
* executable path while the processor internals remain Promise-based.
|
* executable path while the processor internals remain Promise-based.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { Config as EffectConfig, Effect, Layer, Scope } from "effect";
|
import { Config as EffectConfig, Effect, Layer } from "effect";
|
||||||
import {
|
import {
|
||||||
processorLifecycleError,
|
processorLifecycleError,
|
||||||
type FlowRuntimeError,
|
type FlowRuntimeError,
|
||||||
|
|
@ -37,18 +37,16 @@ import type {
|
||||||
import { runFlowProcessorDefinitionScoped } from "./flow-processor.js";
|
import { runFlowProcessorDefinitionScoped } from "./flow-processor.js";
|
||||||
import type { Spec } from "../spec/types.js";
|
import type { Spec } from "../spec/types.js";
|
||||||
|
|
||||||
type ProcessorRunError<Processor> = Processor extends ProcessorRuntime<infer Error, unknown> ? Error : never;
|
|
||||||
type ProcessorRunRequirements<Processor> = Processor extends ProcessorRuntime<unknown, infer Requirements> ? Requirements : never;
|
|
||||||
|
|
||||||
export interface ProcessorProgramOptions<
|
export interface ProcessorProgramOptions<
|
||||||
Config extends ProcessorConfig,
|
Config extends ProcessorConfig,
|
||||||
Error,
|
LoadError,
|
||||||
Requirements,
|
LoadRequirements,
|
||||||
Processor extends ProcessorRuntime<unknown, unknown>,
|
RunError,
|
||||||
|
RunRequirements,
|
||||||
> {
|
> {
|
||||||
readonly id: string;
|
readonly id: string;
|
||||||
readonly make: (config: Config) => Processor;
|
readonly make: (config: Config) => ProcessorRuntime<RunError, RunRequirements>;
|
||||||
readonly loadConfig?: Effect.Effect<Config, Error, Requirements>;
|
readonly loadConfig?: Effect.Effect<Config, LoadError, LoadRequirements>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface FlowProcessorProgramOptions<
|
export interface FlowProcessorProgramOptions<
|
||||||
|
|
@ -68,18 +66,14 @@ export interface FlowProcessorProgramOptions<
|
||||||
) => Layer.Layer<FlowRequirements, Error, LayerRequirements>;
|
) => Layer.Layer<FlowRequirements, Error, LayerRequirements>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function runProcessorScoped<
|
export const runProcessorScoped = Effect.fn("runProcessorScoped")(function* <
|
||||||
Config extends ProcessorConfig,
|
Config extends ProcessorConfig,
|
||||||
Processor extends ProcessorRuntime<unknown, unknown>,
|
RunError,
|
||||||
|
RunRequirements,
|
||||||
>(
|
>(
|
||||||
config: Config,
|
config: Config,
|
||||||
make: (config: Config) => Processor,
|
make: (config: Config) => ProcessorRuntime<RunError, RunRequirements>,
|
||||||
): Effect.Effect<
|
) {
|
||||||
void,
|
|
||||||
ProcessorRunError<Processor> | ProcessorLifecycleError,
|
|
||||||
PubSub | Scope.Scope | ProcessorRunRequirements<Processor>
|
|
||||||
> {
|
|
||||||
return Effect.gen(function* () {
|
|
||||||
const pubsub = yield* PubSub;
|
const pubsub = yield* PubSub;
|
||||||
const runtimeConfig = {
|
const runtimeConfig = {
|
||||||
...config,
|
...config,
|
||||||
|
|
@ -103,23 +97,17 @@ export function runProcessorScoped<
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
|
|
||||||
yield* (
|
yield* processor.startEffect;
|
||||||
processor.startEffect() as Effect.Effect<
|
});
|
||||||
void,
|
|
||||||
ProcessorRunError<Processor> | ProcessorLifecycleError,
|
|
||||||
ProcessorRunRequirements<Processor>
|
|
||||||
>
|
|
||||||
);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
export function makeProcessorProgram<
|
export function makeProcessorProgram<
|
||||||
Config extends ProcessorConfig,
|
Config extends ProcessorConfig,
|
||||||
Error = never,
|
LoadError = never,
|
||||||
Requirements = never,
|
LoadRequirements = never,
|
||||||
Processor extends ProcessorRuntime<unknown, unknown> = ProcessorRuntime,
|
RunError = ProcessorLifecycleError,
|
||||||
|
RunRequirements = never,
|
||||||
>(
|
>(
|
||||||
options: ProcessorProgramOptions<Config, Error, Requirements, Processor>,
|
options: ProcessorProgramOptions<Config, LoadError, LoadRequirements, RunError, RunRequirements>,
|
||||||
) {
|
) {
|
||||||
return Effect.scoped(
|
return Effect.scoped(
|
||||||
Effect.gen(function* () {
|
Effect.gen(function* () {
|
||||||
|
|
@ -147,7 +135,7 @@ export function makeProcessorProgram<
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
const processorEffect = runProcessorScoped<Config, Processor>(
|
const processorEffect = runProcessorScoped<Config, RunError, RunRequirements>(
|
||||||
runtimeConfig,
|
runtimeConfig,
|
||||||
options.make,
|
options.make,
|
||||||
);
|
);
|
||||||
|
|
@ -173,12 +161,37 @@ export function makeFlowProcessorProgram<
|
||||||
FlowRequirements = never,
|
FlowRequirements = never,
|
||||||
LayerRequirements = never,
|
LayerRequirements = never,
|
||||||
>(
|
>(
|
||||||
options: FlowProcessorProgramOptions<Config, Error, FlowRequirements, LayerRequirements>,
|
options: FlowProcessorProgramOptions<Config, Error, FlowRequirements, LayerRequirements> & {
|
||||||
|
readonly layer: (config: Config) => Layer.Layer<FlowRequirements, Error, LayerRequirements>;
|
||||||
|
},
|
||||||
): Effect.Effect<
|
): Effect.Effect<
|
||||||
void,
|
never,
|
||||||
Error | EffectConfig.ConfigError | PubSubError | FlowRuntimeError,
|
Error | EffectConfig.ConfigError | PubSubError | FlowRuntimeError,
|
||||||
LayerRequirements
|
LayerRequirements
|
||||||
> {
|
>;
|
||||||
|
|
||||||
|
export function makeFlowProcessorProgram<
|
||||||
|
Config extends ProcessorConfig,
|
||||||
|
Error = never,
|
||||||
|
FlowRequirements = never,
|
||||||
|
>(
|
||||||
|
options: FlowProcessorProgramOptions<Config, Error, FlowRequirements, FlowRequirements> & {
|
||||||
|
readonly layer?: undefined;
|
||||||
|
},
|
||||||
|
): Effect.Effect<
|
||||||
|
never,
|
||||||
|
Error | EffectConfig.ConfigError | PubSubError | FlowRuntimeError,
|
||||||
|
FlowRequirements
|
||||||
|
>;
|
||||||
|
|
||||||
|
export function makeFlowProcessorProgram<
|
||||||
|
Config extends ProcessorConfig,
|
||||||
|
Error = never,
|
||||||
|
FlowRequirements = never,
|
||||||
|
LayerRequirements = never,
|
||||||
|
>(
|
||||||
|
options: FlowProcessorProgramOptions<Config, Error, FlowRequirements, LayerRequirements>,
|
||||||
|
) {
|
||||||
return Effect.scoped(
|
return Effect.scoped(
|
||||||
Effect.gen(function* () {
|
Effect.gen(function* () {
|
||||||
const config = yield* (
|
const config = yield* (
|
||||||
|
|
@ -226,14 +239,16 @@ export function makeFlowProcessorProgram<
|
||||||
),
|
),
|
||||||
Layer.succeed(FlowRuntime, FlowRuntime.of({ run: runFlowRuntimeScoped })),
|
Layer.succeed(FlowRuntime, FlowRuntime.of({ run: runFlowRuntimeScoped })),
|
||||||
);
|
);
|
||||||
const dependencyLayer = options.layer?.(runtimeConfig) ??
|
if (options.layer !== undefined) {
|
||||||
(Layer.empty as unknown as Layer.Layer<FlowRequirements, Error, LayerRequirements>);
|
return yield* Layer.launch(
|
||||||
const providedProcessorLayer = processorLayer.pipe(
|
processorLayer.pipe(
|
||||||
Layer.provide(dependencyLayer),
|
Layer.provide(options.layer(runtimeConfig)),
|
||||||
Layer.provide(runtimeLayer),
|
Layer.provide(runtimeLayer),
|
||||||
);
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
return yield* Layer.launch(providedProcessorLayer);
|
return yield* Layer.launch(processorLayer.pipe(Layer.provide(runtimeLayer)));
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@
|
||||||
* Python reference: trustgraph-base/trustgraph/base/llm_service.py
|
* Python reference: trustgraph-base/trustgraph/base/llm_service.py
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { Context, Effect } from "effect";
|
import { Context, Effect, Stream } from "effect";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
import {
|
import {
|
||||||
errorMessage,
|
errorMessage,
|
||||||
|
|
@ -69,7 +69,7 @@ export class Llm extends Context.Service<Llm, LlmServiceShape>()(
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
const llmServiceError = (operation: string, cause: unknown) =>
|
const llmServiceError = (operation: string, cause: unknown) =>
|
||||||
new LlmServiceError({
|
LlmServiceError.make({
|
||||||
operation,
|
operation,
|
||||||
message: errorMessage(cause),
|
message: errorMessage(cause),
|
||||||
});
|
});
|
||||||
|
|
@ -135,22 +135,19 @@ const sendStreamingResponse = Effect.fn("LlmService.sendStreamingResponse")(func
|
||||||
) => Effect.Effect<void, MessagingDeliveryError>;
|
) => Effect.Effect<void, MessagingDeliveryError>;
|
||||||
},
|
},
|
||||||
) {
|
) {
|
||||||
const context = yield* Effect.context<never>();
|
yield* Stream.fromAsyncIterable(
|
||||||
yield* Effect.tryPromise({
|
llm.generateContentStream(
|
||||||
try: async () => {
|
msg.system,
|
||||||
for await (const chunk of llm.generateContentStream(
|
msg.prompt,
|
||||||
msg.system,
|
msg.model,
|
||||||
msg.prompt,
|
msg.temperature,
|
||||||
msg.model,
|
),
|
||||||
msg.temperature,
|
(cause) => llmServiceError("generate-content-stream", cause),
|
||||||
)) {
|
).pipe(
|
||||||
await Effect.runPromiseWith(context)(
|
Stream.runForEach((chunk) =>
|
||||||
responseProducer.send(requestId, chunkToResponse(chunk)),
|
responseProducer.send(requestId, chunkToResponse(chunk)),
|
||||||
);
|
),
|
||||||
}
|
);
|
||||||
},
|
|
||||||
catch: (cause) => llmServiceError("generate-content-stream", cause),
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
const onLlmRequest = Effect.fn("LlmService.onRequest")(function* (
|
const onLlmRequest = Effect.fn("LlmService.onRequest")(function* (
|
||||||
|
|
@ -168,16 +165,22 @@ const onLlmRequest = Effect.fn("LlmService.onRequest")(function* (
|
||||||
|
|
||||||
if (msg.streaming === true && llm.supportsStreaming()) {
|
if (msg.streaming === true && llm.supportsStreaming()) {
|
||||||
yield* sendStreamingResponse(llm, requestId, msg, responseProducer).pipe(
|
yield* sendStreamingResponse(llm, requestId, msg, responseProducer).pipe(
|
||||||
Effect.catch((error) =>
|
Effect.catchTags({
|
||||||
Effect.logError("[LlmService] Error processing streaming request", {
|
LlmServiceError: (error) =>
|
||||||
error: error.message,
|
Effect.logError("[LlmService] Error processing streaming request", {
|
||||||
operation: error.operation,
|
error: error.message,
|
||||||
}).pipe(
|
operation: error.operation,
|
||||||
Effect.flatMap(() =>
|
}).pipe(
|
||||||
responseProducer.send(requestId, llmErrorResponse(error)),
|
Effect.flatMap(() =>
|
||||||
|
responseProducer.send(requestId, llmErrorResponse(error)),
|
||||||
|
),
|
||||||
),
|
),
|
||||||
),
|
MessagingDeliveryError: (error) =>
|
||||||
),
|
Effect.logError("[LlmService] Error sending streaming response", {
|
||||||
|
error: error.message,
|
||||||
|
operation: error.operation,
|
||||||
|
}),
|
||||||
|
}),
|
||||||
);
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -62,14 +62,8 @@ export function makeConsumerSpec<T, E = never, R = never>(
|
||||||
return {
|
return {
|
||||||
name,
|
name,
|
||||||
addEffect,
|
addEffect,
|
||||||
add: async (flow, pubsub, definition) => {
|
add: (flow, pubsub, definition, context) =>
|
||||||
const effect = addEffect(flow as Flow<R>, definition) as Effect.Effect<
|
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
|
||||||
void,
|
|
||||||
PubSubError,
|
|
||||||
SpecRuntimeRequirements
|
|
||||||
>;
|
|
||||||
await flow.runInCompatibilityScope(effect, pubsub);
|
|
||||||
},
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -20,8 +20,6 @@ export function makeParameterSpec(name: string): ParameterSpec {
|
||||||
return {
|
return {
|
||||||
name,
|
name,
|
||||||
addEffect,
|
addEffect,
|
||||||
add: async (flow, _pubsub, definition) => {
|
add: (flow, _pubsub, definition) => Effect.runPromise(addEffect(flow, definition)),
|
||||||
await Effect.runPromise(addEffect(flow, definition));
|
|
||||||
},
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,6 @@ import type { Spec } from "./types.js";
|
||||||
import type { Flow, FlowDefinition } from "../processor/flow.js";
|
import type { Flow, FlowDefinition } from "../processor/flow.js";
|
||||||
import {
|
import {
|
||||||
ProducerFactory,
|
ProducerFactory,
|
||||||
type EffectProducer,
|
|
||||||
} from "../messaging/runtime.js";
|
} from "../messaging/runtime.js";
|
||||||
|
|
||||||
declare const ProducerSpecType: unique symbol;
|
declare const ProducerSpecType: unique symbol;
|
||||||
|
|
@ -26,14 +25,13 @@ export function makeProducerSpec<T>(name: string): ProducerSpec<T> {
|
||||||
const topic = definition.topics?.[name] ?? name;
|
const topic = definition.topics?.[name] ?? name;
|
||||||
const factory = yield* ProducerFactory;
|
const factory = yield* ProducerFactory;
|
||||||
const producer = yield* factory.make<T>({ topic });
|
const producer = yield* factory.make<T>({ topic });
|
||||||
flow.registerProducer(name, producer as EffectProducer<unknown>);
|
flow.registerProducer(name, producer);
|
||||||
});
|
});
|
||||||
|
|
||||||
return {
|
return {
|
||||||
name,
|
name,
|
||||||
addEffect,
|
addEffect,
|
||||||
add: async (flow, pubsub, definition) => {
|
add: (flow, pubsub, definition, context) =>
|
||||||
await flow.runInCompatibilityScope(addEffect(flow, definition), pubsub);
|
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
|
||||||
},
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,6 @@ import type { Spec } from "./types.js";
|
||||||
import type { Flow, FlowDefinition } from "../processor/flow.js";
|
import type { Flow, FlowDefinition } from "../processor/flow.js";
|
||||||
import {
|
import {
|
||||||
RequestResponseFactory,
|
RequestResponseFactory,
|
||||||
type EffectRequestResponse,
|
|
||||||
} from "../messaging/runtime.js";
|
} from "../messaging/runtime.js";
|
||||||
|
|
||||||
declare const RequestResponseSpecType: unique symbol;
|
declare const RequestResponseSpecType: unique symbol;
|
||||||
|
|
@ -41,14 +40,13 @@ export function makeRequestResponseSpec<TReq, TRes>(
|
||||||
responseTopic,
|
responseTopic,
|
||||||
subscription: `${flow.processorId}-${flow.name}-${name}`,
|
subscription: `${flow.processorId}-${flow.name}-${name}`,
|
||||||
});
|
});
|
||||||
flow.registerRequestor(name, requestor as EffectRequestResponse<unknown, unknown>);
|
flow.registerRequestor(name, requestor);
|
||||||
});
|
});
|
||||||
|
|
||||||
return {
|
return {
|
||||||
name,
|
name,
|
||||||
addEffect,
|
addEffect,
|
||||||
add: async (flow, pubsub, definition) => {
|
add: (flow, pubsub, definition, context) =>
|
||||||
await flow.runInCompatibilityScope(addEffect(flow, definition), pubsub);
|
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
|
||||||
},
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@
|
||||||
* Python reference: trustgraph-base/trustgraph/base/spec.py and siblings
|
* Python reference: trustgraph-base/trustgraph/base/spec.py and siblings
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Effect, Scope } from "effect";
|
import type { Context, Effect, Scope } from "effect";
|
||||||
import type { PubSubBackend } from "../backend/types.js";
|
import type { PubSubBackend } from "../backend/types.js";
|
||||||
import type {
|
import type {
|
||||||
ConsumerFactory,
|
ConsumerFactory,
|
||||||
|
|
@ -28,5 +28,10 @@ export interface Spec<Requirements = never> {
|
||||||
flow: Flow<Requirements>,
|
flow: Flow<Requirements>,
|
||||||
definition: FlowDefinition,
|
definition: FlowDefinition,
|
||||||
): Effect.Effect<void, SpecRuntimeError, SpecRuntimeRequirements | Requirements>;
|
): Effect.Effect<void, SpecRuntimeError, SpecRuntimeRequirements | Requirements>;
|
||||||
add(flow: Flow, pubsub: PubSubBackend, definition: FlowDefinition): Promise<void>;
|
add(
|
||||||
|
flow: Flow<Requirements>,
|
||||||
|
pubsub: PubSubBackend,
|
||||||
|
definition: FlowDefinition,
|
||||||
|
context: Context.Context<Requirements>,
|
||||||
|
): Promise<void>;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,21 +5,20 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import { cliCommandError, withSocket } from "./util.js";
|
||||||
|
|
||||||
export function registerAgentCommands(program: Command): void {
|
export function registerAgentCommands(program: Command): void {
|
||||||
program
|
program
|
||||||
.command("agent")
|
.command("agent")
|
||||||
.description("Ask the TrustGraph agent a question")
|
.description("Ask the TrustGraph agent a question")
|
||||||
.argument("<question>", "Question to ask")
|
.argument("<question>", "Question to ask")
|
||||||
.action(async (question: string, _opts, cmd) => {
|
.action((question: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket, opts) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flow = socket.flow(opts.flow);
|
const flow = socket.flow(opts.flow);
|
||||||
|
|
||||||
await new Promise<void>((resolve, reject) => {
|
yield* Effect.callback<void, ReturnType<typeof cliCommandError>>((resume) => {
|
||||||
flow.agent(
|
flow.agent(
|
||||||
question,
|
question,
|
||||||
(chunk) => {
|
(chunk) => {
|
||||||
|
|
@ -35,14 +34,13 @@ export function registerAgentCommands(program: Command): void {
|
||||||
if (chunk.length > 0) process.stdout.write(chunk);
|
if (chunk.length > 0) process.stdout.write(chunk);
|
||||||
if (complete) {
|
if (complete) {
|
||||||
process.stdout.write("\n");
|
process.stdout.write("\n");
|
||||||
resolve();
|
resume(Effect.void);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
(err) => reject(new Error(err)),
|
(err) => resume(Effect.fail(cliCommandError("agent", err))),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
} finally {
|
}),
|
||||||
socket.close();
|
)),
|
||||||
}
|
);
|
||||||
});
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,8 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import { cliCommandError, withSocket, writeJson } from "./util.js";
|
||||||
|
|
||||||
export function registerConfigCommands(program: Command): void {
|
export function registerConfigCommands(program: Command): void {
|
||||||
const config = program
|
const config = program
|
||||||
|
|
@ -15,28 +16,26 @@ export function registerConfigCommands(program: Command): void {
|
||||||
config
|
config
|
||||||
.command("show")
|
.command("show")
|
||||||
.description("Show current configuration")
|
.description("Show current configuration")
|
||||||
.action(async (_opts, cmd) => {
|
.action((_opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const cfg = socket.config();
|
const cfg = socket.config();
|
||||||
const resp = await cfg.getConfigAll();
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => cfg.getConfigAll(),
|
||||||
} finally {
|
catch: (error) => cliCommandError("config.show", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
config
|
config
|
||||||
.command("get")
|
.command("get")
|
||||||
.description("Get a configuration value")
|
.description("Get a configuration value")
|
||||||
.argument("<key>", "Config key (format: type/key)")
|
.argument("<key>", "Config key (format: type/key)")
|
||||||
.action(async (key: string, _opts, cmd) => {
|
.action((key: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const cfg = socket.config();
|
const cfg = socket.config();
|
||||||
// Support "type/key" format; fall back to using the whole string as key
|
// Support "type/key" format; fall back to using the whole string as key
|
||||||
const parts = key.split("/");
|
const parts = key.split("/");
|
||||||
|
|
@ -44,72 +43,74 @@ export function registerConfigCommands(program: Command): void {
|
||||||
parts.length >= 2
|
parts.length >= 2
|
||||||
? { type: parts[0], key: parts.slice(1).join("/") }
|
? { type: parts[0], key: parts.slice(1).join("/") }
|
||||||
: { type: "config", key };
|
: { type: "config", key };
|
||||||
const resp = await cfg.getConfig([configKey]);
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => cfg.getConfig([configKey]),
|
||||||
} finally {
|
catch: (error) => cliCommandError("config.get", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
config
|
config
|
||||||
.command("set")
|
.command("set")
|
||||||
.description("Set a configuration value")
|
.description("Set a configuration value")
|
||||||
.argument("<key>", "Config key (format: type/key)")
|
.argument("<key>", "Config key (format: type/key)")
|
||||||
.argument("<value>", "Config value (JSON)")
|
.argument("<value>", "Config value (JSON)")
|
||||||
.action(async (key: string, value: string, _opts, cmd) => {
|
.action((key: string, value: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const cfg = socket.config();
|
const cfg = socket.config();
|
||||||
const parts = key.split("/");
|
const parts = key.split("/");
|
||||||
const configEntry =
|
const configEntry =
|
||||||
parts.length >= 2
|
parts.length >= 2
|
||||||
? { type: parts[0], key: parts.slice(1).join("/"), value }
|
? { type: parts[0], key: parts.slice(1).join("/"), value }
|
||||||
: { type: "config", key, value };
|
: { type: "config", key, value };
|
||||||
const resp = await cfg.putConfig([configEntry]);
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => cfg.putConfig([configEntry]),
|
||||||
} finally {
|
catch: (error) => cliCommandError("config.set", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
config
|
config
|
||||||
.command("list")
|
.command("list")
|
||||||
.description("List configuration keys for a type")
|
.description("List configuration keys for a type")
|
||||||
.argument("[type]", "Config type to list", "config")
|
.argument("[type]", "Config type to list", "config")
|
||||||
.action(async (type: string, _opts, cmd) => {
|
.action((type: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const cfg = socket.config();
|
const cfg = socket.config();
|
||||||
const resp = await cfg.list(type);
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => cfg.list(type),
|
||||||
} finally {
|
catch: (error) => cliCommandError("config.list", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
config
|
config
|
||||||
.command("delete")
|
.command("delete")
|
||||||
.description("Delete a configuration entry")
|
.description("Delete a configuration entry")
|
||||||
.argument("<key>", "Config key (format: type/key)")
|
.argument("<key>", "Config key (format: type/key)")
|
||||||
.action(async (key: string, _opts, cmd) => {
|
.action((key: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const cfg = socket.config();
|
const cfg = socket.config();
|
||||||
const parts = key.split("/");
|
const parts = key.split("/");
|
||||||
const configKey =
|
const configKey =
|
||||||
parts.length >= 2
|
parts.length >= 2
|
||||||
? { type: parts[0], key: parts.slice(1).join("/") }
|
? { type: parts[0], key: parts.slice(1).join("/") }
|
||||||
: { type: "config", key };
|
: { type: "config", key };
|
||||||
const resp = await cfg.deleteConfig(configKey);
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => cfg.deleteConfig(configKey),
|
||||||
} finally {
|
catch: (error) => cliCommandError("config.delete", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,23 +5,24 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import { cliCommandError, withSocket, writeJson } from "./util.js";
|
||||||
|
|
||||||
export function registerEmbeddingsCommands(program: Command): void {
|
export function registerEmbeddingsCommands(program: Command): void {
|
||||||
program
|
program
|
||||||
.command("embeddings")
|
.command("embeddings")
|
||||||
.description("Generate text embeddings")
|
.description("Generate text embeddings")
|
||||||
.argument("<text...>", "Text(s) to embed")
|
.argument("<text...>", "Text(s) to embed")
|
||||||
.action(async (texts: string[], _opts, cmd) => {
|
.action((texts: string[], _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket, opts) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flow = socket.flow(opts.flow);
|
const flow = socket.flow(opts.flow);
|
||||||
const vectors = await flow.embeddings(texts);
|
const vectors = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(vectors, null, 2));
|
try: () => flow.embeddings(texts),
|
||||||
} finally {
|
catch: (error) => cliCommandError("embeddings", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(vectors);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,9 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import * as S from "effect/Schema";
|
||||||
|
import { cliCommandError, withSocket, writeJson } from "./util.js";
|
||||||
|
|
||||||
export function registerFlowCommands(program: Command): void {
|
export function registerFlowCommands(program: Command): void {
|
||||||
const flow = program
|
const flow = program
|
||||||
|
|
@ -15,35 +17,35 @@ export function registerFlowCommands(program: Command): void {
|
||||||
flow
|
flow
|
||||||
.command("list")
|
.command("list")
|
||||||
.description("List active flows")
|
.description("List active flows")
|
||||||
.action(async (_opts, cmd) => {
|
.action((_opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flows = socket.flows();
|
const flows = socket.flows();
|
||||||
const ids = await flows.getFlows();
|
const ids = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(ids, null, 2));
|
try: () => flows.getFlows(),
|
||||||
} finally {
|
catch: (error) => cliCommandError("flow.list", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(ids);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
flow
|
flow
|
||||||
.command("get")
|
.command("get")
|
||||||
.description("Get a flow definition")
|
.description("Get a flow definition")
|
||||||
.argument("<id>", "Flow ID")
|
.argument("<id>", "Flow ID")
|
||||||
.action(async (id: string, _opts, cmd) => {
|
.action((id: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flows = socket.flows();
|
const flows = socket.flows();
|
||||||
const def = await flows.getFlow(id);
|
const def = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(def, null, 2));
|
try: () => flows.getFlow(id),
|
||||||
} finally {
|
catch: (error) => cliCommandError("flow.get", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(def);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
flow
|
flow
|
||||||
.command("start")
|
.command("start")
|
||||||
|
|
@ -52,42 +54,46 @@ export function registerFlowCommands(program: Command): void {
|
||||||
.requiredOption("-b, --blueprint <name>", "Blueprint name")
|
.requiredOption("-b, --blueprint <name>", "Blueprint name")
|
||||||
.option("-d, --description <text>", "Flow description", "")
|
.option("-d, --description <text>", "Flow description", "")
|
||||||
.option("-p, --parameters <json>", "Parameters as JSON")
|
.option("-p, --parameters <json>", "Parameters as JSON")
|
||||||
.action(async (id: string, cmdOpts, cmd) => {
|
.action((id: string, cmdOpts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flows = socket.flows();
|
const flows = socket.flows();
|
||||||
const rawParameters = cmdOpts.parameters as string | undefined;
|
const rawParameters = cmdOpts.parameters as string | undefined;
|
||||||
const params = rawParameters !== undefined && rawParameters.length > 0
|
const params = rawParameters !== undefined && rawParameters.length > 0
|
||||||
? JSON.parse(rawParameters)
|
? yield* S.decodeUnknownEffect(S.UnknownFromJsonString)(rawParameters).pipe(
|
||||||
|
Effect.flatMap(S.decodeUnknownEffect(S.Record(S.String, S.Unknown))),
|
||||||
|
Effect.mapError((error) => cliCommandError("flow.start.parameters", error)),
|
||||||
|
)
|
||||||
: undefined;
|
: undefined;
|
||||||
const resp = await flows.startFlow(
|
const resp = yield* Effect.tryPromise({
|
||||||
id,
|
try: () =>
|
||||||
cmdOpts.blueprint as string,
|
flows.startFlow(
|
||||||
cmdOpts.description as string,
|
id,
|
||||||
params as Record<string, unknown> | undefined,
|
cmdOpts.blueprint as string,
|
||||||
);
|
cmdOpts.description as string,
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
params,
|
||||||
} finally {
|
),
|
||||||
socket.close();
|
catch: (error) => cliCommandError("flow.start", error),
|
||||||
}
|
});
|
||||||
});
|
yield* writeJson(resp);
|
||||||
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
flow
|
flow
|
||||||
.command("stop")
|
.command("stop")
|
||||||
.description("Stop a flow")
|
.description("Stop a flow")
|
||||||
.argument("<id>", "Flow ID")
|
.argument("<id>", "Flow ID")
|
||||||
.action(async (id: string, _opts, cmd) => {
|
.action((id: string, _opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flows = socket.flows();
|
const flows = socket.flows();
|
||||||
const resp = await flows.stopFlow(id);
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => flows.stopFlow(id),
|
||||||
} finally {
|
catch: (error) => cliCommandError("flow.stop", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,8 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import { cliCommandError, withSocket, writeLine } from "./util.js";
|
||||||
|
|
||||||
export function registerGraphRagCommands(program: Command): void {
|
export function registerGraphRagCommands(program: Command): void {
|
||||||
program
|
program
|
||||||
|
|
@ -15,26 +16,27 @@ export function registerGraphRagCommands(program: Command): void {
|
||||||
.option("--entity-limit <n>", "Max entities", "50")
|
.option("--entity-limit <n>", "Max entities", "50")
|
||||||
.option("--triple-limit <n>", "Max triples per entity", "30")
|
.option("--triple-limit <n>", "Max triples per entity", "30")
|
||||||
.option("--collection <name>", "Collection name")
|
.option("--collection <name>", "Collection name")
|
||||||
.action(async (query: string, cmdOpts, cmd) => {
|
.action((query: string, cmdOpts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket, opts) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flow = socket.flow(opts.flow);
|
const flow = socket.flow(opts.flow);
|
||||||
const collection = cmdOpts.collection as string | undefined;
|
const collection = cmdOpts.collection as string | undefined;
|
||||||
const response = await flow.graphRag(
|
const response = yield* Effect.tryPromise({
|
||||||
query,
|
try: () =>
|
||||||
{
|
flow.graphRag(
|
||||||
entityLimit: parseInt(cmdOpts.entityLimit, 10),
|
query,
|
||||||
tripleLimit: parseInt(cmdOpts.tripleLimit, 10),
|
{
|
||||||
},
|
entityLimit: parseInt(cmdOpts.entityLimit, 10),
|
||||||
collection,
|
tripleLimit: parseInt(cmdOpts.tripleLimit, 10),
|
||||||
);
|
},
|
||||||
console.log(response);
|
collection,
|
||||||
} finally {
|
),
|
||||||
socket.close();
|
catch: (error) => cliCommandError("graph-rag", error),
|
||||||
}
|
});
|
||||||
});
|
yield* writeLine(response);
|
||||||
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
program
|
program
|
||||||
.command("document-rag")
|
.command("document-rag")
|
||||||
|
|
@ -42,24 +44,25 @@ export function registerGraphRagCommands(program: Command): void {
|
||||||
.argument("<query>", "Natural language query")
|
.argument("<query>", "Natural language query")
|
||||||
.option("--doc-limit <n>", "Max documents", "20")
|
.option("--doc-limit <n>", "Max documents", "20")
|
||||||
.option("--collection <name>", "Collection name")
|
.option("--collection <name>", "Collection name")
|
||||||
.action(async (query: string, cmdOpts, cmd) => {
|
.action((query: string, cmdOpts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket, opts) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flow = socket.flow(opts.flow);
|
const flow = socket.flow(opts.flow);
|
||||||
const docLimit = cmdOpts.docLimit as string | undefined;
|
const docLimit = cmdOpts.docLimit as string | undefined;
|
||||||
const collection = cmdOpts.collection as string | undefined;
|
const collection = cmdOpts.collection as string | undefined;
|
||||||
const response = await flow.documentRag(
|
const response = yield* Effect.tryPromise({
|
||||||
query,
|
try: () =>
|
||||||
docLimit !== undefined && docLimit.length > 0
|
flow.documentRag(
|
||||||
? parseInt(docLimit, 10)
|
query,
|
||||||
: undefined,
|
docLimit !== undefined && docLimit.length > 0
|
||||||
collection,
|
? parseInt(docLimit, 10)
|
||||||
);
|
: undefined,
|
||||||
console.log(response);
|
collection,
|
||||||
} finally {
|
),
|
||||||
socket.close();
|
catch: (error) => cliCommandError("document-rag", error),
|
||||||
}
|
});
|
||||||
});
|
yield* writeLine(response);
|
||||||
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,8 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import { cliCommandError, withSocket, writeJson } from "./util.js";
|
||||||
|
|
||||||
function basenamePath(filepath: string): string {
|
function basenamePath(filepath: string): string {
|
||||||
const normalized = filepath.replace(/\/+$/, "");
|
const normalized = filepath.replace(/\/+$/, "");
|
||||||
|
|
@ -45,18 +46,18 @@ export function registerLibraryCommands(program: Command): void {
|
||||||
library
|
library
|
||||||
.command("list")
|
.command("list")
|
||||||
.description("List documents in the library")
|
.description("List documents in the library")
|
||||||
.action(async (_opts, cmd) => {
|
.action((_opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const lib = socket.librarian();
|
const lib = socket.librarian();
|
||||||
const docs = await lib.getDocuments();
|
const docs = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(docs, null, 2));
|
try: () => lib.getDocuments(),
|
||||||
} finally {
|
catch: (error) => cliCommandError("library.list", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(docs);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
library
|
library
|
||||||
.command("load")
|
.command("load")
|
||||||
|
|
@ -67,64 +68,68 @@ export function registerLibraryCommands(program: Command): void {
|
||||||
.option("-c, --comments <text>", "Comments", "")
|
.option("-c, --comments <text>", "Comments", "")
|
||||||
.option("--tags <tags...>", "Document tags")
|
.option("--tags <tags...>", "Document tags")
|
||||||
.option("--id <id>", "Optional document ID")
|
.option("--id <id>", "Optional document ID")
|
||||||
.action(async (file: string, cmdOpts, cmd) => {
|
.action((file: string, cmdOpts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const lib = socket.librarian();
|
const lib = socket.librarian();
|
||||||
const data = new Uint8Array(await Bun.file(file).arrayBuffer());
|
const data = new Uint8Array(yield* Effect.tryPromise({
|
||||||
|
try: () => Bun.file(file).arrayBuffer(),
|
||||||
|
catch: (error) => cliCommandError("library.load.read-file", error),
|
||||||
|
}));
|
||||||
const b64 = Buffer.from(data).toString("base64");
|
const b64 = Buffer.from(data).toString("base64");
|
||||||
const mimeType = (cmdOpts.mimeType as string | undefined) ?? guessMimeType(file);
|
const mimeType = (cmdOpts.mimeType as string | undefined) ?? guessMimeType(file);
|
||||||
const title = (cmdOpts.title as string | undefined) ?? basenamePath(file);
|
const title = (cmdOpts.title as string | undefined) ?? basenamePath(file);
|
||||||
const comments = cmdOpts.comments as string;
|
const comments = cmdOpts.comments as string;
|
||||||
const tags: string[] = (cmdOpts.tags as string[] | undefined) ?? [];
|
const tags: string[] = (cmdOpts.tags as string[] | undefined) ?? [];
|
||||||
|
|
||||||
const resp = await lib.loadDocument(
|
const resp = yield* Effect.tryPromise({
|
||||||
b64,
|
try: () =>
|
||||||
mimeType,
|
lib.loadDocument(
|
||||||
title,
|
b64,
|
||||||
comments,
|
mimeType,
|
||||||
tags,
|
title,
|
||||||
cmdOpts.id as string | undefined,
|
comments,
|
||||||
);
|
tags,
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
cmdOpts.id as string | undefined,
|
||||||
} finally {
|
),
|
||||||
socket.close();
|
catch: (error) => cliCommandError("library.load", error),
|
||||||
}
|
});
|
||||||
});
|
yield* writeJson(resp);
|
||||||
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
library
|
library
|
||||||
.command("remove")
|
.command("remove")
|
||||||
.description("Remove a document from the library")
|
.description("Remove a document from the library")
|
||||||
.argument("<id>", "Document ID to remove")
|
.argument("<id>", "Document ID to remove")
|
||||||
.option("--collection <name>", "Collection name")
|
.option("--collection <name>", "Collection name")
|
||||||
.action(async (id: string, cmdOpts, cmd) => {
|
.action((id: string, cmdOpts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const lib = socket.librarian();
|
const lib = socket.librarian();
|
||||||
const resp = await lib.removeDocument(id, cmdOpts.collection as string | undefined);
|
const resp = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(resp, null, 2));
|
try: () => lib.removeDocument(id, cmdOpts.collection as string | undefined),
|
||||||
} finally {
|
catch: (error) => cliCommandError("library.remove", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(resp);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
|
||||||
library
|
library
|
||||||
.command("processing")
|
.command("processing")
|
||||||
.description("List documents currently being processed")
|
.description("List documents currently being processed")
|
||||||
.action(async (_opts, cmd) => {
|
.action((_opts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const lib = socket.librarian();
|
const lib = socket.librarian();
|
||||||
const items = await lib.getProcessing();
|
const items = yield* Effect.tryPromise({
|
||||||
console.log(JSON.stringify(items, null, 2));
|
try: () => lib.getProcessing(),
|
||||||
} finally {
|
catch: (error) => cliCommandError("library.processing", error),
|
||||||
socket.close();
|
});
|
||||||
}
|
yield* writeJson(items);
|
||||||
});
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,8 @@
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import type { Term } from "@trustgraph/client";
|
import type { Term } from "@trustgraph/client";
|
||||||
import { createSocket, getOpts } from "./util.js";
|
import { Effect } from "effect";
|
||||||
|
import { cliCommandError, withSocket, writeJson } from "./util.js";
|
||||||
|
|
||||||
export function registerTriplesCommands(program: Command): void {
|
export function registerTriplesCommands(program: Command): void {
|
||||||
program
|
program
|
||||||
|
|
@ -17,11 +18,9 @@ export function registerTriplesCommands(program: Command): void {
|
||||||
.option("-o, --object <iri>", "Object IRI or literal")
|
.option("-o, --object <iri>", "Object IRI or literal")
|
||||||
.option("-l, --limit <n>", "Max results", "20")
|
.option("-l, --limit <n>", "Max results", "20")
|
||||||
.option("--collection <name>", "Collection name")
|
.option("--collection <name>", "Collection name")
|
||||||
.action(async (cmdOpts, cmd) => {
|
.action((cmdOpts, cmd) =>
|
||||||
const opts = getOpts(cmd);
|
Effect.runPromise(withSocket(cmd, (socket, opts) =>
|
||||||
const socket = await createSocket(opts);
|
Effect.gen(function* () {
|
||||||
|
|
||||||
try {
|
|
||||||
const flow = socket.flow(opts.flow);
|
const flow = socket.flow(opts.flow);
|
||||||
const subject = cmdOpts.subject as string | undefined;
|
const subject = cmdOpts.subject as string | undefined;
|
||||||
const predicate = cmdOpts.predicate as string | undefined;
|
const predicate = cmdOpts.predicate as string | undefined;
|
||||||
|
|
@ -36,16 +35,19 @@ export function registerTriplesCommands(program: Command): void {
|
||||||
? { t: "i", i: object }
|
? { t: "i", i: object }
|
||||||
: undefined;
|
: undefined;
|
||||||
|
|
||||||
const triples = await flow.triplesQuery(
|
const triples = yield* Effect.tryPromise({
|
||||||
s,
|
try: () =>
|
||||||
p,
|
flow.triplesQuery(
|
||||||
o,
|
s,
|
||||||
parseInt(cmdOpts.limit as string, 10),
|
p,
|
||||||
cmdOpts.collection as string | undefined,
|
o,
|
||||||
);
|
parseInt(cmdOpts.limit as string, 10),
|
||||||
console.log(JSON.stringify(triples, null, 2));
|
cmdOpts.collection as string | undefined,
|
||||||
} finally {
|
),
|
||||||
socket.close();
|
catch: (error) => cliCommandError("triples", error),
|
||||||
}
|
});
|
||||||
});
|
yield* writeJson(triples);
|
||||||
|
}),
|
||||||
|
)),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,8 @@
|
||||||
|
|
||||||
import type { Command } from "commander";
|
import type { Command } from "commander";
|
||||||
import { createTrustGraphSocket, type BaseApi } from "@trustgraph/client";
|
import { createTrustGraphSocket, type BaseApi } from "@trustgraph/client";
|
||||||
|
import { Duration, Effect } from "effect";
|
||||||
|
import * as S from "effect/Schema";
|
||||||
|
|
||||||
export interface CliOpts {
|
export interface CliOpts {
|
||||||
gateway: string;
|
gateway: string;
|
||||||
|
|
@ -19,36 +21,76 @@ export function getOpts(cmd: Command): CliOpts {
|
||||||
return root.opts() as CliOpts;
|
return root.opts() as CliOpts;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export class CliCommandError extends S.TaggedErrorClass<CliCommandError>()(
|
||||||
|
"CliCommandError",
|
||||||
|
{
|
||||||
|
message: S.String,
|
||||||
|
operation: S.String,
|
||||||
|
},
|
||||||
|
) {}
|
||||||
|
|
||||||
|
export function cliCommandError(operation: string, error: unknown): CliCommandError {
|
||||||
|
const message = typeof error === "object" && error !== null && "message" in error
|
||||||
|
? String(error.message)
|
||||||
|
: String(error);
|
||||||
|
return CliCommandError.make({ operation, message });
|
||||||
|
}
|
||||||
|
|
||||||
|
export const writeLine = (line: string) =>
|
||||||
|
Effect.sync(() => {
|
||||||
|
process.stdout.write(`${line}\n`);
|
||||||
|
});
|
||||||
|
|
||||||
|
export const writeJson = (value: unknown) =>
|
||||||
|
S.encodeUnknownEffect(S.UnknownFromJsonString)(value).pipe(
|
||||||
|
Effect.mapError((error) => cliCommandError("write-json", error)),
|
||||||
|
Effect.flatMap(writeLine),
|
||||||
|
);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Create a BaseApi socket client and wait for the connection to be established.
|
* Create a BaseApi socket client and wait for the connection to be established.
|
||||||
* The client auto-connects; we listen for the first "connected/authenticated"
|
* The client auto-connects; we listen for the first "connected/authenticated"
|
||||||
* state before handing it back to the caller.
|
* state before handing it back to the caller.
|
||||||
*/
|
*/
|
||||||
export async function createSocket(opts: CliOpts): Promise<BaseApi> {
|
export function createSocketEffect(opts: CliOpts): Effect.Effect<BaseApi, CliCommandError> {
|
||||||
const socket = createTrustGraphSocket(opts.user, opts.token, opts.gateway);
|
const socket = createTrustGraphSocket(opts.user, opts.token, opts.gateway);
|
||||||
|
|
||||||
// Wait for the socket to reach an open state
|
return Effect.callback<void, CliCommandError>((resume) => {
|
||||||
await new Promise<void>((resolve, reject) => {
|
|
||||||
const timeout = setTimeout(() => {
|
|
||||||
unsub();
|
|
||||||
reject(new Error("Timed out waiting for WebSocket connection"));
|
|
||||||
}, 15_000);
|
|
||||||
|
|
||||||
const unsub = socket.onConnectionStateChange((state) => {
|
const unsub = socket.onConnectionStateChange((state) => {
|
||||||
if (
|
if (state.status === "authenticated" || state.status === "unauthenticated") {
|
||||||
state.status === "authenticated" ||
|
|
||||||
state.status === "unauthenticated"
|
|
||||||
) {
|
|
||||||
clearTimeout(timeout);
|
|
||||||
unsub();
|
unsub();
|
||||||
resolve();
|
resume(Effect.void);
|
||||||
} else if (state.status === "failed") {
|
} else if (state.status === "failed") {
|
||||||
clearTimeout(timeout);
|
|
||||||
unsub();
|
unsub();
|
||||||
reject(new Error(state.lastError ?? "WebSocket connection failed"));
|
resume(Effect.fail(cliCommandError("connect", state.lastError ?? "WebSocket connection failed")));
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
});
|
|
||||||
|
|
||||||
return socket;
|
return Effect.sync(() => {
|
||||||
|
unsub();
|
||||||
|
});
|
||||||
|
}).pipe(
|
||||||
|
Effect.timeout(Duration.seconds(15)),
|
||||||
|
Effect.catchTag("TimeoutError", () =>
|
||||||
|
Effect.fail(cliCommandError("connect", "Timed out waiting for WebSocket connection")),
|
||||||
|
),
|
||||||
|
Effect.as(socket),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function createSocket(opts: CliOpts): Promise<BaseApi> {
|
||||||
|
return Effect.runPromise(createSocketEffect(opts));
|
||||||
|
}
|
||||||
|
|
||||||
|
export const withSocket = <A, E, R>(
|
||||||
|
cmd: Command,
|
||||||
|
use: (socket: BaseApi, opts: CliOpts) => Effect.Effect<A, E, R>,
|
||||||
|
) =>
|
||||||
|
Effect.acquireUseRelease(
|
||||||
|
createSocketEffect(getOpts(cmd)),
|
||||||
|
(socket) => use(socket, getOpts(cmd)),
|
||||||
|
(socket) =>
|
||||||
|
Effect.sync(() => {
|
||||||
|
socket.close();
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@
|
||||||
* Python reference: trustgraph-flow/trustgraph/config/service/service.py
|
* Python reference: trustgraph-flow/trustgraph/config/service/service.py
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { Duration, Effect } from "effect";
|
import { Context, Duration, Effect } from "effect";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
import {
|
import {
|
||||||
makeAsyncProcessor,
|
makeAsyncProcessor,
|
||||||
|
|
@ -148,7 +148,7 @@ export type ConfigService = AsyncProcessorRuntime & Record<string, any>;
|
||||||
|
|
||||||
export function makeConfigService(config: ConfigServiceConfig): ConfigService {
|
export function makeConfigService(config: ConfigServiceConfig): ConfigService {
|
||||||
const service = makeAsyncProcessor(config, {
|
const service = makeAsyncProcessor(config, {
|
||||||
run: () => service.run(),
|
run: () => service.run(Context.empty()),
|
||||||
}) as ConfigService;
|
}) as ConfigService;
|
||||||
const baseStop = service.stop;
|
const baseStop = service.stop;
|
||||||
service.store = new Map<string, WorkspaceStore>();
|
service.store = new Map<string, WorkspaceStore>();
|
||||||
|
|
|
||||||
|
|
@ -23,7 +23,7 @@ import {
|
||||||
errorMessage,
|
errorMessage,
|
||||||
} from "@trustgraph/base";
|
} from "@trustgraph/base";
|
||||||
import type { Message } from "@trustgraph/base";
|
import type { Message } from "@trustgraph/base";
|
||||||
import { Config, Duration, Effect } from "effect";
|
import { Config, Context, Duration, Effect } from "effect";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
import { ensureDirectory, joinPath, readTextFile, writeTextFile } from "../runtime/effect-files.js";
|
import { ensureDirectory, joinPath, readTextFile, writeTextFile } from "../runtime/effect-files.js";
|
||||||
|
|
||||||
|
|
@ -118,7 +118,7 @@ const closeResource = (
|
||||||
|
|
||||||
export function makeKnowledgeCoreService(config: KnowledgeCoreServiceConfig): KnowledgeCoreService {
|
export function makeKnowledgeCoreService(config: KnowledgeCoreServiceConfig): KnowledgeCoreService {
|
||||||
const service = makeAsyncProcessor(config, {
|
const service = makeAsyncProcessor(config, {
|
||||||
run: () => service.run(),
|
run: () => service.run(Context.empty()),
|
||||||
}) as KnowledgeCoreService;
|
}) as KnowledgeCoreService;
|
||||||
const baseStop = service.stop;
|
const baseStop = service.stop;
|
||||||
service.cores = new Map<string, KnowledgeCore>();
|
service.cores = new Map<string, KnowledgeCore>();
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ import {
|
||||||
} from "@trustgraph/base";
|
} from "@trustgraph/base";
|
||||||
import { makeProcessorProgram } from "@trustgraph/base";
|
import { makeProcessorProgram } from "@trustgraph/base";
|
||||||
import type { Message } from "@trustgraph/base";
|
import type { Message } from "@trustgraph/base";
|
||||||
import { Duration, Effect, Option } from "effect";
|
import { Context, Duration, Effect, Option } from "effect";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
|
|
||||||
// ---------- Internal state types ----------
|
// ---------- Internal state types ----------
|
||||||
|
|
@ -158,7 +158,7 @@ export type FlowManagerService = AsyncProcessorRuntime & Record<string, any>;
|
||||||
|
|
||||||
export function makeFlowManagerService(config: ProcessorConfig): FlowManagerService {
|
export function makeFlowManagerService(config: ProcessorConfig): FlowManagerService {
|
||||||
const service = makeAsyncProcessor(config, {
|
const service = makeAsyncProcessor(config, {
|
||||||
run: () => service.run(),
|
run: () => service.run(Context.empty()),
|
||||||
}) as FlowManagerService;
|
}) as FlowManagerService;
|
||||||
const baseStop = service.stop;
|
const baseStop = service.stop;
|
||||||
service.flows = new Map<string, FlowInstance>();
|
service.flows = new Map<string, FlowInstance>();
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ import {
|
||||||
type ProcessingMetadata,
|
type ProcessingMetadata,
|
||||||
} from "@trustgraph/base";
|
} from "@trustgraph/base";
|
||||||
import type { Message } from "@trustgraph/base";
|
import type { Message } from "@trustgraph/base";
|
||||||
import { Clock, Config, DateTime, Duration, Effect, Random } from "effect";
|
import { Clock, Config, Context, DateTime, Duration, Effect, Random } from "effect";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
import { makeCollectionManager } from "./collection-manager.js";
|
import { makeCollectionManager } from "./collection-manager.js";
|
||||||
import {
|
import {
|
||||||
|
|
@ -139,7 +139,7 @@ export type LibrarianService = AsyncProcessorRuntime & Record<string, any>;
|
||||||
|
|
||||||
export function makeLibrarianService(config: LibrarianServiceConfig): LibrarianService {
|
export function makeLibrarianService(config: LibrarianServiceConfig): LibrarianService {
|
||||||
const service = makeAsyncProcessor(config, {
|
const service = makeAsyncProcessor(config, {
|
||||||
run: () => service.run(),
|
run: () => service.run(Context.empty()),
|
||||||
}) as LibrarianService;
|
}) as LibrarianService;
|
||||||
const baseStop = service.stop;
|
const baseStop = service.stop;
|
||||||
service.documents = new Map<string, DocumentMetadata>();
|
service.documents = new Map<string, DocumentMetadata>();
|
||||||
|
|
|
||||||
|
|
@ -1,14 +1,21 @@
|
||||||
import {OpenAiClient, OpenAiLanguageModel} from "@effect/ai-openai";
|
import {OpenAiClient, OpenAiLanguageModel} from "@effect/ai-openai";
|
||||||
import {BunHttpServer, BunRuntime} from "@effect/platform-bun";
|
import {BunHttpServer, BunRuntime} from "@effect/platform-bun";
|
||||||
import {createTrustGraphSocket, type BaseApi, type Term as ClientTerm} from "@trustgraph/client";
|
import {createTrustGraphSocket, type BaseApi, type Term as ClientTerm} from "@trustgraph/client";
|
||||||
import {Context, Effect, Layer, Redacted} from "effect";
|
import {Config, Context, Effect, Layer, Redacted} from "effect";
|
||||||
|
import * as O from "effect/Option";
|
||||||
|
import * as Predicate from "effect/Predicate";
|
||||||
import {LanguageModel, McpServer, Prompt, Tool, Toolkit} from "effect/unstable/ai";
|
import {LanguageModel, McpServer, Prompt, Tool, Toolkit} from "effect/unstable/ai";
|
||||||
import {FetchHttpClient, HttpRouter} from "effect/unstable/http";
|
import {FetchHttpClient, HttpRouter} from "effect/unstable/http";
|
||||||
import {HttpApi, HttpApiBuilder, HttpApiEndpoint, HttpApiGroup, HttpApiSchema, OpenApi} from "effect/unstable/httpapi";
|
import {HttpApi, HttpApiBuilder, HttpApiEndpoint, HttpApiGroup, HttpApiSchema, OpenApi} from "effect/unstable/httpapi";
|
||||||
import * as S from "effect/Schema";
|
import * as S from "effect/Schema";
|
||||||
|
|
||||||
const annotateTool = <T extends Tool.Any>(
|
const annotateTool = <Name extends string, Config extends {
|
||||||
tool: T,
|
readonly parameters: S.Top
|
||||||
|
readonly success: S.Top
|
||||||
|
readonly failure: S.Top
|
||||||
|
readonly failureMode: Tool.FailureMode
|
||||||
|
}, Requirements>(
|
||||||
|
tool: Tool.Tool<Name, Config, Requirements>,
|
||||||
annotations: {
|
annotations: {
|
||||||
readonly title: string
|
readonly title: string
|
||||||
readonly readOnly: boolean
|
readonly readOnly: boolean
|
||||||
|
|
@ -17,14 +24,14 @@ const annotateTool = <T extends Tool.Any>(
|
||||||
readonly openWorld: boolean
|
readonly openWorld: boolean
|
||||||
readonly strict?: boolean
|
readonly strict?: boolean
|
||||||
},
|
},
|
||||||
): T =>
|
): Tool.Tool<Name, Config, Requirements> =>
|
||||||
tool
|
tool
|
||||||
.annotate(Tool.Title, annotations.title)
|
.annotate(Tool.Title, annotations.title)
|
||||||
.annotate(Tool.Readonly, annotations.readOnly)
|
.annotate(Tool.Readonly, annotations.readOnly)
|
||||||
.annotate(Tool.Destructive, annotations.destructive)
|
.annotate(Tool.Destructive, annotations.destructive)
|
||||||
.annotate(Tool.Idempotent, annotations.idempotent)
|
.annotate(Tool.Idempotent, annotations.idempotent)
|
||||||
.annotate(Tool.OpenWorld, annotations.openWorld)
|
.annotate(Tool.OpenWorld, annotations.openWorld)
|
||||||
.annotate(Tool.Strict, annotations.strict ?? true) as T
|
.annotate(Tool.Strict, annotations.strict ?? true)
|
||||||
|
|
||||||
class PromptSummary extends S.Class<PromptSummary>("PromptSummary")(
|
class PromptSummary extends S.Class<PromptSummary>("PromptSummary")(
|
||||||
{
|
{
|
||||||
|
|
@ -1217,18 +1224,14 @@ export interface TrustGraphMcpConfigShape {
|
||||||
readonly version: string
|
readonly version: string
|
||||||
readonly mcpPath: HttpRouter.PathInput
|
readonly mcpPath: HttpRouter.PathInput
|
||||||
readonly openAiModel: string
|
readonly openAiModel: string
|
||||||
readonly openAiApiKey: string | undefined
|
readonly openAiApiKey: Redacted.Redacted | undefined
|
||||||
readonly port: number
|
readonly port: number
|
||||||
}
|
}
|
||||||
|
|
||||||
const readNonEmpty = (value: string | undefined): string | undefined =>
|
const readNonEmpty = (value: string | undefined): string | undefined =>
|
||||||
value !== undefined && value.length > 0 ? value : undefined
|
value !== undefined && value.length > 0 ? value : undefined
|
||||||
|
|
||||||
const resolvePort = (value: number | undefined): number => {
|
const parsePort = (raw: string | undefined): number => {
|
||||||
if (value !== undefined) {
|
|
||||||
return value
|
|
||||||
}
|
|
||||||
const raw = readNonEmpty(process.env.PORT)
|
|
||||||
if (raw === undefined) {
|
if (raw === undefined) {
|
||||||
return 3000
|
return 3000
|
||||||
}
|
}
|
||||||
|
|
@ -1236,28 +1239,46 @@ const resolvePort = (value: number | undefined): number => {
|
||||||
return Number.isFinite(parsed) ? parsed : 3000
|
return Number.isFinite(parsed) ? parsed : 3000
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export const loadTrustGraphMcpConfig = Effect.fn("loadTrustGraphMcpConfig")(function*(
|
||||||
|
options: TrustGraphMcpOptions = {},
|
||||||
|
) {
|
||||||
|
const gatewayUrl = O.getOrUndefined(yield* Config.string("GATEWAY_URL").pipe(Config.option))
|
||||||
|
const user = O.getOrUndefined(yield* Config.string("USER_ID").pipe(Config.option))
|
||||||
|
const gatewaySecret = O.getOrUndefined(yield* Config.string("GATEWAY_SECRET").pipe(Config.option))
|
||||||
|
const token = readNonEmpty(gatewaySecret)
|
||||||
|
const flowId = O.getOrUndefined(yield* Config.string("FLOW_ID").pipe(Config.option))
|
||||||
|
const openAiModel = O.getOrUndefined(yield* Config.string("OPENAI_MODEL").pipe(Config.option))
|
||||||
|
const openAiApiKey = O.getOrUndefined(yield* Config.redacted("OPENAI_API_KEY").pipe(Config.option))
|
||||||
|
const openAiToken = O.getOrUndefined(yield* Config.redacted("OPENAI_TOKEN").pipe(Config.option))
|
||||||
|
const port = O.getOrUndefined(yield* Config.string("PORT").pipe(Config.option))
|
||||||
|
|
||||||
|
return {
|
||||||
|
gatewayUrl: options.gatewayUrl ?? gatewayUrl ?? "ws://localhost:8088/api/v1/rpc",
|
||||||
|
user: options.user ?? user ?? "mcp",
|
||||||
|
token: options.token ?? token,
|
||||||
|
flowId: options.flowId ?? flowId ?? "default",
|
||||||
|
name: options.name ?? "trustgraph",
|
||||||
|
version: options.version ?? "0.1.0",
|
||||||
|
mcpPath: options.mcpPath ?? "/mcp",
|
||||||
|
openAiModel: options.openAiModel ?? openAiModel ?? "gpt-4.1",
|
||||||
|
openAiApiKey: options.openAiApiKey === undefined
|
||||||
|
? openAiApiKey ?? openAiToken
|
||||||
|
: Redacted.make(options.openAiApiKey),
|
||||||
|
port: options.port ?? parsePort(readNonEmpty(port)),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
export const resolveTrustGraphMcpConfig = (
|
export const resolveTrustGraphMcpConfig = (
|
||||||
options: TrustGraphMcpOptions = {},
|
options: TrustGraphMcpOptions = {},
|
||||||
): TrustGraphMcpConfigShape => ({
|
): TrustGraphMcpConfigShape => Effect.runSync(loadTrustGraphMcpConfig(options))
|
||||||
gatewayUrl: options.gatewayUrl ?? process.env.GATEWAY_URL ?? "ws://localhost:8088/api/v1/rpc",
|
|
||||||
user: options.user ?? process.env.USER_ID ?? "mcp",
|
|
||||||
token: options.token ?? readNonEmpty(process.env.GATEWAY_SECRET),
|
|
||||||
flowId: options.flowId ?? process.env.FLOW_ID ?? "default",
|
|
||||||
name: options.name ?? "trustgraph",
|
|
||||||
version: options.version ?? "0.1.0",
|
|
||||||
mcpPath: options.mcpPath ?? "/mcp",
|
|
||||||
openAiModel: options.openAiModel ?? process.env.OPENAI_MODEL ?? "gpt-4.1",
|
|
||||||
openAiApiKey: options.openAiApiKey ?? readNonEmpty(process.env.OPENAI_API_KEY) ?? readNonEmpty(process.env.OPENAI_TOKEN),
|
|
||||||
port: resolvePort(options.port),
|
|
||||||
})
|
|
||||||
|
|
||||||
export class TrustGraphMcpConfig extends Context.Service<TrustGraphMcpConfig, TrustGraphMcpConfigShape>()(
|
export class TrustGraphMcpConfig extends Context.Service<TrustGraphMcpConfig, TrustGraphMcpConfigShape>()(
|
||||||
"@trustgraph/mcp/server-effect/TrustGraphMcpConfig",
|
"@trustgraph/mcp/server-effect/TrustGraphMcpConfig",
|
||||||
) {
|
) {
|
||||||
static readonly layer = (options: TrustGraphMcpOptions = {}) =>
|
static readonly layer = (options: TrustGraphMcpOptions = {}) =>
|
||||||
Layer.succeed(
|
Layer.effect(
|
||||||
TrustGraphMcpConfig,
|
TrustGraphMcpConfig,
|
||||||
TrustGraphMcpConfig.of(resolveTrustGraphMcpConfig(options)),
|
loadTrustGraphMcpConfig(options).pipe(Effect.map(TrustGraphMcpConfig.of)),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1278,17 +1299,14 @@ export class TrustGraphSocket extends Context.Service<TrustGraphSocket, BaseApi>
|
||||||
}
|
}
|
||||||
|
|
||||||
const toErrorMessage = (cause: unknown): string => {
|
const toErrorMessage = (cause: unknown): string => {
|
||||||
if (cause instanceof Error && cause.message.length > 0) {
|
if (Predicate.isError(cause) && cause.message.length > 0) {
|
||||||
return cause.message
|
return cause.message
|
||||||
}
|
}
|
||||||
if (typeof cause === "string" && cause.length > 0) {
|
if (typeof cause === "string" && cause.length > 0) {
|
||||||
return cause
|
return cause
|
||||||
}
|
}
|
||||||
if (cause !== null && typeof cause === "object" && "message" in cause) {
|
if (Predicate.isObject(cause) && Predicate.hasProperty(cause, "message") && Predicate.isString(cause.message) && cause.message.length > 0) {
|
||||||
const message = (cause as { readonly message?: unknown }).message
|
return cause.message
|
||||||
if (typeof message === "string" && message.length > 0) {
|
|
||||||
return message
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
return "TrustGraph MCP tool failed"
|
return "TrustGraph MCP tool failed"
|
||||||
}
|
}
|
||||||
|
|
@ -1313,16 +1331,15 @@ const decodeJsonArrayOrFail = <E>(
|
||||||
const asIriTerm = (value: string | undefined): ClientTerm | undefined =>
|
const asIriTerm = (value: string | undefined): ClientTerm | undefined =>
|
||||||
value !== undefined && value.length > 0 ? {t: "i", i: value} : undefined
|
value !== undefined && value.length > 0 ? {t: "i", i: value} : undefined
|
||||||
|
|
||||||
const openAiApiKeyOptions = (apiKey: string | undefined) =>
|
const openAiApiKeyOptions = (apiKey: Redacted.Redacted | undefined) =>
|
||||||
apiKey === undefined
|
apiKey === undefined
|
||||||
? {}
|
? {}
|
||||||
: {apiKey: Redacted.make(apiKey)}
|
: {apiKey}
|
||||||
|
|
||||||
export const makeOpenAiProviderLayer = (
|
const makeOpenAiProviderLayerFromConfig = (
|
||||||
options: TrustGraphMcpOptions = {},
|
config: TrustGraphMcpConfigShape,
|
||||||
) => {
|
) =>
|
||||||
const config = resolveTrustGraphMcpConfig(options)
|
OpenAiLanguageModel.layer({
|
||||||
return OpenAiLanguageModel.layer({
|
|
||||||
model: config.openAiModel,
|
model: config.openAiModel,
|
||||||
config: {
|
config: {
|
||||||
strictJsonSchema: true,
|
strictJsonSchema: true,
|
||||||
|
|
@ -1331,7 +1348,15 @@ export const makeOpenAiProviderLayer = (
|
||||||
Layer.provide(OpenAiClient.layer(openAiApiKeyOptions(config.openAiApiKey))),
|
Layer.provide(OpenAiClient.layer(openAiApiKeyOptions(config.openAiApiKey))),
|
||||||
Layer.provide(FetchHttpClient.layer),
|
Layer.provide(FetchHttpClient.layer),
|
||||||
)
|
)
|
||||||
}
|
|
||||||
|
export const makeOpenAiProviderLayer = (
|
||||||
|
options: TrustGraphMcpOptions = {},
|
||||||
|
) =>
|
||||||
|
Layer.unwrap(
|
||||||
|
loadTrustGraphMcpConfig(options).pipe(
|
||||||
|
Effect.map(makeOpenAiProviderLayerFromConfig),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
Effect.gen(function*() {
|
Effect.gen(function*() {
|
||||||
|
|
@ -1344,33 +1369,32 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
const response = yield* model.generateText({
|
const response = yield* model.generateText({
|
||||||
prompt: Prompt.make(prompt).pipe(Prompt.setSystem(system)),
|
prompt: Prompt.make(prompt).pipe(Prompt.setSystem(system)),
|
||||||
})
|
})
|
||||||
return new TextCompletionSuccess({text: response.text})
|
return TextCompletionSuccess.make({text: response.text})
|
||||||
}),
|
}),
|
||||||
|
|
||||||
graph_rag: ({query, entity_limit, triple_limit, collection}) =>
|
graph_rag: ({query, entity_limit, triple_limit, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => {
|
try: () =>
|
||||||
const response = await socket.flow(config.flowId).graphRag(
|
socket.flow(config.flowId).graphRag(
|
||||||
query,
|
query,
|
||||||
{
|
{
|
||||||
...(entity_limit !== undefined ? {entityLimit: entity_limit} : {}),
|
...(entity_limit !== undefined ? {entityLimit: entity_limit} : {}),
|
||||||
...(triple_limit !== undefined ? {tripleLimit: triple_limit} : {}),
|
...(triple_limit !== undefined ? {tripleLimit: triple_limit} : {}),
|
||||||
},
|
},
|
||||||
collection,
|
collection,
|
||||||
)
|
),
|
||||||
return new GraphRagSuccess({text: response})
|
catch: (cause) => GraphRagError.make({cause, message: toErrorMessage(cause)}),
|
||||||
},
|
}).pipe(
|
||||||
catch: (cause) => new GraphRagError({cause, message: toErrorMessage(cause)}),
|
Effect.map((text) => GraphRagSuccess.make({text})),
|
||||||
}),
|
),
|
||||||
|
|
||||||
document_rag: ({query, doc_limit, collection}) =>
|
document_rag: ({query, doc_limit, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => {
|
try: () => socket.flow(config.flowId).documentRag(query, doc_limit, collection),
|
||||||
const response = await socket.flow(config.flowId).documentRag(query, doc_limit, collection)
|
catch: (cause) => DocumentRagError.make({cause, message: toErrorMessage(cause)}),
|
||||||
return new DocumentRagSuccess({text: response})
|
}).pipe(
|
||||||
},
|
Effect.map((text) => DocumentRagSuccess.make({text})),
|
||||||
catch: (cause) => new DocumentRagError({cause, message: toErrorMessage(cause)}),
|
),
|
||||||
}),
|
|
||||||
|
|
||||||
agent: ({question}) =>
|
agent: ({question}) =>
|
||||||
Effect.callback<AgentSuccess, AgentError>((resume) => {
|
Effect.callback<AgentSuccess, AgentError>((resume) => {
|
||||||
|
|
@ -1382,62 +1406,65 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
(chunk, complete) => {
|
(chunk, complete) => {
|
||||||
fullAnswer += chunk
|
fullAnswer += chunk
|
||||||
if (complete) {
|
if (complete) {
|
||||||
resume(Effect.succeed(new AgentSuccess({text: fullAnswer})))
|
resume(Effect.succeed(AgentSuccess.make({text: fullAnswer})))
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
(cause) => resume(Effect.fail(new AgentError({cause, message: toErrorMessage(cause)}))),
|
(cause) => resume(Effect.fail(AgentError.make({cause, message: toErrorMessage(cause)}))),
|
||||||
)
|
)
|
||||||
}),
|
}),
|
||||||
|
|
||||||
embeddings: ({text}) =>
|
embeddings: ({text}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => {
|
try: () => socket.flow(config.flowId).embeddings([...text]),
|
||||||
const vectors = await socket.flow(config.flowId).embeddings([...text])
|
catch: (cause) => EmbeddingsError.make({cause, message: toErrorMessage(cause)}),
|
||||||
return new EmbeddingsSuccess({vectors})
|
}).pipe(
|
||||||
},
|
Effect.map((vectors) => EmbeddingsSuccess.make({vectors})),
|
||||||
catch: (cause) => new EmbeddingsError({cause, message: toErrorMessage(cause)}),
|
),
|
||||||
}),
|
|
||||||
|
|
||||||
triples_query: ({s, p, o, limit, collection}) =>
|
triples_query: ({s, p, o, limit, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => {
|
try: () =>
|
||||||
const triples = await socket.flow(config.flowId).triplesQuery(
|
socket.flow(config.flowId).triplesQuery(
|
||||||
asIriTerm(s),
|
asIriTerm(s),
|
||||||
asIriTerm(p),
|
asIriTerm(p),
|
||||||
asIriTerm(o),
|
asIriTerm(o),
|
||||||
limit,
|
limit,
|
||||||
collection,
|
collection,
|
||||||
)
|
),
|
||||||
return new TriplesQuerySuccess({triples})
|
catch: (cause) => TriplesQueryError.make({cause, message: toErrorMessage(cause)}),
|
||||||
},
|
}).pipe(
|
||||||
catch: (cause) => new TriplesQueryError({cause, message: toErrorMessage(cause)}),
|
Effect.map((triples) => TriplesQuerySuccess.make({triples})),
|
||||||
}),
|
),
|
||||||
|
|
||||||
graph_embeddings_query: ({query, limit, collection}) =>
|
graph_embeddings_query: ({query, limit, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => {
|
try: () => socket.flow(config.flowId).embeddings([query]),
|
||||||
const vectors = await socket.flow(config.flowId).embeddings([query])
|
catch: (cause) => GraphEmbeddingsQueryError.make({cause, message: toErrorMessage(cause)}),
|
||||||
const entities = await socket.flow(config.flowId).graphEmbeddingsQuery(
|
}).pipe(
|
||||||
vectors[0] ?? [],
|
Effect.flatMap((vectors) =>
|
||||||
limit ?? 10,
|
Effect.tryPromise({
|
||||||
collection,
|
try: () => socket.flow(config.flowId).graphEmbeddingsQuery(
|
||||||
)
|
vectors[0] ?? [],
|
||||||
return new GraphEmbeddingsQuerySuccess({entities})
|
limit ?? 10,
|
||||||
},
|
collection,
|
||||||
catch: (cause) => new GraphEmbeddingsQueryError({cause, message: toErrorMessage(cause)}),
|
),
|
||||||
}),
|
catch: (cause) => GraphEmbeddingsQueryError.make({cause, message: toErrorMessage(cause)}),
|
||||||
|
})
|
||||||
|
),
|
||||||
|
Effect.map((entities) => GraphEmbeddingsQuerySuccess.make({entities})),
|
||||||
|
),
|
||||||
|
|
||||||
get_config_all: () =>
|
get_config_all: () =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.config().getConfigAll(),
|
try: () => socket.config().getConfigAll(),
|
||||||
catch: (cause) => new GetConfigAllError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetConfigAllError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new GetConfigAllError({cause, message: toErrorMessage(cause)}),
|
(cause) => GetConfigAllError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((config) => new GetConfigAllSuccess({config})),
|
Effect.map((config) => GetConfigAllSuccess.make({config})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1445,14 +1472,14 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
get_config: ({keys}) =>
|
get_config: ({keys}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.config().getConfig(keys.map(({type, key}) => ({type, key}))),
|
try: () => socket.config().getConfig(keys.map(({type, key}) => ({type, key}))),
|
||||||
catch: (cause) => new GetConfigError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetConfigError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new GetConfigError({cause, message: toErrorMessage(cause)}),
|
(cause) => GetConfigError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((config) => new GetConfigSuccess({config})),
|
Effect.map((config) => GetConfigSuccess.make({config})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1460,14 +1487,14 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
put_config: ({values}) =>
|
put_config: ({values}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.config().putConfig(values.map(({type, key, value}) => ({type, key, value}))),
|
try: () => socket.config().putConfig(values.map(({type, key, value}) => ({type, key, value}))),
|
||||||
catch: (cause) => new PutConfigError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => PutConfigError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new PutConfigError({cause, message: toErrorMessage(cause)}),
|
(cause) => PutConfigError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new PutConfigSuccess({response})),
|
Effect.map((response) => PutConfigSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1475,56 +1502,58 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
delete_config: ({type, key}) =>
|
delete_config: ({type, key}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.config().deleteConfig({type, key}),
|
try: () => socket.config().deleteConfig({type, key}),
|
||||||
catch: (cause) => new DeleteConfigError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => DeleteConfigError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new DeleteConfigError({cause, message: toErrorMessage(cause)}),
|
(cause) => DeleteConfigError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new DeleteConfigSuccess({response})),
|
Effect.map((response) => DeleteConfigSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
||||||
get_flows: () =>
|
get_flows: () =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => new GetFlowsSuccess({flow_ids: await socket.flows().getFlows()}),
|
try: () => socket.flows().getFlows(),
|
||||||
catch: (cause) => new GetFlowsError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetFlowsError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}),
|
}).pipe(
|
||||||
|
Effect.map((flow_ids) => GetFlowsSuccess.make({flow_ids})),
|
||||||
|
),
|
||||||
|
|
||||||
get_flow: ({flow_id}) =>
|
get_flow: ({flow_id}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.flows().getFlow(flow_id),
|
try: () => socket.flows().getFlow(flow_id),
|
||||||
catch: (cause) => new GetFlowError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetFlowError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new GetFlowError({cause, message: toErrorMessage(cause)}),
|
(cause) => GetFlowError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((flow) => new GetFlowSuccess({flow})),
|
Effect.map((flow) => GetFlowSuccess.make({flow})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
||||||
start_flow: ({flow_id, blueprint_name, description, parameters}) =>
|
start_flow: ({flow_id, blueprint_name, description, parameters}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () =>
|
try: () =>
|
||||||
socket.flows().startFlow(
|
socket.flows().startFlow(
|
||||||
flow_id,
|
flow_id,
|
||||||
blueprint_name,
|
blueprint_name,
|
||||||
description,
|
description,
|
||||||
parameters === undefined ? undefined : {...parameters},
|
parameters === undefined ? undefined : {...parameters},
|
||||||
),
|
),
|
||||||
catch: (cause) => new StartFlowError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => StartFlowError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new StartFlowError({cause, message: toErrorMessage(cause)}),
|
(cause) => StartFlowError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new StartFlowSuccess({response})),
|
Effect.map((response) => StartFlowSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1532,14 +1561,14 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
stop_flow: ({flow_id}) =>
|
stop_flow: ({flow_id}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.flows().stopFlow(flow_id),
|
try: () => socket.flows().stopFlow(flow_id),
|
||||||
catch: (cause) => new StopFlowError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => StopFlowError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new StopFlowError({cause, message: toErrorMessage(cause)}),
|
(cause) => StopFlowError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new StopFlowSuccess({response})),
|
Effect.map((response) => StopFlowSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1547,21 +1576,21 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
get_documents: () =>
|
get_documents: () =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.librarian().getDocuments(),
|
try: () => socket.librarian().getDocuments(),
|
||||||
catch: (cause) => new GetDocumentsError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetDocumentsError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonArrayOrFail(
|
decodeJsonArrayOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new GetDocumentsError({cause, message: toErrorMessage(cause)}),
|
(cause) => GetDocumentsError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((documents) => new GetDocumentsSuccess({documents})),
|
Effect.map((documents) => GetDocumentsSuccess.make({documents})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
||||||
load_document: ({document, mime_type, title, comments, tags, id}) =>
|
load_document: ({document, mime_type, title, comments, tags, id}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () =>
|
try: () =>
|
||||||
socket.librarian().loadDocument(
|
socket.librarian().loadDocument(
|
||||||
document,
|
document,
|
||||||
mime_type,
|
mime_type,
|
||||||
|
|
@ -1570,14 +1599,14 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
tags === undefined ? [] : [...tags],
|
tags === undefined ? [] : [...tags],
|
||||||
id,
|
id,
|
||||||
),
|
),
|
||||||
catch: (cause) => new LoadDocumentError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => LoadDocumentError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new LoadDocumentError({cause, message: toErrorMessage(cause)}),
|
(cause) => LoadDocumentError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new LoadDocumentSuccess({response})),
|
Effect.map((response) => LoadDocumentSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1585,56 +1614,60 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
remove_document: ({id, collection}) =>
|
remove_document: ({id, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.librarian().removeDocument(id, collection),
|
try: () => socket.librarian().removeDocument(id, collection),
|
||||||
catch: (cause) => new RemoveDocumentError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => RemoveDocumentError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new RemoveDocumentError({cause, message: toErrorMessage(cause)}),
|
(cause) => RemoveDocumentError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new RemoveDocumentSuccess({response})),
|
Effect.map((response) => RemoveDocumentSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
||||||
get_prompts: () =>
|
get_prompts: () =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => new GetPromptsSuccess({prompts: await socket.config().getPrompts()}),
|
try: () => socket.config().getPrompts(),
|
||||||
catch: (cause) => new GetPromptsError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetPromptsError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}),
|
}).pipe(
|
||||||
|
Effect.map((prompts) => GetPromptsSuccess.make({prompts})),
|
||||||
|
),
|
||||||
|
|
||||||
get_prompt: ({id}) =>
|
get_prompt: ({id}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.config().getPrompt(id),
|
try: () => socket.config().getPrompt(id),
|
||||||
catch: (cause) => new GetPromptError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetPromptError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new GetPromptError({cause, message: toErrorMessage(cause)}),
|
(cause) => GetPromptError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((prompt) => new GetPromptSuccess({prompt})),
|
Effect.map((prompt) => GetPromptSuccess.make({prompt})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
||||||
get_knowledge_cores: () =>
|
get_knowledge_cores: () =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: async () => new GetKnowledgeCoresSuccess({ids: await socket.knowledge().getKnowledgeCores()}),
|
try: () => socket.knowledge().getKnowledgeCores(),
|
||||||
catch: (cause) => new GetKnowledgeCoresError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => GetKnowledgeCoresError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}),
|
}).pipe(
|
||||||
|
Effect.map((ids) => GetKnowledgeCoresSuccess.make({ids})),
|
||||||
|
),
|
||||||
|
|
||||||
delete_kg_core: ({id, collection}) =>
|
delete_kg_core: ({id, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.knowledge().deleteKgCore(id, collection),
|
try: () => socket.knowledge().deleteKgCore(id, collection),
|
||||||
catch: (cause) => new DeleteKgCoreError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => DeleteKgCoreError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new DeleteKgCoreError({cause, message: toErrorMessage(cause)}),
|
(cause) => DeleteKgCoreError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new DeleteKgCoreSuccess({response})),
|
Effect.map((response) => DeleteKgCoreSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1642,14 +1675,14 @@ export const TrustGraphMcpToolkitLive = TrustGraphMcpToolkit.toLayer(
|
||||||
load_kg_core: ({id, flow, collection}) =>
|
load_kg_core: ({id, flow, collection}) =>
|
||||||
Effect.tryPromise({
|
Effect.tryPromise({
|
||||||
try: () => socket.knowledge().loadKgCore(id, flow, collection),
|
try: () => socket.knowledge().loadKgCore(id, flow, collection),
|
||||||
catch: (cause) => new LoadKgCoreError({cause, message: toErrorMessage(cause)}),
|
catch: (cause) => LoadKgCoreError.make({cause, message: toErrorMessage(cause)}),
|
||||||
}).pipe(
|
}).pipe(
|
||||||
Effect.flatMap((value) =>
|
Effect.flatMap((value) =>
|
||||||
decodeJsonOrFail(
|
decodeJsonOrFail(
|
||||||
value,
|
value,
|
||||||
(cause) => new LoadKgCoreError({cause, message: toErrorMessage(cause)}),
|
(cause) => LoadKgCoreError.make({cause, message: toErrorMessage(cause)}),
|
||||||
).pipe(
|
).pipe(
|
||||||
Effect.map((response) => new LoadKgCoreSuccess({response})),
|
Effect.map((response) => LoadKgCoreSuccess.make({response})),
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
|
@ -1689,13 +1722,12 @@ export const TrustGraphMcpHttpApiRoutes = HttpApiBuilder.layer(
|
||||||
Layer.provide(TrustGraphMcpHttpApiHandlers),
|
Layer.provide(TrustGraphMcpHttpApiHandlers),
|
||||||
)
|
)
|
||||||
|
|
||||||
export const makeTrustGraphMcpHttpLayer = (
|
const makeTrustGraphMcpHttpLayerFromConfig = (
|
||||||
options: TrustGraphMcpOptions = {},
|
config: TrustGraphMcpConfigShape,
|
||||||
) => {
|
) => {
|
||||||
const config = resolveTrustGraphMcpConfig(options)
|
|
||||||
const tools = McpServer.toolkit(TrustGraphMcpToolkit).pipe(
|
const tools = McpServer.toolkit(TrustGraphMcpToolkit).pipe(
|
||||||
Layer.provide(TrustGraphMcpToolkitLive),
|
Layer.provide(TrustGraphMcpToolkitLive),
|
||||||
Layer.provide(makeOpenAiProviderLayer(config)),
|
Layer.provide(makeOpenAiProviderLayerFromConfig(config)),
|
||||||
)
|
)
|
||||||
|
|
||||||
return Layer.mergeAll(
|
return Layer.mergeAll(
|
||||||
|
|
@ -1708,18 +1740,31 @@ export const makeTrustGraphMcpHttpLayer = (
|
||||||
path: config.mcpPath,
|
path: config.mcpPath,
|
||||||
})),
|
})),
|
||||||
Layer.provide(TrustGraphSocket.layer),
|
Layer.provide(TrustGraphSocket.layer),
|
||||||
Layer.provide(TrustGraphMcpConfig.layer(config)),
|
Layer.provide(Layer.succeed(TrustGraphMcpConfig, TrustGraphMcpConfig.of(config))),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
export const makeTrustGraphMcpHttpServerLayer = (
|
export const makeTrustGraphMcpHttpServerLayer = (
|
||||||
options: TrustGraphMcpOptions = {},
|
options: TrustGraphMcpOptions = {},
|
||||||
) => {
|
) =>
|
||||||
const config = resolveTrustGraphMcpConfig(options)
|
Layer.unwrap(
|
||||||
return HttpRouter.serve(makeTrustGraphMcpHttpLayer(config)).pipe(
|
loadTrustGraphMcpConfig(options).pipe(
|
||||||
Layer.provide(BunHttpServer.layer({port: config.port})),
|
Effect.map((config) =>
|
||||||
|
HttpRouter.serve(makeTrustGraphMcpHttpLayerFromConfig(config)).pipe(
|
||||||
|
Layer.provide(BunHttpServer.layer({port: config.port})),
|
||||||
|
)
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
export const makeTrustGraphMcpHttpLayer = (
|
||||||
|
options: TrustGraphMcpOptions = {},
|
||||||
|
) =>
|
||||||
|
Layer.unwrap(
|
||||||
|
loadTrustGraphMcpConfig(options).pipe(
|
||||||
|
Effect.map(makeTrustGraphMcpHttpLayerFromConfig),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
}
|
|
||||||
|
|
||||||
export const runHttp = (options: TrustGraphMcpOptions = {}): void => {
|
export const runHttp = (options: TrustGraphMcpOptions = {}): void => {
|
||||||
Layer.launch(makeTrustGraphMcpHttpServerLayer(options)).pipe(BunRuntime.runMain)
|
Layer.launch(makeTrustGraphMcpHttpServerLayer(options)).pipe(BunRuntime.runMain)
|
||||||
|
|
|
||||||
|
|
@ -1,16 +1,77 @@
|
||||||
/**
|
/**
|
||||||
* TrustGraph MCP server.
|
* TrustGraph MCP stdio compatibility server.
|
||||||
*
|
*
|
||||||
* Exposes TrustGraph capabilities as MCP tools for AI assistants.
|
* This keeps the original @modelcontextprotocol/sdk entry points available,
|
||||||
* Uses the vendored @trustgraph/client for all gateway communication.
|
* while moving gateway calls, callback bridging, JSON encoding, and config
|
||||||
*
|
* reads behind Effect values.
|
||||||
* Python reference: trustgraph-mcp/trustgraph/mcp_server/mcp.py
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
import {McpServer} from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
import {StdioServerTransport} from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||||
import { z } from "zod";
|
import {NodeRuntime} from "@effect/platform-node";
|
||||||
import { createTrustGraphSocket, type BaseApi, type Term } from "@trustgraph/client";
|
import {createTrustGraphSocket, type BaseApi, type Term} from "@trustgraph/client";
|
||||||
|
import {Effect, Layer, ManagedRuntime} from "effect";
|
||||||
|
import * as Predicate from "effect/Predicate";
|
||||||
|
import * as S from "effect/Schema";
|
||||||
|
import {z} from "zod";
|
||||||
|
import {loadTrustGraphMcpConfig} from "./server-effect.js";
|
||||||
|
|
||||||
|
interface ToolTextContent {
|
||||||
|
readonly type: "text"
|
||||||
|
readonly text: string
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ToolTextResult extends Record<string, unknown> {
|
||||||
|
readonly content: Array<ToolTextContent>
|
||||||
|
}
|
||||||
|
|
||||||
|
class StdioMcpError extends S.TaggedErrorClass<StdioMcpError>()(
|
||||||
|
"StdioMcpError",
|
||||||
|
{
|
||||||
|
cause: S.DefectWithStack,
|
||||||
|
message: S.String,
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
||||||
|
const encodeJsonText = S.encodeUnknownEffect(S.UnknownFromJsonString);
|
||||||
|
|
||||||
|
const toErrorMessage = (cause: unknown): string => {
|
||||||
|
if (Predicate.isError(cause) && cause.message.length > 0) {
|
||||||
|
return cause.message;
|
||||||
|
}
|
||||||
|
if (Predicate.isString(cause) && cause.length > 0) {
|
||||||
|
return cause;
|
||||||
|
}
|
||||||
|
if (Predicate.isObject(cause) && Predicate.hasProperty(cause, "message") && Predicate.isString(cause.message) && cause.message.length > 0) {
|
||||||
|
return cause.message;
|
||||||
|
}
|
||||||
|
return "TrustGraph MCP stdio operation failed";
|
||||||
|
};
|
||||||
|
|
||||||
|
const stdioMcpError = (cause: unknown) =>
|
||||||
|
StdioMcpError.make({cause, message: toErrorMessage(cause)});
|
||||||
|
|
||||||
|
const textResult = (text: string): ToolTextResult => ({
|
||||||
|
content: [{type: "text", text}],
|
||||||
|
});
|
||||||
|
|
||||||
|
const gatewayRequest = <A>(request: () => Promise<A>) =>
|
||||||
|
Effect.tryPromise({
|
||||||
|
try: request,
|
||||||
|
catch: stdioMcpError,
|
||||||
|
});
|
||||||
|
|
||||||
|
const jsonText = (value: unknown) =>
|
||||||
|
encodeJsonText(value).pipe(
|
||||||
|
Effect.mapError(stdioMcpError),
|
||||||
|
);
|
||||||
|
|
||||||
|
const runTextTool = (effect: Effect.Effect<string, StdioMcpError>) =>
|
||||||
|
Effect.runPromise(effect.pipe(Effect.map(textResult)));
|
||||||
|
|
||||||
|
const runJsonTool = (effect: Effect.Effect<unknown, StdioMcpError>) =>
|
||||||
|
Effect.runPromise(effect.pipe(Effect.flatMap(jsonText), Effect.map(textResult)));
|
||||||
|
|
||||||
export function createMcpServer(config: {
|
export function createMcpServer(config: {
|
||||||
gatewayUrl: string;
|
gatewayUrl: string;
|
||||||
|
|
@ -34,7 +95,6 @@ export function createMcpServer(config: {
|
||||||
|
|
||||||
// ===================== Flow-scoped tools =====================
|
// ===================== Flow-scoped tools =====================
|
||||||
|
|
||||||
// --- Text Completion ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"text_completion",
|
"text_completion",
|
||||||
"Run a text completion using the configured LLM",
|
"Run a text completion using the configured LLM",
|
||||||
|
|
@ -42,14 +102,10 @@ export function createMcpServer(config: {
|
||||||
system: z.string().describe("System prompt"),
|
system: z.string().describe("System prompt"),
|
||||||
prompt: z.string().describe("User prompt"),
|
prompt: z.string().describe("User prompt"),
|
||||||
},
|
},
|
||||||
async ({ system, prompt }) => {
|
({system, prompt}) =>
|
||||||
const flow = socket.flow(flowId);
|
runTextTool(gatewayRequest(() => socket.flow(flowId).textCompletion(system, prompt))),
|
||||||
const response = await flow.textCompletion(system, prompt);
|
|
||||||
return { content: [{ type: "text" as const, text: response }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// --- Graph RAG ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"graph_rag",
|
"graph_rag",
|
||||||
"Query the knowledge graph using RAG",
|
"Query the knowledge graph using RAG",
|
||||||
|
|
@ -59,21 +115,21 @@ export function createMcpServer(config: {
|
||||||
triple_limit: z.number().optional().describe("Max triples per entity"),
|
triple_limit: z.number().optional().describe("Max triples per entity"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ query, entity_limit, triple_limit, collection }) => {
|
({query, entity_limit, triple_limit, collection}) =>
|
||||||
const flow = socket.flow(flowId);
|
runTextTool(
|
||||||
const response = await flow.graphRag(
|
gatewayRequest(() =>
|
||||||
query,
|
socket.flow(flowId).graphRag(
|
||||||
{
|
query,
|
||||||
...(entity_limit !== undefined ? { entityLimit: entity_limit } : {}),
|
{
|
||||||
...(triple_limit !== undefined ? { tripleLimit: triple_limit } : {}),
|
...(entity_limit !== undefined ? {entityLimit: entity_limit} : {}),
|
||||||
},
|
...(triple_limit !== undefined ? {tripleLimit: triple_limit} : {}),
|
||||||
collection,
|
},
|
||||||
);
|
collection,
|
||||||
return { content: [{ type: "text" as const, text: response }] };
|
)
|
||||||
},
|
),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
|
|
||||||
// --- Document RAG ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"document_rag",
|
"document_rag",
|
||||||
"Query documents using RAG",
|
"Query documents using RAG",
|
||||||
|
|
@ -82,56 +138,45 @@ export function createMcpServer(config: {
|
||||||
doc_limit: z.number().optional().describe("Max documents to retrieve"),
|
doc_limit: z.number().optional().describe("Max documents to retrieve"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ query, doc_limit, collection }) => {
|
({query, doc_limit, collection}) =>
|
||||||
const flow = socket.flow(flowId);
|
runTextTool(gatewayRequest(() => socket.flow(flowId).documentRag(query, doc_limit, collection))),
|
||||||
const response = await flow.documentRag(query, doc_limit, collection);
|
|
||||||
return { content: [{ type: "text" as const, text: response }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// --- Agent ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"agent",
|
"agent",
|
||||||
"Ask the TrustGraph agent a question",
|
"Ask the TrustGraph agent a question",
|
||||||
{
|
{
|
||||||
question: z.string().describe("Question for the agent"),
|
question: z.string().describe("Question for the agent"),
|
||||||
},
|
},
|
||||||
async ({ question }) => {
|
({question}) =>
|
||||||
const flow = socket.flow(flowId);
|
runTextTool(
|
||||||
let fullAnswer = "";
|
Effect.callback<string, StdioMcpError>((resume) => {
|
||||||
|
let fullAnswer = "";
|
||||||
await new Promise<void>((resolve, reject) => {
|
socket.flow(flowId).agent(
|
||||||
flow.agent(
|
question,
|
||||||
question,
|
() => {},
|
||||||
() => {}, // think — ignore for MCP
|
() => {},
|
||||||
() => {}, // observe — ignore for MCP
|
(chunk, complete) => {
|
||||||
(chunk, complete) => {
|
fullAnswer += chunk;
|
||||||
fullAnswer += chunk;
|
if (complete) {
|
||||||
if (complete) resolve();
|
resume(Effect.succeed(fullAnswer));
|
||||||
},
|
}
|
||||||
(err) => reject(new Error(err)),
|
},
|
||||||
);
|
(cause) => resume(Effect.fail(stdioMcpError(cause))),
|
||||||
});
|
);
|
||||||
|
}),
|
||||||
return { content: [{ type: "text" as const, text: fullAnswer }] };
|
),
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// --- Embeddings ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"embeddings",
|
"embeddings",
|
||||||
"Generate text embeddings",
|
"Generate text embeddings",
|
||||||
{
|
{
|
||||||
text: z.array(z.string()).describe("Texts to embed"),
|
text: z.array(z.string()).describe("Texts to embed"),
|
||||||
},
|
},
|
||||||
async ({ text }) => {
|
({text}) => runJsonTool(gatewayRequest(() => socket.flow(flowId).embeddings(text))),
|
||||||
const flow = socket.flow(flowId);
|
|
||||||
const vectors = await flow.embeddings(text);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(vectors) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// --- Triples Query ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"triples_query",
|
"triples_query",
|
||||||
"Query the knowledge graph for triples matching a pattern",
|
"Query the knowledge graph for triples matching a pattern",
|
||||||
|
|
@ -142,17 +187,16 @@ export function createMcpServer(config: {
|
||||||
limit: z.number().optional().describe("Max results"),
|
limit: z.number().optional().describe("Max results"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ s, p, o, limit, collection }) => {
|
({s, p, o, limit, collection}) => {
|
||||||
const flow = socket.flow(flowId);
|
const sTerm: Term | undefined = s !== undefined && s.length > 0 ? {t: "i", i: s} : undefined;
|
||||||
const sTerm: Term | undefined = s !== undefined && s.length > 0 ? { t: "i", i: s } : undefined;
|
const pTerm: Term | undefined = p !== undefined && p.length > 0 ? {t: "i", i: p} : undefined;
|
||||||
const pTerm: Term | undefined = p !== undefined && p.length > 0 ? { t: "i", i: p } : undefined;
|
const oTerm: Term | undefined = o !== undefined && o.length > 0 ? {t: "i", i: o} : undefined;
|
||||||
const oTerm: Term | undefined = o !== undefined && o.length > 0 ? { t: "i", i: o } : undefined;
|
return runJsonTool(
|
||||||
const triples = await flow.triplesQuery(sTerm, pTerm, oTerm, limit, collection);
|
gatewayRequest(() => socket.flow(flowId).triplesQuery(sTerm, pTerm, oTerm, limit, collection)),
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(triples, null, 2) }] };
|
);
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
// --- Graph Embeddings Query ---
|
|
||||||
server.tool(
|
server.tool(
|
||||||
"graph_embeddings_query",
|
"graph_embeddings_query",
|
||||||
"Find entities similar to a text query using vector embeddings",
|
"Find entities similar to a text query using vector embeddings",
|
||||||
|
|
@ -161,17 +205,20 @@ export function createMcpServer(config: {
|
||||||
limit: z.number().optional().describe("Max results"),
|
limit: z.number().optional().describe("Max results"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ query, limit, collection }) => {
|
({query, limit, collection}) =>
|
||||||
const flow = socket.flow(flowId);
|
runJsonTool(
|
||||||
// First embed the query, then search
|
gatewayRequest(() => socket.flow(flowId).embeddings([query])).pipe(
|
||||||
const vectors = await flow.embeddings([query]);
|
Effect.flatMap((vectors) =>
|
||||||
const entities = await flow.graphEmbeddingsQuery(
|
gatewayRequest(() =>
|
||||||
vectors[0],
|
socket.flow(flowId).graphEmbeddingsQuery(
|
||||||
limit ?? 10,
|
vectors[0] ?? [],
|
||||||
collection,
|
limit ?? 10,
|
||||||
);
|
collection,
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(entities, null, 2) }] };
|
)
|
||||||
},
|
)
|
||||||
|
),
|
||||||
|
),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
|
|
||||||
// ===================== Config tools =====================
|
// ===================== Config tools =====================
|
||||||
|
|
@ -180,11 +227,7 @@ export function createMcpServer(config: {
|
||||||
"get_config_all",
|
"get_config_all",
|
||||||
"Get all configuration values",
|
"Get all configuration values",
|
||||||
{},
|
{},
|
||||||
async () => {
|
() => runJsonTool(gatewayRequest(() => socket.config().getConfigAll())),
|
||||||
const cfg = socket.config();
|
|
||||||
const resp = await cfg.getConfigAll();
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -198,11 +241,7 @@ export function createMcpServer(config: {
|
||||||
}),
|
}),
|
||||||
).describe("Config keys to retrieve"),
|
).describe("Config keys to retrieve"),
|
||||||
},
|
},
|
||||||
async ({ keys }) => {
|
({keys}) => runJsonTool(gatewayRequest(() => socket.config().getConfig(keys))),
|
||||||
const cfg = socket.config();
|
|
||||||
const resp = await cfg.getConfig(keys);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -217,11 +256,7 @@ export function createMcpServer(config: {
|
||||||
}),
|
}),
|
||||||
).describe("Key-value entries to set"),
|
).describe("Key-value entries to set"),
|
||||||
},
|
},
|
||||||
async ({ values }) => {
|
({values}) => runJsonTool(gatewayRequest(() => socket.config().putConfig(values))),
|
||||||
const cfg = socket.config();
|
|
||||||
const resp = await cfg.putConfig(values);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -231,11 +266,7 @@ export function createMcpServer(config: {
|
||||||
type: z.string().describe("Config type"),
|
type: z.string().describe("Config type"),
|
||||||
key: z.string().describe("Config key"),
|
key: z.string().describe("Config key"),
|
||||||
},
|
},
|
||||||
async ({ type, key }) => {
|
({type, key}) => runJsonTool(gatewayRequest(() => socket.config().deleteConfig({type, key}))),
|
||||||
const cfg = socket.config();
|
|
||||||
const resp = await cfg.deleteConfig({ type, key });
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// ===================== Flow management tools =====================
|
// ===================== Flow management tools =====================
|
||||||
|
|
@ -244,11 +275,7 @@ export function createMcpServer(config: {
|
||||||
"get_flows",
|
"get_flows",
|
||||||
"List all available flows",
|
"List all available flows",
|
||||||
{},
|
{},
|
||||||
async () => {
|
() => runJsonTool(gatewayRequest(() => socket.flows().getFlows())),
|
||||||
const flows = socket.flows();
|
|
||||||
const ids = await flows.getFlows();
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(ids, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -257,11 +284,7 @@ export function createMcpServer(config: {
|
||||||
{
|
{
|
||||||
flow_id: z.string().describe("Flow ID to retrieve"),
|
flow_id: z.string().describe("Flow ID to retrieve"),
|
||||||
},
|
},
|
||||||
async ({ flow_id }) => {
|
({flow_id}) => runJsonTool(gatewayRequest(() => socket.flows().getFlow(flow_id))),
|
||||||
const flows = socket.flows();
|
|
||||||
const def = await flows.getFlow(flow_id);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(def, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -273,11 +296,10 @@ export function createMcpServer(config: {
|
||||||
description: z.string().describe("Flow description"),
|
description: z.string().describe("Flow description"),
|
||||||
parameters: z.record(z.unknown()).optional().describe("Optional flow parameters"),
|
parameters: z.record(z.unknown()).optional().describe("Optional flow parameters"),
|
||||||
},
|
},
|
||||||
async ({ flow_id, blueprint_name, description, parameters }) => {
|
({flow_id, blueprint_name, description, parameters}) =>
|
||||||
const flows = socket.flows();
|
runJsonTool(
|
||||||
const resp = await flows.startFlow(flow_id, blueprint_name, description, parameters);
|
gatewayRequest(() => socket.flows().startFlow(flow_id, blueprint_name, description, parameters)),
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp, null, 2) }] };
|
),
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -286,11 +308,7 @@ export function createMcpServer(config: {
|
||||||
{
|
{
|
||||||
flow_id: z.string().describe("Flow ID to stop"),
|
flow_id: z.string().describe("Flow ID to stop"),
|
||||||
},
|
},
|
||||||
async ({ flow_id }) => {
|
({flow_id}) => runJsonTool(gatewayRequest(() => socket.flows().stopFlow(flow_id))),
|
||||||
const flows = socket.flows();
|
|
||||||
const resp = await flows.stopFlow(flow_id);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// ===================== Library (document) tools =====================
|
// ===================== Library (document) tools =====================
|
||||||
|
|
@ -299,11 +317,7 @@ export function createMcpServer(config: {
|
||||||
"get_documents",
|
"get_documents",
|
||||||
"List all documents in the library",
|
"List all documents in the library",
|
||||||
{},
|
{},
|
||||||
async () => {
|
() => runJsonTool(gatewayRequest(() => socket.librarian().getDocuments())),
|
||||||
const lib = socket.librarian();
|
|
||||||
const docs = await lib.getDocuments();
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(docs, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -317,18 +331,19 @@ export function createMcpServer(config: {
|
||||||
tags: z.array(z.string()).optional().describe("Document tags"),
|
tags: z.array(z.string()).optional().describe("Document tags"),
|
||||||
id: z.string().optional().describe("Optional document ID"),
|
id: z.string().optional().describe("Optional document ID"),
|
||||||
},
|
},
|
||||||
async ({ document, mime_type, title, comments, tags, id }) => {
|
({document, mime_type, title, comments, tags, id}) =>
|
||||||
const lib = socket.librarian();
|
runJsonTool(
|
||||||
const resp = await lib.loadDocument(
|
gatewayRequest(() =>
|
||||||
document,
|
socket.librarian().loadDocument(
|
||||||
mime_type,
|
document,
|
||||||
title,
|
mime_type,
|
||||||
comments ?? "",
|
title,
|
||||||
tags ?? [],
|
comments ?? "",
|
||||||
id,
|
tags ?? [],
|
||||||
);
|
id,
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp, null, 2) }] };
|
)
|
||||||
},
|
),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -338,11 +353,7 @@ export function createMcpServer(config: {
|
||||||
id: z.string().describe("Document ID to remove"),
|
id: z.string().describe("Document ID to remove"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ id, collection }) => {
|
({id, collection}) => runJsonTool(gatewayRequest(() => socket.librarian().removeDocument(id, collection))),
|
||||||
const lib = socket.librarian();
|
|
||||||
const resp = await lib.removeDocument(id, collection);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// ===================== Prompt tools =====================
|
// ===================== Prompt tools =====================
|
||||||
|
|
@ -351,11 +362,7 @@ export function createMcpServer(config: {
|
||||||
"get_prompts",
|
"get_prompts",
|
||||||
"List available prompt templates",
|
"List available prompt templates",
|
||||||
{},
|
{},
|
||||||
async () => {
|
() => runJsonTool(gatewayRequest(() => socket.config().getPrompts())),
|
||||||
const cfg = socket.config();
|
|
||||||
const prompts = await cfg.getPrompts();
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(prompts, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -364,11 +371,7 @@ export function createMcpServer(config: {
|
||||||
{
|
{
|
||||||
id: z.string().describe("Prompt template ID"),
|
id: z.string().describe("Prompt template ID"),
|
||||||
},
|
},
|
||||||
async ({ id }) => {
|
({id}) => runJsonTool(gatewayRequest(() => socket.config().getPrompt(id))),
|
||||||
const cfg = socket.config();
|
|
||||||
const prompt = await cfg.getPrompt(id);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(prompt, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
// ===================== Knowledge core tools =====================
|
// ===================== Knowledge core tools =====================
|
||||||
|
|
@ -377,11 +380,7 @@ export function createMcpServer(config: {
|
||||||
"get_knowledge_cores",
|
"get_knowledge_cores",
|
||||||
"List available knowledge graph cores",
|
"List available knowledge graph cores",
|
||||||
{},
|
{},
|
||||||
async () => {
|
() => runJsonTool(gatewayRequest(() => socket.knowledge().getKnowledgeCores())),
|
||||||
const knowledge = socket.knowledge();
|
|
||||||
const cores = await knowledge.getKnowledgeCores();
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(cores, null, 2) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -391,11 +390,7 @@ export function createMcpServer(config: {
|
||||||
id: z.string().describe("Knowledge core ID"),
|
id: z.string().describe("Knowledge core ID"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ id, collection }) => {
|
({id, collection}) => runJsonTool(gatewayRequest(() => socket.knowledge().deleteKgCore(id, collection))),
|
||||||
const knowledge = socket.knowledge();
|
|
||||||
const resp = await knowledge.deleteKgCore(id, collection);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
server.tool(
|
server.tool(
|
||||||
|
|
@ -406,31 +401,42 @@ export function createMcpServer(config: {
|
||||||
flow: z.string().describe("Flow to use for loading"),
|
flow: z.string().describe("Flow to use for loading"),
|
||||||
collection: z.string().optional().describe("Collection name"),
|
collection: z.string().optional().describe("Collection name"),
|
||||||
},
|
},
|
||||||
async ({ id, flow, collection }) => {
|
({id, flow, collection}) => runJsonTool(gatewayRequest(() => socket.knowledge().loadKgCore(id, flow, collection))),
|
||||||
const knowledge = socket.knowledge();
|
|
||||||
const resp = await knowledge.loadKgCore(id, flow, collection);
|
|
||||||
return { content: [{ type: "text" as const, text: JSON.stringify(resp) }] };
|
|
||||||
},
|
|
||||||
);
|
);
|
||||||
|
|
||||||
return { server, socket };
|
return {server, socket};
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function run(): Promise<void> {
|
export const runProgram = Effect.gen(function*() {
|
||||||
const { server, socket } = createMcpServer({
|
const config = yield* loadTrustGraphMcpConfig();
|
||||||
gatewayUrl: process.env.GATEWAY_URL ?? "ws://localhost:8088/api/v1/rpc",
|
const serverConfig = {
|
||||||
user: process.env.USER_ID ?? "mcp",
|
gatewayUrl: config.gatewayUrl,
|
||||||
flowId: process.env.FLOW_ID ?? "default",
|
user: config.user,
|
||||||
...(process.env.GATEWAY_SECRET !== undefined
|
flowId: config.flowId,
|
||||||
? { token: process.env.GATEWAY_SECRET }
|
...(config.token === undefined ? {} : {token: config.token}),
|
||||||
: {}),
|
};
|
||||||
});
|
const {server, socket} = createMcpServer(serverConfig);
|
||||||
|
|
||||||
const transport = new StdioServerTransport();
|
const transport = new StdioServerTransport();
|
||||||
await server.connect(transport);
|
|
||||||
|
|
||||||
process.on("SIGINT", () => {
|
yield* Effect.tryPromise({
|
||||||
socket.close();
|
try: () => server.connect(transport),
|
||||||
process.exit(0);
|
catch: stdioMcpError,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
yield* Effect.sync(() => {
|
||||||
|
process.on("SIGINT", () => {
|
||||||
|
socket.close();
|
||||||
|
process.exit(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const stdioRuntime = ManagedRuntime.make(Layer.empty);
|
||||||
|
|
||||||
|
export function run(): Promise<void> {
|
||||||
|
return stdioRuntime.runPromise(runProgram);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function runMain(): void {
|
||||||
|
NodeRuntime.runMain(runProgram);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue