apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-07-01 09:29:38 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

refactor(ts): make port effect native

Browse source
This commit is contained in:
elpresidank 2026-06-06 10:33:10 -05:00
parent 2868ced2d3
commit b6759e75df
113 changed files with 4140 additions and 4554 deletions
Download patch file Download diff file Expand all files Collapse all files

127
ts/EFFECT_NATIVE_REWRITE_PLAN.md Normal file
View file

@ -0,0 +1,127 @@
# TrustGraph TS Effect-Native Rewrite Plan
## Summary
Bring the TypeScript port to a genuinely Effect-native shape while keeping existing TrustGraph capabilities working. The current native MCP rewrite should be preserved. The remaining work is to remove Promise-first TrustGraph APIs and replace manual host/framework plumbing with Effect v4-native backend, messaging, processor, HTTP/RPC, and CLI layers.
The implementation should proceed in green checkpoints rather than one broad sweep. Each stage should leave the repo closer to Effect-native and should run the smallest relevant gate before moving on.
## Non-Negotiable Architecture Decisions
- TrustGraph-authored production APIs should be Effect-first, not Promise-first.
- Remove exported Promise APIs from backend, messaging, processor, flow service, gateway, CLI, and service runner surfaces.
- Do not keep dual Promise and Effect methods on the same TrustGraph objects.
- Do not rely on Fastify, `@fastify/websocket`, Commander, or production `Effect.runPromise` / `ManagedRuntime` compatibility bridges.
- External JavaScript SDKs that are inherently Promise-based may be wrapped with `Effect.tryPromise`, but that Promise shape must stay behind Effect APIs.
- Gateway HTTP and WebSocket behavior should move to Effect v4 native modules: `effect/unstable/httpapi`, `effect/unstable/http`, `effect/unstable/rpc`, `effect/unstable/socket`, and platform Bun/Node HTTP/socket layers.
- CLI behavior should move to `effect/unstable/cli`.
- The MCP package should stay on the native `effect/unstable/ai/McpServer` implementation and must not reintroduce server-side `@modelcontextprotocol/sdk` or `zod`.
## Current Grounding
- `bun run check:tsgo` is green.
- `bun run build` is green.
- `bun run --cwd packages/mcp test` is green.
- `bun run test` currently fails only in `@trustgraph/base`, in `packages/base/src/__tests__/nats-backend.test.ts`.
- The NATS failures are a symptom of the backend still sitting between old Promise-shaped abstractions and newer JetStream calls. The rewrite should fix the contract, not only update the mock.
- The Effect v4 source of truth for APIs and examples is available at `/home/elpresidank/YeeBois/projects/beep-effect/.repos/effect-v4`.
## Stage 1: Backend And NATS
- Redefine `Message`, `BackendProducer`, `BackendConsumer`, and `PubSubBackend` around `Effect.Effect` return values.
- Make NATS connection, JetStream manager/client, stream initialization, and durable consumer handles scoped resources.
- Replace deprecated receive plumbing with modern JetStream consumer APIs.
- Preserve behavior for stream creation, durable consumer creation, headers, JSON/schema encode/decode, receive timeout returning `null`, ack/nak, close/drain, and tagged `PubSubError` mapping.
- Update NATS tests and in-memory fake backends to the new Effect-native interface.
Verification:
- `bun run --cwd packages/base test src/__tests__/nats-backend.test.ts`
- `bun run --cwd packages/base test`
- `bun run check:tsgo`
## Stage 2: Messaging And Processor Runtime
- Remove Promise facades from `makeProducer`, `makeConsumer`, `makeRequestResponse`, processor `start` / `stop` / `run`, and flow compatibility wrappers.
- Expose Effect values, scoped constructors, Context services, Layers, and finalizers as the only TrustGraph API shape.
- Convert message handlers, config handlers, shutdown callbacks, producer send/flush/close, consumer receive/ack/nak/close, and request/response operations to Effect functions.
- Remove internal `ManagedRuntime.make(Layer.empty)` compatibility runtimes from production source.
Verification:
- `bun run --cwd packages/base test`
- `bun run check:tsgo`
## Stage 3: Flow Service Call Sites
- Convert flow service state methods that currently call `Effect.runPromise` into Effect-valued service methods.
- Keep public feature behavior unchanged for config, flow manager, librarian, knowledge cores, retrieval, embeddings, triples, agent, MCP tool, and text-completion services.
- Convert test fakes to the same Effect-native backend/messaging contracts.
Verification:
- `bun run --cwd packages/flow test`
- `bun run test`
- `bun run check:tsgo`
## Stage 4: Gateway HTTP And RPC
- Remove Fastify and `@fastify/websocket` from `packages/flow`.
- Rebuild the gateway with Effect v4 `HttpApi` groups/endpoints and native RPC/socket layers.
- Preserve existing behavior:
- `POST /api/v1/workbench/dispatch`
- `POST /api/v1/:kind`
- `POST /api/v1/flow/:flow/service/:kind`
- `POST /api/v1/flow/:flow/load`
- `GET /api/v1/rpc`
- `GET /api/v1/metrics`
- bearer auth behavior and RPC token behavior
- existing response/error shapes expected by clients and workbench
- Use native Effect HTTP test utilities or route-level protocol tests instead of Fastify injection.
Verification:
- `bun run --cwd packages/flow test`
- `bun run --cwd packages/client test`
- `bun run test`
- `bun run build`
## Stage 5: CLI And Runner Scripts
- Remove Commander from `packages/cli`.
- Rebuild CLI commands with `effect/unstable/cli` and Effect-native socket/API clients.
- Convert service runner scripts to launch Effect programs directly with platform runtime `runMain` style execution.
- Remove `run(): Promise<void>` exports from flow services; export Effect programs/layers and Effect-native `runMain` helpers instead.
- Leave script-only demo/seed tools as follow-up only if they are outside production package source, but do not let production packages depend on Promise facades.
Verification:
- `bun run --cwd packages/cli test`
- `bun run check:tsgo`
- `bun run build`
- `bun run test`
## Stage 6: Cleanup And Acceptance
- Remove obsolete dependencies from package manifests and lockfile:
- `fastify`
- `@fastify/websocket`
- `commander`
- any legacy dependency made unnecessary by the rewrite
- Preserve unrelated dirty work. Do not revert user changes.
- Use parallel agents for bounded audits or disjoint rewrite slices when useful.
- Use Graphiti memory if available; if unavailable, continue safely and report it skipped.
Final verification:
- `bun run check:tsgo`
- `bun run build`
- `bun run lint`
- `bun run test`
- `bun run --cwd packages/mcp test`
- `bun run workbench:qa` after installing the matching Playwright browser if needed
- `git diff --check`
- `rg -n "fastify|@fastify/websocket|commander" packages package.json bun.lock` has no production dependency/use hits
- `rg -n "Effect\\.runPromise|ManagedRuntime\\.make|Promise<|async function main" packages/base/src packages/flow/src packages/cli/src scripts -g "*.ts"` has no production-source hits except tests or unavoidable external type declarations that are not TrustGraph APIs
Do not mark the goal complete until all required gates are green, or until a real external blocker is reported with the exact failing command, error, and smallest next action.

37
ts/GOAL.md Normal file
View file

@ -0,0 +1,37 @@
# Goal Prompt
```text
/goal In `/home/elpresidank/YeeBois/dev/trustgraph/ts`, implement the Effect-native rewrite described in `EFFECT_NATIVE_REWRITE_PLAN.md`.
Follow the plan as the source of truth. The desired end state is that production TrustGraph APIs are Effect-first throughout backend/pubsub, messaging, processor/runtime, flow services, gateway, CLI, and service runners.
Hard constraints:
- Preserve existing features and wire behavior.
- Remove Promise-returning TrustGraph production APIs instead of keeping dual Promise/Effect methods.
- Remove production `Effect.runPromise` / `ManagedRuntime` compatibility bridges.
- Remove Fastify, `@fastify/websocket`, and Commander.
- Rebuild gateway surfaces with Effect v4 native `effect/unstable/httpapi`, `effect/unstable/http`, `effect/unstable/rpc`, and socket/platform layers.
- Rebuild CLI surfaces with `effect/unstable/cli`.
- Keep the existing native `effect/unstable/ai/McpServer` MCP rewrite; do not reintroduce server-side `@modelcontextprotocol/sdk` or `zod` in `packages/mcp`.
- Use `/home/elpresidank/YeeBois/projects/beep-effect/.repos/effect-v4` as the Effect v4 source reference.
- Preserve unrelated dirty work.
Work in staged gates:
1. Backend and NATS Effect API.
2. Messaging and processor API cleanup.
3. Flow service call-site migration.
4. Gateway HttpApi/RPC migration.
5. CLI/script migration.
6. Dependency cleanup and final verification.
Run the smallest relevant tests after each stage. Final done means all of these are green:
`bun run check:tsgo`
`bun run build`
`bun run lint`
`bun run test`
`bun run --cwd packages/mcp test`
`bun run workbench:qa`
`git diff --check`
Also verify no production Fastify/Commander use remains, and no production `Effect.runPromise`, `ManagedRuntime.make`, Promise API, or `async function main` remains in TrustGraph package/source surfaces except tests or unavoidable external type declarations.
```

277
ts/bun.lock
View file

@ -5,12 +5,12 @@
"": { "": {
"name": "trustgraph-ts", "name": "trustgraph-ts",
"dependencies": { "dependencies": {
"effect": "4.0.0-beta.75", "effect": "4.0.0-beta.78",
}, },
"devDependencies": { "devDependencies": {
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/bun": "^1.3.13", "@types/bun": "^1.3.13",
"@types/node": "^25.7.0", "@types/node": "^25.7.0",
"@typescript/native-preview": "7.0.0-dev.20260511.1", "@typescript/native-preview": "7.0.0-dev.20260511.1",
@ -27,19 +27,19 @@
"name": "@trustgraph/base", "name": "@trustgraph/base",
"version": "0.1.0", "version": "0.1.0",
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"effect": "4.0.0-beta.75", "effect": "4.0.0-beta.78",
"nats": "^2.29.0", "nats": "^2.29.0",
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6", "vitest": "^4.1.6",
@ -52,21 +52,20 @@
"tg": "dist/index.js", "tg": "dist/index.js",
}, },
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@trustgraph/base": "workspace:*", "@trustgraph/base": "workspace:*",
"@trustgraph/client": "workspace:*", "@trustgraph/client": "workspace:*",
"commander": "^13.1.0",
"ws": "^8.18.0", "ws": "^8.18.0",
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/ws": "^8.5.0", "@types/ws": "^8.5.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6", "vitest": "^4.1.6",
@ -76,10 +75,10 @@
"name": "@trustgraph/client", "name": "@trustgraph/client",
"version": "0.1.0", "version": "0.1.0",
"dependencies": { "dependencies": {
"effect": "4.0.0-beta.75", "effect": "4.0.0-beta.78",
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"@types/ws": "^8.5.0", "@types/ws": "^8.5.0",
"happy-dom": "^20.0.0", "happy-dom": "^20.0.0",
@ -97,32 +96,30 @@
"name": "@trustgraph/flow", "name": "@trustgraph/flow",
"version": "0.1.0", "version": "0.1.0",
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/platform-node": "4.0.0-beta.75", "@effect/platform-node": "4.0.0-beta.78",
"@effect/platform-node-shared": "4.0.0-beta.75", "@effect/platform-node-shared": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@fastify/websocket": "^11.0.0",
"@mistralai/mistralai": "^1.0.0", "@mistralai/mistralai": "^1.0.0",
"@modelcontextprotocol/sdk": "^1.12.0", "@modelcontextprotocol/sdk": "^1.12.0",
"@qdrant/js-client-rest": "^1.13.0", "@qdrant/js-client-rest": "^1.13.0",
"@trustgraph/base": "workspace:*", "@trustgraph/base": "workspace:*",
"effect": "4.0.0-beta.75", "effect": "4.0.0-beta.78",
"falkordb": "^5.0.0", "falkordb": "^5.0.0",
"fastify": "^5.2.0",
"ollama": "^0.6.3", "ollama": "^0.6.3",
"openai": "^4.85.0", "openai": "^4.85.0",
"pdfjs-dist": "^5.6.205", "pdfjs-dist": "^5.6.205",
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6", "vitest": "^4.1.6",
@ -132,25 +129,23 @@
"name": "@trustgraph/mcp", "name": "@trustgraph/mcp",
"version": "0.1.0", "version": "0.1.0",
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/platform-node": "4.0.0-beta.75", "@effect/platform-node": "4.0.0-beta.78",
"@effect/platform-node-shared": "4.0.0-beta.75", "@effect/platform-node-shared": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@modelcontextprotocol/sdk": "^1.8.0",
"@trustgraph/base": "workspace:*", "@trustgraph/base": "workspace:*",
"@trustgraph/client": "workspace:*", "@trustgraph/client": "workspace:*",
"zod": "^3.23.0",
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6", "vitest": "^4.1.6",
@ -160,18 +155,18 @@
"name": "@trustgraph/workbench", "name": "@trustgraph/workbench",
"version": "0.1.0", "version": "0.1.0",
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/platform-node": "4.0.0-beta.75", "@effect/platform-node": "4.0.0-beta.78",
"@effect/platform-node-shared": "4.0.0-beta.75", "@effect/platform-node-shared": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@tanstack/react-query": "^5.75.0", "@tanstack/react-query": "^5.75.0",
"@trustgraph/client": "workspace:*", "@trustgraph/client": "workspace:*",
"clsx": "^2.1.0", "clsx": "^2.1.0",
@ -186,7 +181,7 @@
"zustand": "^5.0.0", "zustand": "^5.0.0",
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@playwright/test": "^1.57.0", "@playwright/test": "^1.57.0",
"@tailwindcss/vite": "^4.1.0", "@tailwindcss/vite": "^4.1.0",
"@types/react": "^19.1.0", "@types/react": "^19.1.0",
@ -237,43 +232,43 @@
"@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "7.27.1", "@babel/helper-validator-identifier": "7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="], "@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "7.27.1", "@babel/helper-validator-identifier": "7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@effect/ai-anthropic": ["@effect/ai-anthropic@4.0.0-beta.75", "", { "peerDependencies": { "effect": "^4.0.0-beta.75" } }, "sha512-6o2F1RgMKXSenS0mDRde9o9rNh4N+gDnI0KS95DD+BMnArTnBSEcArTkskfogKC6b9vtzExpTU+xo6TNj3cdKw=="], "@effect/ai-anthropic": ["@effect/ai-anthropic@4.0.0-beta.78", "", { "peerDependencies": { "effect": "^4.0.0-beta.78" } }, "sha512-FZzwvKx2k+UIDJyv+FWMtC2CKRxJMWTII+z8EmMcT6tw1uzkkn+ouIyJ32AU+lfveKScNV1SXL4ml3VdcYGy/g=="],
"@effect/ai-openai": ["@effect/ai-openai@4.0.0-beta.75", "", { "peerDependencies": { "effect": "^4.0.0-beta.75" } }, "sha512-Qp6h8TwAdIxiU6ASW3h4R8b0XgnJv/+gviQsPs9kkhRr73O3ctvBnDeq3G0zyFCoKPx57PUDKKdkR0GAE8vZsw=="], "@effect/ai-openai": ["@effect/ai-openai@4.0.0-beta.78", "", { "peerDependencies": { "effect": "^4.0.0-beta.78" } }, "sha512-tx953rRkLqW2BeEkWK12/nRBPO0b1eS6pI+2YyWI0nQvX2JTTijrGlBv/qDVa5kxDkLm63+tA04xnxgZMlA8NA=="],
"@effect/ai-openrouter": ["@effect/ai-openrouter@4.0.0-beta.75", "", { "peerDependencies": { "effect": "^4.0.0-beta.75" } }, "sha512-+guPiVUytZPlCn0NsvJhZMrc10YzJ2nDLj1iNZnBMvQvDNKJsZp+O5R0sGc35/y61eomsMCqDwdG7hbOHFaEkA=="], "@effect/ai-openrouter": ["@effect/ai-openrouter@4.0.0-beta.78", "", { "peerDependencies": { "effect": "^4.0.0-beta.78" } }, "sha512-9GIRU9stAnDU5EJ5ZghUWrQXaE+rECCWI/eKVfYeC7UqjZmmmJmTcEbid3tvz2NMsnvIn0ymeKsJAohWCys39w=="],
"@effect/atom-react": ["@effect/atom-react@4.0.0-beta.75", "", { "peerDependencies": { "effect": "^4.0.0-beta.75", "react": "^19.2.4", "scheduler": "*" } }, "sha512-h5fkV3IJx3BGAqf+ehd/voLvpMUONMY3fH/p7SIRAooxx9b7Sl2UeNIlSAm9rzkBHWIQ1kYnQsEz86eZfP1QEw=="], "@effect/atom-react": ["@effect/atom-react@4.0.0-beta.78", "", { "peerDependencies": { "effect": "^4.0.0-beta.78", "react": "^19.2.4", "scheduler": "*" } }, "sha512-cgxDXJaD0wlbQXbp6tiEmmY+yajwurB0ynkFG20RVucvH4LsQMB3ogiHe0mt42wGggfbVYMEDxgBpQdqDRY8yA=="],
"@effect/openapi-generator": ["@effect/openapi-generator@4.0.0-beta.75", "", { "peerDependencies": { "@effect/platform-node": "^4.0.0-beta.75", "effect": "^4.0.0-beta.75" }, "bin": { "openapigen": "dist/bin.js" } }, "sha512-kuVREGa2Es8F2cvAcC4kwI6FYPARVHeC66U+OXii7lKHMJO0MnB5kqxbjCJHS4TevCH6zX6cL4nRDdEuc5XcvA=="], "@effect/openapi-generator": ["@effect/openapi-generator@4.0.0-beta.78", "", { "peerDependencies": { "@effect/platform-node": "^4.0.0-beta.78", "effect": "^4.0.0-beta.78" }, "bin": { "openapigen": "dist/bin.js" } }, "sha512-qhKRcZCNQ5b0Klrct+AC/tPQgIDBxVsD0MkQLIzqvLU3qRHaNd5yHo7kxFf/DuhCyyL++xZfbHsPdq3VdLIByg=="],
"@effect/opentelemetry": ["@effect/opentelemetry@4.0.0-beta.75", "", { "peerDependencies": { "@opentelemetry/api": "^1.9", "@opentelemetry/api-logs": ">=0.203.0 <0.300.0", "@opentelemetry/resources": "^2.0.0", "@opentelemetry/sdk-logs": ">=0.203.0 <0.300.0", "@opentelemetry/sdk-metrics": "^2.0.0", "@opentelemetry/sdk-trace-base": "^2.0.0", "@opentelemetry/sdk-trace-node": "^2.0.0", "@opentelemetry/sdk-trace-web": "^2.0.0", "@opentelemetry/semantic-conventions": "^1.33.0", "effect": "^4.0.0-beta.75" }, "optionalPeers": ["@opentelemetry/api", "@opentelemetry/api-logs", "@opentelemetry/resources", "@opentelemetry/sdk-logs", "@opentelemetry/sdk-metrics", "@opentelemetry/sdk-trace-base", "@opentelemetry/sdk-trace-node", "@opentelemetry/sdk-trace-web"] }, "sha512-dvO2HQcJSj4F91CP9pK/n1w1zat3fIWcXYBMeP9rM+Wxwx7ZzUy99PF88vdrPiqbR+iQ9/FYnRxndjg4+svlsA=="], "@effect/opentelemetry": ["@effect/opentelemetry@4.0.0-beta.78", "", { "peerDependencies": { "@opentelemetry/api": "^1.9", "@opentelemetry/api-logs": ">=0.203.0 <0.300.0", "@opentelemetry/resources": "^2.0.0", "@opentelemetry/sdk-logs": ">=0.203.0 <0.300.0", "@opentelemetry/sdk-metrics": "^2.0.0", "@opentelemetry/sdk-trace-base": "^2.0.0", "@opentelemetry/sdk-trace-node": "^2.0.0", "@opentelemetry/sdk-trace-web": "^2.0.0", "@opentelemetry/semantic-conventions": "^1.33.0", "effect": "^4.0.0-beta.78" }, "optionalPeers": ["@opentelemetry/api", "@opentelemetry/api-logs", "@opentelemetry/resources", "@opentelemetry/sdk-logs", "@opentelemetry/sdk-metrics", "@opentelemetry/sdk-trace-base", "@opentelemetry/sdk-trace-node", "@opentelemetry/sdk-trace-web"] }, "sha512-OJGnlNkxfhUmZ/8aLIfQly8ic2tntcnwidAP0BdrTUKa1/sbZjq5xTrhVUjvmehFra2Thsef0k4UPTgsOrBG1A=="],
"@effect/platform-browser": ["@effect/platform-browser@4.0.0-beta.75", "", { "dependencies": { "multipasta": "^0.2.7" }, "peerDependencies": { "effect": "^4.0.0-beta.75" } }, "sha512-AOLG8d8FMEEhu9TMiYFGuSTMUr+NL9OjVbBDeKciWJxEl+aBcK+ch5MLzBwGMdUztsD6txB9MYRPQXIg5D+97A=="], "@effect/platform-browser": ["@effect/platform-browser@4.0.0-beta.78", "", { "dependencies": { "multipasta": "^0.2.7" }, "peerDependencies": { "effect": "^4.0.0-beta.78" } }, "sha512-8r9MVuZ8xJRyVyi+C8SKSYLbMsHr7qOiUgLV6lKMECuAWyMhlbK/7Ka9SQGr0ZPqOe5ShLEvV7DevnGkG+owAQ=="],
"@effect/platform-bun": ["@effect/platform-bun@4.0.0-beta.75", "", { "dependencies": { "@effect/platform-node-shared": "^4.0.0-beta.75" }, "peerDependencies": { "effect": "^4.0.0-beta.75" } }, "sha512-OIIi1kI/DFkzgIHPzy7kqLI1VVbRVpq3HkgH72LHPjVt9mIO3+YXvt8rnjSmTGpSpjEzzBsas8rZ4cnFbYKESA=="], "@effect/platform-bun": ["@effect/platform-bun@4.0.0-beta.78", "", { "dependencies": { "@effect/platform-node-shared": "^4.0.0-beta.78" }, "peerDependencies": { "effect": "^4.0.0-beta.78" } }, "sha512-lmPCL1G7SlkCWCguX3rDPS7kKuvJ/AN4pjS7IXb/5SoauHPd67iUdc1ZbB7o6lwTChJaIfWNNPkUWygiaUeJiA=="],
"@effect/platform-node": ["@effect/platform-node@4.0.0-beta.75", "", { "dependencies": { "@effect/platform-node-shared": "^4.0.0-beta.75", "mime": "^4.1.0", "undici": "^8.2.0" }, "peerDependencies": { "effect": "^4.0.0-beta.75", "ioredis": "^5.7.0" } }, "sha512-3A7yOMwQUV8p743QRWC2Qqao3ASmolbh2i4TGbJZ3bZLBGVcRQqBSg1sg2zpSFihf0R+RZg/dlV6OYFIanK/zA=="], "@effect/platform-node": ["@effect/platform-node@4.0.0-beta.78", "", { "dependencies": { "@effect/platform-node-shared": "^4.0.0-beta.78", "mime": "^4.1.0", "undici": "^8.2.0" }, "peerDependencies": { "effect": "^4.0.0-beta.78", "ioredis": "^5.7.0" } }, "sha512-8ONrIS5/R9dq+0BJ6v3kUXNEkfjU6S3GzIYCH5gmHdiriRvIoBhXYNAITfRvZpfx1JPrKuP70cHyuQDjmJcDkQ=="],
"@effect/platform-node-shared": ["@effect/platform-node-shared@4.0.0-beta.75", "", { "dependencies": { "@types/ws": "^8.18.1", "ws": "^8.20.0" }, "peerDependencies": { "effect": "^4.0.0-beta.75" } }, "sha512-ERxuFDuCuMAiSHLrNVTvSXPnUajcZV4PUVibXbk1lwiymhbkxRRBs1Zeuicwho8jZdpn7niXCDI24CiB9PIozA=="], "@effect/platform-node-shared": ["@effect/platform-node-shared@4.0.0-beta.78", "", { "dependencies": { "@types/ws": "^8.18.1", "ws": "^8.20.0" }, "peerDependencies": { "effect": "^4.0.0-beta.78" } }, "sha512-mo0ddTPATyCMyqzQasYDL7+NI29vozoMplom+qu9f/onDTd4xG5hvEEfGxfL0Ljygui6keG/YE/E9OZVf2z5WA=="],
"@effect/tsgo": ["@effect/tsgo@0.13.0", "", { "optionalDependencies": { "@effect/tsgo-darwin-arm64": "0.13.0", "@effect/tsgo-darwin-x64": "0.13.0", "@effect/tsgo-linux-arm": "0.13.0", "@effect/tsgo-linux-arm64": "0.13.0", "@effect/tsgo-linux-x64": "0.13.0", "@effect/tsgo-win32-arm64": "0.13.0", "@effect/tsgo-win32-x64": "0.13.0" }, "bin": { "effect-tsgo": "dist/effect-tsgo.js" } }, "sha512-oOBoz8iFVhrBpvr0R6vLE01ydPLbyRu1eN8eU97E17T+LUDMgcgm4AqdTu1pUaCIhRAWIZcmRxD+Jvd38940yQ=="], "@effect/tsgo": ["@effect/tsgo@0.14.0", "", { "optionalDependencies": { "@effect/tsgo-darwin-arm64": "0.14.0", "@effect/tsgo-darwin-x64": "0.14.0", "@effect/tsgo-linux-arm": "0.14.0", "@effect/tsgo-linux-arm64": "0.14.0", "@effect/tsgo-linux-x64": "0.14.0", "@effect/tsgo-win32-arm64": "0.14.0", "@effect/tsgo-win32-x64": "0.14.0" }, "bin": { "effect-tsgo": "dist/effect-tsgo.js" } }, "sha512-UThRMJJwXtCWDlQyNTfPTAWTyG3nuODZWu3txLRXoHg/C8q2IlvKD+ap6RUU1kK+SR+pTa73mZTtkScQAD8N7g=="],
"@effect/tsgo-darwin-arm64": ["@effect/tsgo-darwin-arm64@0.13.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-DGpDMwmE+fVx+/w7DurQJz1iGPiIp4kUoIZ/iUb0zBdPuhycH5bqm8x4lMEYKdO9D6k4VIDY3zeSoPAs3yqGfQ=="], "@effect/tsgo-darwin-arm64": ["@effect/tsgo-darwin-arm64@0.14.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-bNJDM6CrIYIEz2PLMpQjxHAvPzcaiP0IRfLr1a+0Kfr1+bd/Ck5qpt+tLsukWcg+WTiiIJqwp0zP1gBuJA3SIA=="],
"@effect/tsgo-darwin-x64": ["@effect/tsgo-darwin-x64@0.13.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-KfoQu1SeKUdoegK1M3B9ZqxNtJqZM+Odqlyg48G6K7CQ4tBPlpOwAschVr4h4IBANMEQNFuQADPGy0gSSZwQvQ=="], "@effect/tsgo-darwin-x64": ["@effect/tsgo-darwin-x64@0.14.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-daJpJhYhG25t8N8cN/u6F6v88UFwttVx6McnyV6VS4qNaOI3F8cYKHkndL6LoouC4rjgEli0BplA7jSxSSvSbQ=="],
"@effect/tsgo-linux-arm": ["@effect/tsgo-linux-arm@0.13.0", "", { "os": "linux", "cpu": "arm" }, "sha512-txp7VxQIYXBpJo66G77JKoki6ouEJk/HokEIrp+mBHN6BNx8O5h6kjUtLCeFbIiaKuX5UrqHN7Yvx/vj4qLkSQ=="], "@effect/tsgo-linux-arm": ["@effect/tsgo-linux-arm@0.14.0", "", { "os": "linux", "cpu": "arm" }, "sha512-8+3akhHgNCIZu/vXBKd30h+eSmgQge4RmU/7YMw1e+UA30Ch5UVEgPiJDMqtDDCCALL8RKpHbeFKY/cP6N+C4Q=="],
"@effect/tsgo-linux-arm64": ["@effect/tsgo-linux-arm64@0.13.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-9KadPsq9b5sQqh6pRRTr0mNcRVM8Jk9l3Y9SgsmwnfEjxHFoo/NThDQNgfctdBawOCwImQdm/YtC5oj7AwCFrA=="], "@effect/tsgo-linux-arm64": ["@effect/tsgo-linux-arm64@0.14.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-z67ZXpcPpxZgcatfZSWI7LEgNh+ba5LrqbuHovh9l2ywtOpiXf7d5kU2VPlEA8YVfU7dhnJviDGY8hgJQIIt5g=="],
"@effect/tsgo-linux-x64": ["@effect/tsgo-linux-x64@0.13.0", "", { "os": "linux", "cpu": "x64" }, "sha512-Osp1yIFPmibHvLTkVHk86RODr/hP44yXQU8NqyGauFDm93FdFEOn8v96UkcXziST76v0wReKvB/Ng39g7lafSw=="], "@effect/tsgo-linux-x64": ["@effect/tsgo-linux-x64@0.14.0", "", { "os": "linux", "cpu": "x64" }, "sha512-nJ/4cU0aWnof2A23/Cmry/M4Ek6F0/Igv+BCX09x5EC8sbn4hGIJ5rlQQqjhj5XF7Zd1BEkN5oxhLzwA6i1veA=="],
"@effect/tsgo-win32-arm64": ["@effect/tsgo-win32-arm64@0.13.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-c2nUvQyW/iAqYKjn/BrFKt2BxWQGlW8fGdq7MRTLs6cOrwwa4XnqeVsomyqIyeBcXB5s3HilRolF4xnFvJpTKw=="], "@effect/tsgo-win32-arm64": ["@effect/tsgo-win32-arm64@0.14.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-lSwV3IP7MvCr/ElWIBywmjfdRmfq98lFeIVR7NpuuSSicVvpAPv4ACpw6z93SUKSPjJrw7jMvZn9MOrwKuPx8g=="],
"@effect/tsgo-win32-x64": ["@effect/tsgo-win32-x64@0.13.0", "", { "os": "win32", "cpu": "x64" }, "sha512-c+tQDZ+oOGj1PVgoTa4SswoqbF5t9fMHse0cKh3QMJPxh1nkJ+E7cqAyFcpL8vy2apKZQUB1UbGEAXCQRm1u1Q=="], "@effect/tsgo-win32-x64": ["@effect/tsgo-win32-x64@0.14.0", "", { "os": "win32", "cpu": "x64" }, "sha512-5yRE1NAtBCd1IYSU2rLe/W9NGpyTOqhsJvCs4ZV81j3U6xNk/+kk1jVfMPZXZjal4cIRWI1UVxbt4uSN4pw/lw=="],
"@effect/vitest": ["@effect/vitest@4.0.0-beta.75", "", { "peerDependencies": { "effect": "^4.0.0-beta.75", "vitest": "^3.0.0 || ^4.0.0" } }, "sha512-9R6s/9j/tZUc8lMmkOKXfgwnSRhk5EvKS89Rwwj8TaSYh+n9ZCDvvepxGcE9DpNghqDd7fC3Gcbgk0c56hSs3Q=="], "@effect/vitest": ["@effect/vitest@4.0.0-beta.78", "", { "peerDependencies": { "effect": "^4.0.0-beta.78", "vitest": "^3.0.0 || ^4.0.0" } }, "sha512-5KQsQYrQ/o7mfOVAxRtNnfD9M0W4OI6yQd0n/m2N7OOLxTdX4FwN4s/X4obykBC7ZEwH+bzMrFJiB4pq9lrQKQ=="],
"@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.27.7", "", { "os": "aix", "cpu": "ppc64" }, "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg=="], "@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.27.7", "", { "os": "aix", "cpu": "ppc64" }, "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg=="],
@ -331,20 +326,6 @@
"@falkordb/graph": ["@falkordb/graph@2.0.1", "", { "peerDependencies": { "@falkordb/client": "1.6.0" } }, "sha512-pwLdV4lbgBgNQlwSa4qTHwQHNTXCKVlDvH+a7cjgF5bcnDQyLtunk6FouUjBX0y7SKkSY/+Dk+RwV1EAU5e7Nw=="], "@falkordb/graph": ["@falkordb/graph@2.0.1", "", { "peerDependencies": { "@falkordb/client": "1.6.0" } }, "sha512-pwLdV4lbgBgNQlwSa4qTHwQHNTXCKVlDvH+a7cjgF5bcnDQyLtunk6FouUjBX0y7SKkSY/+Dk+RwV1EAU5e7Nw=="],
"@fastify/ajv-compiler": ["@fastify/ajv-compiler@4.0.5", "", { "dependencies": { "ajv": "8.18.0", "ajv-formats": "3.0.1", "fast-uri": "3.1.0" } }, "sha512-KoWKW+MhvfTRWL4qrhUwAAZoaChluo0m0vbiJlGMt2GXvL4LVPQEjt8kSpHI3IBq5Rez8fg+XeH3cneztq+C7A=="],
"@fastify/error": ["@fastify/error@4.2.0", "", {}, "sha512-RSo3sVDXfHskiBZKBPRgnQTtIqpi/7zhJOEmAxCiBcM7d0uwdGdxLlsCaLzGs8v8NnxIRlfG0N51p5yFaOentQ=="],
"@fastify/fast-json-stringify-compiler": ["@fastify/fast-json-stringify-compiler@5.0.3", "", { "dependencies": { "fast-json-stringify": "6.3.0" } }, "sha512-uik7yYHkLr6fxd8hJSZ8c+xF4WafPK+XzneQDPU+D10r5X19GW8lJcom2YijX2+qtFF1ENJlHXKFM9ouXNJYgQ=="],
"@fastify/forwarded": ["@fastify/forwarded@3.0.1", "", {}, "sha512-JqDochHFqXs3C3Ml3gOY58zM7OqO9ENqPo0UqAjAjH8L01fRZqwX9iLeX34//kiJubF7r2ZQHtBRU36vONbLlw=="],
"@fastify/merge-json-schemas": ["@fastify/merge-json-schemas@0.2.1", "", { "dependencies": { "dequal": "2.0.3" } }, "sha512-OA3KGBCy6KtIvLf8DINC5880o5iBlDX4SxzLQS8HorJAbqluzLRn80UXU0bxZn7UOFhFgpRJDasfwn9nG4FG4A=="],
"@fastify/proxy-addr": ["@fastify/proxy-addr@5.1.0", "", { "dependencies": { "@fastify/forwarded": "3.0.1", "ipaddr.js": "2.3.0" } }, "sha512-INS+6gh91cLUjB+PVHfu1UqcB76Sqtpyp7bnL+FYojhjygvOPA9ctiD/JDKsyD9Xgu4hUhCSJBPig/w7duNajw=="],
"@fastify/websocket": ["@fastify/websocket@11.2.0", "", { "dependencies": { "duplexify": "4.1.3", "fastify-plugin": "5.1.0", "ws": "8.20.0" } }, "sha512-3HrDPbAG1CzUCqnslgJxppvzaAZffieOVbLp1DAy1huCSynUWPifSvfdEDUR8HlJLp3sp1A36uOM2tJogADS8w=="],
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "4.12.10" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="], "@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "4.12.10" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
"@ioredis/commands": ["@ioredis/commands@1.10.0", "", {}, "sha512-UmeW7z4LfctwoQ5wkhVzgq8tXkreED2xZGpX+Bg+zA+WJFZCT6c062AfCK/Dfk81xZnnwdhJCUMkitihRaoC2Q=="], "@ioredis/commands": ["@ioredis/commands@1.10.0", "", {}, "sha512-UmeW7z4LfctwoQ5wkhVzgq8tXkreED2xZGpX+Bg+zA+WJFZCT6c062AfCK/Dfk81xZnnwdhJCUMkitihRaoC2Q=="],
@ -407,8 +388,6 @@
"@pdf-lib/upng": ["@pdf-lib/upng@1.0.1", "", { "dependencies": { "pako": "^1.0.10" } }, "sha512-dQK2FUMQtowVP00mtIksrlZhdFXQZPC+taih1q4CvPZ5vqdxR/LKBaFg0oAfzd1GlHZXXSPdQfzQnt+ViGvEIQ=="], "@pdf-lib/upng": ["@pdf-lib/upng@1.0.1", "", { "dependencies": { "pako": "^1.0.10" } }, "sha512-dQK2FUMQtowVP00mtIksrlZhdFXQZPC+taih1q4CvPZ5vqdxR/LKBaFg0oAfzd1GlHZXXSPdQfzQnt+ViGvEIQ=="],
"@pinojs/redact": ["@pinojs/redact@0.4.0", "", {}, "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg=="],
"@playwright/test": ["@playwright/test@1.60.0", "", { "dependencies": { "playwright": "1.60.0" }, "bin": { "playwright": "cli.js" } }, "sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag=="], "@playwright/test": ["@playwright/test@1.60.0", "", { "dependencies": { "playwright": "1.60.0" }, "bin": { "playwright": "cli.js" } }, "sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag=="],
"@qdrant/js-client-rest": ["@qdrant/js-client-rest@1.17.0", "", { "dependencies": { "@qdrant/openapi-typescript-fetch": "1.2.6", "undici": "6.24.1" }, "peerDependencies": { "typescript": "5.9.3" } }, "sha512-aZFQeirWVqWAa1a8vJ957LMzcXkFHGbsoRhzc8AkGfg6V0jtK8PlG8/eyyc2xhYsR961FDDx1Tx6nyE0K7lS+A=="], "@qdrant/js-client-rest": ["@qdrant/js-client-rest@1.17.0", "", { "dependencies": { "@qdrant/openapi-typescript-fetch": "1.2.6", "undici": "6.24.1" }, "peerDependencies": { "typescript": "5.9.3" } }, "sha512-aZFQeirWVqWAa1a8vJ957LMzcXkFHGbsoRhzc8AkGfg6V0jtK8PlG8/eyyc2xhYsR961FDDx1Tx6nyE0K7lS+A=="],
@ -605,8 +584,6 @@
"abort-controller": ["abort-controller@3.0.0", "", { "dependencies": { "event-target-shim": "5.0.1" } }, "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg=="], "abort-controller": ["abort-controller@3.0.0", "", { "dependencies": { "event-target-shim": "5.0.1" } }, "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg=="],
"abstract-logging": ["abstract-logging@2.0.1", "", {}, "sha512-2BjRTZxTPvheOvGbBslFSYOUkr+SjPtOnrLP33f+VIWLzezQpZcqVg7ja3L4dBXmzzgwT+a029jRx5PCi3JuiA=="],
"accepts": ["accepts@2.0.0", "", { "dependencies": { "mime-types": "3.0.2", "negotiator": "1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="], "accepts": ["accepts@2.0.0", "", { "dependencies": { "mime-types": "3.0.2", "negotiator": "1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="],
"accessor-fn": ["accessor-fn@1.5.3", "", {}, "sha512-rkAofCwe/FvYFUlMB0v0gWmhqtfAtV1IUkdPbfhTUyYniu5LrC0A0UJkTH0Jv3S8SvwkmfuAlY+mQIJATdocMA=="], "accessor-fn": ["accessor-fn@1.5.3", "", {}, "sha512-rkAofCwe/FvYFUlMB0v0gWmhqtfAtV1IUkdPbfhTUyYniu5LrC0A0UJkTH0Jv3S8SvwkmfuAlY+mQIJATdocMA=="],
@ -621,10 +598,6 @@
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="], "asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"atomic-sleep": ["atomic-sleep@1.0.0", "", {}, "sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ=="],
"avvio": ["avvio@9.2.0", "", { "dependencies": { "@fastify/error": "4.2.0", "fastq": "1.20.1" } }, "sha512-2t/sy01ArdHHE0vRH5Hsay+RtCZt3dLPji7W7/MMOCEgze5b7SNDC4j5H6FnVgPkI1MTNFGzHdHrVXDDl7QSSQ=="],
"bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="], "bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="],
"baseline-browser-mapping": ["baseline-browser-mapping@2.10.15", "", { "bin": { "baseline-browser-mapping": "dist/cli.cjs" } }, "sha512-1nfKCq9wuAZFTkA2ey/3OXXx7GzFjLdkTiFVNwlJ9WqdI706CZRIhEqjuwanjMIja+84jDLa9rcyZDPDiVkASQ=="], "baseline-browser-mapping": ["baseline-browser-mapping@2.10.15", "", { "bin": { "baseline-browser-mapping": "dist/cli.cjs" } }, "sha512-1nfKCq9wuAZFTkA2ey/3OXXx7GzFjLdkTiFVNwlJ9WqdI706CZRIhEqjuwanjMIja+84jDLa9rcyZDPDiVkASQ=="],
@ -667,8 +640,6 @@
"comma-separated-tokens": ["comma-separated-tokens@2.0.3", "", {}, "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg=="], "comma-separated-tokens": ["comma-separated-tokens@2.0.3", "", {}, "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg=="],
"commander": ["commander@13.1.0", "", {}, "sha512-/rFeCpNJQbhSZjGVwO9RFV3xPqbnERS8MmIQzCtD/zl6gpJuV/bMLuN92oG3F7d8oDEHHRrujSXNUr8fpjntKw=="],
"content-disposition": ["content-disposition@1.0.1", "", {}, "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q=="], "content-disposition": ["content-disposition@1.0.1", "", {}, "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q=="],
"content-type": ["content-type@1.0.5", "", {}, "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA=="], "content-type": ["content-type@1.0.5", "", {}, "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA=="],
@ -741,18 +712,14 @@
"dunder-proto": ["dunder-proto@1.0.1", "", { "dependencies": { "call-bind-apply-helpers": "1.0.2", "es-errors": "1.3.0", "gopd": "1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="], "dunder-proto": ["dunder-proto@1.0.1", "", { "dependencies": { "call-bind-apply-helpers": "1.0.2", "es-errors": "1.3.0", "gopd": "1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="],
"duplexify": ["duplexify@4.1.3", "", { "dependencies": { "end-of-stream": "1.4.5", "inherits": "2.0.4", "readable-stream": "3.6.2", "stream-shift": "1.0.3" } }, "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA=="],
"ee-first": ["ee-first@1.1.1", "", {}, "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="], "ee-first": ["ee-first@1.1.1", "", {}, "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="],
"effect": ["effect@4.0.0-beta.75", "", { "dependencies": { "@standard-schema/spec": "^1.1.0", "fast-check": "^4.8.0", "find-my-way-ts": "^0.1.6", "ini": "^7.0.0", "kubernetes-types": "^1.30.0", "msgpackr": "^2.0.1", "multipasta": "^0.2.7", "toml": "^4.1.1", "uuid": "^14.0.0", "yaml": "^2.9.0" } }, "sha512-KrDuecxuN8rC7dytVs5GFfUhtC+/D8pOyfVb1LNZGjJY7qigQG9xXJ4R1Bp+tBo7sKbZOXBswCfuQ3ombaEldQ=="], "effect": ["effect@4.0.0-beta.78", "", { "dependencies": { "@standard-schema/spec": "^1.1.0", "fast-check": "^4.8.0", "find-my-way-ts": "^0.1.6", "ini": "^7.0.0", "kubernetes-types": "^1.30.0", "msgpackr": "^2.0.1", "multipasta": "^0.2.7", "toml": "^4.1.1", "uuid": "^14.0.0", "yaml": "^2.9.0" } }, "sha512-j79Rl9QpHwMz/ZJWLNpZoiVj9N7zHqiLKN5EcYd/A8J1oqejILWQLfc4HPlvqHqKC8SK55LJ+X4gy4ONJ+JpfQ=="],
"electron-to-chromium": ["electron-to-chromium@1.5.331", "", {}, "sha512-IbxXrsTlD3hRodkLnbxAPP4OuJYdWCeM3IOdT+CpcMoIwIoDfCmRpEtSPfwBXxVkg9xmBeY7Lz2Eo2TDn/HC3Q=="], "electron-to-chromium": ["electron-to-chromium@1.5.331", "", {}, "sha512-IbxXrsTlD3hRodkLnbxAPP4OuJYdWCeM3IOdT+CpcMoIwIoDfCmRpEtSPfwBXxVkg9xmBeY7Lz2Eo2TDn/HC3Q=="],
"encodeurl": ["encodeurl@2.0.0", "", {}, "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg=="], "encodeurl": ["encodeurl@2.0.0", "", {}, "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg=="],
"end-of-stream": ["end-of-stream@1.4.5", "", { "dependencies": { "once": "1.4.0" } }, "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg=="],
"enhanced-resolve": ["enhanced-resolve@5.20.1", "", { "dependencies": { "graceful-fs": "4.2.11", "tapable": "2.3.2" } }, "sha512-Qohcme7V1inbAfvjItgw0EaxVX5q2rdVEZHRBrEQdRZTssLDGsL8Lwrznl8oQ/6kuTJONLaDcGjkNP247XEhcA=="], "enhanced-resolve": ["enhanced-resolve@5.20.1", "", { "dependencies": { "graceful-fs": "4.2.11", "tapable": "2.3.2" } }, "sha512-Qohcme7V1inbAfvjItgw0EaxVX5q2rdVEZHRBrEQdRZTssLDGsL8Lwrznl8oQ/6kuTJONLaDcGjkNP247XEhcA=="],
"entities": ["entities@7.0.1", "", {}, "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA=="], "entities": ["entities@7.0.1", "", {}, "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA=="],
@ -797,28 +764,14 @@
"fast-check": ["fast-check@4.8.0", "", { "dependencies": { "pure-rand": "^8.0.0" } }, "sha512-GOJ158CUMnN6cSahsv4+ExARvIDuzzinFjkp0E9WtiBa5zcVeLozVkWaE4IzFcc+Y48Wp1EDlUZsXRyAztQcSg=="], "fast-check": ["fast-check@4.8.0", "", { "dependencies": { "pure-rand": "^8.0.0" } }, "sha512-GOJ158CUMnN6cSahsv4+ExARvIDuzzinFjkp0E9WtiBa5zcVeLozVkWaE4IzFcc+Y48Wp1EDlUZsXRyAztQcSg=="],
"fast-decode-uri-component": ["fast-decode-uri-component@1.0.1", "", {}, "sha512-WKgKWg5eUxvRZGwW8FvfbaH7AXSh2cL+3j5fMGzUMCxWBJ3dV3a7Wz8y2f/uQ0e3B6WmodD3oS54jTQ9HVTIIg=="],
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="], "fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-json-stringify": ["fast-json-stringify@6.3.0", "", { "dependencies": { "@fastify/merge-json-schemas": "0.2.1", "ajv": "8.18.0", "ajv-formats": "3.0.1", "fast-uri": "3.1.0", "json-schema-ref-resolver": "3.0.0", "rfdc": "1.4.1" } }, "sha512-oRCntNDY/329HJPlmdNLIdogNtt6Vyjb1WuT01Soss3slIdyUp8kAcDU3saQTOquEK8KFVfwIIF7FebxUAu+yA=="],
"fast-querystring": ["fast-querystring@1.1.2", "", { "dependencies": { "fast-decode-uri-component": "1.0.1" } }, "sha512-g6KuKWmFXc0fID8WWH0jit4g0AGBoJhCkJMb1RmbsSEUNvQ+ZC8D6CUZ+GtF8nMzSPXnhiePyyqqipzNNEnHjg=="],
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="], "fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
"fastify": ["fastify@5.8.4", "", { "dependencies": { "@fastify/ajv-compiler": "4.0.5", "@fastify/error": "4.2.0", "@fastify/fast-json-stringify-compiler": "5.0.3", "@fastify/proxy-addr": "5.1.0", "abstract-logging": "2.0.1", "avvio": "9.2.0", "fast-json-stringify": "6.3.0", "find-my-way": "9.5.0", "light-my-request": "6.6.0", "pino": "10.3.1", "process-warning": "5.0.0", "rfdc": "1.4.1", "secure-json-parse": "4.1.0", "semver": "7.7.4", "toad-cache": "3.7.0" } }, "sha512-sa42J1xylbBAYUWALSBoyXKPDUvM3OoNOibIefA+Oha57FryXKKCZarA1iDntOCWp3O35voZLuDg2mdODXtPzQ=="],
"fastify-plugin": ["fastify-plugin@5.1.0", "", {}, "sha512-FAIDA8eovSt5qcDgcBvDuX/v0Cjz0ohGhENZ/wpc3y+oZCY2afZ9Baqql3g/lC+OHRnciQol4ww7tuthOb9idw=="],
"fastq": ["fastq@1.20.1", "", { "dependencies": { "reusify": "1.1.0" } }, "sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw=="],
"fdir": ["fdir@6.5.0", "", { "optionalDependencies": { "picomatch": "4.0.4" } }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], "fdir": ["fdir@6.5.0", "", { "optionalDependencies": { "picomatch": "4.0.4" } }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
"finalhandler": ["finalhandler@2.1.1", "", { "dependencies": { "debug": "4.4.3", "encodeurl": "2.0.0", "escape-html": "1.0.3", "on-finished": "2.4.1", "parseurl": "1.3.3", "statuses": "2.0.2" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="], "finalhandler": ["finalhandler@2.1.1", "", { "dependencies": { "debug": "4.4.3", "encodeurl": "2.0.0", "escape-html": "1.0.3", "on-finished": "2.4.1", "parseurl": "1.3.3", "statuses": "2.0.2" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="],
"find-my-way": ["find-my-way@9.5.0", "", { "dependencies": { "fast-deep-equal": "3.1.3", "fast-querystring": "1.1.2", "safe-regex2": "5.1.0" } }, "sha512-VW2RfnmscZO5KgBY5XVyKREMW5nMZcxDy+buTOsL+zIPnBlbKm+00sgzoQzq1EVh4aALZLfKdwv6atBGcjvjrQ=="],
"find-my-way-ts": ["find-my-way-ts@0.1.6", "", {}, "sha512-a85L9ZoXtNAey3Y6Z+eBWW658kO/MwR7zIafkIUPUMf3isZG0NCs2pjW2wtjxAKuJPxMAsHUIP4ZPGv0o5gyTA=="], "find-my-way-ts": ["find-my-way-ts@0.1.6", "", {}, "sha512-a85L9ZoXtNAey3Y6Z+eBWW658kO/MwR7zIafkIUPUMf3isZG0NCs2pjW2wtjxAKuJPxMAsHUIP4ZPGv0o5gyTA=="],
"float-tooltip": ["float-tooltip@1.7.5", "", { "dependencies": { "d3-selection": "3.0.0", "kapsule": "1.16.3", "preact": "10.29.1" } }, "sha512-/kXzuDnnBqyyWyhDMH7+PfP8J/oXiAavGzcRxASOMRHFuReDtofizLLJsf7nnDLAfEaMW4pVWaXrAjtnglpEkg=="], "float-tooltip": ["float-tooltip@1.7.5", "", { "dependencies": { "d3-selection": "3.0.0", "kapsule": "1.16.3", "preact": "10.29.1" } }, "sha512-/kXzuDnnBqyyWyhDMH7+PfP8J/oXiAavGzcRxASOMRHFuReDtofizLLJsf7nnDLAfEaMW4pVWaXrAjtnglpEkg=="],
@ -889,7 +842,7 @@
"ip-address": ["ip-address@10.1.0", "", {}, "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q=="], "ip-address": ["ip-address@10.1.0", "", {}, "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q=="],
"ipaddr.js": ["ipaddr.js@2.3.0", "", {}, "sha512-Zv/pA+ciVFbCSBBjGfaKUya/CcGmUHzTydLMaTwrUUEM2DIEO3iZvueGxmacvmN50fGpGVKeTXpb2LcYQxeVdg=="], "ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
"is-alphabetical": ["is-alphabetical@2.0.1", "", {}, "sha512-FWyyY60MeTNyeSRpkM2Iry0G9hpr7/9kD40mD/cGQEuilcZYS4okz8SN2Q6rLCJ8gbCt6fN+rC+6tMGS99LaxQ=="], "is-alphabetical": ["is-alphabetical@2.0.1", "", {}, "sha512-FWyyY60MeTNyeSRpkM2Iry0G9hpr7/9kD40mD/cGQEuilcZYS4okz8SN2Q6rLCJ8gbCt6fN+rC+6tMGS99LaxQ=="],
@ -915,8 +868,6 @@
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="], "jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],
"json-schema-ref-resolver": ["json-schema-ref-resolver@3.0.0", "", { "dependencies": { "dequal": "2.0.3" } }, "sha512-hOrZIVL5jyYFjzk7+y7n5JDzGlU8rfWDuYyHwGa2WA8/pcmMHezp2xsVwxrebD/Q9t8Nc5DboieySDpCp4WG4A=="],
"json-schema-traverse": ["json-schema-traverse@1.0.0", "", {}, "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug=="], "json-schema-traverse": ["json-schema-traverse@1.0.0", "", {}, "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug=="],
"json-schema-typed": ["json-schema-typed@8.0.2", "", {}, "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA=="], "json-schema-typed": ["json-schema-typed@8.0.2", "", {}, "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA=="],
@ -927,8 +878,6 @@
"kubernetes-types": ["kubernetes-types@1.30.0", "", {}, "sha512-Dew1okvhM/SQcIa2rcgujNndZwU8VnSapDgdxlYoB84ZlpAD43U6KLAFqYo17ykSFGHNPrg0qry0bP+GJd9v7Q=="], "kubernetes-types": ["kubernetes-types@1.30.0", "", {}, "sha512-Dew1okvhM/SQcIa2rcgujNndZwU8VnSapDgdxlYoB84ZlpAD43U6KLAFqYo17ykSFGHNPrg0qry0bP+GJd9v7Q=="],
"light-my-request": ["light-my-request@6.6.0", "", { "dependencies": { "cookie": "1.1.1", "process-warning": "4.0.1", "set-cookie-parser": "2.7.2" } }, "sha512-CHYbu8RtboSIoVsHZ6Ye4cj4Aw/yg2oAFimlF7mNvfDV192LR7nDiKtSIfCuLT7KokPSTn/9kfVLm5OGN0A28A=="],
"lightningcss": ["lightningcss@1.32.0", "", { "dependencies": { "detect-libc": "2.1.2" }, "optionalDependencies": { "lightningcss-android-arm64": "1.32.0", "lightningcss-darwin-arm64": "1.32.0", "lightningcss-darwin-x64": "1.32.0", "lightningcss-freebsd-x64": "1.32.0", "lightningcss-linux-arm-gnueabihf": "1.32.0", "lightningcss-linux-arm64-gnu": "1.32.0", "lightningcss-linux-arm64-musl": "1.32.0", "lightningcss-linux-x64-gnu": "1.32.0", "lightningcss-linux-x64-musl": "1.32.0", "lightningcss-win32-arm64-msvc": "1.32.0", "lightningcss-win32-x64-msvc": "1.32.0" } }, "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ=="], "lightningcss": ["lightningcss@1.32.0", "", { "dependencies": { "detect-libc": "2.1.2" }, "optionalDependencies": { "lightningcss-android-arm64": "1.32.0", "lightningcss-darwin-arm64": "1.32.0", "lightningcss-darwin-x64": "1.32.0", "lightningcss-freebsd-x64": "1.32.0", "lightningcss-linux-arm-gnueabihf": "1.32.0", "lightningcss-linux-arm64-gnu": "1.32.0", "lightningcss-linux-arm64-musl": "1.32.0", "lightningcss-linux-x64-gnu": "1.32.0", "lightningcss-linux-x64-musl": "1.32.0", "lightningcss-win32-arm64-msvc": "1.32.0", "lightningcss-win32-x64-msvc": "1.32.0" } }, "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ=="],
"lightningcss-android-arm64": ["lightningcss-android-arm64@1.32.0", "", { "os": "android", "cpu": "arm64" }, "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg=="], "lightningcss-android-arm64": ["lightningcss-android-arm64@1.32.0", "", { "os": "android", "cpu": "arm64" }, "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg=="],
@ -1067,8 +1016,6 @@
"ollama": ["ollama@0.6.3", "", { "dependencies": { "whatwg-fetch": "^3.6.20" } }, "sha512-KEWEhIqE5wtfzEIZbDCLH51VFZ6Z3ZSa6sIOg/E/tBV8S51flyqBOXi+bRxlOYKDf8i327zG9eSTb8IJxvm3Zg=="], "ollama": ["ollama@0.6.3", "", { "dependencies": { "whatwg-fetch": "^3.6.20" } }, "sha512-KEWEhIqE5wtfzEIZbDCLH51VFZ6Z3ZSa6sIOg/E/tBV8S51flyqBOXi+bRxlOYKDf8i327zG9eSTb8IJxvm3Zg=="],
"on-exit-leak-free": ["on-exit-leak-free@2.1.2", "", {}, "sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA=="],
"on-finished": ["on-finished@2.4.1", "", { "dependencies": { "ee-first": "1.1.1" } }, "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg=="], "on-finished": ["on-finished@2.4.1", "", { "dependencies": { "ee-first": "1.1.1" } }, "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg=="],
"once": ["once@1.4.0", "", { "dependencies": { "wrappy": "1.0.2" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="], "once": ["once@1.4.0", "", { "dependencies": { "wrappy": "1.0.2" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="],
@ -1095,12 +1042,6 @@
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="], "picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"pino": ["pino@10.3.1", "", { "dependencies": { "@pinojs/redact": "0.4.0", "atomic-sleep": "1.0.0", "on-exit-leak-free": "2.1.2", "pino-abstract-transport": "3.0.0", "pino-std-serializers": "7.1.0", "process-warning": "5.0.0", "quick-format-unescaped": "4.0.4", "real-require": "0.2.0", "safe-stable-stringify": "2.5.0", "sonic-boom": "4.2.1", "thread-stream": "4.0.0" }, "bin": { "pino": "bin.js" } }, "sha512-r34yH/GlQpKZbU1BvFFqOjhISRo1MNx1tWYsYvmj6KIRHSPMT2+yHOEb1SG6NMvRoHRF0a07kCOox/9yakl1vg=="],
"pino-abstract-transport": ["pino-abstract-transport@3.0.0", "", { "dependencies": { "split2": "4.2.0" } }, "sha512-wlfUczU+n7Hy/Ha5j9a/gZNy7We5+cXp8YL+X+PG8S0KXxw7n/JXA3c46Y0zQznIJ83URJiwy7Lh56WLokNuxg=="],
"pino-std-serializers": ["pino-std-serializers@7.1.0", "", {}, "sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw=="],
"pkce-challenge": ["pkce-challenge@5.0.1", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="], "pkce-challenge": ["pkce-challenge@5.0.1", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="],
"playwright": ["playwright@1.60.0", "", { "dependencies": { "playwright-core": "1.60.0" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA=="], "playwright": ["playwright@1.60.0", "", { "dependencies": { "playwright-core": "1.60.0" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA=="],
@ -1111,8 +1052,6 @@
"preact": ["preact@10.29.1", "", {}, "sha512-gQCLc/vWroE8lIpleXtdJhTFDogTdZG9AjMUpVkDf2iTCNwYNWA+u16dL41TqUDJO4gm2IgrcMv3uTpjd4Pwmg=="], "preact": ["preact@10.29.1", "", {}, "sha512-gQCLc/vWroE8lIpleXtdJhTFDogTdZG9AjMUpVkDf2iTCNwYNWA+u16dL41TqUDJO4gm2IgrcMv3uTpjd4Pwmg=="],
"process-warning": ["process-warning@5.0.0", "", {}, "sha512-a39t9ApHNx2L4+HBnQKqxxHNs1r7KF+Intd8Q/g1bUh6q0WIp9voPXJ/x0j+ZL45KF1pJd9+q2jLIRMfvEshkA=="],
"prop-types": ["prop-types@15.8.1", "", { "dependencies": { "loose-envify": "1.4.0", "object-assign": "4.1.1", "react-is": "16.13.1" } }, "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg=="], "prop-types": ["prop-types@15.8.1", "", { "dependencies": { "loose-envify": "1.4.0", "object-assign": "4.1.1", "react-is": "16.13.1" } }, "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg=="],
"property-information": ["property-information@7.1.0", "", {}, "sha512-TwEZ+X+yCJmYfL7TPUOcvBZ4QfoT5YenQiJuX//0th53DE6w0xxLEtfK3iyryQFddXuvkIk51EEgrJQ0WJkOmQ=="], "property-information": ["property-information@7.1.0", "", {}, "sha512-TwEZ+X+yCJmYfL7TPUOcvBZ4QfoT5YenQiJuX//0th53DE6w0xxLEtfK3iyryQFddXuvkIk51EEgrJQ0WJkOmQ=="],
@ -1123,8 +1062,6 @@
"qs": ["qs@6.15.0", "", { "dependencies": { "side-channel": "1.1.0" } }, "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ=="], "qs": ["qs@6.15.0", "", { "dependencies": { "side-channel": "1.1.0" } }, "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ=="],
"quick-format-unescaped": ["quick-format-unescaped@4.0.4", "", {}, "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg=="],
"range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="], "range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],
"raw-body": ["raw-body@3.0.2", "", { "dependencies": { "bytes": "3.1.2", "http-errors": "2.0.1", "iconv-lite": "0.7.2", "unpipe": "1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="], "raw-body": ["raw-body@3.0.2", "", { "dependencies": { "bytes": "3.1.2", "http-errors": "2.0.1", "iconv-lite": "0.7.2", "unpipe": "1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="],
@ -1147,10 +1084,6 @@
"react-router": ["react-router@7.14.0", "", { "dependencies": { "cookie": "1.1.1", "set-cookie-parser": "2.7.2" }, "optionalDependencies": { "react-dom": "19.2.4" }, "peerDependencies": { "react": "19.2.4" } }, "sha512-m/xR9N4LQLmAS0ZhkY2nkPA1N7gQ5TUVa5n8TgANuDTARbn1gt+zLPXEm7W0XDTbrQ2AJSJKhoa6yx1D8BcpxQ=="], "react-router": ["react-router@7.14.0", "", { "dependencies": { "cookie": "1.1.1", "set-cookie-parser": "2.7.2" }, "optionalDependencies": { "react-dom": "19.2.4" }, "peerDependencies": { "react": "19.2.4" } }, "sha512-m/xR9N4LQLmAS0ZhkY2nkPA1N7gQ5TUVa5n8TgANuDTARbn1gt+zLPXEm7W0XDTbrQ2AJSJKhoa6yx1D8BcpxQ=="],
"readable-stream": ["readable-stream@3.6.2", "", { "dependencies": { "inherits": "2.0.4", "string_decoder": "1.3.0", "util-deprecate": "1.0.2" } }, "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA=="],
"real-require": ["real-require@0.2.0", "", {}, "sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg=="],
"redis-errors": ["redis-errors@1.2.0", "", {}, "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w=="], "redis-errors": ["redis-errors@1.2.0", "", {}, "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w=="],
"redis-parser": ["redis-parser@3.0.0", "", { "dependencies": { "redis-errors": "^1.0.0" } }, "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A=="], "redis-parser": ["redis-parser@3.0.0", "", { "dependencies": { "redis-errors": "^1.0.0" } }, "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A=="],
@ -1163,29 +1096,15 @@
"resolve-pkg-maps": ["resolve-pkg-maps@1.0.0", "", {}, "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw=="], "resolve-pkg-maps": ["resolve-pkg-maps@1.0.0", "", {}, "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw=="],
"ret": ["ret@0.5.0", "", {}, "sha512-I1XxrZSQ+oErkRR4jYbAyEEu2I0avBvvMM5JN+6EBprOGRCs63ENqZ3vjavq8fBw2+62G5LF5XelKwuJpcvcxw=="],
"reusify": ["reusify@1.1.0", "", {}, "sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw=="],
"rfdc": ["rfdc@1.4.1", "", {}, "sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA=="],
"rollup": ["rollup@4.60.1", "", { "dependencies": { "@types/estree": "1.0.8" }, "optionalDependencies": { "@rollup/rollup-android-arm-eabi": "4.60.1", "@rollup/rollup-android-arm64": "4.60.1", "@rollup/rollup-darwin-arm64": "4.60.1", "@rollup/rollup-darwin-x64": "4.60.1", "@rollup/rollup-freebsd-arm64": "4.60.1", "@rollup/rollup-freebsd-x64": "4.60.1", "@rollup/rollup-linux-arm-gnueabihf": "4.60.1", "@rollup/rollup-linux-arm-musleabihf": "4.60.1", "@rollup/rollup-linux-arm64-gnu": "4.60.1", "@rollup/rollup-linux-arm64-musl": "4.60.1", "@rollup/rollup-linux-loong64-gnu": "4.60.1", "@rollup/rollup-linux-loong64-musl": "4.60.1", "@rollup/rollup-linux-ppc64-gnu": "4.60.1", "@rollup/rollup-linux-ppc64-musl": "4.60.1", "@rollup/rollup-linux-riscv64-gnu": "4.60.1", "@rollup/rollup-linux-riscv64-musl": "4.60.1", "@rollup/rollup-linux-s390x-gnu": "4.60.1", "@rollup/rollup-linux-x64-gnu": "4.60.1", "@rollup/rollup-linux-x64-musl": "4.60.1", "@rollup/rollup-openbsd-x64": "4.60.1", "@rollup/rollup-openharmony-arm64": "4.60.1", "@rollup/rollup-win32-arm64-msvc": "4.60.1", "@rollup/rollup-win32-ia32-msvc": "4.60.1", "@rollup/rollup-win32-x64-gnu": "4.60.1", "@rollup/rollup-win32-x64-msvc": "4.60.1", "fsevents": "2.3.3" }, "bin": { "rollup": "dist/bin/rollup" } }, "sha512-VmtB2rFU/GroZ4oL8+ZqXgSA38O6GR8KSIvWmEFv63pQ0G6KaBH9s07PO8XTXP4vI+3UJUEypOfjkGfmSBBR0w=="], "rollup": ["rollup@4.60.1", "", { "dependencies": { "@types/estree": "1.0.8" }, "optionalDependencies": { "@rollup/rollup-android-arm-eabi": "4.60.1", "@rollup/rollup-android-arm64": "4.60.1", "@rollup/rollup-darwin-arm64": "4.60.1", "@rollup/rollup-darwin-x64": "4.60.1", "@rollup/rollup-freebsd-arm64": "4.60.1", "@rollup/rollup-freebsd-x64": "4.60.1", "@rollup/rollup-linux-arm-gnueabihf": "4.60.1", "@rollup/rollup-linux-arm-musleabihf": "4.60.1", "@rollup/rollup-linux-arm64-gnu": "4.60.1", "@rollup/rollup-linux-arm64-musl": "4.60.1", "@rollup/rollup-linux-loong64-gnu": "4.60.1", "@rollup/rollup-linux-loong64-musl": "4.60.1", "@rollup/rollup-linux-ppc64-gnu": "4.60.1", "@rollup/rollup-linux-ppc64-musl": "4.60.1", "@rollup/rollup-linux-riscv64-gnu": "4.60.1", "@rollup/rollup-linux-riscv64-musl": "4.60.1", "@rollup/rollup-linux-s390x-gnu": "4.60.1", "@rollup/rollup-linux-x64-gnu": "4.60.1", "@rollup/rollup-linux-x64-musl": "4.60.1", "@rollup/rollup-openbsd-x64": "4.60.1", "@rollup/rollup-openharmony-arm64": "4.60.1", "@rollup/rollup-win32-arm64-msvc": "4.60.1", "@rollup/rollup-win32-ia32-msvc": "4.60.1", "@rollup/rollup-win32-x64-gnu": "4.60.1", "@rollup/rollup-win32-x64-msvc": "4.60.1", "fsevents": "2.3.3" }, "bin": { "rollup": "dist/bin/rollup" } }, "sha512-VmtB2rFU/GroZ4oL8+ZqXgSA38O6GR8KSIvWmEFv63pQ0G6KaBH9s07PO8XTXP4vI+3UJUEypOfjkGfmSBBR0w=="],
"router": ["router@2.2.0", "", { "dependencies": { "debug": "4.4.3", "depd": "2.0.0", "is-promise": "4.0.0", "parseurl": "1.3.3", "path-to-regexp": "8.4.2" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="], "router": ["router@2.2.0", "", { "dependencies": { "debug": "4.4.3", "depd": "2.0.0", "is-promise": "4.0.0", "parseurl": "1.3.3", "path-to-regexp": "8.4.2" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="],
"safe-buffer": ["safe-buffer@5.2.1", "", {}, "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ=="],
"safe-regex2": ["safe-regex2@5.1.0", "", { "dependencies": { "ret": "0.5.0" }, "bin": { "safe-regex2": "bin/safe-regex2.js" } }, "sha512-pNHAuBW7TrcleFHsxBr5QMi/Iyp0ENjUKz7GCcX1UO7cMh+NmVK6HxQckNL1tJp1XAJVjG6B8OKIPqodqj9rtw=="],
"safe-stable-stringify": ["safe-stable-stringify@2.5.0", "", {}, "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA=="],
"safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="], "safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="],
"scheduler": ["scheduler@0.27.0", "", {}, "sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q=="], "scheduler": ["scheduler@0.27.0", "", {}, "sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q=="],
"secure-json-parse": ["secure-json-parse@4.1.0", "", {}, "sha512-l4KnYfEyqYJxDwlNVyRfO2E4NTHfMKAWdUuA8J0yve2Dz/E/PdBepY03RvyJpssIpRFwJoCD55wA+mEDs6ByWA=="], "semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="],
"semver": ["semver@7.7.4", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA=="],
"send": ["send@1.2.1", "", { "dependencies": { "debug": "4.4.3", "encodeurl": "2.0.0", "escape-html": "1.0.3", "etag": "1.8.1", "fresh": "2.0.0", "http-errors": "2.0.1", "mime-types": "3.0.2", "ms": "2.1.3", "on-finished": "2.4.1", "range-parser": "1.2.1", "statuses": "2.0.2" } }, "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ=="], "send": ["send@1.2.1", "", { "dependencies": { "debug": "4.4.3", "encodeurl": "2.0.0", "escape-html": "1.0.3", "etag": "1.8.1", "fresh": "2.0.0", "http-errors": "2.0.1", "mime-types": "3.0.2", "ms": "2.1.3", "on-finished": "2.4.1", "range-parser": "1.2.1", "statuses": "2.0.2" } }, "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ=="],
@ -1209,14 +1128,10 @@
"siginfo": ["siginfo@2.0.0", "", {}, "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g=="], "siginfo": ["siginfo@2.0.0", "", {}, "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g=="],
"sonic-boom": ["sonic-boom@4.2.1", "", { "dependencies": { "atomic-sleep": "1.0.0" } }, "sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q=="],
"source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="], "source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
"space-separated-tokens": ["space-separated-tokens@2.0.2", "", {}, "sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q=="], "space-separated-tokens": ["space-separated-tokens@2.0.2", "", {}, "sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q=="],
"split2": ["split2@4.2.0", "", {}, "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg=="],
"stackback": ["stackback@0.0.2", "", {}, "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw=="], "stackback": ["stackback@0.0.2", "", {}, "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw=="],
"standard-as-callback": ["standard-as-callback@2.1.0", "", {}, "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A=="], "standard-as-callback": ["standard-as-callback@2.1.0", "", {}, "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A=="],
@ -1225,10 +1140,6 @@
"std-env": ["std-env@4.1.0", "", {}, "sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ=="], "std-env": ["std-env@4.1.0", "", {}, "sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ=="],
"stream-shift": ["stream-shift@1.0.3", "", {}, "sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ=="],
"string_decoder": ["string_decoder@1.3.0", "", { "dependencies": { "safe-buffer": "5.2.1" } }, "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA=="],
"stringify-entities": ["stringify-entities@4.0.4", "", { "dependencies": { "character-entities-html4": "2.1.0", "character-entities-legacy": "3.0.0" } }, "sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg=="], "stringify-entities": ["stringify-entities@4.0.4", "", { "dependencies": { "character-entities-html4": "2.1.0", "character-entities-legacy": "3.0.0" } }, "sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg=="],
"style-to-js": ["style-to-js@1.1.21", "", { "dependencies": { "style-to-object": "1.0.14" } }, "sha512-RjQetxJrrUJLQPHbLku6U/ocGtzyjbJMP9lCNK7Ag0CNh690nSH8woqWH9u16nMjYBAok+i7JO1NP2pOy8IsPQ=="], "style-to-js": ["style-to-js@1.1.21", "", { "dependencies": { "style-to-object": "1.0.14" } }, "sha512-RjQetxJrrUJLQPHbLku6U/ocGtzyjbJMP9lCNK7Ag0CNh690nSH8woqWH9u16nMjYBAok+i7JO1NP2pOy8IsPQ=="],
@ -1241,8 +1152,6 @@
"tapable": ["tapable@2.3.2", "", {}, "sha512-1MOpMXuhGzGL5TTCZFItxCc0AARf1EZFQkGqMm7ERKj8+Hgr5oLvJOVFcC+lRmR8hCe2S3jC4T5D7Vg/d7/fhA=="], "tapable": ["tapable@2.3.2", "", {}, "sha512-1MOpMXuhGzGL5TTCZFItxCc0AARf1EZFQkGqMm7ERKj8+Hgr5oLvJOVFcC+lRmR8hCe2S3jC4T5D7Vg/d7/fhA=="],
"thread-stream": ["thread-stream@4.0.0", "", { "dependencies": { "real-require": "0.2.0" } }, "sha512-4iMVL6HAINXWf1ZKZjIPcz5wYaOdPhtO8ATvZ+Xqp3BTdaqtAwQkNmKORqcIo5YkQqGXq5cwfswDwMqqQNrpJA=="],
"tinybench": ["tinybench@2.9.0", "", {}, "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg=="], "tinybench": ["tinybench@2.9.0", "", {}, "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg=="],
"tinycolor2": ["tinycolor2@1.6.0", "", {}, "sha512-XPaBkWQJdsf3pLKJV9p4qN/S+fm2Oj8AIPo1BTUhg5oxkvm9+SVEGFdhyOz7tTdUTfvxMiAs4sp6/eZO2Ew+pw=="], "tinycolor2": ["tinycolor2@1.6.0", "", {}, "sha512-XPaBkWQJdsf3pLKJV9p4qN/S+fm2Oj8AIPo1BTUhg5oxkvm9+SVEGFdhyOz7tTdUTfvxMiAs4sp6/eZO2Ew+pw=="],
@ -1253,8 +1162,6 @@
"tinyrainbow": ["tinyrainbow@3.1.0", "", {}, "sha512-Bf+ILmBgretUrdJxzXM0SgXLZ3XfiaUuOj/IKQHuTXip+05Xn+uyEYdVg0kYDipTBcLrCVyUzAPz7QmArb0mmw=="], "tinyrainbow": ["tinyrainbow@3.1.0", "", {}, "sha512-Bf+ILmBgretUrdJxzXM0SgXLZ3XfiaUuOj/IKQHuTXip+05Xn+uyEYdVg0kYDipTBcLrCVyUzAPz7QmArb0mmw=="],
"toad-cache": ["toad-cache@3.7.0", "", {}, "sha512-/m8M+2BJUpoJdgAHoG+baCwBT+tf2VraSfkBgl0Y00qIWt41DJ8R5B8nsEw0I58YwF5IZH6z24/2TobDKnqSWw=="],
"toidentifier": ["toidentifier@1.0.1", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="], "toidentifier": ["toidentifier@1.0.1", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="],
"toml": ["toml@4.1.1", "", {}, "sha512-EBJnVBr3dTXdA89WVFoAIPUqkBjxPMwRqsfuo1r240tKFHXv3zgca4+NJib/h6TyvGF7vOawz0jGuryJCdNHrw=="], "toml": ["toml@4.1.1", "", {}, "sha512-EBJnVBr3dTXdA89WVFoAIPUqkBjxPMwRqsfuo1r240tKFHXv3zgca4+NJib/h6TyvGF7vOawz0jGuryJCdNHrw=="],
@ -1297,8 +1204,6 @@
"update-browserslist-db": ["update-browserslist-db@1.2.3", "", { "dependencies": { "escalade": "3.2.0", "picocolors": "1.1.1" }, "peerDependencies": { "browserslist": "4.28.2" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w=="], "update-browserslist-db": ["update-browserslist-db@1.2.3", "", { "dependencies": { "escalade": "3.2.0", "picocolors": "1.1.1" }, "peerDependencies": { "browserslist": "4.28.2" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w=="],
"util-deprecate": ["util-deprecate@1.0.2", "", {}, "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw=="],
"uuid": ["uuid@14.0.0", "", { "bin": { "uuid": "dist-node/bin/uuid" } }, "sha512-Qo+uWgilfSmAhXCMav1uYFynlQO7fMFiMVZsQqZRMIXp0O7rR7qjkj+cPvBHLgBqi960QCoo/PH2/6ZtVqKvrg=="], "uuid": ["uuid@14.0.0", "", { "bin": { "uuid": "dist-node/bin/uuid" } }, "sha512-Qo+uWgilfSmAhXCMav1uYFynlQO7fMFiMVZsQqZRMIXp0O7rR7qjkj+cPvBHLgBqi960QCoo/PH2/6ZtVqKvrg=="],
"vary": ["vary@1.1.2", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="], "vary": ["vary@1.1.2", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="],
@ -1341,10 +1246,6 @@
"zwitch": ["zwitch@2.0.4", "", {}, "sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A=="], "zwitch": ["zwitch@2.0.4", "", {}, "sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A=="],
"@babel/core/semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="],
"@babel/helper-compilation-targets/semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="],
"@qdrant/js-client-rest/undici": ["undici@6.24.1", "", {}, "sha512-sC+b0tB1whOCzbtlx20fx3WgCXwkW627p4EA9uM+/tNNPkSS+eSEld6pAs9nDv7WbY1UUljBMYPtu9BCOrCWKA=="], "@qdrant/js-client-rest/undici": ["undici@6.24.1", "", {}, "sha512-sC+b0tB1whOCzbtlx20fx3WgCXwkW627p4EA9uM+/tNNPkSS+eSEld6pAs9nDv7WbY1UUljBMYPtu9BCOrCWKA=="],
"@trustgraph/base/@types/node": ["@types/node@22.19.17", "", { "dependencies": { "undici-types": "6.21.0" } }, "sha512-wGdMcf+vPYM6jikpS/qhg6WiqSV/OhG+jeeHT/KlVqxYfD40iYJf9/AE1uQxVWFvU7MipKRkRv8NSHiCGgPr8Q=="], "@trustgraph/base/@types/node": ["@types/node@22.19.17", "", { "dependencies": { "undici-types": "6.21.0" } }, "sha512-wGdMcf+vPYM6jikpS/qhg6WiqSV/OhG+jeeHT/KlVqxYfD40iYJf9/AE1uQxVWFvU7MipKRkRv8NSHiCGgPr8Q=="],
@ -1367,8 +1268,6 @@
"ioredis/cluster-key-slot": ["cluster-key-slot@1.1.1", "", {}, "sha512-rwHwUfXL40Chm1r08yrhU3qpUvdVlgkKNeyeGPOxnW8/SyVDvgRaed/Uz54AqWNaTCAThlj6QAs3TZcKI0xDEw=="], "ioredis/cluster-key-slot": ["cluster-key-slot@1.1.1", "", {}, "sha512-rwHwUfXL40Chm1r08yrhU3qpUvdVlgkKNeyeGPOxnW8/SyVDvgRaed/Uz54AqWNaTCAThlj6QAs3TZcKI0xDEw=="],
"light-my-request/process-warning": ["process-warning@4.0.1", "", {}, "sha512-3c2LzQ3rY9d0hc1emcsHhfT9Jwz0cChib/QN89oME2R451w5fy3f0afAhERFZAwrbDU43wk12d0ORBpDVME50Q=="],
"lru-cache/yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="], "lru-cache/yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "5.26.5" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="], "openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "5.26.5" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
@ -1377,8 +1276,6 @@
"playwright/fsevents": ["fsevents@2.3.2", "", { "os": "darwin" }, "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA=="], "playwright/fsevents": ["fsevents@2.3.2", "", { "os": "darwin" }, "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA=="],
"proxy-addr/ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
"vite/@types/node": ["@types/node@22.19.17", "", { "dependencies": { "undici-types": "6.21.0" } }, "sha512-wGdMcf+vPYM6jikpS/qhg6WiqSV/OhG+jeeHT/KlVqxYfD40iYJf9/AE1uQxVWFvU7MipKRkRv8NSHiCGgPr8Q=="], "vite/@types/node": ["@types/node@22.19.17", "", { "dependencies": { "undici-types": "6.21.0" } }, "sha512-wGdMcf+vPYM6jikpS/qhg6WiqSV/OhG+jeeHT/KlVqxYfD40iYJf9/AE1uQxVWFvU7MipKRkRv8NSHiCGgPr8Q=="],
"vite/esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="], "vite/esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="],

8
ts/package.json
View file

@ -44,9 +44,9 @@
"llm:mistral": "bun scripts/run-llm-mistral.ts" "llm:mistral": "bun scripts/run-llm-mistral.ts"
}, },
"devDependencies": { "devDependencies": {
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/bun": "^1.3.13", "@types/bun": "^1.3.13",
"@types/node": "^25.7.0", "@types/node": "^25.7.0",
"@typescript/native-preview": "7.0.0-dev.20260511.1", "@typescript/native-preview": "7.0.0-dev.20260511.1",
@ -60,7 +60,7 @@
"vitest": "^4.1.6" "vitest": "^4.1.6"
}, },
"dependencies": { "dependencies": {
"effect": "4.0.0-beta.75" "effect": "4.0.0-beta.78"
}, },
"packageManager": "bun@1.3.13", "packageManager": "bun@1.3.13",
"workspaces": [ "workspaces": [

20
ts/packages/base/package.json
View file

@ -20,19 +20,19 @@
"test": "bunx --bun vitest run" "test": "bunx --bun vitest run"
}, },
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"effect": "4.0.0-beta.75", "effect": "4.0.0-beta.78",
"nats": "^2.29.0" "nats": "^2.29.0"
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6" "vitest": "^4.1.6"

128
ts/packages/base/src/__tests__/consumer.test.ts
View file

@ -1,4 +1,5 @@
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { Effect } from "effect";
import { makeConsumer, type ConsumerOptions, type FlowContext } from "../messaging/consumer.js"; import { makeConsumer, type ConsumerOptions, type FlowContext } from "../messaging/consumer.js";
import type { import type {
PubSubBackend, PubSubBackend,
@ -25,14 +26,17 @@ function createMockBackendConsumer<T>(): BackendConsumer<T> & {
acknowledge: ReturnType<typeof vi.fn>; acknowledge: ReturnType<typeof vi.fn>;
negativeAcknowledge: ReturnType<typeof vi.fn>; negativeAcknowledge: ReturnType<typeof vi.fn>;
unsubscribe: ReturnType<typeof vi.fn>; unsubscribe: ReturnType<typeof vi.fn>;
close: ReturnType<typeof vi.fn>; close: Effect.Effect<void>;
closeMock: ReturnType<typeof vi.fn>;
} { } {
const closeMock = vi.fn();
return { return {
receive: vi.fn().mockResolvedValue(null), receive: vi.fn().mockReturnValue(Effect.succeed(null)),
acknowledge: vi.fn().mockResolvedValue(undefined), acknowledge: vi.fn().mockReturnValue(Effect.void),
negativeAcknowledge: vi.fn().mockResolvedValue(undefined), negativeAcknowledge: vi.fn().mockReturnValue(Effect.void),
unsubscribe: vi.fn().mockResolvedValue(undefined), unsubscribe: vi.fn().mockReturnValue(Effect.void),
close: vi.fn().mockResolvedValue(undefined), close: Effect.sync(closeMock),
closeMock,
}; };
} }
@ -41,9 +45,9 @@ function createMockPubSub<T>(
backendConsumer: BackendConsumer<T>, backendConsumer: BackendConsumer<T>,
): PubSubBackend { ): PubSubBackend {
return { return {
createProducer: vi.fn().mockResolvedValue({} as BackendProducer<unknown>), createProducer: vi.fn().mockReturnValue(Effect.succeed({} as BackendProducer<unknown>)),
createConsumer: vi.fn().mockResolvedValue(backendConsumer), createConsumer: vi.fn().mockReturnValue(Effect.succeed(backendConsumer)),
close: vi.fn().mockResolvedValue(undefined), close: Effect.void,
}; };
} }
@ -94,7 +98,7 @@ describe("Consumer", () => {
expect(consumer).toMatchObject({ expect(consumer).toMatchObject({
start: expect.any(Function), start: expect.any(Function),
stop: expect.any(Function), stop: expect.any(Object),
}); });
}); });
@ -111,13 +115,13 @@ describe("Consumer", () => {
expect(consumer).toMatchObject({ expect(consumer).toMatchObject({
start: expect.any(Function), start: expect.any(Function),
stop: expect.any(Function), stop: expect.any(Object),
}); });
}); });
// ── start() creates consumer and calls handler ───────────────── // ── start() creates consumer and calls handler ─────────────────
it("starts a scoped consumer and invokes handler for received messages", async () => { it("starts a scoped consumer and invokes handler for received messages", async () => {
const handler = vi.fn().mockResolvedValue(undefined); const handler = vi.fn().mockReturnValue(Effect.void);
const msg = createMockMessage({ data: "hello" }, { id: "1" }); const msg = createMockMessage({ data: "hello" }, { id: "1" });
const consumer = makeConsumer({ const consumer = makeConsumer({
@ -127,11 +131,11 @@ describe("Consumer", () => {
handler, handler,
}); });
backendConsumer.receive.mockResolvedValueOnce(msg).mockResolvedValue(null); backendConsumer.receive.mockReturnValueOnce(Effect.succeed(msg)).mockReturnValue(Effect.succeed(null));
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await advanceUntil(() => handler.mock.calls.length > 0); await advanceUntil(() => handler.mock.calls.length > 0);
await consumer.stop(); await Effect.runPromise(consumer.stop);
expect(pubsub.createConsumer).toHaveBeenCalledWith({ expect(pubsub.createConsumer).toHaveBeenCalledWith({
topic: "topic-a", topic: "topic-a",
@ -143,7 +147,7 @@ describe("Consumer", () => {
// ── Messages are acknowledged after successful handling ──────── // ── Messages are acknowledged after successful handling ────────
it("acknowledges messages after successful handling", async () => { it("acknowledges messages after successful handling", async () => {
const handler = vi.fn().mockResolvedValue(undefined); const handler = vi.fn().mockReturnValue(Effect.void);
const msg = createMockMessage("payload"); const msg = createMockMessage("payload");
const consumer = makeConsumer({ const consumer = makeConsumer({
@ -153,11 +157,11 @@ describe("Consumer", () => {
handler, handler,
}); });
backendConsumer.receive.mockResolvedValueOnce(msg).mockResolvedValue(null); backendConsumer.receive.mockReturnValueOnce(Effect.succeed(msg)).mockReturnValue(Effect.succeed(null));
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await advanceUntil(() => backendConsumer.acknowledge.mock.calls.length > 0); await advanceUntil(() => backendConsumer.acknowledge.mock.calls.length > 0);
await consumer.stop(); await Effect.runPromise(consumer.stop);
expect(backendConsumer.acknowledge).toHaveBeenCalledWith(msg); expect(backendConsumer.acknowledge).toHaveBeenCalledWith(msg);
expect(backendConsumer.negativeAcknowledge).not.toHaveBeenCalled(); expect(backendConsumer.negativeAcknowledge).not.toHaveBeenCalled();
@ -165,7 +169,7 @@ describe("Consumer", () => {
// ── Messages are negatively acknowledged on handler error ────── // ── Messages are negatively acknowledged on handler error ──────
it("negatively acknowledges messages when the handler throws", async () => { it("negatively acknowledges messages when the handler throws", async () => {
const handler = vi.fn().mockRejectedValue("handler boom"); const handler = vi.fn().mockReturnValue(Effect.fail("handler boom"));
const msg = createMockMessage("bad-payload"); const msg = createMockMessage("bad-payload");
const consumer = makeConsumer({ const consumer = makeConsumer({
@ -175,11 +179,11 @@ describe("Consumer", () => {
handler, handler,
}); });
backendConsumer.receive.mockResolvedValueOnce(msg).mockResolvedValue(null); backendConsumer.receive.mockReturnValueOnce(Effect.succeed(msg)).mockReturnValue(Effect.succeed(null));
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await advanceUntil(() => backendConsumer.negativeAcknowledge.mock.calls.length > 0); await advanceUntil(() => backendConsumer.negativeAcknowledge.mock.calls.length > 0);
await consumer.stop(); await Effect.runPromise(consumer.stop);
expect(backendConsumer.negativeAcknowledge).toHaveBeenCalledWith(msg); expect(backendConsumer.negativeAcknowledge).toHaveBeenCalledWith(msg);
expect(backendConsumer.acknowledge).not.toHaveBeenCalled(); expect(backendConsumer.acknowledge).not.toHaveBeenCalled();
@ -188,12 +192,17 @@ describe("Consumer", () => {
// ── TooManyRequestsError triggers retry ──────────────────────── // ── TooManyRequestsError triggers retry ────────────────────────
it("retries the handler on TooManyRequestsError", async () => { it("retries the handler on TooManyRequestsError", async () => {
let handlerCalls = 0; let handlerCalls = 0;
const handler = vi.fn().mockImplementation(async () => { const handler = vi.fn().mockImplementation(() => {
handlerCalls++; return Effect.sync(() => {
if (handlerCalls === 1) { handlerCalls++;
throw tooManyRequestsError("rate limited"); return handlerCalls;
} }).pipe(
// Second call succeeds Effect.flatMap((attempt) =>
attempt === 1
? Effect.fail(tooManyRequestsError("rate limited"))
: Effect.void
),
);
}); });
const msg = createMockMessage("rate-limited-payload"); const msg = createMockMessage("rate-limited-payload");
@ -206,17 +215,17 @@ describe("Consumer", () => {
rateLimitRetryMs: 500, rateLimitRetryMs: 500,
}); });
backendConsumer.receive.mockResolvedValueOnce(msg).mockResolvedValue(null); backendConsumer.receive.mockReturnValueOnce(Effect.succeed(msg)).mockReturnValue(Effect.succeed(null));
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {}); const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await vi.advanceTimersByTimeAsync(600); await vi.advanceTimersByTimeAsync(600);
await advanceUntil(() => handler.mock.calls.length >= 2); await advanceUntil(() => handler.mock.calls.length >= 2);
await consumer.stop(); await Effect.runPromise(consumer.stop);
// Handler called twice: first throws TooManyRequestsError, second succeeds // Handler called twice: first throws TooManyRequestsError, second succeeds
expect(handler).toHaveBeenCalledTimes(2); expect(handlerCalls).toBe(2);
// Message should be acknowledged (retry succeeded) // Message should be acknowledged (retry succeeded)
expect(backendConsumer.acknowledge).toHaveBeenCalledWith(msg); expect(backendConsumer.acknowledge).toHaveBeenCalledWith(msg);
@ -225,11 +234,17 @@ describe("Consumer", () => {
it("retries repeated TooManyRequestsError until success within the timeout", async () => { it("retries repeated TooManyRequestsError until success within the timeout", async () => {
let handlerCalls = 0; let handlerCalls = 0;
const handler = vi.fn().mockImplementation(async () => { const handler = vi.fn().mockImplementation(() => {
handlerCalls++; return Effect.sync(() => {
if (handlerCalls <= 2) { handlerCalls++;
throw tooManyRequestsError("rate limited"); return handlerCalls;
} }).pipe(
Effect.flatMap((attempt) =>
attempt <= 2
? Effect.fail(tooManyRequestsError("rate limited"))
: Effect.void
),
);
}); });
const msg = createMockMessage("rate-limited-payload"); const msg = createMockMessage("rate-limited-payload");
@ -243,22 +258,27 @@ describe("Consumer", () => {
rateLimitTimeoutMs: 2_000, rateLimitTimeoutMs: 2_000,
}); });
backendConsumer.receive.mockResolvedValueOnce(msg).mockResolvedValue(null); backendConsumer.receive.mockReturnValueOnce(Effect.succeed(msg)).mockReturnValue(Effect.succeed(null));
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await vi.advanceTimersByTimeAsync(1_100); await vi.advanceTimersByTimeAsync(1_100);
await advanceUntil(() => backendConsumer.acknowledge.mock.calls.length > 0); await advanceUntil(() => backendConsumer.acknowledge.mock.calls.length > 0);
await consumer.stop(); await Effect.runPromise(consumer.stop);
expect(handler).toHaveBeenCalledTimes(3); expect(handlerCalls).toBe(3);
expect(backendConsumer.acknowledge).toHaveBeenCalledWith(msg); expect(backendConsumer.acknowledge).toHaveBeenCalledWith(msg);
expect(backendConsumer.negativeAcknowledge).not.toHaveBeenCalled(); expect(backendConsumer.negativeAcknowledge).not.toHaveBeenCalled();
}); });
it("negatively acknowledges when rate-limit retry timeout elapses", async () => { it("negatively acknowledges when rate-limit retry timeout elapses", async () => {
const handler = vi.fn().mockImplementation(async () => { let handlerCalls = 0;
throw tooManyRequestsError("rate limited"); const handler = vi.fn().mockReturnValue(
}); Effect.sync(() => {
handlerCalls++;
}).pipe(
Effect.flatMap(() => Effect.fail(tooManyRequestsError("rate limited"))),
),
);
const msg = createMockMessage("rate-limited-payload"); const msg = createMockMessage("rate-limited-payload");
const consumer = makeConsumer({ const consumer = makeConsumer({
@ -270,21 +290,21 @@ describe("Consumer", () => {
rateLimitTimeoutMs: 1_000, rateLimitTimeoutMs: 1_000,
}); });
backendConsumer.receive.mockResolvedValueOnce(msg).mockResolvedValue(null); backendConsumer.receive.mockReturnValueOnce(Effect.succeed(msg)).mockReturnValue(Effect.succeed(null));
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await vi.advanceTimersByTimeAsync(1_100); await vi.advanceTimersByTimeAsync(1_100);
await advanceUntil(() => backendConsumer.negativeAcknowledge.mock.calls.length > 0); await advanceUntil(() => backendConsumer.negativeAcknowledge.mock.calls.length > 0);
await consumer.stop(); await Effect.runPromise(consumer.stop);
expect(handler).toHaveBeenCalledTimes(2); expect(handlerCalls).toBeGreaterThanOrEqual(2);
expect(backendConsumer.negativeAcknowledge).toHaveBeenCalledWith(msg); expect(backendConsumer.negativeAcknowledge).toHaveBeenCalledWith(msg);
expect(backendConsumer.acknowledge).not.toHaveBeenCalled(); expect(backendConsumer.acknowledge).not.toHaveBeenCalled();
}); });
// ── stop() closes the backend ────────────────────────────────── // ── stop() closes the backend ──────────────────────────────────
it("stop() sets running=false and closes the backend", async () => { it("stop() sets running=false and closes the backend", async () => {
backendConsumer.receive.mockResolvedValue(null); backendConsumer.receive.mockReturnValue(Effect.succeed(null));
const consumer = makeConsumer({ const consumer = makeConsumer({
pubsub, pubsub,
@ -293,10 +313,10 @@ describe("Consumer", () => {
handler: vi.fn(), handler: vi.fn(),
}); });
await consumer.start(flowCtx); await Effect.runPromise(consumer.start(flowCtx));
await consumer.stop(); await Effect.runPromise(consumer.stop);
expect(backendConsumer.close).toHaveBeenCalled(); expect(backendConsumer.closeMock).toHaveBeenCalled();
await expect(consumer.stop()).resolves.toBeUndefined(); await expect(Effect.runPromise(consumer.stop)).resolves.toBeUndefined();
}); });
}); });

74
ts/packages/base/src/__tests__/embeddings-service.test.ts
View file

@ -62,13 +62,15 @@ const waitFor = (condition: () => boolean, label: string) =>
class RecordingProducer<T> implements BackendProducer<T> { class RecordingProducer<T> implements BackendProducer<T> {
readonly sent: Array<{ readonly message: T; readonly properties?: Record<string, string> }> = []; readonly sent: Array<{ readonly message: T; readonly properties?: Record<string, string> }> = [];
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.sent.push(properties === undefined ? { message } : { message, properties });
});
} }
async flush(): Promise<void> {} readonly flush: Effect.Effect<void> = Effect.void;
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
class PushConsumer<T> implements BackendConsumer<T> { class PushConsumer<T> implements BackendConsumer<T> {
@ -87,32 +89,38 @@ class PushConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || this.closed) { const message = this.messages.shift();
return message ?? null; if (message !== undefined || this.closed) {
} return Promise.resolve(message ?? null);
return await new Promise((resolve) => { }
this.waiters.push(resolve); return new Promise((resolve) => {
this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
} });
} }
class EmbeddingsBackend implements PubSubBackend { class EmbeddingsBackend implements PubSubBackend {
@ -121,24 +129,28 @@ class EmbeddingsBackend implements PubSubBackend {
readonly producersByTopic = new Map<string, RecordingProducer<unknown>>(); readonly producersByTopic = new Map<string, RecordingProducer<unknown>>();
closeCount = 0; closeCount = 0;
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
const producer = new RecordingProducer<unknown>(); return Effect.sync(() => {
this.producersByTopic.set(options.topic, producer); const producer = new RecordingProducer<unknown>();
return producer as BackendProducer<T>; this.producersByTopic.set(options.topic, producer);
return producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
if (options.topic === topics.configPush) { return Effect.sync(() => {
return this.configConsumer as unknown as BackendConsumer<T>; if (options.topic === topics.configPush) {
} return this.configConsumer as unknown as BackendConsumer<T>;
const consumer = new PushConsumer<unknown>(); }
this.consumersByTopic.set(options.topic, consumer); const consumer = new PushConsumer<unknown>();
return consumer as BackendConsumer<T>; this.consumersByTopic.set(options.topic, consumer);
return consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
pushConfig(): void { pushConfig(): void {
this.configConsumer.push( this.configConsumer.push(

82
ts/packages/base/src/__tests__/flow-processor-runtime.test.ts
View file

@ -58,17 +58,19 @@ class RecordingProducer<T> implements BackendProducer<T> {
closeCount = 0; closeCount = 0;
flushCount = 0; flushCount = 0;
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.sent.push(properties === undefined ? { message } : { message, properties });
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class PushConsumer<T> implements BackendConsumer<T> { class PushConsumer<T> implements BackendConsumer<T> {
@ -88,33 +90,39 @@ class PushConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || this.closed) { const message = this.messages.shift();
return message ?? null; if (message !== undefined || this.closed) {
} return Promise.resolve(message ?? null);
return await new Promise((resolve) => { }
this.waiters.push(resolve); return new Promise((resolve) => {
this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
this.closeCount += 1; this.closeCount += 1;
} });
} }
class FlowProcessorBackend implements PubSubBackend { class FlowProcessorBackend implements PubSubBackend {
@ -124,24 +132,28 @@ class FlowProcessorBackend implements PubSubBackend {
readonly producers: Array<RecordingProducer<unknown>> = []; readonly producers: Array<RecordingProducer<unknown>> = [];
closeCount = 0; closeCount = 0;
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions.push(options); return Effect.sync(() => {
const producer = new RecordingProducer<unknown>(); this.producerOptions.push(options);
this.producers.push(producer); const producer = new RecordingProducer<unknown>();
return producer as BackendProducer<T>; this.producers.push(producer);
return producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions.push(options); return Effect.sync(() => {
if (options.topic === topics.configPush) { this.consumerOptions.push(options);
return this.configConsumer as unknown as BackendConsumer<T>; if (options.topic === topics.configPush) {
} return this.configConsumer as unknown as BackendConsumer<T>;
return new PushConsumer<T>(); }
return new PushConsumer<T>();
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
pushConfig(version: number, flows: Record<string, unknown>): void { pushConfig(version: number, flows: Record<string, unknown>): void {
this.pushFlowConfig(version, flows); this.pushFlowConfig(version, flows);
@ -159,9 +171,9 @@ class TestFlowProcessor extends FlowProcessor {
) { ) {
super(config); super(config);
this.registerSpecification(makeProducerSpec<string>("output")); this.registerSpecification(makeProducerSpec<string>("output"));
this.registerConfigHandler(async (_config, version) => { this.registerConfigHandler((_config, version) => Effect.sync(() => {
this.events.push(`handler:${version}`); this.events.push(`handler:${version}`);
}); }));
} }
} }

92
ts/packages/base/src/__tests__/flow-spec-runtime.test.ts
View file

@ -4,7 +4,6 @@ import * as S from "effect/Schema";
import * as TestClock from "effect/testing/TestClock"; import * as TestClock from "effect/testing/TestClock";
import { import {
makeConsumerSpec, makeConsumerSpec,
makeConsumerSpecFromPromise,
Flow, Flow,
MessagingRuntimeLive, MessagingRuntimeLive,
makeParameterSpec, makeParameterSpec,
@ -34,18 +33,20 @@ class RecordingProducer<T> implements BackendProducer<T> {
constructor(private readonly onSend?: (message: T, properties?: Record<string, string>) => void) {} constructor(private readonly onSend?: (message: T, properties?: Record<string, string>) => void) {}
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.onSend?.(message, properties); this.sent.push(properties === undefined ? { message } : { message, properties });
this.onSend?.(message, properties);
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class ScriptedConsumer<T> implements BackendConsumer<T> { class ScriptedConsumer<T> implements BackendConsumer<T> {
@ -72,33 +73,39 @@ class ScriptedConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || !this.waitForMessages || this.closed) { const message = this.messages.shift();
return message ?? null; if (message !== undefined || !this.waitForMessages || this.closed) {
} return Promise.resolve(message ?? null);
return await new Promise((resolve) => { }
this.waiters.push(resolve); return new Promise((resolve) => {
this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
this.closeCount += 1; this.closeCount += 1;
} });
} }
class RuntimeBackend implements PubSubBackend { class RuntimeBackend implements PubSubBackend {
@ -114,19 +121,23 @@ class RuntimeBackend implements PubSubBackend {
this.producer = new RecordingProducer<unknown>(onSend); this.producer = new RecordingProducer<unknown>(onSend);
} }
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions = options; return Effect.sync(() => {
return this.producer as BackendProducer<T>; this.producerOptions = options;
return this.producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions = options; return Effect.sync(() => {
return this.consumer as BackendConsumer<T>; this.consumerOptions = options;
return this.consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
const fastMessagingConfig = ConfigProvider.layer( const fastMessagingConfig = ConfigProvider.layer(
@ -187,7 +198,7 @@ describe("Effect-native flow specifications", () => {
); );
it.effect( it.effect(
"runs Promise handlers through the explicit makeConsumerSpec compatibility helper", "runs Effect handlers through makeConsumerSpec",
Effect.fnUntraced(function* () { Effect.fnUntraced(function* () {
const message = createMessage("payload", { id: "request-1" }); const message = createMessage("payload", { id: "request-1" });
const consumer = new ScriptedConsumer<string>([message]); const consumer = new ScriptedConsumer<string>([message]);
@ -199,11 +210,11 @@ describe("Effect-native flow specifications", () => {
backend, backend,
{}, {},
[ [
makeConsumerSpecFromPromise<string>( makeConsumerSpec<string>(
"input", "input",
async (value, properties, flowContext: FlowContext) => { (value, properties, flowContext: FlowContext) => Effect.sync(() => {
handled.push(`${flowContext.name}:${properties.id}:${value}`); handled.push(`${flowContext.name}:${properties.id}:${value}`);
}, }),
), ),
], ],
); );
@ -226,7 +237,7 @@ describe("Effect-native flow specifications", () => {
); );
it.effect( it.effect(
"registers request-response specs through Effect queues and keeps the Promise facade working", "registers request-response specs through Effect queues and exposes Effect requestors",
Effect.fnUntraced(function* () { Effect.fnUntraced(function* () {
const responseConsumer = new ScriptedConsumer<string>([], true); const responseConsumer = new ScriptedConsumer<string>([], true);
const backend = new RuntimeBackend( const backend = new RuntimeBackend(
@ -257,10 +268,8 @@ describe("Effect-native flow specifications", () => {
yield* flow.startEffect; yield* flow.startEffect;
const duplicateSpecError = yield* flow.requestorEffect(duplicateRequestResponseSpec).pipe(Effect.flip); const duplicateSpecError = yield* flow.requestorEffect(duplicateRequestResponseSpec).pipe(Effect.flip);
expect(duplicateSpecError._tag).toBe("FlowResourceNotFoundError"); expect(duplicateSpecError._tag).toBe("FlowResourceNotFoundError");
const requestor = flow.requestor(requestResponseSpec); const requestor = yield* flow.requestor(requestResponseSpec);
const fiber = yield* Effect.promise(() => const fiber = yield* requestor.request("request", { timeoutMs: 250 }).pipe(Effect.forkChild);
requestor.request("request", { timeoutMs: 250 }),
).pipe(Effect.forkChild);
yield* TestClock.adjust(Duration.millis(5)); yield* TestClock.adjust(Duration.millis(5));
return yield* Fiber.join(fiber); return yield* Fiber.join(fiber);
}), }),
@ -299,7 +308,8 @@ describe("Effect-native flow specifications", () => {
const legacyParameter = yield* flow.parameterEffect("present"); const legacyParameter = yield* flow.parameterEffect("present");
const parameterError = yield* flow.parameterEffect("missing-parameter").pipe(Effect.flip); const parameterError = yield* flow.parameterEffect("missing-parameter").pipe(Effect.flip);
const invalidParameterError = yield* flow.parameterEffect(invalidParameter).pipe(Effect.flip); const invalidParameterError = yield* flow.parameterEffect(invalidParameter).pipe(Effect.flip);
return { producerError, parameter, legacyParameter, parameterError, invalidParameterError }; const legacyProducerError = yield* flow.producer("missing-producer").pipe(Effect.flip);
return { producerError, legacyProducerError, parameter, legacyParameter, parameterError, invalidParameterError };
}), }),
), ),
); );
@ -313,10 +323,10 @@ describe("Effect-native flow specifications", () => {
expect(errors.parameterError.resourceType).toBe("parameter"); expect(errors.parameterError.resourceType).toBe("parameter");
expect(errors.invalidParameterError._tag).toBe("FlowParameterDecodeError"); expect(errors.invalidParameterError._tag).toBe("FlowParameterDecodeError");
expect(errors.invalidParameterError.parameterName).toBe("present"); expect(errors.invalidParameterError.parameterName).toBe("present");
expect(errors.legacyProducerError._tag).toBe("FlowResourceNotFoundError");
expect(flow.parameter(presentParameter)).toBe(42); expect(flow.parameter(presentParameter)).toBe(42);
expect(flow.parameter("present")).toBe(42); expect(flow.parameter("present")).toBe(42);
expect(() => flow.parameter(invalidParameter)).toThrow("failed schema decoding"); expect(() => flow.parameter(invalidParameter)).toThrow("failed schema decoding");
expect(() => flow.producer("missing-producer")).toThrow("not found");
}), }),
); );
}); });

110
ts/packages/base/src/__tests__/messaging-runtime.test.ts
View file

@ -36,18 +36,20 @@ class RecordingProducer<T> implements BackendProducer<T> {
constructor(private readonly onSend?: (message: T, properties?: Record<string, string>) => void) {} constructor(private readonly onSend?: (message: T, properties?: Record<string, string>) => void) {}
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.onSend?.(message, properties); this.sent.push(properties === undefined ? { message } : { message, properties });
this.onSend?.(message, properties);
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class ScriptedConsumer<T> implements BackendConsumer<T> { class ScriptedConsumer<T> implements BackendConsumer<T> {
@ -64,27 +66,33 @@ class ScriptedConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.sync(() => {
if (message !== undefined) { const message = this.messages.shift();
return message; if (message !== undefined) {
} return message;
return null; }
return null;
});
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class RuntimeBackend implements PubSubBackend { class RuntimeBackend implements PubSubBackend {
@ -100,19 +108,23 @@ class RuntimeBackend implements PubSubBackend {
this.producer = new RecordingProducer<unknown>(onSend); this.producer = new RecordingProducer<unknown>(onSend);
} }
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions = options; return Effect.sync(() => {
return this.producer as BackendProducer<T>; this.producerOptions = options;
return this.producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions = options; return Effect.sync(() => {
return this.consumer as BackendConsumer<T>; this.consumerOptions = options;
return this.consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class ConsumerHandle { class ConsumerHandle {
@ -123,31 +135,33 @@ class ConcurrentConsumerBackend implements PubSubBackend {
readonly consumerOptions: Array<CreateConsumerOptions> = []; readonly consumerOptions: Array<CreateConsumerOptions> = [];
readonly consumers: Array<ConsumerHandle> = []; readonly consumers: Array<ConsumerHandle> = [];
async createProducer<T>(_options: CreateProducerOptions<T>): Promise<BackendProducer<T>> { createProducer<T>(_options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>> {
return { return Effect.succeed({
send: async () => {}, send: () => Effect.void,
flush: async () => {}, flush: Effect.void,
close: async () => {}, close: Effect.void,
}; });
} }
async createConsumer<T>(options: CreateConsumerOptions<T>): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions<T>): Effect.Effect<BackendConsumer<T>> {
const handle = new ConsumerHandle(); return Effect.sync(() => {
this.consumerOptions.push(options); const handle = new ConsumerHandle();
this.consumers.push(handle); this.consumerOptions.push(options);
this.consumers.push(handle);
return { return {
receive: async () => null, receive: () => Effect.succeed(null),
acknowledge: async () => {}, acknowledge: () => Effect.void,
negativeAcknowledge: async () => {}, negativeAcknowledge: () => Effect.void,
unsubscribe: async () => {}, unsubscribe: Effect.void,
close: async () => { close: Effect.sync(() => {
handle.closeCount += 1; handle.closeCount += 1;
}, }),
}; };
});
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
const flowContext: FlowContext = { const flowContext: FlowContext = {

52
ts/packages/base/src/__tests__/nats-backend.test.ts
View file

@ -1,4 +1,5 @@
import { beforeEach, describe, expect, it, vi } from "vitest"; import { beforeEach, describe, expect, it, vi } from "vitest";
import { Effect } from "effect";
import { makeNatsBackend } from "../backend/nats.js"; import { makeNatsBackend } from "../backend/nats.js";
const natsMock = vi.hoisted(() => { const natsMock = vi.hoisted(() => {
@ -34,6 +35,7 @@ const natsMock = vi.hoisted(() => {
const publish = vi.fn(); const publish = vi.fn();
const consumersGet = vi.fn(); const consumersGet = vi.fn();
const consumersInfo = vi.fn();
const consumersAdd = vi.fn(); const consumersAdd = vi.fn();
const streamsInfo = vi.fn(); const streamsInfo = vi.fn();
const streamsAdd = vi.fn(); const streamsAdd = vi.fn();
@ -50,6 +52,7 @@ const natsMock = vi.hoisted(() => {
connect, connect,
consumersAdd, consumersAdd,
consumersGet, consumersGet,
consumersInfo,
decoder, decoder,
drain, drain,
encoder, encoder,
@ -86,6 +89,7 @@ function resetNatsMock(): void {
natsMock.publish.mockResolvedValue({ duplicate: false, seq: 1, stream: "tg_test" }); natsMock.publish.mockResolvedValue({ duplicate: false, seq: 1, stream: "tg_test" });
natsMock.consumersGet.mockResolvedValue({ next: natsMock.next }); natsMock.consumersGet.mockResolvedValue({ next: natsMock.next });
natsMock.consumersInfo.mockResolvedValue({ name: "worker" });
natsMock.consumersAdd.mockResolvedValue(undefined); natsMock.consumersAdd.mockResolvedValue(undefined);
natsMock.streamsInfo.mockResolvedValue({ config: { name: "tg_test" } }); natsMock.streamsInfo.mockResolvedValue({ config: { name: "tg_test" } });
natsMock.streamsAdd.mockResolvedValue(undefined); natsMock.streamsAdd.mockResolvedValue(undefined);
@ -108,7 +112,7 @@ function resetNatsMock(): void {
}), }),
jetstreamManager: () => jetstreamManager: () =>
Promise.resolve({ Promise.resolve({
consumers: { add: natsMock.consumersAdd }, consumers: { add: natsMock.consumersAdd, info: natsMock.consumersInfo },
streams: { streams: {
add: natsMock.streamsAdd, add: natsMock.streamsAdd,
info: natsMock.streamsInfo, info: natsMock.streamsInfo,
@ -126,7 +130,7 @@ describe("NATS backend", () => {
natsMock.streamsInfo.mockRejectedValueOnce(makeNatsError("404", 404)); natsMock.streamsInfo.mockRejectedValueOnce(makeNatsError("404", 404));
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
await backend.createProducer<string>({ topic: "tg.test.topic" }); await Effect.runPromise(backend.createProducer<string>({ topic: "tg.test.topic" }));
expect(natsMock.streamsAdd).toHaveBeenCalledWith({ expect(natsMock.streamsAdd).toHaveBeenCalledWith({
name: "tg_test", name: "tg_test",
@ -137,11 +141,11 @@ describe("NATS backend", () => {
it("caches initialized streams through the Effect mutable set", async () => { it("caches initialized streams through the Effect mutable set", async () => {
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
await backend.createProducer<string>({ topic: "tg.test.topic" }); await Effect.runPromise(backend.createProducer<string>({ topic: "tg.test.topic" }));
await backend.createConsumer<string>({ await Effect.runPromise(backend.createConsumer<string>({
topic: "tg.test.other", topic: "tg.test.other",
subscription: "worker", subscription: "worker",
}); }));
expect(natsMock.streamsInfo).toHaveBeenCalledTimes(1); expect(natsMock.streamsInfo).toHaveBeenCalledTimes(1);
expect(natsMock.streamsInfo).toHaveBeenCalledWith("tg_test"); expect(natsMock.streamsInfo).toHaveBeenCalledWith("tg_test");
@ -151,7 +155,9 @@ describe("NATS backend", () => {
natsMock.streamsInfo.mockRejectedValueOnce(makeNatsError("PERMISSIONS_VIOLATION")); natsMock.streamsInfo.mockRejectedValueOnce(makeNatsError("PERMISSIONS_VIOLATION"));
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
const error = await backend.createProducer<string>({ topic: "tg.test.topic" }).catch((caught: unknown) => caught); const error = await Effect.runPromise(
backend.createProducer<string>({ topic: "tg.test.topic" }),
).catch((caught: unknown) => caught);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "PubSubError", _tag: "PubSubError",
@ -161,15 +167,13 @@ describe("NATS backend", () => {
}); });
it("creates durable consumers only when consumer lookup returns a JetStream 404", async () => { it("creates durable consumers only when consumer lookup returns a JetStream 404", async () => {
natsMock.consumersGet natsMock.consumersInfo.mockRejectedValueOnce(makeNatsError("404", 404));
.mockRejectedValueOnce(makeNatsError("404", 404))
.mockResolvedValueOnce({ next: natsMock.next });
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
await backend.createConsumer<string>({ await Effect.runPromise(backend.createConsumer<string>({
topic: "tg.test.topic", topic: "tg.test.topic",
subscription: "worker", subscription: "worker",
}); }));
expect(natsMock.consumersAdd).toHaveBeenCalledWith("tg_test", { expect(natsMock.consumersAdd).toHaveBeenCalledWith("tg_test", {
ack_policy: "explicit", ack_policy: "explicit",
@ -180,13 +184,13 @@ describe("NATS backend", () => {
}); });
it("does not create durable consumers for non-missing lookup failures", async () => { it("does not create durable consumers for non-missing lookup failures", async () => {
natsMock.consumersGet.mockRejectedValueOnce(makeNatsError("PERMISSIONS_VIOLATION")); natsMock.consumersInfo.mockRejectedValueOnce(makeNatsError("PERMISSIONS_VIOLATION"));
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
const error = await backend.createConsumer<string>({ const error = await Effect.runPromise(backend.createConsumer<string>({
topic: "tg.test.topic", topic: "tg.test.topic",
subscription: "worker", subscription: "worker",
}).catch((caught: unknown) => caught); })).catch((caught: unknown) => caught);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "PubSubError", _tag: "PubSubError",
@ -200,9 +204,11 @@ describe("NATS backend", () => {
throw "invalid header"; throw "invalid header";
}); });
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
const producer = await backend.createProducer<string>({ topic: "tg.test.topic" }); const producer = await Effect.runPromise(backend.createProducer<string>({ topic: "tg.test.topic" }));
const error = await producer.send("hello", { bad: "value" }).catch((caught: unknown) => caught); const error = await Effect.runPromise(
producer.send("hello", { bad: "value" }),
).catch((caught: unknown) => caught);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "PubSubError", _tag: "PubSubError",
@ -219,19 +225,23 @@ describe("NATS backend", () => {
throw "nak failed"; throw "nak failed";
}); });
const backend = makeNatsBackend("nats://test"); const backend = makeNatsBackend("nats://test");
const consumer = await backend.createConsumer<string>({ const consumer = await Effect.runPromise(backend.createConsumer<string>({
topic: "tg.test.topic", topic: "tg.test.topic",
subscription: "worker", subscription: "worker",
}); }));
const message = await consumer.receive(1); const message = await Effect.runPromise(consumer.receive(1));
expect(message).not.toBeNull(); expect(message).not.toBeNull();
if (message === null) { if (message === null) {
return; return;
} }
const ackError = await consumer.acknowledge(message).catch((caught: unknown) => caught); const ackError = await Effect.runPromise(
const nakError = await consumer.negativeAcknowledge(message).catch((caught: unknown) => caught); consumer.acknowledge(message),
).catch((caught: unknown) => caught);
const nakError = await Effect.runPromise(
consumer.negativeAcknowledge(message),
).catch((caught: unknown) => caught);
expect(ackError).toMatchObject({ expect(ackError).toMatchObject({
_tag: "PubSubError", _tag: "PubSubError",

57
ts/packages/base/src/__tests__/producer.test.ts
View file

@ -1,6 +1,8 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { Effect } from "effect";
import { import {
makeProducer, makeProducer,
pubSubError,
type BackendConsumer, type BackendConsumer,
type BackendProducer, type BackendProducer,
type CreateConsumerOptions, type CreateConsumerOptions,
@ -15,30 +17,33 @@ class ProducerBackend implements PubSubBackend {
flushCount = 0; flushCount = 0;
failFlush = false; failFlush = false;
async createProducer<T>(options: CreateProducerOptions<T>): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>> {
this.producerTopics.push(options.topic); return Effect.sync(() => {
this.producerTopics.push(options.topic);
return { return {
send: async (message, properties) => { send: (message, properties) => Effect.sync(() => {
this.sent.push(properties === undefined ? { message } : { message, properties }); this.sent.push(properties === undefined ? { message } : { message, properties });
}, }),
flush: async () => { flush: Effect.suspend(() => {
this.flushCount += 1; this.flushCount += 1;
if (this.failFlush) { if (this.failFlush) {
return Promise.reject("flush failed"); return Effect.fail(pubSubError("flush", "flush failed"));
} }
}, return Effect.void;
close: async () => { }),
this.closeCount += 1; close: Effect.sync(() => {
}, this.closeCount += 1;
}; }),
};
});
} }
createConsumer<T>(_options: CreateConsumerOptions<T>): Promise<BackendConsumer<T>> { createConsumer<T>(_options: CreateConsumerOptions<T>): Effect.Effect<BackendConsumer<T>> {
return Promise.reject("consumer not supported"); return Effect.fail(pubSubError("create-consumer", "consumer not supported"));
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
describe("Producer", () => { describe("Producer", () => {
@ -46,9 +51,9 @@ describe("Producer", () => {
const backend = new ProducerBackend(); const backend = new ProducerBackend();
const producer = makeProducer<string>(backend, "tg.test.producer"); const producer = makeProducer<string>(backend, "tg.test.producer");
await producer.start(); await Effect.runPromise(producer.start);
await producer.send("message-1", "hello"); await Effect.runPromise(producer.send("message-1", "hello"));
await producer.stop(); await Effect.runPromise(producer.stop);
expect(backend.producerTopics).toEqual(["tg.test.producer"]); expect(backend.producerTopics).toEqual(["tg.test.producer"]);
expect(backend.sent).toEqual([ expect(backend.sent).toEqual([
@ -56,9 +61,11 @@ describe("Producer", () => {
]); ]);
expect(backend.flushCount).toBe(1); expect(backend.flushCount).toBe(1);
expect(backend.closeCount).toBe(1); expect(backend.closeCount).toBe(1);
await expect(producer.stop()).resolves.toBeUndefined(); await expect(Effect.runPromise(producer.stop)).resolves.toBeUndefined();
const error = await producer.send("message-2", "late").catch((caught: unknown) => caught); const error = await Effect.runPromise(
producer.send("message-2", "late"),
).catch((caught: unknown) => caught);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "MessagingLifecycleError", _tag: "MessagingLifecycleError",
operation: "send", operation: "send",
@ -70,10 +77,10 @@ describe("Producer", () => {
const backend = new ProducerBackend(); const backend = new ProducerBackend();
const producer = makeProducer<string>(backend, "tg.test.producer"); const producer = makeProducer<string>(backend, "tg.test.producer");
await producer.start(); await Effect.runPromise(producer.start);
backend.failFlush = true; backend.failFlush = true;
const error = await producer.stop().catch((caught: unknown) => caught); const error = await Effect.runPromise(producer.stop).catch((caught: unknown) => caught);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "MessagingDeliveryError", _tag: "MessagingDeliveryError",

85
ts/packages/base/src/__tests__/request-response.test.ts
View file

@ -1,4 +1,5 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { Effect } from "effect";
import { import {
makeRequestResponse, makeRequestResponse,
type BackendConsumer, type BackendConsumer,
@ -23,18 +24,20 @@ class RecordingProducer<T> implements BackendProducer<T> {
constructor(private readonly onSend?: (message: T, properties?: Record<string, string>) => void) {} constructor(private readonly onSend?: (message: T, properties?: Record<string, string>) => void) {}
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.onSend?.(message, properties); this.sent.push(properties === undefined ? { message } : { message, properties });
this.onSend?.(message, properties);
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class WaitingConsumer<T> implements BackendConsumer<T> { class WaitingConsumer<T> implements BackendConsumer<T> {
@ -55,32 +58,38 @@ class WaitingConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || this.closed) return message ?? null; const message = this.messages.shift();
if (message !== undefined || this.closed) return Promise.resolve(message ?? null);
return await new Promise((resolve) => { return new Promise((resolve) => {
this.waiters.push(resolve); this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
this.closeCount += 1; this.closeCount += 1;
} });
} }
class RuntimeBackend implements PubSubBackend { class RuntimeBackend implements PubSubBackend {
@ -96,19 +105,23 @@ class RuntimeBackend implements PubSubBackend {
this.producer = new RecordingProducer<unknown>(onSend); this.producer = new RecordingProducer<unknown>(onSend);
} }
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions = options; return Effect.sync(() => {
return this.producer as BackendProducer<T>; this.producerOptions = options;
return this.producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions = options; return Effect.sync(() => {
return this.consumer as BackendConsumer<T>; this.consumerOptions = options;
return this.consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
describe("RequestResponse compatibility facade", () => { describe("RequestResponse compatibility facade", () => {
@ -127,9 +140,9 @@ describe("RequestResponse compatibility facade", () => {
subscription: "sub", subscription: "sub",
}); });
await requestor.start(); await Effect.runPromise(requestor.start);
const response = await requestor.request("request", { timeoutMs: 250 }); const response = await Effect.runPromise(requestor.request("request", { timeoutMs: 250 }));
await requestor.stop(); await Effect.runPromise(requestor.stop);
expect(response).toBe("response"); expect(response).toBe("response");
expect(backend.producerOptions).toEqual({ topic: "request-topic" }); expect(backend.producerOptions).toEqual({ topic: "request-topic" });
@ -150,9 +163,11 @@ describe("RequestResponse compatibility facade", () => {
subscription: "sub", subscription: "sub",
}); });
await requestor.start(); await Effect.runPromise(requestor.start);
const error = await requestor.request("request", { timeoutMs: 5 }).catch((caught: unknown) => caught); const error = await Effect.runPromise(
await requestor.stop(); requestor.request("request", { timeoutMs: 5 }),
).catch((caught: unknown) => caught);
await Effect.runPromise(requestor.stop);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "MessagingTimeoutError", _tag: "MessagingTimeoutError",
@ -171,7 +186,9 @@ describe("RequestResponse compatibility facade", () => {
subscription: "sub", subscription: "sub",
}); });
const error = await requestor.request("request").catch((caught: unknown) => caught); const error = await Effect.runPromise(
requestor.request("request"),
).catch((caught: unknown) => caught);
expect(error).toMatchObject({ expect(error).toMatchObject({
_tag: "MessagingLifecycleError", _tag: "MessagingLifecycleError",

88
ts/packages/base/src/__tests__/runtime-services.test.ts
View file

@ -4,6 +4,7 @@ import * as S from "effect/Schema";
import { import {
PubSub, PubSub,
makeAsyncProcessor, makeAsyncProcessor,
pubSubError,
runProcessorScoped, runProcessorScoped,
type BackendConsumer, type BackendConsumer,
type BackendProducer, type BackendProducer,
@ -24,39 +25,45 @@ class FakeProducer<T> implements BackendProducer<T> {
closeCount = 0; closeCount = 0;
flushCount = 0; flushCount = 0;
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push( return Effect.sync(() => {
properties === undefined this.sent.push(
? { message } properties === undefined
: { message, properties }, ? { message }
); : { message, properties },
);
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class FakeConsumer<T> implements BackendConsumer<T> { class FakeConsumer<T> implements BackendConsumer<T> {
closeCount = 0; closeCount = 0;
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
return null; return Effect.succeed(null);
} }
async acknowledge(): Promise<void> {} acknowledge(): Effect.Effect<void> {
return Effect.void;
}
async negativeAcknowledge(): Promise<void> {} negativeAcknowledge(): Effect.Effect<void> {
return Effect.void;
}
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class FakePubSubBackend implements PubSubBackend { class FakePubSubBackend implements PubSubBackend {
@ -64,24 +71,30 @@ class FakePubSubBackend implements PubSubBackend {
producerOptions: CreateProducerOptions | null = null; producerOptions: CreateProducerOptions | null = null;
consumerOptions: CreateConsumerOptions | null = null; consumerOptions: CreateConsumerOptions | null = null;
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions = options; return Effect.sync(() => {
return new FakeProducer<T>(); this.producerOptions = options;
return new FakeProducer<T>();
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions = options; return Effect.sync(() => {
return new FakeConsumer<T>(); this.consumerOptions = options;
return new FakeConsumer<T>();
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class FailingProducerBackend extends FakePubSubBackend { class FailingProducerBackend extends FakePubSubBackend {
override async createProducer<T>(): Promise<BackendProducer<T>> { override createProducer<T>(): Effect.Effect<BackendProducer<T>> {
throw RuntimeServicesTestError.make({ message: "producer unavailable" }); return Effect.fail(
pubSubError("createProducer:tg.test.failure", RuntimeServicesTestError.make({ message: "producer unavailable" })),
);
} }
} }
@ -90,23 +103,19 @@ const makeRecordingProcessor = (
events: Array<string>, events: Array<string>,
) => { ) => {
const processor = makeAsyncProcessor(config, { const processor = makeAsyncProcessor(config, {
run: async (runtime) => { run: (runtime) => Effect.sync(() => {
events.push(`run:${runtime.config.manageProcessSignals === false ? "effect-signals" : "class-signals"}`); events.push(`run:${runtime.config.manageProcessSignals === false ? "effect-signals" : "class-signals"}`);
}, }),
}); });
const stop = processor.stop; processor.onShutdown(() => Effect.sync(() => {
processor.stop = async () => {
events.push("stop"); events.push("stop");
await stop(); }));
};
return processor; return processor;
}; };
const makeFailingProcessor = (config: ProcessorConfig) => const makeFailingProcessor = (config: ProcessorConfig) =>
makeAsyncProcessor(config, { makeAsyncProcessor(config, {
run: async () => { run: () => Effect.fail(RuntimeServicesTestError.make({ message: "processor failed" })),
throw RuntimeServicesTestError.make({ message: "processor failed" });
},
}); });
const makeNativeRecordingProcessor = ( const makeNativeRecordingProcessor = (
@ -122,8 +131,9 @@ const makeNativeRecordingProcessor = (
}), }),
}); });
processor.onShutdown(() => { processor.onShutdown(() => {
events.push("native-stop"); return Effect.sync(() => {
return Promise.resolve(); events.push("native-stop");
});
}); });
return processor; return processor;
}; };
@ -138,7 +148,7 @@ describe("Effect runtime services", () => {
Effect.gen(function* () { Effect.gen(function* () {
const pubsub = yield* PubSub; const pubsub = yield* PubSub;
const producer = yield* pubsub.createProducer<string>({ topic: "tg.test.topic" }); const producer = yield* pubsub.createProducer<string>({ topic: "tg.test.topic" });
yield* Effect.promise(() => producer.send("hello", { id: "1" })); yield* producer.send("hello", { id: "1" });
expect(backend.producerOptions).toEqual({ topic: "tg.test.topic" }); expect(backend.producerOptions).toEqual({ topic: "tg.test.topic" });
expect(pubsub.backend).toBe(backend); expect(pubsub.backend).toBe(backend);

345
ts/packages/base/src/backend/nats.ts
View file

@ -36,7 +36,7 @@ import type {
CreateConsumerOptions, CreateConsumerOptions,
Message, Message,
} from "./types.js"; } from "./types.js";
import { pubSubError } from "../errors.js"; import { pubSubError, type PubSubError } from "../errors.js";
const sc = StringCodec(); const sc = StringCodec();
@ -113,7 +113,7 @@ function makeNatsProducer<T>(
): BackendProducer<T> { ): BackendProducer<T> {
const makePublishOptions = ( const makePublishOptions = (
properties: Record<string, string> | undefined, properties: Record<string, string> | undefined,
): Effect.Effect<Partial<JetStreamPublishOptions>, ReturnType<typeof pubSubError>> => { ): Effect.Effect<Partial<JetStreamPublishOptions>, PubSubError> => {
if (properties === undefined || Object.keys(properties).length === 0) { if (properties === undefined || Object.keys(properties).length === 0) {
return Effect.succeed({}); return Effect.succeed({});
} }
@ -131,35 +131,32 @@ function makeNatsProducer<T>(
}; };
return { return {
send: (message, properties) => send: Effect.fn(`NatsProducer.send:${subject}`)(function*(message: T, properties?: Record<string, string>) {
Effect.runPromise( const encoded = schema !== undefined
Effect.gen(function* () { ? yield* S.encodeUnknownEffect(schema)(message).pipe(
const encoded = schema !== undefined Effect.mapError((error) => pubSubError(`encode:${subject}`, error)),
? yield* S.encodeUnknownEffect(schema)(message).pipe( )
Effect.mapError((error) => pubSubError(`encode:${subject}`, error)), : message;
) const json = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(encoded).pipe(
: message; Effect.mapError((error) => pubSubError(`encode-json:${subject}`, error)),
const json = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(encoded).pipe( );
Effect.mapError((error) => pubSubError(`encode-json:${subject}`, error)), const data = sc.encode(json);
); const opts = yield* makePublishOptions(properties);
const data = sc.encode(json);
const opts = yield* makePublishOptions(properties);
yield* Effect.tryPromise({ yield* Effect.tryPromise({
try: () => js.publish(subject, data, opts), try: () => js.publish(subject, data, opts),
catch: (error) => pubSubError(`publish:${subject}`, error), catch: (error) => pubSubError(`publish:${subject}`, error),
}); });
}), }),
),
// NATS publishes are flushed on the connection level. // NATS publishes are flushed on the connection level.
flush: () => Promise.resolve(), flush: Effect.void,
// No per-producer cleanup needed for NATS. // No per-producer cleanup needed for NATS.
close: () => Promise.resolve(), close: Effect.void,
}; };
} }
interface InitializableBackendConsumer<T> extends BackendConsumer<T> { interface InitializableBackendConsumer<T> extends BackendConsumer<T> {
readonly init: () => Promise<void>; readonly init: Effect.Effect<void, PubSubError>;
} }
function makeNatsConsumer<T>( function makeNatsConsumer<T>(
@ -173,115 +170,111 @@ function makeNatsConsumer<T>(
): InitializableBackendConsumer<T> { ): InitializableBackendConsumer<T> {
let consumer: NatsJsConsumer | null = null; let consumer: NatsJsConsumer | null = null;
const isReceiveTimeoutError = (error: unknown): boolean => {
const code = P.isObject(error) ? (error as { readonly code?: unknown }).code : undefined;
return code === 408 || code === "408" || code === ErrorCode.Timeout;
};
return { return {
init: () => init: Effect.gen(function* () {
Effect.runPromise( yield* Effect.tryPromise({
Effect.gen(function* () { try: () => jsm.consumers.info(streamName, subscription),
const existing = yield* Effect.tryPromise({ catch: (error) => natsLookupError(`consumer-info:${streamName}:${subscription}`, error),
try: () => js.consumers.get(streamName, subscription), }).pipe(
catch: (error) => natsLookupError(`get-consumer:${streamName}:${subscription}`, error), Effect.catchIf(
}).pipe( isMissingLookupError,
Effect.catchIf( () =>
isMissingLookupError, Effect.gen(function* () {
() => const deliverPolicy =
Effect.gen(function* () { initialPosition === "earliest"
const deliverPolicy = ? DeliverPolicy.All
initialPosition === "earliest" : DeliverPolicy.New;
? DeliverPolicy.All
: DeliverPolicy.New;
yield* Effect.tryPromise({ yield* Effect.tryPromise({
try: () => try: () =>
jsm.consumers.add(streamName, { jsm.consumers.add(streamName, {
durable_name: subscription, durable_name: subscription,
ack_policy: AckPolicy.Explicit, ack_policy: AckPolicy.Explicit,
deliver_policy: deliverPolicy, deliver_policy: deliverPolicy,
filter_subject: subject, filter_subject: subject,
}), }),
catch: (error) => pubSubError(`add-consumer:${streamName}:${subscription}`, error), catch: (error) => pubSubError(`add-consumer:${streamName}:${subscription}`, error),
}); });
}),
(error) => Effect.fail(pubSubError(error.operation, error.cause)),
),
);
consumer = yield* Effect.tryPromise({
try: () => js.consumers.get(streamName, subscription),
catch: (error) => pubSubError(`get-consumer:${streamName}:${subscription}`, error),
});
}),
receive: Effect.fn(`NatsConsumer.receive:${subject}`)(function*(timeoutMs = 2000) {
const current = consumer;
if (current === null) {
return yield* pubSubError("receive", "Consumer not initialized");
}
return yield* Effect.tryPromise({ const msg = yield* Effect.tryPromise({
try: () => js.consumers.get(streamName, subscription), try: () => current.next({ expires: timeoutMs }),
catch: (error) => pubSubError(`get-consumer:${streamName}:${subscription}`, error), catch: (error) =>
}); isReceiveTimeoutError(error)
}), ? pubSubError(`receive-timeout:${subject}`, error)
(error) => Effect.fail(pubSubError(error.operation, error.cause)), : pubSubError(`receive:${subject}`, error),
), }).pipe(
Effect.catchIf(
(error) => error.operation === `receive-timeout:${subject}`,
() => Effect.succeed(null),
),
);
if (msg === null) return null;
const parsed = yield* S.decodeUnknownEffect(S.UnknownFromJsonString)(sc.decode(msg.data)).pipe(
Effect.mapError((error) => pubSubError(`decode-json:${subject}`, error)),
);
const decoded = schema !== undefined
? yield* S.decodeUnknownEffect(schema)(parsed).pipe(
Effect.mapError((error) => pubSubError(`decode-schema:${subject}`, error)),
)
: yield* S.decodeUnknownEffect(S.Any)(parsed).pipe(
Effect.mapError((error) => pubSubError(`decode-any:${subject}`, error)),
); );
consumer = existing; return makeNatsMessage(msg, decoded);
}), }),
), acknowledge: Effect.fn(`NatsConsumer.acknowledge:${subject}`)(function*(message: Message<T>) {
receive: (timeoutMs = 2000) => if (!isNatsMessage(message)) {
Effect.runPromise( return yield* pubSubError(
Effect.gen(function* () { `acknowledge:${subject}`,
const current = consumer; "Message was not produced by NATS backend",
if (current === null) { );
return yield* pubSubError("receive", "Consumer not initialized"); }
} yield* Effect.try({
try: () => {
// Pull a single message with a timeout using the pull-based API. message._jsMsg.ack();
// consumer.next() returns a JsMsg or null when the timeout expires. },
const msg = yield* Effect.tryPromise({ catch: (error) => pubSubError(`acknowledge:${subject}`, error),
try: () => current.next({ expires: timeoutMs }), });
catch: (error) => pubSubError(`receive:${subject}`, error), }),
}); negativeAcknowledge: Effect.fn(`NatsConsumer.negativeAcknowledge:${subject}`)(function*(message: Message<T>) {
if (msg === null) return null; if (!isNatsMessage(message)) {
return yield* pubSubError(
const parsed = yield* S.decodeUnknownEffect(S.UnknownFromJsonString)(sc.decode(msg.data)).pipe( `negative-acknowledge:${subject}`,
Effect.mapError((error) => pubSubError(`decode-json:${subject}`, error)), "Message was not produced by NATS backend",
); );
const decoded = schema !== undefined }
? yield* S.decodeUnknownEffect(schema)(parsed).pipe( yield* Effect.try({
Effect.mapError((error) => pubSubError(`decode-schema:${subject}`, error)), try: () => {
) message._jsMsg.nak();
: yield* S.decodeUnknownEffect(S.Any)(parsed).pipe( },
Effect.mapError((error) => pubSubError(`decode-any:${subject}`, error)), catch: (error) => pubSubError(`negative-acknowledge:${subject}`, error),
); });
return makeNatsMessage(msg, decoded); }),
}), unsubscribe: Effect.sync(() => {
),
acknowledge: (message) =>
Effect.runPromise(
Effect.gen(function* () {
if (!isNatsMessage(message)) {
return yield* pubSubError(`acknowledge:${subject}`, "Message was not produced by NATS backend");
}
yield* Effect.try({
try: () => {
message._jsMsg.ack();
},
catch: (error) => pubSubError(`acknowledge:${subject}`, error),
});
}),
),
negativeAcknowledge: (message) =>
Effect.runPromise(
Effect.gen(function* () {
if (!isNatsMessage(message)) {
return yield* pubSubError(
`negative-acknowledge:${subject}`,
"Message was not produced by NATS backend",
);
}
yield* Effect.try({
try: () => {
message._jsMsg.nak();
},
catch: (error) => pubSubError(`negative-acknowledge:${subject}`, error),
});
}),
),
unsubscribe: () => {
// The pull-based consumer does not have a persistent subscription to drain.
// Clearing the reference is sufficient; the durable consumer persists server-side.
consumer = null; consumer = null;
return Promise.resolve(); }),
}, close: Effect.sync(() => {
close: () => {
consumer = null; consumer = null;
return Promise.resolve(); }),
},
}; };
} }
@ -319,7 +312,9 @@ export function makeNatsBackend(url = "nats://localhost:4222"): PubSubBackend {
const wildcardSubject = `${parts.slice(0, 2).join(".")}.>`; const wildcardSubject = `${parts.slice(0, 2).join(".")}.>`;
const manager = jsm; const manager = jsm;
if (manager === null) return yield* pubSubError("ensure-stream", "NATS backend not connected"); if (manager === null) {
return yield* pubSubError("ensure-stream", "NATS backend not connected");
}
yield* Effect.tryPromise({ yield* Effect.tryPromise({
try: () => manager.streams.info(streamName), try: () => manager.streams.info(streamName),
@ -344,56 +339,48 @@ export function makeNatsBackend(url = "nats://localhost:4222"): PubSubBackend {
}); });
return { return {
createProducer: <T>(options: CreateProducerOptions<T>) => createProducer: Effect.fn("NatsBackend.createProducer")(function*<T>(options: CreateProducerOptions<T>) {
Effect.runPromise( yield* ensureConnected();
Effect.gen(function* () { yield* ensureStream(options.topic);
yield* ensureConnected(); const client = js;
yield* ensureStream(options.topic); if (client === null) {
const client = js; return yield* pubSubError("create-producer", "NATS backend not connected");
if (client === null) return yield* pubSubError("create-producer", "NATS backend not connected"); }
return makeNatsProducer<T>(client, options.topic, options.schema); return makeNatsProducer<T>(client, options.topic, options.schema);
}), }),
), createConsumer: Effect.fn("NatsBackend.createConsumer")(function*<T>(options: CreateConsumerOptions<T>) {
createConsumer: <T>(options: CreateConsumerOptions<T>) => yield* ensureConnected();
Effect.runPromise( const streamName = yield* ensureStream(options.topic);
Effect.gen(function* () { const client = js;
yield* ensureConnected(); const manager = jsm;
const streamName = yield* ensureStream(options.topic); if (client === null || manager === null) {
const client = js; return yield* pubSubError("create-consumer", "NATS backend not connected");
const manager = jsm; }
if (client === null || manager === null) { const consumer = makeNatsConsumer<T>(
return yield* pubSubError("create-consumer", "NATS backend not connected"); client,
} manager,
const consumer = makeNatsConsumer<T>( options.topic,
client, options.subscription,
manager, options.initialPosition ?? "latest",
options.topic, streamName,
options.subscription, options.schema,
options.initialPosition ?? "latest", );
streamName, yield* consumer.init.pipe(
options.schema, Effect.mapError((error) => pubSubError(`init-consumer:${options.topic}`, error)),
); );
yield* Effect.tryPromise({ return consumer;
try: () => consumer.init(), }),
catch: (error) => pubSubError(`init-consumer:${options.topic}`, error), close: Effect.gen(function* () {
}); const conn = connection;
return consumer; if (conn !== null) {
}), yield* Effect.tryPromise({
), try: () => conn.drain(),
close: () => catch: (error) => pubSubError("close", error),
Effect.runPromise( });
Effect.gen(function* () { connection = null;
const conn = connection; js = null;
if (conn !== null) { jsm = null;
yield* Effect.tryPromise({ }
try: () => conn.drain(), }),
catch: (error) => pubSubError("close", error),
});
connection = null;
js = null;
jsm = null;
}
}),
),
}; };
} }

29
ts/packages/base/src/backend/pubsub.ts
View file

@ -1,8 +1,8 @@
/** /**
* Effect-native pub/sub capability for runtime composition. * Effect-native pub/sub capability for runtime composition.
* *
* The existing Promise-based backend protocol stays available as the * The backend protocol is Effect-native; this service provides the
* compatibility bridge while service code moves to `Context.Service`/Layers. * Context.Service/Layer boundary used by runtime composition.
*/ */
import { Config, Context, Effect, Layer } from "effect"; import { Config, Context, Effect, Layer } from "effect";
@ -15,17 +15,17 @@ import type {
PubSubBackend, PubSubBackend,
} from "./types.js"; } from "./types.js";
import { makeNatsBackend } from "./nats.js"; import { makeNatsBackend } from "./nats.js";
import { pubSubError } from "../errors.js"; import type { PubSubError } from "../errors.js";
export interface PubSubService { export interface PubSubService {
readonly backend: PubSubBackend; readonly backend: PubSubBackend;
readonly createProducer: <T>( readonly createProducer: <T>(
options: CreateProducerOptions<T>, options: CreateProducerOptions<T>,
) => Effect.Effect<BackendProducer<T>, ReturnType<typeof pubSubError>>; ) => Effect.Effect<BackendProducer<T>, PubSubError>;
readonly createConsumer: <T>( readonly createConsumer: <T>(
options: CreateConsumerOptions<T>, options: CreateConsumerOptions<T>,
) => Effect.Effect<BackendConsumer<T>, ReturnType<typeof pubSubError>>; ) => Effect.Effect<BackendConsumer<T>, PubSubError>;
readonly close: Effect.Effect<void, ReturnType<typeof pubSubError>>; readonly close: Effect.Effect<void, PubSubError>;
} }
export class PubSub extends Context.Service<PubSub, PubSubService>()("@trustgraph/base/backend/pubsub") { export class PubSub extends Context.Service<PubSub, PubSubService>()("@trustgraph/base/backend/pubsub") {
@ -41,20 +41,9 @@ export class PubSub extends Context.Service<PubSub, PubSubService>()("@trustgrap
export function makePubSubService(backend: PubSubBackend): PubSubService { export function makePubSubService(backend: PubSubBackend): PubSubService {
return { return {
backend, backend,
createProducer: <T>(options: CreateProducerOptions<T>) => createProducer: <T>(options: CreateProducerOptions<T>) => backend.createProducer<T>(options),
Effect.tryPromise({ createConsumer: <T>(options: CreateConsumerOptions<T>) => backend.createConsumer<T>(options),
try: () => backend.createProducer<T>(options), close: backend.close,
catch: (error) => pubSubError(`createProducer:${options.topic}`, error),
}),
createConsumer: <T>(options: CreateConsumerOptions<T>) =>
Effect.tryPromise({
try: () => backend.createConsumer<T>(options),
catch: (error) => pubSubError(`createConsumer:${options.topic}`, error),
}),
close: Effect.tryPromise({
try: () => backend.close(),
catch: (error) => pubSubError("close", error),
}),
}; };
} }

24
ts/packages/base/src/backend/types.ts
View file

@ -5,7 +5,9 @@
* (NATS, Pulsar, Redis Streams) implements these interfaces. * (NATS, Pulsar, Redis Streams) implements these interfaces.
*/ */
import type { Effect } from "effect";
import type * as S from "effect/Schema"; import type * as S from "effect/Schema";
import type { PubSubError } from "../errors.js";
export interface Message<T = unknown> { export interface Message<T = unknown> {
value(): T; value(): T;
@ -13,17 +15,17 @@ export interface Message<T = unknown> {
} }
export interface BackendProducer<T = unknown> { export interface BackendProducer<T = unknown> {
send(message: T, properties?: Record<string, string>): Promise<void>; send(message: T, properties?: Record<string, string>): Effect.Effect<void, PubSubError>;
flush(): Promise<void>; flush: Effect.Effect<void, PubSubError>;
close(): Promise<void>; close: Effect.Effect<void, PubSubError>;
} }
export interface BackendConsumer<T = unknown> { export interface BackendConsumer<T = unknown> {
receive(timeoutMs?: number): Promise<Message<T> | null>; receive(timeoutMs?: number): Effect.Effect<Message<T> | null, PubSubError>;
acknowledge(message: Message<T>): Promise<void>; acknowledge(message: Message<T>): Effect.Effect<void, PubSubError>;
negativeAcknowledge(message: Message<T>): Promise<void>; negativeAcknowledge(message: Message<T>): Effect.Effect<void, PubSubError>;
unsubscribe(): Promise<void>; unsubscribe: Effect.Effect<void, PubSubError>;
close(): Promise<void>; close: Effect.Effect<void, PubSubError>;
} }
export type ConsumerType = "shared" | "exclusive" | "failover"; export type ConsumerType = "shared" | "exclusive" | "failover";
@ -43,7 +45,7 @@ export interface CreateConsumerOptions<T = unknown> {
} }
export interface PubSubBackend { export interface PubSubBackend {
createProducer<T>(options: CreateProducerOptions<T>): Promise<BackendProducer<T>>; createProducer<T>(options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>, PubSubError>;
createConsumer<T>(options: CreateConsumerOptions<T>): Promise<BackendConsumer<T>>; createConsumer<T>(options: CreateConsumerOptions<T>): Effect.Effect<BackendConsumer<T>, PubSubError>;
close(): Promise<void>; close: Effect.Effect<void, PubSubError>;
} }

2
ts/packages/base/src/errors.ts
View file

@ -104,7 +104,7 @@ export class MessagingTimeoutError extends S.TaggedErrorClass<MessagingTimeoutEr
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
timeoutMs: S.Number, timeoutMs: S.Finite,
}, },
) {} ) {}

106
ts/packages/base/src/messaging/consumer.ts
View file

@ -12,18 +12,22 @@ import {
TooManyRequestsError, TooManyRequestsError,
messagingHandlerError, messagingHandlerError,
messagingLifecycleError, messagingLifecycleError,
type MessagingLifecycleError,
} from "../errors.js"; } from "../errors.js";
import { Effect, Exit, Layer, ManagedRuntime, Scope } from "effect"; import { Config as EffectConfig, Effect, Exit, Scope } from "effect";
import * as P from "effect/Predicate"; import * as P from "effect/Predicate";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { loadMessagingRuntimeConfig } from "../runtime/index.ts"; import { loadMessagingRuntimeConfig } from "../runtime/index.ts";
import { makeEffectConsumerFromPubSub, type EffectConsumer } from "./runtime.js"; import {
makeEffectConsumerFromPubSub,
type EffectConsumer,
} from "./runtime.js";
export type MessageHandler<T> = ( export type MessageHandler<T> = (
message: T, message: T,
properties: Record<string, string>, properties: Record<string, string>,
flow: FlowContext, flow: FlowContext,
) => Promise<void>; ) => Effect.Effect<void, TooManyRequestsError | MessagingHandlerError>;
export interface FlowContext<Requirements = never> { export interface FlowContext<Requirements = never> {
id: string; id: string;
@ -47,8 +51,10 @@ declare const ConsumerMessageType: unique symbol;
export interface Consumer<T> { export interface Consumer<T> {
readonly [ConsumerMessageType]?: (_: T) => T; readonly [ConsumerMessageType]?: (_: T) => T;
readonly start: (flow: FlowContext) => Promise<void>; readonly start: (
readonly stop: () => Promise<void>; flow: FlowContext,
) => Effect.Effect<void, MessagingLifecycleError | EffectConfig.ConfigError>;
readonly stop: Effect.Effect<void, MessagingLifecycleError>;
} }
interface ConsumerRuntime { interface ConsumerRuntime {
@ -56,8 +62,6 @@ interface ConsumerRuntime {
readonly consumer: EffectConsumer; readonly consumer: EffectConsumer;
} }
const consumerRuntime = ManagedRuntime.make(Layer.empty);
export function makeConsumer<T>(options: ConsumerOptions<T>): Consumer<T> { export function makeConsumer<T>(options: ConsumerOptions<T>): Consumer<T> {
let runtime: ConsumerRuntime | null = null; let runtime: ConsumerRuntime | null = null;
const isTooManyRequestsError = S.is(TooManyRequestsError); const isTooManyRequestsError = S.is(TooManyRequestsError);
@ -67,62 +71,58 @@ export function makeConsumer<T>(options: ConsumerOptions<T>): Consumer<T> {
properties: Record<string, string>, properties: Record<string, string>,
flow: FlowContext, flow: FlowContext,
): Effect.Effect<void, TooManyRequestsError | MessagingHandlerError> => ): Effect.Effect<void, TooManyRequestsError | MessagingHandlerError> =>
Effect.tryPromise({ options.handler(message, properties, flow).pipe(
try: () => options.handler(message, properties, flow), Effect.mapError((error) =>
catch: (error) =>
isTooManyRequestsError(error) isTooManyRequestsError(error)
? error ? error
: messagingHandlerError(options.topic, options.subscription, error), : messagingHandlerError(options.topic, options.subscription, error)
}); ),
);
return { return {
start: (flow) => start: (flow) =>
P.isNotNull(runtime) P.isNotNull(runtime)
? Promise.resolve() ? Effect.void
: consumerRuntime.runPromise( : Effect.gen(function* () {
Effect.gen(function* () { const scope = yield* Scope.make();
const scope = yield* Scope.make(); const startConsumer = Effect.gen(function* () {
const startConsumer = Effect.gen(function* () { const config = yield* loadMessagingRuntimeConfig();
const config = yield* loadMessagingRuntimeConfig(); const consumer = yield* makeEffectConsumerFromPubSub<T, TooManyRequestsError | MessagingHandlerError, never>(
const consumer = yield* makeEffectConsumerFromPubSub<T, TooManyRequestsError | MessagingHandlerError, never>( PubSub.fromBackend(options.pubsub),
PubSub.fromBackend(options.pubsub), config,
config, {
{ topic: options.topic,
topic: options.topic, subscription: options.subscription,
subscription: options.subscription, handler: runHandler,
handler: runHandler, ...(options.concurrency === undefined ? {} : { concurrency: options.concurrency }),
...(options.concurrency === undefined ? {} : { concurrency: options.concurrency }), initialPosition: options.initialPosition ?? "latest",
initialPosition: options.initialPosition ?? "latest", ...(options.rateLimitRetryMs === undefined ? {} : { rateLimitRetryMs: options.rateLimitRetryMs }),
...(options.rateLimitRetryMs === undefined ? {} : { rateLimitRetryMs: options.rateLimitRetryMs }), ...(options.rateLimitTimeoutMs === undefined
...(options.rateLimitTimeoutMs === undefined ? {}
? {} : { rateLimitTimeoutMs: options.rateLimitTimeoutMs }),
: { rateLimitTimeoutMs: options.rateLimitTimeoutMs }), },
}, flow,
flow, ).pipe(
).pipe( Scope.provide(scope),
Scope.provide(scope), Effect.mapError((error) =>
Effect.mapError((error) => messagingLifecycleError(`${options.topic}:${options.subscription}`, "create-consumer", error)
messagingLifecycleError(`${options.topic}:${options.subscription}`, "create-consumer", error) ),
),
);
runtime = { scope, consumer };
});
yield* startConsumer.pipe(
Effect.onError((cause) => Scope.close(scope, Exit.failCause(cause))),
); );
}), runtime = { scope, consumer };
), });
stop: () => {
yield* startConsumer.pipe(
Effect.onError((cause) => Scope.close(scope, Exit.failCause(cause))),
);
}),
stop: Effect.suspend(() => {
const current = runtime; const current = runtime;
runtime = null; runtime = null;
return current === null return current === null
? Promise.resolve() ? Effect.void
: consumerRuntime.runPromise( : current.consumer.stop.pipe(
current.consumer.stop.pipe( Effect.ensuring(Scope.close(current.scope, Exit.void)),
Effect.ensuring(Scope.close(current.scope, Exit.void)),
),
); );
}, }),
}; };
} }

79
ts/packages/base/src/messaging/producer.ts
View file

@ -9,12 +9,16 @@ import type { ProducerMetrics } from "../metrics/index.ts";
import { Effect, Exit, Scope } from "effect"; import { Effect, Exit, Scope } from "effect";
import { PubSub } from "../backend/pubsub.js"; import { PubSub } from "../backend/pubsub.js";
import { makeEffectProducerFromPubSub, type EffectProducer } from "./runtime.js"; import { makeEffectProducerFromPubSub, type EffectProducer } from "./runtime.js";
import { messagingLifecycleError } from "../errors.js"; import {
messagingLifecycleError,
type MessagingDeliveryError,
type MessagingLifecycleError,
} from "../errors.js";
export interface Producer<T> { export interface Producer<T> {
readonly start: () => Promise<void>; readonly start: Effect.Effect<void, MessagingLifecycleError>;
readonly send: (id: string, message: T) => Promise<void>; readonly send: (id: string, message: T) => Effect.Effect<void, MessagingDeliveryError | MessagingLifecycleError>;
readonly stop: () => Promise<void>; readonly stop: Effect.Effect<void, MessagingDeliveryError>;
} }
interface ProducerRuntime<T> { interface ProducerRuntime<T> {
@ -29,49 +33,46 @@ export function makeProducer<T>(
): Producer<T> { ): Producer<T> {
let runtime: ProducerRuntime<T> | null = null; let runtime: ProducerRuntime<T> | null = null;
const start = Effect.fn(`Producer.start:${topic}`)(function* () {
if (runtime !== null) return;
const scope = yield* Scope.make();
const startProducer = Effect.gen(function* () {
const producer = yield* makeEffectProducerFromPubSub<T>(
PubSub.fromBackend(pubsub),
{
topic,
...(metrics === undefined ? {} : { metrics }),
},
).pipe(
Scope.provide(scope),
Effect.mapError((error) => messagingLifecycleError(topic, "create-producer", error)),
);
runtime = { scope, producer };
});
yield* startProducer.pipe(
Effect.onError((cause) => Scope.close(scope, Exit.failCause(cause))),
);
});
return { return {
start: () => start: start(),
runtime !== null
? Promise.resolve()
: Effect.runPromise(
Effect.gen(function* () {
const scope = yield* Scope.make();
const startProducer = Effect.gen(function* () {
const producer = yield* makeEffectProducerFromPubSub<T>(
PubSub.fromBackend(pubsub),
{
topic,
...(metrics === undefined ? {} : { metrics }),
},
).pipe(
Scope.provide(scope),
Effect.mapError((error) => messagingLifecycleError(topic, "create-producer", error)),
);
runtime = { scope, producer };
});
yield* startProducer.pipe(
Effect.onError((cause) => Scope.close(scope, Exit.failCause(cause))),
);
}),
),
send: (id, message) => { send: (id, message) => {
const current = runtime; const current = runtime;
return current === null return current === null
? Effect.runPromise(Effect.fail(messagingLifecycleError(topic, "send", "Producer not started"))) ? Effect.fail(messagingLifecycleError(topic, "send", "Producer not started"))
: Effect.runPromise(current.producer.send(id, message)); : current.producer.send(id, message);
}, },
stop: () => { stop: Effect.suspend(() => {
const current = runtime; const current = runtime;
runtime = null; runtime = null;
return current === null return current === null
? Promise.resolve() ? Effect.void
: Effect.runPromise( : current.producer.flush.pipe(
current.producer.flush.pipe( Effect.ensuring(Scope.close(current.scope, Exit.void)),
Effect.ensuring(Scope.close(current.scope, Exit.void)),
),
); );
}, }),
}; };
} }

127
ts/packages/base/src/messaging/request-response.ts
View file

@ -7,12 +7,22 @@
* Python reference: trustgraph-base/trustgraph/base/request_response_spec.py * Python reference: trustgraph-base/trustgraph/base/request_response_spec.py
*/ */
import { Effect, Exit, Scope } from "effect"; import { Config as EffectConfig, Effect, Exit, Scope } from "effect";
import type { PubSubBackend } from "../backend/types.js"; import type { PubSubBackend } from "../backend/types.js";
import { PubSub } from "../backend/pubsub.js"; import { PubSub } from "../backend/pubsub.js";
import { messagingDeliveryError, messagingLifecycleError } from "../errors.js"; import {
messagingLifecycleError,
type MessagingDeliveryError,
type MessagingLifecycleError,
type MessagingTimeoutError,
type PubSubError,
} from "../errors.js";
import { loadMessagingRuntimeConfig } from "../runtime/index.ts"; import { loadMessagingRuntimeConfig } from "../runtime/index.ts";
import { makeEffectRequestResponseFromPubSub, type EffectRequestResponse } from "./runtime.js"; import {
makeEffectRequestResponseFromPubSub,
type EffectRequestOptions,
type EffectRequestResponse,
} from "./runtime.js";
export interface RequestResponseOptions { export interface RequestResponseOptions {
pubsub: PubSubBackend; pubsub: PubSubBackend;
@ -22,15 +32,12 @@ export interface RequestResponseOptions {
} }
export interface RequestResponse<TReq, TRes> { export interface RequestResponse<TReq, TRes> {
readonly start: () => Promise<void>; readonly start: Effect.Effect<void, PubSubError | EffectConfig.ConfigError>;
readonly stop: () => Promise<void>; readonly stop: Effect.Effect<void>;
readonly request: ( readonly request: <E = never, R = never>(
request: TReq, request: TReq,
options?: { options?: EffectRequestOptions<TRes, E, R>,
timeoutMs?: number; ) => Effect.Effect<TRes, MessagingDeliveryError | MessagingLifecycleError | MessagingTimeoutError | E, R>;
recipient?: (response: TRes) => Promise<boolean>;
},
) => Promise<TRes>;
} }
interface RequestResponseRuntime<TReq, TRes> { interface RequestResponseRuntime<TReq, TRes> {
@ -43,44 +50,43 @@ export function makeRequestResponse<TReq, TRes>(
): RequestResponse<TReq, TRes> { ): RequestResponse<TReq, TRes> {
let runtime: RequestResponseRuntime<TReq, TRes> | null = null; let runtime: RequestResponseRuntime<TReq, TRes> | null = null;
const start = Effect.fn(`RequestResponse.start:${options.requestTopic}:${options.responseTopic}`)(function* () {
if (runtime !== null) return;
const scope = yield* Scope.make();
const startRuntime = Effect.gen(function* () {
const config = yield* loadMessagingRuntimeConfig();
const requestor = yield* makeEffectRequestResponseFromPubSub<TReq, TRes>(
PubSub.fromBackend(options.pubsub),
config,
{
requestTopic: options.requestTopic,
responseTopic: options.responseTopic,
subscription: options.subscription,
},
).pipe(Scope.provide(scope));
runtime = { scope, requestor };
});
yield* startRuntime.pipe(
Effect.catch((error) =>
Scope.close(scope, Exit.fail(error)).pipe(
Effect.flatMap(() => Effect.fail(error)),
),
),
);
});
return { return {
start: () => start: start(),
runtime !== null stop: Effect.suspend(() => {
? Promise.resolve()
: Effect.runPromise(
Effect.gen(function* () {
const scope = yield* Scope.make();
const startRuntime = Effect.gen(function* () {
const config = yield* loadMessagingRuntimeConfig();
const requestor = yield* makeEffectRequestResponseFromPubSub<TReq, TRes>(
PubSub.fromBackend(options.pubsub),
config,
{
requestTopic: options.requestTopic,
responseTopic: options.responseTopic,
subscription: options.subscription,
},
).pipe(Scope.provide(scope));
runtime = { scope, requestor };
});
yield* startRuntime.pipe(
Effect.catch((error) =>
Scope.close(scope, Exit.fail(error)).pipe(
Effect.flatMap(() => Effect.fail(error)),
),
),
);
}),
),
stop: () => {
const current = runtime; const current = runtime;
runtime = null; runtime = null;
return current === null return current === null
? Promise.resolve() ? Effect.void
: Effect.runPromise(Scope.close(current.scope, Exit.void)); : Scope.close(current.scope, Exit.void);
}, }),
/** /**
* Send a request and wait for responses. * Send a request and wait for responses.
* *
@ -93,34 +99,21 @@ export function makeRequestResponse<TReq, TRes>(
request: (request, requestOptions) => { request: (request, requestOptions) => {
const current = runtime; const current = runtime;
if (current === null) { if (current === null) {
return Effect.runPromise( return Effect.fail(
Effect.fail( messagingLifecycleError(
messagingLifecycleError( `${options.requestTopic}:${options.responseTopic}`,
`${options.requestTopic}:${options.responseTopic}`, "request",
"request", "RequestResponse not started",
"RequestResponse not started",
),
), ),
); );
} }
const timeoutMs = requestOptions?.timeoutMs ?? 300_000; const timeoutMs = requestOptions?.timeoutMs ?? 300_000;
const recipient = requestOptions?.recipient;
return Effect.runPromise( return current.requestor.request(request, {
current.requestor.request(request, { ...requestOptions,
timeoutMs, timeoutMs,
...(recipient === undefined });
? {}
: {
recipient: (response) =>
Effect.tryPromise({
try: () => recipient(response),
catch: (error) => messagingDeliveryError(options.responseTopic, "recipient", error),
}),
}),
}),
);
}, },
}; };
} }

48
ts/packages/base/src/messaging/runtime.ts
View file

@ -168,10 +168,8 @@ export function makeEffectProducerHandle<T>(
): EffectProducer<T> { ): EffectProducer<T> {
return { return {
send: Effect.fn(`Producer.send:${options.topic}`)((id: string, message: T) => send: Effect.fn(`Producer.send:${options.topic}`)((id: string, message: T) =>
Effect.tryPromise({ backend.send(message, { id }).pipe(
try: () => backend.send(message, { id }), Effect.mapError((error) => messagingDeliveryError(options.topic, "send", error)),
catch: (error) => messagingDeliveryError(options.topic, "send", error),
}).pipe(
Effect.tap(() => Effect.tap(() =>
options.metrics === undefined options.metrics === undefined
? Effect.void ? Effect.void
@ -179,14 +177,12 @@ export function makeEffectProducerHandle<T>(
), ),
), ),
), ),
flush: Effect.tryPromise({ flush: backend.flush.pipe(
try: () => backend.flush(), Effect.mapError((error) => messagingDeliveryError(options.topic, "flush", error)),
catch: (error) => messagingDeliveryError(options.topic, "flush", error), ),
}), close: backend.close.pipe(
close: Effect.tryPromise({ Effect.mapError((error) => messagingDeliveryError(options.topic, "close", error)),
try: () => backend.close(), ),
catch: (error) => messagingDeliveryError(options.topic, "close", error),
}),
}; };
} }
@ -219,40 +215,36 @@ const closeConsumerBackend = <T>(
topic: string, topic: string,
subscription: string, subscription: string,
) => ) =>
Effect.tryPromise({ backend.close.pipe(
try: () => backend.close(), Effect.mapError((error) => messagingLifecycleError(`${topic}:${subscription}`, "close-consumer", error)),
catch: (error) => messagingLifecycleError(`${topic}:${subscription}`, "close-consumer", error), );
});
const acknowledgeMessage = <T>( const acknowledgeMessage = <T>(
backend: BackendConsumer<T>, backend: BackendConsumer<T>,
message: Message<T>, message: Message<T>,
topic: string, topic: string,
) => ) =>
Effect.tryPromise({ backend.acknowledge(message).pipe(
try: () => backend.acknowledge(message), Effect.mapError((error) => messagingDeliveryError(topic, "acknowledge", error)),
catch: (error) => messagingDeliveryError(topic, "acknowledge", error), );
});
const negativeAcknowledgeMessage = <T>( const negativeAcknowledgeMessage = <T>(
backend: BackendConsumer<T>, backend: BackendConsumer<T>,
message: Message<T>, message: Message<T>,
topic: string, topic: string,
) => ) =>
Effect.tryPromise({ backend.negativeAcknowledge(message).pipe(
try: () => backend.negativeAcknowledge(message), Effect.mapError((error) => messagingDeliveryError(topic, "negative-acknowledge", error)),
catch: (error) => messagingDeliveryError(topic, "negative-acknowledge", error), );
});
const receiveMessage = <T>( const receiveMessage = <T>(
backend: BackendConsumer<T>, backend: BackendConsumer<T>,
topic: string, topic: string,
timeoutMs: number, timeoutMs: number,
) => ) =>
Effect.tryPromise({ backend.receive(timeoutMs).pipe(
try: () => backend.receive(timeoutMs), Effect.mapError((error) => messagingDeliveryError(topic, "receive", error)),
catch: (error) => messagingDeliveryError(topic, "receive", error), );
});
const handleMessageWithRetry = Effect.fn("handleMessageWithRetry")(function* <T, E, R>( const handleMessageWithRetry = Effect.fn("handleMessageWithRetry")(function* <T, E, R>(
options: EffectConsumerOptions<T, E, R>, options: EffectConsumerOptions<T, E, R>,

94
ts/packages/base/src/processor/async-processor.ts
View file

@ -8,7 +8,7 @@
import type { PubSubBackend } from "../backend/types.js"; import type { PubSubBackend } from "../backend/types.js";
import { makeNatsBackend } from "../backend/nats.js"; import { makeNatsBackend } from "../backend/nats.js";
import { Context, Effect, Layer, ManagedRuntime } from "effect"; import { Cause, Config as EffectConfig, Context, Effect } from "effect";
import { processorLifecycleError, type ProcessorLifecycleError } from "../errors.js"; import { processorLifecycleError, type ProcessorLifecycleError } from "../errors.js";
import { loadProcessorRuntimeConfig } from "../runtime/config.js"; import { loadProcessorRuntimeConfig } from "../runtime/config.js";
@ -23,7 +23,7 @@ export interface ProcessorConfig {
export type ConfigHandler = ( export type ConfigHandler = (
config: Record<string, unknown>, config: Record<string, unknown>,
version: number, version: number,
) => Promise<void>; ) => Effect.Effect<void, Cause.UnknownError>;
export type EffectConfigHandler<E = never, R = never> = ( export type EffectConfigHandler<E = never, R = never> = (
config: Record<string, unknown>, config: Record<string, unknown>,
@ -36,8 +36,10 @@ declare const processorRunRequirementsType: unique symbol;
export interface ProcessorRuntime<RunError = ProcessorLifecycleError, RunRequirements = never> { export interface ProcessorRuntime<RunError = ProcessorLifecycleError, RunRequirements = never> {
readonly [processorRunErrorType]?: RunError; readonly [processorRunErrorType]?: RunError;
readonly [processorRunRequirementsType]?: RunRequirements; readonly [processorRunRequirementsType]?: RunRequirements;
readonly start: (context: Context.Context<RunRequirements>) => Promise<void>; readonly start: (
readonly stop: () => Promise<void>; context: Context.Context<RunRequirements>,
) => Effect.Effect<void, RunError | ProcessorLifecycleError>;
readonly stop: Effect.Effect<void, ProcessorLifecycleError>;
startEffect: Effect.Effect<void, RunError | ProcessorLifecycleError, RunRequirements>; startEffect: Effect.Effect<void, RunError | ProcessorLifecycleError, RunRequirements>;
stopEffect: Effect.Effect<void, ProcessorLifecycleError>; stopEffect: Effect.Effect<void, ProcessorLifecycleError>;
} }
@ -48,12 +50,16 @@ export interface AsyncProcessorRuntime<
> extends ProcessorRuntime<RunError, RunRequirements> { > extends ProcessorRuntime<RunError, RunRequirements> {
readonly config: ProcessorConfig; readonly config: ProcessorConfig;
readonly pubsub: PubSubBackend; readonly pubsub: PubSubBackend;
readonly configHandlers: ConfigHandler[]; readonly configHandlers: Array<EffectConfigHandler<RunError | ProcessorLifecycleError, RunRequirements>>;
readonly running: boolean; readonly running: boolean;
readonly isRunning: () => boolean; readonly isRunning: () => boolean;
readonly registerConfigHandler: (handler: ConfigHandler) => void; readonly registerConfigHandler: (
readonly onShutdown: (callback: () => Promise<void>) => void; handler: EffectConfigHandler<RunError | ProcessorLifecycleError, RunRequirements>,
readonly run: (context: Context.Context<RunRequirements>) => Promise<void>; ) => void;
readonly onShutdown: (callback: () => Effect.Effect<void, Cause.UnknownError>) => void;
readonly run: (
context: Context.Context<RunRequirements>,
) => Effect.Effect<void, RunError | ProcessorLifecycleError>;
runEffect: Effect.Effect<void, RunError | ProcessorLifecycleError, RunRequirements>; runEffect: Effect.Effect<void, RunError | ProcessorLifecycleError, RunRequirements>;
} }
@ -63,7 +69,7 @@ export interface AsyncProcessorRuntimeOptions<
> { > {
readonly run?: ( readonly run?: (
processor: AsyncProcessorRuntime<RunError, RunRequirements>, processor: AsyncProcessorRuntime<RunError, RunRequirements>,
) => Promise<void>; ) => Effect.Effect<void, RunError, RunRequirements>;
readonly runEffect?: ( readonly runEffect?: (
processor: AsyncProcessorRuntime<RunError, RunRequirements>, processor: AsyncProcessorRuntime<RunError, RunRequirements>,
) => Effect.Effect<void, RunError, RunRequirements>; ) => Effect.Effect<void, RunError, RunRequirements>;
@ -74,8 +80,6 @@ interface RegisteredSignalHandler {
readonly handler: () => void; readonly handler: () => void;
} }
const asyncProcessorRuntime = ManagedRuntime.make(Layer.empty);
export function makeAsyncProcessor< export function makeAsyncProcessor<
RunError = ProcessorLifecycleError, RunError = ProcessorLifecycleError,
RunRequirements = never, RunRequirements = never,
@ -85,8 +89,8 @@ export function makeAsyncProcessor<
): AsyncProcessorRuntime<RunError, RunRequirements> { ): AsyncProcessorRuntime<RunError, RunRequirements> {
const pubsub = config.pubsub ?? makeNatsBackend(config.pubsubUrl ?? "nats://localhost:4222"); const pubsub = config.pubsub ?? makeNatsBackend(config.pubsubUrl ?? "nats://localhost:4222");
const ownsPubSub = config.pubsub === undefined; const ownsPubSub = config.pubsub === undefined;
const configHandlers: ConfigHandler[] = []; const configHandlers: Array<EffectConfigHandler<RunError | ProcessorLifecycleError, RunRequirements>> = [];
const shutdownCallbacks: Array<() => Promise<void>> = []; const shutdownCallbacks: Array<() => Effect.Effect<void, Cause.UnknownError>> = [];
let running = false; let running = false;
let signalHandlers: RegisteredSignalHandler[] = []; let signalHandlers: RegisteredSignalHandler[] = [];
@ -96,12 +100,16 @@ export function makeAsyncProcessor<
} }
const shutdown = () => { const shutdown = () => {
void asyncProcessorRuntime.runPromise( Effect.runFork(
Effect.log(`[${config.id}] Shutting down...`).pipe( Effect.log(`[${config.id}] Shutting down...`).pipe(
Effect.flatMap(() => processor.stopEffect), Effect.flatMap(() => processor.stop),
Effect.mapError((error) => processorLifecycleError(config.id, "signal-shutdown", error)), Effect.mapError((error) => processorLifecycleError(config.id, "signal-shutdown", error)),
Effect.match({
onFailure: () => process.exit(1),
onSuccess: () => process.exit(0),
}),
), ),
).then(() => process.exit(0), () => process.exit(1)); );
}; };
const handlers: RegisteredSignalHandler[] = [ const handlers: RegisteredSignalHandler[] = [
{ signal: "SIGINT", handler: shutdown }, { signal: "SIGINT", handler: shutdown },
@ -131,8 +139,10 @@ export function makeAsyncProcessor<
registerConfigHandler: (handler) => { registerConfigHandler: (handler) => {
configHandlers.push(handler); configHandlers.push(handler);
}, },
start: (context) => asyncProcessorRuntime.runPromise(Effect.provide(processor.startEffect, context)), start: (context) => Effect.provide(processor.startEffect, context),
stop: () => asyncProcessorRuntime.runPromise(processor.stopEffect), get stop() {
return processor.stopEffect;
},
onShutdown: (callback) => { onShutdown: (callback) => {
shutdownCallbacks.push(callback); shutdownCallbacks.push(callback);
}, },
@ -161,30 +171,27 @@ export function makeAsyncProcessor<
}); });
for (const cb of shutdownCallbacks) { for (const cb of shutdownCallbacks) {
yield* Effect.tryPromise({ yield* cb().pipe(
try: () => cb(), Effect.mapError((error) => processorLifecycleError(config.id, "shutdown-callback", error)),
catch: (error) => processorLifecycleError(config.id, "shutdown-callback", error), );
});
} }
if (ownsPubSub) { if (ownsPubSub) {
yield* Effect.tryPromise({ yield* pubsub.close.pipe(
try: () => pubsub.close(), Effect.mapError((error) => processorLifecycleError(config.id, "close-pubsub", error)),
catch: (error) => processorLifecycleError(config.id, "close-pubsub", error), );
});
} }
}); });
return stopProcessor(); return stopProcessor();
}, },
run: (context) => asyncProcessorRuntime.runPromise(Effect.provide(processor.runEffect, context)), run: (context) => Effect.provide(processor.runEffect, context),
get runEffect() { get runEffect() {
if (options.runEffect !== undefined) { if (options.runEffect !== undefined) {
return options.runEffect(processor); return options.runEffect(processor);
} }
return Effect.tryPromise({ return options.run?.(processor).pipe(
try: () => options.run?.(processor) ?? Promise.resolve(), Effect.mapError((error) => processorLifecycleError(config.id, "start", error)),
catch: (error) => processorLifecycleError(config.id, "start", error), ) ?? Effect.void;
});
}, },
}; };
@ -201,21 +208,18 @@ export const AsyncProcessor = Object.assign(
return makeAsyncProcessor(config); return makeAsyncProcessor(config);
}, },
{ {
launch<T extends ProcessorRuntime<unknown, never>>( launch<RunError, T extends ProcessorRuntime<RunError, never>>(
this: new (config: ProcessorConfig) => T, this: new (config: ProcessorConfig) => T,
id: string, id: string,
): Promise<void> { ): Effect.Effect<void, ProcessorLifecycleError | EffectConfig.ConfigError> {
const ProcessorCtor = this; const ProcessorCtor = this;
return asyncProcessorRuntime.runPromise( return Effect.gen(function* () {
Effect.gen(function* () { const config = yield* loadProcessorRuntimeConfig(id);
const config = yield* loadProcessorRuntimeConfig(id); const processor = new ProcessorCtor(config);
const processor = new ProcessorCtor(config); yield* processor.start(Context.empty()).pipe(
yield* Effect.tryPromise({ Effect.mapError((error) => processorLifecycleError(id, "launch", error)),
try: () => processor.start(Context.empty()), );
catch: (error) => processorLifecycleError(id, "launch", error), });
});
}),
);
}, },
}, },
) as unknown as { ) as unknown as {
@ -225,8 +229,8 @@ export const AsyncProcessor = Object.assign(
<RunError = ProcessorLifecycleError, RunRequirements = never>( <RunError = ProcessorLifecycleError, RunRequirements = never>(
config: ProcessorConfig, config: ProcessorConfig,
): AsyncProcessor<RunError, RunRequirements>; ): AsyncProcessor<RunError, RunRequirements>;
launch<T extends ProcessorRuntime<unknown, never>>( launch<RunError, T extends ProcessorRuntime<RunError, never>>(
this: new (config: ProcessorConfig) => T, this: new (config: ProcessorConfig) => T,
id: string, id: string,
): Promise<void>; ): Effect.Effect<void, ProcessorLifecycleError | EffectConfig.ConfigError>;
}; };

73
ts/packages/base/src/processor/flow-processor.ts
View file

@ -10,7 +10,6 @@
import { import {
makeAsyncProcessor, makeAsyncProcessor,
type AsyncProcessorRuntime, type AsyncProcessorRuntime,
type ConfigHandler,
type EffectConfigHandler, type EffectConfigHandler,
type ProcessorRuntime, type ProcessorRuntime,
type ProcessorConfig, type ProcessorConfig,
@ -38,7 +37,7 @@ import {
} from "../messaging/runtime.js"; } from "../messaging/runtime.js";
import { makePubSubService, PubSub } from "../backend/pubsub.js"; import { makePubSubService, PubSub } from "../backend/pubsub.js";
import { loadMessagingRuntimeConfig } from "../runtime/index.ts"; import { loadMessagingRuntimeConfig } from "../runtime/index.ts";
import { Context, Duration, Effect, Exit, Layer, ManagedRuntime, Scope } from "effect"; import { Config as EffectConfig, Context, Duration, Effect, Exit, Scope } from "effect";
import * as MutableHashMap from "effect/MutableHashMap"; import * as MutableHashMap from "effect/MutableHashMap";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
@ -75,22 +74,38 @@ type FlowProcessorRuntimeRequirements<FlowRequirements> =
| Scope.Scope | Scope.Scope
| FlowRequirements; | FlowRequirements;
type FlowProcessorRunError =
| PubSubError
| FlowRuntimeError
| ProcessorLifecycleError
| EffectConfig.ConfigError;
export type FlowProcessorStartEffect<FlowRequirements> = Effect.Effect< export type FlowProcessorStartEffect<FlowRequirements> = Effect.Effect<
void, void,
PubSubError | FlowRuntimeError | ProcessorLifecycleError, FlowProcessorRunError,
FlowProcessorRuntimeRequirements<FlowRequirements> FlowProcessorRuntimeRequirements<FlowRequirements>
>; >;
export interface FlowProcessorRuntime<FlowRequirements = never> export interface FlowProcessorRuntime<FlowRequirements = never>
extends ProcessorRuntime< extends ProcessorRuntime<
PubSubError | FlowRuntimeError | ProcessorLifecycleError, FlowProcessorRunError,
FlowProcessorRuntimeRequirements<FlowRequirements> FlowProcessorRuntimeRequirements<FlowRequirements>
> { > {
readonly config: ProcessorConfig; readonly config: ProcessorConfig;
readonly pubsub: PubSubBackend; readonly pubsub: PubSubBackend;
readonly configHandlers: ConfigHandler[]; readonly configHandlers: ReadonlyArray<
EffectConfigHandler<
FlowProcessorRunError,
FlowProcessorRuntimeRequirements<FlowRequirements>
>
>;
readonly isRunning: () => boolean; readonly isRunning: () => boolean;
readonly registerConfigHandler: (handler: ConfigHandler) => void; readonly registerConfigHandler: (
handler: EffectConfigHandler<
FlowProcessorRunError,
FlowProcessorRuntimeRequirements<FlowRequirements>
>,
) => void;
readonly registerSpecification: (spec: Spec<FlowRequirements>) => void; readonly registerSpecification: (spec: Spec<FlowRequirements>) => void;
readonly specifications: ReadonlyArray<Spec<FlowRequirements>>; readonly specifications: ReadonlyArray<Spec<FlowRequirements>>;
} }
@ -103,7 +118,7 @@ export interface MakeFlowProcessorOptions<FlowRequirements = never> {
} }
const ConfigPushSchema = S.Struct({ const ConfigPushSchema = S.Struct({
version: S.Number, version: S.Finite,
config: S.Record(S.String, S.Unknown), config: S.Record(S.String, S.Unknown),
}); });
@ -162,10 +177,8 @@ export function runFlowProcessorDefinitionScoped<
if (consumer === null) { if (consumer === null) {
return Effect.void; return Effect.void;
} }
return Effect.tryPromise({ return consumer.close.pipe(
try: () => consumer.close(), Effect.mapError((error) => pubSubError("close:config-push", error)),
catch: (error) => pubSubError("close:config-push", error),
}).pipe(
Effect.catch((error) => Effect.catch((error) =>
Effect.logError(`[${options.id}] Failed to close config consumer`, { Effect.logError(`[${options.id}] Failed to close config consumer`, {
error: error.message, error: error.message,
@ -253,10 +266,9 @@ export function runFlowProcessorDefinitionScoped<
return; return;
} }
const msg = yield* Effect.tryPromise({ const msg = yield* consumer.receive(2000).pipe(
try: () => consumer.receive(2000), Effect.mapError((error) => pubSubError("receive:config-push", error)),
catch: (error) => pubSubError("receive:config-push", error), );
});
if (msg === null) { if (msg === null) {
return; return;
} }
@ -270,10 +282,9 @@ export function runFlowProcessorDefinitionScoped<
yield* handler(push.config, push.version); yield* handler(push.config, push.version);
} }
yield* Effect.tryPromise({ yield* consumer.acknowledge(msg).pipe(
try: () => consumer.acknowledge(msg), Effect.mapError((error) => pubSubError("acknowledge:config-push", error)),
catch: (error) => pubSubError("acknowledge:config-push", error), );
});
}); });
const processNextConfigPushSafelyEffect = Effect.fn("FlowProcessor.processNextConfigPushSafely")(function* () { const processNextConfigPushSafelyEffect = Effect.fn("FlowProcessor.processNextConfigPushSafely")(function* () {
@ -324,29 +335,19 @@ export function makeFlowProcessor<FlowRequirements = never>(
const specifications: Array<Spec<FlowRequirements>> = [ const specifications: Array<Spec<FlowRequirements>> = [
...(options.specifications ?? []), ...(options.specifications ?? []),
]; ];
const compatibilityRuntime = ManagedRuntime.make(Layer.empty);
let processor: FlowProcessorRuntime<FlowRequirements>; let processor: FlowProcessorRuntime<FlowRequirements>;
const base: AsyncProcessorRuntime< const base: AsyncProcessorRuntime<
PubSubError | FlowRuntimeError | ProcessorLifecycleError, FlowProcessorRunError,
FlowProcessorRuntimeRequirements<FlowRequirements> FlowProcessorRuntimeRequirements<FlowRequirements>
> = makeAsyncProcessor(config, { > = makeAsyncProcessor(config, {
runEffect: (runtime) => { runEffect: (runtime) =>
const configHandlers = runtime.configHandlers.map( runFlowProcessorDefinitionScoped({
(handler): EffectConfigHandler<PubSubError> =>
(pushedConfig, version) =>
Effect.tryPromise({
try: () => handler(pushedConfig, version),
catch: (error) => pubSubError("config-handler", error),
}),
);
return runFlowProcessorDefinitionScoped({
id: runtime.config.id, id: runtime.config.id,
pubsub: runtime.pubsub, pubsub: runtime.pubsub,
specifications, specifications,
configHandlers, configHandlers: runtime.configHandlers,
isRunning: runtime.isRunning, isRunning: runtime.isRunning,
}); }),
},
}); });
const makeStartEffect = (): FlowProcessorStartEffect<FlowRequirements> => { const makeStartEffect = (): FlowProcessorStartEffect<FlowRequirements> => {
@ -381,7 +382,7 @@ export function makeFlowProcessor<FlowRequirements = never>(
get startEffect() { get startEffect() {
return makeStartEffect(); return makeStartEffect();
}, },
start: (context) => compatibilityRuntime.runPromise(startProcessorEffect(context)), start: (context) => startProcessorEffect(context),
}; };
return processor; return processor;

166
ts/packages/base/src/processor/flow.ts
View file

@ -4,7 +4,7 @@
* Python reference: trustgraph-base/trustgraph/base/flow.py * Python reference: trustgraph-base/trustgraph/base/flow.py
*/ */
import { Config as EffectConfig, Context, Effect, Exit, Layer, ManagedRuntime, Scope } from "effect"; import { Config as EffectConfig, Context, Effect, Exit, Scope } from "effect";
import * as MutableHashMap from "effect/MutableHashMap"; import * as MutableHashMap from "effect/MutableHashMap";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
@ -15,6 +15,9 @@ import {
flowResourceNotFoundError, flowResourceNotFoundError,
type FlowParameterDecodeError, type FlowParameterDecodeError,
type FlowResourceNotFoundError, type FlowResourceNotFoundError,
type MessagingDeliveryError,
type MessagingLifecycleError,
type MessagingTimeoutError,
type PubSubError, type PubSubError,
} from "../errors.js"; } from "../errors.js";
import { import {
@ -43,26 +46,26 @@ export interface FlowDefinition {
} }
export interface FlowProducer<T> { export interface FlowProducer<T> {
readonly send: (id: string, message: T) => Promise<void>; readonly send: (id: string, message: T) => Effect.Effect<void, MessagingDeliveryError>;
readonly flush: () => Promise<void>; readonly flush: Effect.Effect<void, MessagingDeliveryError>;
readonly stop: () => Promise<void>; readonly stop: Effect.Effect<void, MessagingDeliveryError>;
} }
export interface FlowConsumer { export interface FlowConsumer {
readonly stop: () => Promise<void>; readonly stop: Effect.Effect<void, MessagingLifecycleError>;
} }
export interface FlowRequestOptions<TRes> { export interface FlowRequestOptions<TRes, E = never, R = never> {
readonly timeoutMs?: number; readonly timeoutMs?: number;
readonly recipient?: (response: TRes) => Promise<boolean>; readonly recipient?: (response: TRes) => Effect.Effect<boolean, E, R>;
} }
export interface FlowRequestor<TReq, TRes> { export interface FlowRequestor<TReq, TRes> {
readonly request: ( readonly request: <E = never, R = never>(
request: TReq, request: TReq,
options?: FlowRequestOptions<TRes>, options?: FlowRequestOptions<TRes, E, R>,
) => Promise<TRes>; ) => Effect.Effect<TRes, MessagingDeliveryError | MessagingLifecycleError | MessagingTimeoutError | E, R>;
readonly stop: () => Promise<void>; readonly stop: Effect.Effect<void, MessagingLifecycleError | MessagingDeliveryError>;
} }
type FlowParameterError = FlowResourceNotFoundError | FlowParameterDecodeError; type FlowParameterError = FlowResourceNotFoundError | FlowParameterDecodeError;
@ -71,19 +74,14 @@ export interface Flow<Requirements = never> {
readonly name: string; readonly name: string;
readonly processorId: string; readonly processorId: string;
startEffect: Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>; startEffect: Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>;
start: (context: Context.Context<Requirements>) => Promise<void>; start: (context: Context.Context<Requirements>) => Effect.Effect<void, PubSubError | EffectConfig.ConfigError>;
stop: () => Promise<void>; stop: Effect.Effect<void>;
stopEffect: Effect.Effect<void>; stopEffect: Effect.Effect<void>;
runInCompatibilityScopeEffect: <A, E>( runInRuntimeScopeEffect: <A, E>(
effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>, effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>,
runtimePubsub: PubSubBackend, runtimePubsub: PubSubBackend,
context: Context.Context<Requirements>, context: Context.Context<Requirements>,
) => Effect.Effect<A, E | EffectConfig.ConfigError>; ) => Effect.Effect<A, E | EffectConfig.ConfigError>;
runInCompatibilityScope: <A, E>(
effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>,
runtimePubsub: PubSubBackend,
context: Context.Context<Requirements>,
) => Promise<A>;
clearResources: () => void; clearResources: () => void;
registerProducer: <T>(registerName: string, producer: EffectProducer<T>) => void; registerProducer: <T>(registerName: string, producer: EffectProducer<T>) => void;
registerConsumer: (registerName: string, consumer: EffectConsumer) => void; registerConsumer: (registerName: string, consumer: EffectConsumer) => void;
@ -108,13 +106,15 @@ export interface Flow<Requirements = never> {
(parameterName: string): Effect.Effect<unknown, FlowResourceNotFoundError>; (parameterName: string): Effect.Effect<unknown, FlowResourceNotFoundError>;
}; };
producer: { producer: {
<T>(producerSpec: ProducerSpec<T>): FlowProducer<T>; <T>(producerSpec: ProducerSpec<T>): Effect.Effect<FlowProducer<T>, FlowResourceNotFoundError>;
(producerName: string): FlowProducer<never>; (producerName: string): Effect.Effect<FlowProducer<never>, FlowResourceNotFoundError>;
}; };
consumer: (consumerName: string) => FlowConsumer; consumer: (consumerName: string) => Effect.Effect<FlowConsumer, FlowResourceNotFoundError>;
requestor: { requestor: {
<TReq, TRes>(requestorSpec: RequestResponseSpec<TReq, TRes>): FlowRequestor<TReq, TRes>; <TReq, TRes>(
(requestorName: string): FlowRequestor<never, unknown>; requestorSpec: RequestResponseSpec<TReq, TRes>,
): Effect.Effect<FlowRequestor<TReq, TRes>, FlowResourceNotFoundError>;
(requestorName: string): Effect.Effect<FlowRequestor<never, unknown>, FlowResourceNotFoundError>;
}; };
parameter: { parameter: {
<T>(parameterSpec: ParameterSpec<T>): T; <T>(parameterSpec: ParameterSpec<T>): T;
@ -133,21 +133,20 @@ export function makeFlow<Requirements = never>(
const consumers = MutableHashMap.empty<string, EffectConsumer>(); const consumers = MutableHashMap.empty<string, EffectConsumer>();
const requestors = MutableHashMap.empty<string, EffectRequestResponse<never, unknown>>(); const requestors = MutableHashMap.empty<string, EffectRequestResponse<never, unknown>>();
const parameters = MutableHashMap.empty<string, unknown>(); const parameters = MutableHashMap.empty<string, unknown>();
let compatibilityScope: Scope.Closeable | null = null; let runtimeScope: Scope.Closeable | null = null;
const compatibilityRuntime = ManagedRuntime.make(Layer.empty);
const ensureCompatibilityScopeEffect = Effect.fn("Flow.ensureCompatibilityScope")(function* () { const ensureRuntimeScopeEffect = Effect.fn("Flow.ensureRuntimeScope")(function* () {
if (compatibilityScope !== null) { if (runtimeScope !== null) {
return compatibilityScope; return runtimeScope;
} }
const scope = yield* Scope.make(); const scope = yield* Scope.make();
compatibilityScope = scope; runtimeScope = scope;
return scope; return scope;
}); });
const toEffectRequestOptions = <TRes>( const toEffectRequestOptions = <TRes, E, R>(
options: FlowRequestOptions<TRes> | undefined, options: FlowRequestOptions<TRes, E, R> | undefined,
): EffectRequestOptions<TRes> | undefined => { ): EffectRequestOptions<TRes, E, R> | undefined => {
if (options === undefined) { if (options === undefined) {
return undefined; return undefined;
} }
@ -157,7 +156,7 @@ export function makeFlow<Requirements = never>(
...(recipient === undefined ...(recipient === undefined
? {} ? {}
: { : {
recipient: (response: TRes) => Effect.promise(() => recipient(response)), recipient,
}), }),
}; };
}; };
@ -198,12 +197,6 @@ export function makeFlow<Requirements = never>(
: Effect.succeed(producer); : Effect.succeed(producer);
}; };
const getProducer = (producerName: string): EffectProducer<never> => {
const producer = O.getOrUndefined(MutableHashMap.get(producers, producerName));
if (producer === undefined) throw flowResourceNotFoundError(name, "producer", producerName);
return producer;
};
const getRequestorEffect = ( const getRequestorEffect = (
requestorName: string, requestorName: string,
): Effect.Effect<EffectRequestResponse<never, unknown>, FlowResourceNotFoundError> => { ): Effect.Effect<EffectRequestResponse<never, unknown>, FlowResourceNotFoundError> => {
@ -213,31 +206,21 @@ export function makeFlow<Requirements = never>(
: Effect.succeed(requestor); : Effect.succeed(requestor);
}; };
const getRequestor = (
requestorName: string,
): EffectRequestResponse<never, unknown> => {
const requestor = O.getOrUndefined(MutableHashMap.get(requestors, requestorName));
if (requestor === undefined) throw flowResourceNotFoundError(name, "requestor", requestorName);
return requestor;
};
const toFlowProducer = <T>(producer: EffectProducer<T>): FlowProducer<T> => ({ const toFlowProducer = <T>(producer: EffectProducer<T>): FlowProducer<T> => ({
send: (id, message) => compatibilityRuntime.runPromise(producer.send(id, message)), send: producer.send,
flush: () => compatibilityRuntime.runPromise(producer.flush), flush: producer.flush,
stop: () => compatibilityRuntime.runPromise(producer.flush.pipe(Effect.flatMap(() => producer.close))), stop: producer.flush.pipe(Effect.flatMap(() => producer.close)),
}); });
const toFlowRequestor = <TReq, TRes>( const toFlowRequestor = <TReq, TRes>(
requestor: EffectRequestResponse<TReq, TRes>, requestor: EffectRequestResponse<TReq, TRes>,
): FlowRequestor<TReq, TRes> => ({ ): FlowRequestor<TReq, TRes> => ({
request: (request, options) => request: (request, options) =>
compatibilityRuntime.runPromise( requestor.request(
requestor.request( request,
request, toEffectRequestOptions(options),
toEffectRequestOptions(options),
),
), ),
stop: () => compatibilityRuntime.runPromise(requestor.stop), stop: requestor.stop,
}); });
function producerEffect<T>( function producerEffect<T>(
@ -303,32 +286,26 @@ export function makeFlow<Requirements = never>(
return decodeParameter(parameter, value); return decodeParameter(parameter, value);
} }
function producer<T>(producerSpec: ProducerSpec<T>): FlowProducer<T>; function producer<T>(producerSpec: ProducerSpec<T>): Effect.Effect<FlowProducer<T>, FlowResourceNotFoundError>;
function producer(producerName: string): FlowProducer<never>; function producer(producerName: string): Effect.Effect<FlowProducer<never>, FlowResourceNotFoundError>;
function producer<T>(producer: string | ProducerSpec<T>) { function producer<T>(producer: string | ProducerSpec<T>) {
if (typeof producer === "string") { if (typeof producer === "string") {
return toFlowProducer(getProducer(producer)); return getProducerEffect(producer).pipe(Effect.map(toFlowProducer));
} }
if (!MutableHashMap.has(producers, producer.name)) { return producer.producerEffect(flow).pipe(Effect.map(toFlowProducer));
throw flowResourceNotFoundError(name, "producer", producer.name);
}
return toFlowProducer(compatibilityRuntime.runSync(producer.producerEffect(flow)));
} }
function requestor<TReq, TRes>( function requestor<TReq, TRes>(
requestorSpec: RequestResponseSpec<TReq, TRes>, requestorSpec: RequestResponseSpec<TReq, TRes>,
): FlowRequestor<TReq, TRes>; ): Effect.Effect<FlowRequestor<TReq, TRes>, FlowResourceNotFoundError>;
function requestor(requestorName: string): FlowRequestor<never, unknown>; function requestor(requestorName: string): Effect.Effect<FlowRequestor<never, unknown>, FlowResourceNotFoundError>;
function requestor<TReq, TRes>( function requestor<TReq, TRes>(
requestor: string | RequestResponseSpec<TReq, TRes>, requestor: string | RequestResponseSpec<TReq, TRes>,
) { ) {
if (typeof requestor === "string") { if (typeof requestor === "string") {
return toFlowRequestor(getRequestor(requestor)); return getRequestorEffect(requestor).pipe(Effect.map(toFlowRequestor));
} }
if (!MutableHashMap.has(requestors, requestor.name)) { return requestor.requestorEffect(flow).pipe(Effect.map(toFlowRequestor));
throw flowResourceNotFoundError(name, "requestor", requestor.name);
}
return toFlowRequestor(compatibilityRuntime.runSync(requestor.requestorEffect(flow)));
} }
const flow: Flow<Requirements> = { const flow: Flow<Requirements> = {
@ -339,33 +316,31 @@ export function makeFlow<Requirements = never>(
yield* spec.addEffect(flow, definition); yield* spec.addEffect(flow, definition);
} }
}).pipe(Effect.withSpan("Flow.startEffect")), }).pipe(Effect.withSpan("Flow.startEffect")),
start(context: Context.Context<Requirements>): Promise<void> { start(context: Context.Context<Requirements>): Effect.Effect<void, PubSubError | EffectConfig.ConfigError> {
return compatibilityRuntime.runPromise( return Effect.gen(function* () {
Effect.gen(function* () { if (runtimeScope !== null) {
if (compatibilityScope !== null) { yield* flow.stop;
yield* flow.stopEffect; }
} yield* flow.runInRuntimeScopeEffect(flow.startEffect, pubsub, context);
yield* flow.runInCompatibilityScopeEffect(flow.startEffect, pubsub, context); });
}),
);
}, },
stop(): Promise<void> { get stop() {
return compatibilityRuntime.runPromise(flow.stopEffect); return flow.stopEffect;
}, },
stopEffect: Effect.gen(function* () { stopEffect: Effect.gen(function* () {
const scope = compatibilityScope; const scope = runtimeScope;
compatibilityScope = null; runtimeScope = null;
if (scope !== null) { if (scope !== null) {
yield* Scope.close(scope, Exit.void); yield* Scope.close(scope, Exit.void);
} }
flow.clearResources(); flow.clearResources();
}).pipe(Effect.withSpan("Flow.stopEffect")), }).pipe(Effect.withSpan("Flow.stopEffect")),
runInCompatibilityScopeEffect: Effect.fn("Flow.runInCompatibilityScopeEffect")(function* <A, E>( runInRuntimeScopeEffect: Effect.fn("Flow.runInRuntimeScopeEffect")(function* <A, E>(
effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>, effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>,
runtimePubsub: PubSubBackend, runtimePubsub: PubSubBackend,
context: Context.Context<Requirements>, context: Context.Context<Requirements>,
) { ) {
const scope = yield* ensureCompatibilityScopeEffect(); const scope = yield* ensureRuntimeScopeEffect();
const pubsubService = makePubSubService(runtimePubsub); const pubsubService = makePubSubService(runtimePubsub);
const messagingConfig = yield* loadMessagingRuntimeConfig(); const messagingConfig = yield* loadMessagingRuntimeConfig();
return yield* Effect.provide( return yield* Effect.provide(
@ -381,13 +356,6 @@ export function makeFlow<Requirements = never>(
context, context,
); );
}), }),
runInCompatibilityScope<A, E>(
effect: Effect.Effect<A, E, SpecRuntimeRequirements | Requirements>,
runtimePubsub: PubSubBackend,
context: Context.Context<Requirements>,
): Promise<A> {
return compatibilityRuntime.runPromise(flow.runInCompatibilityScopeEffect(effect, runtimePubsub, context));
},
clearResources(): void { clearResources(): void {
MutableHashMap.clear(producers); MutableHashMap.clear(producers);
MutableHashMap.clear(consumers); MutableHashMap.clear(consumers);
@ -416,12 +384,12 @@ export function makeFlow<Requirements = never>(
requestorEffect, requestorEffect,
parameterEffect, parameterEffect,
producer, producer,
consumer(consumerName: string): FlowConsumer { consumer(consumerName: string): Effect.Effect<FlowConsumer, FlowResourceNotFoundError> {
const c = O.getOrUndefined(MutableHashMap.get(consumers, consumerName)); return flow.consumerEffect(consumerName).pipe(
if (c === undefined) throw flowResourceNotFoundError(name, "consumer", consumerName); Effect.map((c) => ({
return { stop: c.stop,
stop: () => compatibilityRuntime.runPromise(c.stop), })),
}; );
}, },
requestor, requestor,
parameter, parameter,

6
ts/packages/base/src/processor/program.ts
View file

@ -7,7 +7,6 @@
import { Config as EffectConfig, Effect, Layer } from "effect"; import { Config as EffectConfig, Effect, Layer } from "effect";
import { import {
processorLifecycleError,
type FlowRuntimeError, type FlowRuntimeError,
type ProcessorLifecycleError, type ProcessorLifecycleError,
type PubSubError, type PubSubError,
@ -83,10 +82,7 @@ export const runProcessorScoped = Effect.fn("runProcessorScoped")(function* <
const processor = make(runtimeConfig); const processor = make(runtimeConfig);
yield* Effect.addFinalizer(() => yield* Effect.addFinalizer(() =>
Effect.tryPromise({ processor.stop.pipe(
try: () => processor.stop(),
catch: (error) => processorLifecycleError(config.id, "stop", error),
}).pipe(
Effect.catch((error) => Effect.catch((error) =>
Effect.logError("[Processor] Failed to stop processor", { Effect.logError("[Processor] Failed to stop processor", {
error: error.message, error: error.message,

56
ts/packages/base/src/schema/messages.ts
View file

@ -11,7 +11,7 @@ const UnknownRecord = S.Record(S.String, S.Unknown);
const MutableArray = <A extends S.Top>(schema: A) => schema.pipe(S.Array, S.mutable); const MutableArray = <A extends S.Top>(schema: A) => schema.pipe(S.Array, S.mutable);
const OptionalMutableArray = <A extends S.Top>(schema: A) => schema.pipe(S.Array, S.mutable, S.optionalKey); const OptionalMutableArray = <A extends S.Top>(schema: A) => schema.pipe(S.Array, S.mutable, S.optionalKey);
const StringArray = MutableArray(S.String); const StringArray = MutableArray(S.String);
const NumberArray = MutableArray(S.Number); const NumberArray = MutableArray(S.Finite);
const NumberArrays = MutableArray(NumberArray); const NumberArrays = MutableArray(NumberArray);
// Text completion // Text completion
@ -19,7 +19,7 @@ export const TextCompletionRequest = S.Struct({
system: S.String, system: S.String,
prompt: S.String, prompt: S.String,
model: S.optionalKey(S.String), model: S.optionalKey(S.String),
temperature: S.optionalKey(S.Number), temperature: S.optionalKey(S.Finite),
streaming: S.optionalKey(S.Boolean), streaming: S.optionalKey(S.Boolean),
}); });
export type TextCompletionRequest = typeof TextCompletionRequest.Type; export type TextCompletionRequest = typeof TextCompletionRequest.Type;
@ -27,8 +27,8 @@ export type TextCompletionRequest = typeof TextCompletionRequest.Type;
export const TextCompletionResponse = S.Struct({ export const TextCompletionResponse = S.Struct({
response: S.String, response: S.String,
model: S.optionalKey(S.String), model: S.optionalKey(S.String),
inToken: S.optionalKey(S.Number), inToken: S.optionalKey(S.Finite),
outToken: S.optionalKey(S.Number), outToken: S.optionalKey(S.Finite),
error: S.optionalKey(TgError), error: S.optionalKey(TgError),
endOfStream: S.optionalKey(S.Boolean), endOfStream: S.optionalKey(S.Boolean),
}); });
@ -51,10 +51,10 @@ export type EmbeddingsResponse = typeof EmbeddingsResponse.Type;
export const GraphRagRequest = S.Struct({ export const GraphRagRequest = S.Struct({
query: S.String, query: S.String,
collection: S.optionalKey(S.String), collection: S.optionalKey(S.String),
entityLimit: S.optionalKey(S.Number), entityLimit: S.optionalKey(S.Finite),
tripleLimit: S.optionalKey(S.Number), tripleLimit: S.optionalKey(S.Finite),
maxSubgraphSize: S.optionalKey(S.Number), maxSubgraphSize: S.optionalKey(S.Finite),
maxPathLength: S.optionalKey(S.Number), maxPathLength: S.optionalKey(S.Finite),
streaming: S.optionalKey(S.Boolean), streaming: S.optionalKey(S.Boolean),
}); });
export type GraphRagRequest = typeof GraphRagRequest.Type; export type GraphRagRequest = typeof GraphRagRequest.Type;
@ -126,7 +126,7 @@ export const TriplesQueryRequest = S.Struct({
p: S.optionalKey(Term), p: S.optionalKey(Term),
o: S.optionalKey(Term), o: S.optionalKey(Term),
collection: S.optionalKey(S.String), collection: S.optionalKey(S.String),
limit: S.optionalKey(S.Number), limit: S.optionalKey(S.Finite),
}); });
export type TriplesQueryRequest = typeof TriplesQueryRequest.Type; export type TriplesQueryRequest = typeof TriplesQueryRequest.Type;
@ -140,7 +140,7 @@ export type TriplesQueryResponse = typeof TriplesQueryResponse.Type;
export const GraphEmbeddingsRequest = S.Struct({ export const GraphEmbeddingsRequest = S.Struct({
vectors: NumberArrays, vectors: NumberArrays,
user: S.optionalKey(S.String), user: S.optionalKey(S.String),
limit: S.optionalKey(S.Number), limit: S.optionalKey(S.Finite),
collection: S.optionalKey(S.String), collection: S.optionalKey(S.String),
}); });
export type GraphEmbeddingsRequest = typeof GraphEmbeddingsRequest.Type; export type GraphEmbeddingsRequest = typeof GraphEmbeddingsRequest.Type;
@ -154,7 +154,7 @@ export type GraphEmbeddingsResponse = typeof GraphEmbeddingsResponse.Type;
// Document embeddings query // Document embeddings query
export const DocumentEmbeddingsRequest = S.Struct({ export const DocumentEmbeddingsRequest = S.Struct({
vectors: NumberArrays, vectors: NumberArrays,
limit: S.optionalKey(S.Number), limit: S.optionalKey(S.Finite),
user: S.optionalKey(S.String), user: S.optionalKey(S.String),
collection: S.optionalKey(S.String), collection: S.optionalKey(S.String),
}); });
@ -162,7 +162,7 @@ export type DocumentEmbeddingsRequest = typeof DocumentEmbeddingsRequest.Type;
const DocumentEmbeddingChunk = S.Struct({ const DocumentEmbeddingChunk = S.Struct({
chunkId: S.String, chunkId: S.String,
score: S.Number, score: S.Finite,
content: S.optionalKey(S.String), content: S.optionalKey(S.String),
}); });
@ -193,7 +193,7 @@ export const ConfigRequest = S.StructWithRest(
export type ConfigRequest = typeof ConfigRequest.Type; export type ConfigRequest = typeof ConfigRequest.Type;
export const ConfigResponse = S.Struct({ export const ConfigResponse = S.Struct({
version: S.optionalKey(S.Number), version: S.optionalKey(S.Finite),
values: S.optionalKey(S.Unknown), values: S.optionalKey(S.Unknown),
directory: S.optionalKey(StringArray), directory: S.optionalKey(StringArray),
config: S.optionalKey(UnknownRecord), config: S.optionalKey(UnknownRecord),
@ -266,7 +266,7 @@ export type Triples = typeof Triples.Type;
// Document metadata // Document metadata
export const DocumentMetadata = S.Struct({ export const DocumentMetadata = S.Struct({
id: S.String, id: S.String,
time: S.Number, time: S.Finite,
kind: S.String, kind: S.String,
title: S.String, title: S.String,
comments: S.String, comments: S.String,
@ -284,7 +284,7 @@ export const ProcessingMetadata = S.Struct({
id: S.String, id: S.String,
documentId: S.String, documentId: S.String,
"document-id": S.optionalKey(S.String), "document-id": S.optionalKey(S.String),
time: S.Number, time: S.Finite,
flow: S.String, flow: S.String,
user: S.String, user: S.String,
collection: S.String, collection: S.String,
@ -329,10 +329,10 @@ export const LibrarianRequest = S.StructWithRest(
content: S.optionalKey(S.String), content: S.optionalKey(S.String),
user: S.optionalKey(S.String), user: S.optionalKey(S.String),
collection: S.optionalKey(S.String), collection: S.optionalKey(S.String),
"total-size": S.optionalKey(S.Number), "total-size": S.optionalKey(S.Finite),
"chunk-size": S.optionalKey(S.Number), "chunk-size": S.optionalKey(S.Finite),
"upload-id": S.optionalKey(S.String), "upload-id": S.optionalKey(S.String),
"chunk-index": S.optionalKey(S.Number), "chunk-index": S.optionalKey(S.Finite),
}), }),
[UnknownRecord], [UnknownRecord],
); );
@ -342,10 +342,10 @@ const UploadSessionInfo = S.Struct({
"upload-id": S.String, "upload-id": S.String,
"document-id": S.String, "document-id": S.String,
"document-metadata-json": S.String, "document-metadata-json": S.String,
"total-size": S.Number, "total-size": S.Finite,
"chunk-size": S.Number, "chunk-size": S.Finite,
"total-chunks": S.Number, "total-chunks": S.Finite,
"chunks-received": S.Number, "chunks-received": S.Finite,
"created-at": S.String, "created-at": S.String,
}); });
@ -363,12 +363,12 @@ export const LibrarianResponse = S.StructWithRest(
"document-id": S.optionalKey(S.String), "document-id": S.optionalKey(S.String),
"object-id": S.optionalKey(S.String), "object-id": S.optionalKey(S.String),
"upload-id": S.optionalKey(S.String), "upload-id": S.optionalKey(S.String),
"chunk-size": S.optionalKey(S.Number), "chunk-size": S.optionalKey(S.Finite),
"chunk-index": S.optionalKey(S.Number), "chunk-index": S.optionalKey(S.Finite),
"total-chunks": S.optionalKey(S.Number), "total-chunks": S.optionalKey(S.Finite),
"chunks-received": S.optionalKey(S.Number), "chunks-received": S.optionalKey(S.Finite),
"bytes-received": S.optionalKey(S.Number), "bytes-received": S.optionalKey(S.Finite),
"total-bytes": S.optionalKey(S.Number), "total-bytes": S.optionalKey(S.Finite),
"upload-state": S.optionalKey(S.String), "upload-state": S.optionalKey(S.String),
"received-chunks": S.optionalKey(NumberArray), "received-chunks": S.optionalKey(NumberArray),
"missing-chunks": S.optionalKey(NumberArray), "missing-chunks": S.optionalKey(NumberArray),

8
ts/packages/base/src/schema/primitives.ts
View file

@ -84,16 +84,16 @@ export type RowSchema = typeof RowSchema.Type;
export const LlmResult = S.Struct({ export const LlmResult = S.Struct({
text: S.String, text: S.String,
inToken: S.Number, inToken: S.Finite,
outToken: S.Number, outToken: S.Finite,
model: S.String, model: S.String,
}); });
export type LlmResult = typeof LlmResult.Type; export type LlmResult = typeof LlmResult.Type;
export const LlmChunk = S.Struct({ export const LlmChunk = S.Struct({
text: S.String, text: S.String,
inToken: S.NullOr(S.Number), inToken: S.NullOr(S.Finite),
outToken: S.NullOr(S.Number), outToken: S.NullOr(S.Finite),
model: S.String, model: S.String,
isFinal: S.Boolean, isFinal: S.Boolean,
}); });

47
ts/packages/base/src/services/llm-service.ts
View file

@ -32,19 +32,19 @@ export class LlmServiceError extends S.TaggedErrorClass<LlmServiceError>()(
}, },
) {} ) {}
export interface LlmProvider { export interface LlmProvider<ProviderError = never> {
readonly generateContent: ( readonly generateContent: (
system: string, system: string,
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
) => Promise<LlmResult>; ) => Effect.Effect<LlmResult, ProviderError>;
readonly generateContentStream: ( readonly generateContentStream: (
system: string, system: string,
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
) => AsyncGenerator<LlmChunk>; ) => Stream.Stream<LlmChunk, ProviderError>;
readonly supportsStreaming: () => boolean; readonly supportsStreaming: () => boolean;
} }
@ -60,7 +60,7 @@ export interface LlmServiceShape {
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
) => AsyncGenerator<LlmChunk>; ) => Stream.Stream<LlmChunk, LlmServiceError>;
readonly supportsStreaming: () => boolean; readonly supportsStreaming: () => boolean;
} }
@ -74,24 +74,28 @@ const llmServiceError = (operation: string, cause: unknown) =>
message: errorMessage(cause), message: errorMessage(cause),
}); });
export const makeLlmServiceShape = (provider: LlmProvider): LlmServiceShape => ({ export const makeLlmServiceShape = <ProviderError>(
provider: LlmProvider<ProviderError>,
): LlmServiceShape => ({
generateContent: Effect.fn("Llm.generateContent")(( generateContent: Effect.fn("Llm.generateContent")((
system, system,
prompt, prompt,
model, model,
temperature, temperature,
) => ) =>
Effect.tryPromise({ provider.generateContent(system, prompt, model, temperature).pipe(
try: () => provider.generateContent(system, prompt, model, temperature), Effect.mapError((cause) => llmServiceError("generate-content", cause)),
catch: (cause) => llmServiceError("generate-content", cause), ),
}),
), ),
generateContentStream: ( generateContentStream: (
system, system,
prompt, prompt,
model, model,
temperature, temperature,
) => provider.generateContentStream(system, prompt, model, temperature), ) =>
provider.generateContentStream(system, prompt, model, temperature).pipe(
Stream.mapError((cause) => llmServiceError("generate-content-stream", cause)),
),
supportsStreaming: () => provider.supportsStreaming(), supportsStreaming: () => provider.supportsStreaming(),
}); });
@ -137,14 +141,11 @@ const sendStreamingResponse = Effect.fn("LlmService.sendStreamingResponse")(func
) => Effect.Effect<void, MessagingDeliveryError>; ) => Effect.Effect<void, MessagingDeliveryError>;
}, },
) { ) {
yield* Stream.fromAsyncIterable( yield* llm.generateContentStream(
llm.generateContentStream( msg.system,
msg.system, msg.prompt,
msg.prompt, msg.model,
msg.model, msg.temperature,
msg.temperature,
),
(cause) => llmServiceError("generate-content-stream", cause),
).pipe( ).pipe(
Stream.runForEach((chunk) => Stream.runForEach((chunk) =>
responseProducer.send(requestId, chunkToResponse(chunk)), responseProducer.send(requestId, chunkToResponse(chunk)),
@ -215,12 +216,14 @@ export const makeLlmSpecs = (): ReadonlyArray<Spec<Llm>> => [
makeParameterSpec("temperature"), makeParameterSpec("temperature"),
]; ];
export type LlmService = FlowProcessorRuntime<Llm> & LlmProvider; export type LlmService<ProviderError = never> =
& FlowProcessorRuntime<Llm>
& LlmProvider<ProviderError>;
export function makeLlmService( export function makeLlmService<ProviderError>(
config: ProcessorConfig, config: ProcessorConfig,
provider: LlmProvider, provider: LlmProvider<ProviderError>,
): LlmService { ): LlmService<ProviderError> {
const service = makeFlowProcessor(config, { const service = makeFlowProcessor(config, {
specifications: makeLlmSpecs(), specifications: makeLlmSpecs(),
provide: (effect) => provide: (effect) =>

28
ts/packages/base/src/spec/consumer-spec.ts
View file

@ -5,24 +5,17 @@
*/ */
import { Effect } from "effect"; import { Effect } from "effect";
import * as S from "effect/Schema";
import type { Spec } from "./types.js"; import type { Spec } from "./types.js";
import type { SpecRuntimeRequirements } from "./types.js"; import type { SpecRuntimeRequirements } from "./types.js";
import type { Flow, FlowDefinition } from "../processor/flow.js"; import type { Flow, FlowDefinition } from "../processor/flow.js";
import { type MessageHandler } from "../messaging/consumer.js";
import { import {
ConsumerFactory, ConsumerFactory,
type EffectMessageHandler, type EffectMessageHandler,
} from "../messaging/runtime.js"; } from "../messaging/runtime.js";
import { import {
messagingHandlerError,
TooManyRequestsError,
type MessagingHandlerError,
type PubSubError, type PubSubError,
} from "../errors.js"; } from "../errors.js";
const isTooManyRequestsError = S.is(TooManyRequestsError);
declare const ConsumerSpecType: unique symbol; declare const ConsumerSpecType: unique symbol;
export interface ConsumerSpec<T, E = never, R = never> extends Spec<R> { export interface ConsumerSpec<T, E = never, R = never> extends Spec<R> {
@ -62,26 +55,5 @@ export function makeConsumerSpec<T, E = never, R = never>(
return { return {
name, name,
addEffect, addEffect,
add: (flow, pubsub, definition, context) =>
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
}; };
} }
export function makeConsumerSpecFromPromise<T>(
name: string,
handler: MessageHandler<T>,
concurrency = 1,
): ConsumerSpec<T, TooManyRequestsError | MessagingHandlerError> {
return makeConsumerSpec<T, TooManyRequestsError | MessagingHandlerError>(
name,
(message, properties, flow) =>
Effect.tryPromise({
try: () => handler(message, properties, flow),
catch: (error) =>
isTooManyRequestsError(error)
? error
: messagingHandlerError(name, `${flow.id}-${flow.name}-${name}`, error),
}),
concurrency,
);
}

2
ts/packages/base/src/spec/index.ts
View file

@ -1,5 +1,5 @@
export type { Spec, SpecRuntimeError, SpecRuntimeRequirements } from "./types.js"; export type { Spec, SpecRuntimeError, SpecRuntimeRequirements } from "./types.js";
export { makeConsumerSpec, makeConsumerSpecFromPromise, type ConsumerSpec } from "./consumer-spec.js"; export { makeConsumerSpec, type ConsumerSpec } from "./consumer-spec.js";
export { makeProducerSpec, type ProducerSpec } from "./producer-spec.js"; export { makeProducerSpec, type ProducerSpec } from "./producer-spec.js";
export { makeParameterSpec, type ParameterSpec } from "./parameter-spec.js"; export { makeParameterSpec, type ParameterSpec } from "./parameter-spec.js";
export { makeRequestResponseSpec, type RequestResponseSpec } from "./request-response-spec.js"; export { makeRequestResponseSpec, type RequestResponseSpec } from "./request-response-spec.js";

16
ts/packages/base/src/spec/parameter-spec.ts
View file

@ -4,9 +4,8 @@
* Python reference: trustgraph-base/trustgraph/base/parameter_spec.py * Python reference: trustgraph-base/trustgraph/base/parameter_spec.py
*/ */
import { Effect, type Context } from "effect"; import { Effect } from "effect";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import type { PubSubBackend } from "../backend/types.js";
import type { SpecRuntimeRequirements } from "./types.js"; import type { SpecRuntimeRequirements } from "./types.js";
import type { Flow, FlowDefinition } from "../processor/flow.js"; import type { Flow, FlowDefinition } from "../processor/flow.js";
import type { PubSubError } from "../errors.js"; import type { PubSubError } from "../errors.js";
@ -23,12 +22,6 @@ export interface ParameterSpec<T = unknown> {
flow: Flow<Requirements>, flow: Flow<Requirements>,
definition: FlowDefinition, definition: FlowDefinition,
) => Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>; ) => Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>;
readonly add: <Requirements = never>(
flow: Flow<Requirements>,
pubsub: PubSubBackend,
definition: FlowDefinition,
context: Context.Context<Requirements>,
) => Promise<void>;
} }
export function makeParameterSpec(name: string): ParameterSpec<unknown>; export function makeParameterSpec(name: string): ParameterSpec<unknown>;
@ -51,12 +44,5 @@ export function makeParameterSpec<T>(
name, name,
schema: parameterSchema, schema: parameterSchema,
addEffect, addEffect,
add: <Requirements = never>(
flow: Flow<Requirements>,
pubsub: PubSubBackend,
definition: FlowDefinition,
context: Context.Context<Requirements>,
) =>
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
}; };
} }

11
ts/packages/base/src/spec/producer-spec.ts
View file

@ -4,10 +4,9 @@
* Python reference: trustgraph-base/trustgraph/base/producer_spec.py * Python reference: trustgraph-base/trustgraph/base/producer_spec.py
*/ */
import { Effect, type Context } from "effect"; import { Effect } from "effect";
import type { SpecRuntimeRequirements } from "./types.js"; import type { SpecRuntimeRequirements } from "./types.js";
import type { Flow, FlowDefinition } from "../processor/flow.js"; import type { Flow, FlowDefinition } from "../processor/flow.js";
import type { PubSubBackend } from "../backend/types.js";
import { import {
flowResourceNotFoundError, flowResourceNotFoundError,
type FlowResourceNotFoundError, type FlowResourceNotFoundError,
@ -27,12 +26,6 @@ export interface ProducerSpec<T> {
flow: Flow<Requirements>, flow: Flow<Requirements>,
definition: FlowDefinition, definition: FlowDefinition,
) => Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>; ) => Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>;
readonly add: <Requirements = never>(
flow: Flow<Requirements>,
pubsub: PubSubBackend,
definition: FlowDefinition,
context: Context.Context<Requirements>,
) => Promise<void>;
readonly producerEffect: <Requirements = never>( readonly producerEffect: <Requirements = never>(
flow: Flow<Requirements>, flow: Flow<Requirements>,
) => Effect.Effect<EffectProducer<T>, FlowResourceNotFoundError>; ) => Effect.Effect<EffectProducer<T>, FlowResourceNotFoundError>;
@ -84,7 +77,5 @@ export function makeProducerSpec<T>(name: string): ProducerSpec<T> {
name, name,
producerEffect, producerEffect,
addEffect, addEffect,
add: (flow, pubsub, definition, context) =>
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
}; };
} }

11
ts/packages/base/src/spec/request-response-spec.ts
View file

@ -7,10 +7,9 @@
* Python reference: trustgraph-base/trustgraph/base/prompt_client_spec.py * Python reference: trustgraph-base/trustgraph/base/prompt_client_spec.py
*/ */
import { Effect, type Context } from "effect"; import { Effect } from "effect";
import type { SpecRuntimeRequirements } from "./types.js"; import type { SpecRuntimeRequirements } from "./types.js";
import type { Flow, FlowDefinition } from "../processor/flow.js"; import type { Flow, FlowDefinition } from "../processor/flow.js";
import type { PubSubBackend } from "../backend/types.js";
import { import {
flowResourceNotFoundError, flowResourceNotFoundError,
type FlowResourceNotFoundError, type FlowResourceNotFoundError,
@ -33,12 +32,6 @@ export interface RequestResponseSpec<TReq, TRes> {
flow: Flow<Requirements>, flow: Flow<Requirements>,
definition: FlowDefinition, definition: FlowDefinition,
) => Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>; ) => Effect.Effect<void, PubSubError, SpecRuntimeRequirements | Requirements>;
readonly add: <Requirements = never>(
flow: Flow<Requirements>,
pubsub: PubSubBackend,
definition: FlowDefinition,
context: Context.Context<Requirements>,
) => Promise<void>;
readonly requestorEffect: <Requirements = never>( readonly requestorEffect: <Requirements = never>(
flow: Flow<Requirements>, flow: Flow<Requirements>,
) => Effect.Effect<EffectRequestResponse<TReq, TRes>, FlowResourceNotFoundError>; ) => Effect.Effect<EffectRequestResponse<TReq, TRes>, FlowResourceNotFoundError>;
@ -99,7 +92,5 @@ export function makeRequestResponseSpec<TReq, TRes>(
name, name,
requestorEffect, requestorEffect,
addEffect, addEffect,
add: (flow, pubsub, definition, context) =>
flow.runInCompatibilityScope(addEffect(flow, definition), pubsub, context),
}; };
} }

9
ts/packages/base/src/spec/types.ts
View file

@ -4,8 +4,7 @@
* Python reference: trustgraph-base/trustgraph/base/spec.py and siblings * Python reference: trustgraph-base/trustgraph/base/spec.py and siblings
*/ */
import type { Context, Effect, Scope } from "effect"; import type { Effect, Scope } from "effect";
import type { PubSubBackend } from "../backend/types.js";
import type { import type {
ConsumerFactory, ConsumerFactory,
ProducerFactory, ProducerFactory,
@ -28,10 +27,4 @@ export interface Spec<Requirements = never> {
flow: Flow<Requirements>, flow: Flow<Requirements>,
definition: FlowDefinition, definition: FlowDefinition,
): Effect.Effect<void, SpecRuntimeError, SpecRuntimeRequirements | Requirements>; ): Effect.Effect<void, SpecRuntimeError, SpecRuntimeRequirements | Requirements>;
add(
flow: Flow<Requirements>,
pubsub: PubSubBackend,
definition: FlowDefinition,
context: Context.Context<Requirements>,
): Promise<void>;
} }

19
ts/packages/cli/package.json
View file

@ -12,21 +12,20 @@
"test": "bunx --bun vitest run --passWithNoTests --exclude=dist/**" "test": "bunx --bun vitest run --passWithNoTests --exclude=dist/**"
}, },
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.78",
"@trustgraph/base": "workspace:*", "@trustgraph/base": "workspace:*",
"@trustgraph/client": "workspace:*", "@trustgraph/client": "workspace:*",
"commander": "^13.1.0",
"@effect/ai-anthropic": "4.0.0-beta.75",
"@effect/ai-openai": "4.0.0-beta.75",
"@effect/ai-openrouter": "4.0.0-beta.75",
"@effect/atom-react": "4.0.0-beta.75",
"@effect/openapi-generator": "4.0.0-beta.75",
"@effect/opentelemetry": "4.0.0-beta.75",
"@effect/platform-browser": "4.0.0-beta.75",
"@effect/platform-bun": "4.0.0-beta.75",
"ws": "^8.18.0" "ws": "^8.18.0"
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/ws": "^8.5.0", "@types/ws": "^8.5.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6" "vitest": "^4.1.6"

25
ts/packages/cli/src/commands/agent.ts
View file

@ -4,21 +4,19 @@
* Python reference: trustgraph-cli/trustgraph/cli/invoke_agent.py * Python reference: trustgraph-cli/trustgraph/cli/invoke_agent.py
*/ */
import type { Command } from "commander";
import { Effect } from "effect"; import { Effect } from "effect";
import * as Argument from "effect/unstable/cli/Argument";
import * as Command from "effect/unstable/cli/Command";
import { cliCommandError, withSocket } from "./util.js"; import { cliCommandError, withSocket } from "./util.js";
export function registerAgentCommands(program: Command): void { export const agentCommand = Command.make("agent", {
program question: Argument.string("question").pipe(Argument.withDescription("Question to ask")),
.command("agent") }, ({ question }) =>
.description("Ask the TrustGraph agent a question") withSocket((socket, opts) =>
.argument("<question>", "Question to ask") Effect.gen(function* () {
.action((question: string, _opts, cmd) =>
Effect.runPromise(withSocket(cmd, (socket, opts) =>
Effect.gen(function* () {
const flow = socket.flow(opts.flow); const flow = socket.flow(opts.flow);
yield* Effect.callback<void, ReturnType<typeof cliCommandError>>((resume) => { yield* Effect.callback<void, ReturnType<typeof cliCommandError>>((resume) => {
flow.agent( flow.agent(
question, question,
(chunk) => { (chunk) => {
@ -40,7 +38,6 @@ export function registerAgentCommands(program: Command): void {
(err) => resume(Effect.fail(cliCommandError("agent", err))), (err) => resume(Effect.fail(cliCommandError("agent", err))),
); );
}); });
}), }),
)), ),
); ).pipe(Command.withDescription("Ask the TrustGraph agent a question"));
}

156
ts/packages/cli/src/commands/config.ts
View file

@ -4,38 +4,29 @@
* Python reference: trustgraph-cli/trustgraph/cli/show_config.py etc. * Python reference: trustgraph-cli/trustgraph/cli/show_config.py etc.
*/ */
import type { Command } from "commander";
import { Effect } from "effect"; import { Effect } from "effect";
import * as Argument from "effect/unstable/cli/Argument";
import * as Command from "effect/unstable/cli/Command";
import { cliCommandError, withSocket, writeJson } from "./util.js"; import { cliCommandError, withSocket, writeJson } from "./util.js";
export function registerConfigCommands(program: Command): void { const show = Command.make("show", {}, () =>
const config = program withSocket((socket) =>
.command("config") Effect.gen(function* () {
.description("Configuration management");
config
.command("show")
.description("Show current configuration")
.action((_opts, cmd) =>
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const cfg = socket.config(); const cfg = socket.config();
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => cfg.getConfigAll(), try: () => cfg.getConfigAll(),
catch: (error) => cliCommandError("config.show", error), catch: (error) => cliCommandError("config.show", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Show current configuration"));
config const get = Command.make("get", {
.command("get") key: Argument.string("key").pipe(Argument.withDescription("Config key (format: type/key)")),
.description("Get a configuration value") }, ({ key }) =>
.argument("<key>", "Config key (format: type/key)") withSocket((socket) =>
.action((key: string, _opts, cmd) => Effect.gen(function* () {
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const cfg = socket.config(); const cfg = socket.config();
// Support "type/key" format; fall back to using the whole string as key // Support "type/key" format; fall back to using the whole string as key
const parts = key.split("/"); const parts = key.split("/");
@ -43,74 +34,75 @@ export function registerConfigCommands(program: Command): void {
parts.length >= 2 parts.length >= 2
? { type: parts[0], key: parts.slice(1).join("/") } ? { type: parts[0], key: parts.slice(1).join("/") }
: { type: "config", key }; : { type: "config", key };
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => cfg.getConfig([configKey]), try: () => cfg.getConfig([configKey]),
catch: (error) => cliCommandError("config.get", error), catch: (error) => cliCommandError("config.get", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Get a configuration value"));
config const set = Command.make("set", {
.command("set") key: Argument.string("key").pipe(Argument.withDescription("Config key (format: type/key)")),
.description("Set a configuration value") value: Argument.string("value").pipe(Argument.withDescription("Config value (JSON)")),
.argument("<key>", "Config key (format: type/key)") }, ({ key, value }) =>
.argument("<value>", "Config value (JSON)") withSocket((socket) =>
.action((key: string, value: string, _opts, cmd) => Effect.gen(function* () {
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const cfg = socket.config(); const cfg = socket.config();
const parts = key.split("/"); const parts = key.split("/");
const configEntry = const configEntry =
parts.length >= 2 parts.length >= 2
? { type: parts[0], key: parts.slice(1).join("/"), value } ? { type: parts[0], key: parts.slice(1).join("/"), value }
: { type: "config", key, value }; : { type: "config", key, value };
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => cfg.putConfig([configEntry]), try: () => cfg.putConfig([configEntry]),
catch: (error) => cliCommandError("config.set", error), catch: (error) => cliCommandError("config.set", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Set a configuration value"));
config const list = Command.make("list", {
.command("list") type: Argument.string("type").pipe(
.description("List configuration keys for a type") Argument.withDescription("Config type to list"),
.argument("[type]", "Config type to list", "config") Argument.withDefault("config"),
.action((type: string, _opts, cmd) => ),
Effect.runPromise(withSocket(cmd, (socket) => }, ({ type }) =>
Effect.gen(function* () { withSocket((socket) =>
Effect.gen(function* () {
const cfg = socket.config(); const cfg = socket.config();
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => cfg.list(type), try: () => cfg.list(type),
catch: (error) => cliCommandError("config.list", error), catch: (error) => cliCommandError("config.list", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("List configuration keys for a type"));
config const deleteCommand = Command.make("delete", {
.command("delete") key: Argument.string("key").pipe(Argument.withDescription("Config key (format: type/key)")),
.description("Delete a configuration entry") }, ({ key }) =>
.argument("<key>", "Config key (format: type/key)") withSocket((socket) =>
.action((key: string, _opts, cmd) => Effect.gen(function* () {
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const cfg = socket.config(); const cfg = socket.config();
const parts = key.split("/"); const parts = key.split("/");
const configKey = const configKey =
parts.length >= 2 parts.length >= 2
? { type: parts[0], key: parts.slice(1).join("/") } ? { type: parts[0], key: parts.slice(1).join("/") }
: { type: "config", key }; : { type: "config", key };
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => cfg.deleteConfig(configKey), try: () => cfg.deleteConfig(configKey),
catch: (error) => cliCommandError("config.delete", error), catch: (error) => cliCommandError("config.delete", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Delete a configuration entry"));
}
export const configCommand = Command.make("config").pipe(
Command.withDescription("Configuration management"),
Command.withSubcommands([show, get, set, list, deleteCommand]),
);

36
ts/packages/cli/src/commands/embeddings.ts
View file

@ -4,25 +4,25 @@
* Generate text embeddings using the configured embedding model. * Generate text embeddings using the configured embedding model.
*/ */
import type { Command } from "commander";
import { Effect } from "effect"; import { Effect } from "effect";
import * as Argument from "effect/unstable/cli/Argument";
import * as Command from "effect/unstable/cli/Command";
import { cliCommandError, withSocket, writeJson } from "./util.js"; import { cliCommandError, withSocket, writeJson } from "./util.js";
export function registerEmbeddingsCommands(program: Command): void { export const embeddingsCommand = Command.make("embeddings", {
program texts: Argument.string("text").pipe(
.command("embeddings") Argument.withDescription("Text(s) to embed"),
.description("Generate text embeddings") Argument.variadic({ min: 1 }),
.argument("<text...>", "Text(s) to embed") ),
.action((texts: string[], _opts, cmd) => }, ({ texts }) =>
Effect.runPromise(withSocket(cmd, (socket, opts) => withSocket((socket, opts) =>
Effect.gen(function* () { Effect.gen(function* () {
const flow = socket.flow(opts.flow); const flow = socket.flow(opts.flow);
const vectors = yield* Effect.tryPromise({ const vectors = yield* Effect.tryPromise({
try: () => flow.embeddings(texts), try: () => flow.embeddings(Array.from(texts)),
catch: (error) => cliCommandError("embeddings", error), catch: (error) => cliCommandError("embeddings", error),
}); });
yield* writeJson(vectors); yield* writeJson(vectors);
}), }),
)), ),
); ).pipe(Command.withDescription("Generate text embeddings"));
}

157
ts/packages/cli/src/commands/flow.ts
View file

@ -4,96 +4,99 @@
* Python reference: trustgraph-cli/trustgraph/cli/start_flow.py, stop_flow.py, etc. * Python reference: trustgraph-cli/trustgraph/cli/start_flow.py, stop_flow.py, etc.
*/ */
import type { Command } from "commander";
import { Effect } from "effect"; import { Effect } from "effect";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import * as Argument from "effect/unstable/cli/Argument";
import * as Command from "effect/unstable/cli/Command";
import * as Flag from "effect/unstable/cli/Flag";
import { cliCommandError, withSocket, writeJson } from "./util.js"; import { cliCommandError, withSocket, writeJson } from "./util.js";
export function registerFlowCommands(program: Command): void { const list = Command.make("list", {}, () =>
const flow = program withSocket((socket) =>
.command("flow") Effect.gen(function* () {
.description("Flow management");
flow
.command("list")
.description("List active flows")
.action((_opts, cmd) =>
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const flows = socket.flows(); const flows = socket.flows();
const ids = yield* Effect.tryPromise({ const ids = yield* Effect.tryPromise({
try: () => flows.getFlows(), try: () => flows.getFlows(),
catch: (error) => cliCommandError("flow.list", error), catch: (error) => cliCommandError("flow.list", error),
}); });
yield* writeJson(ids); yield* writeJson(ids);
}), }),
)), ),
); ).pipe(Command.withDescription("List active flows"));
flow const get = Command.make("get", {
.command("get") id: Argument.string("id").pipe(Argument.withDescription("Flow ID")),
.description("Get a flow definition") }, ({ id }) =>
.argument("<id>", "Flow ID") withSocket((socket) =>
.action((id: string, _opts, cmd) => Effect.gen(function* () {
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const flows = socket.flows(); const flows = socket.flows();
const def = yield* Effect.tryPromise({ const def = yield* Effect.tryPromise({
try: () => flows.getFlow(id), try: () => flows.getFlow(id),
catch: (error) => cliCommandError("flow.get", error), catch: (error) => cliCommandError("flow.get", error),
}); });
yield* writeJson(def); yield* writeJson(def);
}), }),
)), ),
); ).pipe(Command.withDescription("Get a flow definition"));
flow const start = Command.make("start", {
.command("start") id: Argument.string("id").pipe(Argument.withDescription("Flow ID")),
.description("Start a flow") blueprint: Flag.string("blueprint").pipe(
.argument("<id>", "Flow ID") Flag.withAlias("b"),
.requiredOption("-b, --blueprint <name>", "Blueprint name") Flag.withDescription("Blueprint name"),
.option("-d, --description <text>", "Flow description", "") ),
.option("-p, --parameters <json>", "Parameters as JSON") description: Flag.string("description").pipe(
.action((id: string, cmdOpts, cmd) => Flag.withAlias("d"),
Effect.runPromise(withSocket(cmd, (socket) => Flag.withDescription("Flow description"),
Effect.gen(function* () { Flag.withDefault(""),
),
parameters: Flag.string("parameters").pipe(
Flag.withAlias("p"),
Flag.withDescription("Parameters as JSON"),
Flag.optional,
),
}, ({ id, blueprint, description, parameters }) =>
withSocket((socket) =>
Effect.gen(function* () {
const flows = socket.flows(); const flows = socket.flows();
const rawParameters = cmdOpts.parameters as string | undefined; const rawParameters = parameters._tag === "Some" ? parameters.value : undefined;
const params = rawParameters !== undefined && rawParameters.length > 0 const params = rawParameters !== undefined && rawParameters.length > 0
? yield* S.decodeUnknownEffect(S.UnknownFromJsonString)(rawParameters).pipe( ? yield* S.decodeUnknownEffect(S.UnknownFromJsonString)(rawParameters).pipe(
Effect.flatMap(S.decodeUnknownEffect(S.Record(S.String, S.Unknown))), Effect.flatMap(S.decodeUnknownEffect(S.Record(S.String, S.Unknown))),
Effect.mapError((error) => cliCommandError("flow.start.parameters", error)), Effect.mapError((error) => cliCommandError("flow.start.parameters", error)),
) )
: undefined; : undefined;
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => try: () =>
flows.startFlow( flows.startFlow(
id, id,
cmdOpts.blueprint as string, blueprint,
cmdOpts.description as string, description,
params, params,
), ),
catch: (error) => cliCommandError("flow.start", error), catch: (error) => cliCommandError("flow.start", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Start a flow"));
flow const stop = Command.make("stop", {
.command("stop") id: Argument.string("id").pipe(Argument.withDescription("Flow ID")),
.description("Stop a flow") }, ({ id }) =>
.argument("<id>", "Flow ID") withSocket((socket) =>
.action((id: string, _opts, cmd) => Effect.gen(function* () {
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const flows = socket.flows(); const flows = socket.flows();
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => flows.stopFlow(id), try: () => flows.stopFlow(id),
catch: (error) => cliCommandError("flow.stop", error), catch: (error) => cliCommandError("flow.stop", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Stop a flow"));
}
export const flowCommand = Command.make("flow").pipe(
Command.withDescription("Flow management"),
Command.withSubcommands([list, get, start, stop]),
);

119
ts/packages/cli/src/commands/graph-rag.ts
View file

@ -4,65 +4,72 @@
* Python reference: trustgraph-cli/trustgraph/cli/invoke_graph_rag.py * Python reference: trustgraph-cli/trustgraph/cli/invoke_graph_rag.py
*/ */
import type { Command } from "commander";
import { Effect } from "effect"; import { Effect } from "effect";
import * as O from "effect/Option";
import * as Argument from "effect/unstable/cli/Argument";
import * as Command from "effect/unstable/cli/Command";
import * as Flag from "effect/unstable/cli/Flag";
import { cliCommandError, withSocket, writeLine } from "./util.js"; import { cliCommandError, withSocket, writeLine } from "./util.js";
export function registerGraphRagCommands(program: Command): void { export const graphRagCommand = Command.make("graph-rag", {
program query: Argument.string("query").pipe(Argument.withDescription("Natural language query")),
.command("graph-rag") entityLimit: Flag.integer("entity-limit").pipe(
.description("Query the knowledge graph using RAG") Flag.withDescription("Max entities"),
.argument("<query>", "Natural language query") Flag.withDefault(50),
.option("--entity-limit <n>", "Max entities", "50") ),
.option("--triple-limit <n>", "Max triples per entity", "30") tripleLimit: Flag.integer("triple-limit").pipe(
.option("--collection <name>", "Collection name") Flag.withDescription("Max triples per entity"),
.action((query: string, cmdOpts, cmd) => Flag.withDefault(30),
Effect.runPromise(withSocket(cmd, (socket, opts) => ),
Effect.gen(function* () { collection: Flag.string("collection").pipe(
Flag.withDescription("Collection name"),
Flag.optional,
),
}, ({ query, entityLimit, tripleLimit, collection }) =>
withSocket((socket, opts) =>
Effect.gen(function* () {
const flow = socket.flow(opts.flow); const flow = socket.flow(opts.flow);
const collection = cmdOpts.collection as string | undefined; const response = yield* Effect.tryPromise({
const response = yield* Effect.tryPromise({ try: () =>
try: () => flow.graphRag(
flow.graphRag( query,
query, {
{ entityLimit,
entityLimit: parseInt(cmdOpts.entityLimit, 10), tripleLimit,
tripleLimit: parseInt(cmdOpts.tripleLimit, 10), },
}, O.getOrUndefined(collection),
collection, ),
), catch: (error) => cliCommandError("graph-rag", error),
catch: (error) => cliCommandError("graph-rag", error), });
}); yield* writeLine(response);
yield* writeLine(response); }),
}), ),
)), ).pipe(Command.withDescription("Query the knowledge graph using RAG"));
);
program export const documentRagCommand = Command.make("document-rag", {
.command("document-rag") query: Argument.string("query").pipe(Argument.withDescription("Natural language query")),
.description("Query documents using RAG") docLimit: Flag.integer("doc-limit").pipe(
.argument("<query>", "Natural language query") Flag.withDescription("Max documents"),
.option("--doc-limit <n>", "Max documents", "20") Flag.withDefault(20),
.option("--collection <name>", "Collection name") ),
.action((query: string, cmdOpts, cmd) => collection: Flag.string("collection").pipe(
Effect.runPromise(withSocket(cmd, (socket, opts) => Flag.withDescription("Collection name"),
Effect.gen(function* () { Flag.optional,
),
}, ({ query, docLimit, collection }) =>
withSocket((socket, opts) =>
Effect.gen(function* () {
const flow = socket.flow(opts.flow); const flow = socket.flow(opts.flow);
const docLimit = cmdOpts.docLimit as string | undefined; const response = yield* Effect.tryPromise({
const collection = cmdOpts.collection as string | undefined; try: () =>
const response = yield* Effect.tryPromise({ flow.documentRag(
try: () => query,
flow.documentRag( docLimit,
query, O.getOrUndefined(collection),
docLimit !== undefined && docLimit.length > 0 ),
? parseInt(docLimit, 10) catch: (error) => cliCommandError("document-rag", error),
: undefined, });
collection, yield* writeLine(response);
), }),
catch: (error) => cliCommandError("document-rag", error), ),
}); ).pipe(Command.withDescription("Query documents using RAG"));
yield* writeLine(response);
}),
)),
);
}

187
ts/packages/cli/src/commands/library.ts
View file

@ -4,8 +4,11 @@
* Manages documents stored in the TrustGraph library. * Manages documents stored in the TrustGraph library.
*/ */
import type { Command } from "commander";
import { Effect, Match } from "effect"; import { Effect, Match } from "effect";
import * as O from "effect/Option";
import * as Argument from "effect/unstable/cli/Argument";
import * as Command from "effect/unstable/cli/Command";
import * as Flag from "effect/unstable/cli/Flag";
import { cliCommandError, withSocket, writeJson } from "./util.js"; import { cliCommandError, withSocket, writeJson } from "./util.js";
function basenamePath(filepath: string): string { function basenamePath(filepath: string): string {
@ -30,98 +33,106 @@ export function guessMimeType(filepath: string): string {
); );
} }
export function registerLibraryCommands(program: Command): void { const list = Command.make("list", {}, () =>
const library = program withSocket((socket) =>
.command("library") Effect.gen(function* () {
.description("Document library management");
library
.command("list")
.description("List documents in the library")
.action((_opts, cmd) =>
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const lib = socket.librarian(); const lib = socket.librarian();
const docs = yield* Effect.tryPromise({ const docs = yield* Effect.tryPromise({
try: () => lib.getDocuments(), try: () => lib.getDocuments(),
catch: (error) => cliCommandError("library.list", error), catch: (error) => cliCommandError("library.list", error),
}); });
yield* writeJson(docs); yield* writeJson(docs);
}), }),
)), ),
); ).pipe(Command.withDescription("List documents in the library"));
library const load = Command.make("load", {
.command("load") file: Argument.string("file").pipe(Argument.withDescription("Path to the file to load")),
.description("Load a document into the library") title: Flag.string("title").pipe(
.argument("<file>", "Path to the file to load") Flag.withAlias("t"),
.option("-t, --title <title>", "Document title") Flag.withDescription("Document title"),
.option("-m, --mime-type <type>", "MIME type (auto-detected if omitted)") Flag.optional,
.option("-c, --comments <text>", "Comments", "") ),
.option("--tags <tags...>", "Document tags") mimeType: Flag.string("mime-type").pipe(
.option("--id <id>", "Optional document ID") Flag.withAlias("m"),
.action((file: string, cmdOpts, cmd) => Flag.withDescription("MIME type (auto-detected if omitted)"),
Effect.runPromise(withSocket(cmd, (socket) => Flag.optional,
Effect.gen(function* () { ),
comments: Flag.string("comments").pipe(
Flag.withAlias("c"),
Flag.withDescription("Comments"),
Flag.withDefault(""),
),
tags: Flag.string("tags").pipe(
Flag.withDescription("Document tags"),
Flag.atMost(Number.MAX_SAFE_INTEGER),
),
id: Flag.string("id").pipe(
Flag.withDescription("Optional document ID"),
Flag.optional,
),
}, ({ file, title, mimeType, comments, tags, id }) =>
withSocket((socket) =>
Effect.gen(function* () {
const lib = socket.librarian(); const lib = socket.librarian();
const data = new Uint8Array(yield* Effect.tryPromise({ const data = new Uint8Array(yield* Effect.tryPromise({
try: () => Bun.file(file).arrayBuffer(), try: () => Bun.file(file).arrayBuffer(),
catch: (error) => cliCommandError("library.load.read-file", error), catch: (error) => cliCommandError("library.load.read-file", error),
})); }));
const b64 = Buffer.from(data).toString("base64"); const b64 = Buffer.from(data).toString("base64");
const mimeType = (cmdOpts.mimeType as string | undefined) ?? guessMimeType(file); const resolvedMimeType = O.getOrUndefined(mimeType) ?? guessMimeType(file);
const title = (cmdOpts.title as string | undefined) ?? basenamePath(file); const resolvedTitle = O.getOrUndefined(title) ?? basenamePath(file);
const comments = cmdOpts.comments as string;
const tags: string[] = (cmdOpts.tags as string[] | undefined) ?? [];
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => try: () =>
lib.loadDocument( lib.loadDocument(
b64, b64,
mimeType, resolvedMimeType,
title, resolvedTitle,
comments, comments,
tags, Array.from(tags),
cmdOpts.id as string | undefined, O.getOrUndefined(id),
), ),
catch: (error) => cliCommandError("library.load", error), catch: (error) => cliCommandError("library.load", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Load a document into the library"));
library const remove = Command.make("remove", {
.command("remove") id: Argument.string("id").pipe(Argument.withDescription("Document ID to remove")),
.description("Remove a document from the library") collection: Flag.string("collection").pipe(
.argument("<id>", "Document ID to remove") Flag.withDescription("Collection name"),
.option("--collection <name>", "Collection name") Flag.optional,
.action((id: string, cmdOpts, cmd) => ),
Effect.runPromise(withSocket(cmd, (socket) => }, ({ id, collection }) =>
Effect.gen(function* () { withSocket((socket) =>
Effect.gen(function* () {
const lib = socket.librarian(); const lib = socket.librarian();
const resp = yield* Effect.tryPromise({ const resp = yield* Effect.tryPromise({
try: () => lib.removeDocument(id, cmdOpts.collection as string | undefined), try: () => lib.removeDocument(id, O.getOrUndefined(collection)),
catch: (error) => cliCommandError("library.remove", error), catch: (error) => cliCommandError("library.remove", error),
}); });
yield* writeJson(resp); yield* writeJson(resp);
}), }),
)), ),
); ).pipe(Command.withDescription("Remove a document from the library"));
library const processing = Command.make("processing", {}, () =>
.command("processing") withSocket((socket) =>
.description("List documents currently being processed") Effect.gen(function* () {
.action((_opts, cmd) =>
Effect.runPromise(withSocket(cmd, (socket) =>
Effect.gen(function* () {
const lib = socket.librarian(); const lib = socket.librarian();
const items = yield* Effect.tryPromise({ const items = yield* Effect.tryPromise({
try: () => lib.getProcessing(), try: () => lib.getProcessing(),
catch: (error) => cliCommandError("library.processing", error), catch: (error) => cliCommandError("library.processing", error),
}); });
yield* writeJson(items); yield* writeJson(items);
}), }),
)), ),
); ).pipe(Command.withDescription("List documents currently being processed"));
}
export const libraryCommand = Command.make("library").pipe(
Command.withDescription("Document library management"),
Command.withSubcommands([list, load, remove, processing]),
);

93
ts/packages/cli/src/commands/triples.ts
View file

@ -4,50 +4,67 @@
* Query the knowledge graph for subject-predicate-object triples. * Query the knowledge graph for subject-predicate-object triples.
*/ */
import type { Command } from "commander";
import type { Term } from "@trustgraph/client"; import type { Term } from "@trustgraph/client";
import { Effect } from "effect"; import { Effect } from "effect";
import * as O from "effect/Option";
import * as Command from "effect/unstable/cli/Command";
import * as Flag from "effect/unstable/cli/Flag";
import { cliCommandError, withSocket, writeJson } from "./util.js"; import { cliCommandError, withSocket, writeJson } from "./util.js";
export function registerTriplesCommands(program: Command): void { export const triplesCommand = Command.make("triples", {
program subject: Flag.string("subject").pipe(
.command("triples") Flag.withAlias("s"),
.description("Query knowledge graph triples") Flag.withDescription("Subject IRI"),
.option("-s, --subject <iri>", "Subject IRI") Flag.optional,
.option("-p, --predicate <iri>", "Predicate IRI") ),
.option("-o, --object <iri>", "Object IRI or literal") predicate: Flag.string("predicate").pipe(
.option("-l, --limit <n>", "Max results", "20") Flag.withAlias("p"),
.option("--collection <name>", "Collection name") Flag.withDescription("Predicate IRI"),
.action((cmdOpts, cmd) => Flag.optional,
Effect.runPromise(withSocket(cmd, (socket, opts) => ),
Effect.gen(function* () { object: Flag.string("object").pipe(
Flag.withAlias("o"),
Flag.withDescription("Object IRI or literal"),
Flag.optional,
),
limit: Flag.integer("limit").pipe(
Flag.withAlias("l"),
Flag.withDescription("Max results"),
Flag.withDefault(20),
),
collection: Flag.string("collection").pipe(
Flag.withDescription("Collection name"),
Flag.optional,
),
}, ({ subject, predicate, object, limit, collection }) =>
withSocket((socket, opts) =>
Effect.gen(function* () {
const flow = socket.flow(opts.flow); const flow = socket.flow(opts.flow);
const subject = cmdOpts.subject as string | undefined; const subjectValue = O.getOrUndefined(subject);
const predicate = cmdOpts.predicate as string | undefined; const predicateValue = O.getOrUndefined(predicate);
const object = cmdOpts.object as string | undefined; const objectValue = O.getOrUndefined(object);
const s: Term | undefined = subject !== undefined && subject.length > 0 const s: Term | undefined = subjectValue !== undefined && subjectValue.length > 0
? { t: "i", i: subject } ? { t: "i", i: subjectValue }
: undefined; : undefined;
const p: Term | undefined = predicate !== undefined && predicate.length > 0 const p: Term | undefined = predicateValue !== undefined && predicateValue.length > 0
? { t: "i", i: predicate } ? { t: "i", i: predicateValue }
: undefined; : undefined;
const o: Term | undefined = object !== undefined && object.length > 0 const o: Term | undefined = objectValue !== undefined && objectValue.length > 0
? { t: "i", i: object } ? { t: "i", i: objectValue }
: undefined; : undefined;
const triples = yield* Effect.tryPromise({ const triples = yield* Effect.tryPromise({
try: () => try: () =>
flow.triplesQuery( flow.triplesQuery(
s, s,
p, p,
o, o,
parseInt(cmdOpts.limit as string, 10), limit,
cmdOpts.collection as string | undefined, O.getOrUndefined(collection),
), ),
catch: (error) => cliCommandError("triples", error), catch: (error) => cliCommandError("triples", error),
}); });
yield* writeJson(triples); yield* writeJson(triples);
}), }),
)), ),
); ).pipe(Command.withDescription("Query knowledge graph triples"));
}

63
ts/packages/cli/src/commands/util.ts
View file

@ -2,10 +2,12 @@
* Shared CLI utilities. * Shared CLI utilities.
*/ */
import type { Command } from "commander";
import { createTrustGraphSocket, type BaseApi } from "@trustgraph/client"; import { createTrustGraphSocket, type BaseApi } from "@trustgraph/client";
import { Duration, Effect } from "effect"; import { Duration, Effect } from "effect";
import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import * as Command from "effect/unstable/cli/Command";
import * as Flag from "effect/unstable/cli/Flag";
export interface CliOpts { export interface CliOpts {
gateway: string; gateway: string;
@ -14,12 +16,42 @@ export interface CliOpts {
flow: string; flow: string;
} }
export function getOpts(cmd: Command): CliOpts { export const rootCommand = Command.make("tg").pipe(
// Walk up to root command to get global options Command.withDescription("TrustGraph CLI - interact with TrustGraph services"),
let root = cmd; Command.withSharedFlags({
while (root.parent !== null) root = root.parent; gateway: Flag.string("gateway").pipe(
return root.opts() as CliOpts; Flag.withAlias("g"),
} Flag.withDescription("Gateway WebSocket URL"),
Flag.withDefault("ws://localhost:8088/api/v1/rpc"),
),
user: Flag.string("user").pipe(
Flag.withAlias("u"),
Flag.withDescription("User identifier"),
Flag.withDefault("cli"),
),
token: Flag.string("token").pipe(
Flag.withAlias("t"),
Flag.withDescription("Authentication token"),
Flag.optional,
),
flow: Flag.string("flow").pipe(
Flag.withAlias("f"),
Flag.withDescription("Flow ID"),
Flag.withDefault("default"),
),
}),
);
export const getOpts = Effect.gen(function* () {
const opts = yield* rootCommand;
const base = {
gateway: opts.gateway,
user: opts.user,
flow: opts.flow,
};
const token = O.getOrUndefined(opts.token);
return token === undefined ? base : { ...base, token } satisfies CliOpts;
});
export class CliCommandError extends S.TaggedErrorClass<CliCommandError>()( export class CliCommandError extends S.TaggedErrorClass<CliCommandError>()(
"CliCommandError", "CliCommandError",
@ -78,19 +110,16 @@ export function createSocketEffect(opts: CliOpts): Effect.Effect<BaseApi, CliCom
); );
} }
export function createSocket(opts: CliOpts): Promise<BaseApi> { export const withSocket = Effect.fn("withSocket")(function* <A, E, R>(
return Effect.runPromise(createSocketEffect(opts));
}
export const withSocket = <A, E, R>(
cmd: Command,
use: (socket: BaseApi, opts: CliOpts) => Effect.Effect<A, E, R>, use: (socket: BaseApi, opts: CliOpts) => Effect.Effect<A, E, R>,
) => ) {
Effect.acquireUseRelease( const opts = yield* getOpts;
createSocketEffect(getOpts(cmd)), return yield* Effect.acquireUseRelease(
(socket) => use(socket, getOpts(cmd)), createSocketEffect(opts),
(socket) => use(socket, opts),
(socket) => (socket) =>
Effect.sync(() => { Effect.sync(() => {
socket.close(); socket.close();
}), }),
); );
});

55
ts/packages/cli/src/index.ts
View file

@ -1,5 +1,7 @@
#!/usr/bin/env node #!/usr/bin/env node
/** @effect-diagnostics strictEffectProvide:skip-file */
/** /**
* Unified TrustGraph CLI. * Unified TrustGraph CLI.
* *
@ -9,32 +11,33 @@
* Python reference: trustgraph-cli/trustgraph/cli/ * Python reference: trustgraph-cli/trustgraph/cli/
*/ */
import { Command } from "commander"; import { BunRuntime, BunServices } from "@effect/platform-bun";
import { registerAgentCommands } from "./commands/agent.js"; import { Effect } from "effect";
import { registerGraphRagCommands } from "./commands/graph-rag.js"; import * as Command from "effect/unstable/cli/Command";
import { registerConfigCommands } from "./commands/config.js"; import { agentCommand } from "./commands/agent.js";
import { registerFlowCommands } from "./commands/flow.js"; import { configCommand } from "./commands/config.js";
import { registerLibraryCommands } from "./commands/library.js"; import { embeddingsCommand } from "./commands/embeddings.js";
import { registerTriplesCommands } from "./commands/triples.js"; import { flowCommand } from "./commands/flow.js";
import { registerEmbeddingsCommands } from "./commands/embeddings.js"; import { graphRagCommand, documentRagCommand } from "./commands/graph-rag.js";
import { libraryCommand } from "./commands/library.js";
import { triplesCommand } from "./commands/triples.js";
import { rootCommand } from "./commands/util.js";
const program = new Command(); export const cli = rootCommand.pipe(
Command.withSubcommands([
agentCommand,
graphRagCommand,
documentRagCommand,
configCommand,
flowCommand,
libraryCommand,
triplesCommand,
embeddingsCommand,
]),
);
program export const program = Command.run(cli, { version: "0.1.0" }).pipe(
.name("tg") Effect.provide(BunServices.layer),
.description("TrustGraph CLI — interact with TrustGraph services") );
.version("0.1.0")
.option("-g, --gateway <url>", "Gateway WebSocket URL", "ws://localhost:8088/api/v1/rpc")
.option("-u, --user <id>", "User identifier", "cli")
.option("-t, --token <token>", "Authentication token")
.option("-f, --flow <id>", "Flow ID", "default");
registerAgentCommands(program); BunRuntime.runMain(program);
registerGraphRagCommands(program);
registerConfigCommands(program);
registerFlowCommands(program);
registerLibraryCommands(program);
registerTriplesCommands(program);
registerEmbeddingsCommands(program);
program.parse();

4
ts/packages/client/package.json
View file

@ -12,7 +12,7 @@
"test": "bunx --bun vitest run" "test": "bunx --bun vitest run"
}, },
"dependencies": { "dependencies": {
"effect": "4.0.0-beta.75" "effect": "4.0.0-beta.78"
}, },
"peerDependencies": { "peerDependencies": {
"ws": "^8.0.0" "ws": "^8.0.0"
@ -23,7 +23,7 @@
} }
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"@types/ws": "^8.5.0", "@types/ws": "^8.5.0",

30
ts/packages/flow/package.json
View file

@ -11,32 +11,30 @@
"test": "bunx --bun vitest run" "test": "bunx --bun vitest run"
}, },
"dependencies": { "dependencies": {
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/platform-node": "4.0.0-beta.75", "@effect/platform-node": "4.0.0-beta.78",
"@effect/platform-node-shared": "4.0.0-beta.75", "@effect/platform-node-shared": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@fastify/websocket": "^11.0.0",
"@mistralai/mistralai": "^1.0.0", "@mistralai/mistralai": "^1.0.0",
"@modelcontextprotocol/sdk": "^1.12.0", "@modelcontextprotocol/sdk": "^1.12.0",
"@qdrant/js-client-rest": "^1.13.0", "@qdrant/js-client-rest": "^1.13.0",
"@trustgraph/base": "workspace:*", "@trustgraph/base": "workspace:*",
"effect": "4.0.0-beta.75", "effect": "4.0.0-beta.78",
"falkordb": "^5.0.0", "falkordb": "^5.0.0",
"fastify": "^5.2.0",
"ollama": "^0.6.3", "ollama": "^0.6.3",
"openai": "^4.85.0", "openai": "^4.85.0",
"pdfjs-dist": "^5.6.205" "pdfjs-dist": "^5.6.205"
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6" "vitest": "^4.1.6"

82
ts/packages/flow/src/__tests__/chunking-service.test.ts
View file

@ -59,17 +59,19 @@ class RecordingProducer<T> implements BackendProducer<T> {
closeCount = 0; closeCount = 0;
flushCount = 0; flushCount = 0;
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.sent.push(properties === undefined ? { message } : { message, properties });
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class PushConsumer<T> implements BackendConsumer<T> { class PushConsumer<T> implements BackendConsumer<T> {
@ -89,33 +91,39 @@ class PushConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || this.closed) { const message = this.messages.shift();
return message ?? null; if (message !== undefined || this.closed) {
} return Promise.resolve(message ?? null);
return await new Promise((resolve) => { }
this.waiters.push(resolve); return new Promise<Message<T> | null>((resolve) => {
this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
this.closeCount += 1; this.closeCount += 1;
} });
} }
class ChunkingBackend implements PubSubBackend { class ChunkingBackend implements PubSubBackend {
@ -126,26 +134,30 @@ class ChunkingBackend implements PubSubBackend {
readonly consumerOptions: Array<CreateConsumerOptions> = []; readonly consumerOptions: Array<CreateConsumerOptions> = [];
closeCount = 0; closeCount = 0;
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions.push(options); return Effect.sync(() => {
const producer = new RecordingProducer<unknown>(); this.producerOptions.push(options);
this.producersByTopic.set(options.topic, producer); const producer = new RecordingProducer<unknown>();
return producer as BackendProducer<T>; this.producersByTopic.set(options.topic, producer);
return producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions.push(options); return Effect.sync(() => {
if (options.topic === topics.configPush) { this.consumerOptions.push(options);
return this.configConsumer as unknown as BackendConsumer<T>; if (options.topic === topics.configPush) {
} return this.configConsumer as unknown as BackendConsumer<T>;
const consumer = new PushConsumer<unknown>(); }
this.consumersByTopic.set(options.topic, consumer); const consumer = new PushConsumer<unknown>();
return consumer as BackendConsumer<T>; this.consumersByTopic.set(options.topic, consumer);
return consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
pushConfig(): void { pushConfig(): void {
this.configConsumer.push( this.configConsumer.push(

83
ts/packages/flow/src/__tests__/config-service.test.ts
View file

@ -20,29 +20,29 @@ import type {
class NoopPubSub implements PubSubBackend { class NoopPubSub implements PubSubBackend {
readonly sentByTopic = new Map<string, Array<unknown>>(); readonly sentByTopic = new Map<string, Array<unknown>>();
async createProducer<T>(options: CreateProducerOptions<T>): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>> {
return { return Effect.succeed({
send: async (message) => { send: (message) => Effect.sync(() => {
const sent = this.sentByTopic.get(options.topic) ?? []; const sent = this.sentByTopic.get(options.topic) ?? [];
sent.push(message); sent.push(message);
this.sentByTopic.set(options.topic, sent); this.sentByTopic.set(options.topic, sent);
}, }),
flush: async () => undefined, flush: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async createConsumer<T>(_options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(_options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
return { return Effect.succeed({
receive: async () => null, receive: () => Effect.succeed(null),
acknowledge: async () => undefined, acknowledge: () => Effect.void,
negativeAcknowledge: async () => undefined, negativeAcknowledge: () => Effect.void,
unsubscribe: async () => undefined, unsubscribe: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
const makeService = (persistPath?: string) => const makeService = (persistPath?: string) =>
@ -59,9 +59,9 @@ describe("ConfigService operations", () => {
const putRequest: ConfigRequest = { operation: "put" }; const putRequest: ConfigRequest = { operation: "put" };
const deleteRequest: ConfigRequest = { operation: "delete" }; const deleteRequest: ConfigRequest = { operation: "delete" };
const putError = await service.handlePut(putRequest) const putError = await Effect.runPromise(service.handlePutEffect(putRequest))
.catch((caught: unknown) => caught); .catch((caught: unknown) => caught);
const deleteError = await service.handleDelete(deleteRequest) const deleteError = await Effect.runPromise(service.handleDeleteEffect(deleteRequest))
.catch((caught: unknown) => caught); .catch((caught: unknown) => caught);
expect(putError).toBeInstanceOf(ConfigServiceError); expect(putError).toBeInstanceOf(ConfigServiceError);
@ -81,7 +81,7 @@ describe("ConfigService operations", () => {
], ],
}; };
await service.handlePut(putRequest); await Effect.runPromise(service.handlePutEffect(putRequest));
const persisted = await Bun.file(persistPath).json(); const persisted = await Bun.file(persistPath).json();
await rm(dir, { recursive: true, force: true }); await rm(dir, { recursive: true, force: true });
@ -107,7 +107,7 @@ describe("ConfigService operations", () => {
); );
const service = makeService(persistPath); const service = makeService(persistPath);
await service.loadFromDisk(); await Effect.runPromise(service.loadFromDiskEffect);
const getRequest: ConfigRequest = { const getRequest: ConfigRequest = {
operation: "get", operation: "get",
keys: ["prompt", "system"], keys: ["prompt", "system"],
@ -131,7 +131,10 @@ describe("ConfigService operations", () => {
{ operation: "put", values: [{ workspace: "beta", type: "prompt", key: "c", value: "three" }] }, { operation: "put", values: [{ workspace: "beta", type: "prompt", key: "c", value: "three" }] },
]; ];
await Promise.all(requests.map((request) => service.handlePut(request))); await Effect.runPromise(Effect.all(requests.map((request) => service.handlePutEffect(request)), {
concurrency: "unbounded",
discard: true,
}));
expect(service.handleGet({ operation: "get", keys: ["prompt"] })).toEqual({ expect(service.handleGet({ operation: "get", keys: ["prompt"] })).toEqual({
version: 3, version: 3,
@ -150,53 +153,53 @@ describe("ConfigService operations", () => {
it("dispatches all config operations through the Match-backed handler", async () => { it("dispatches all config operations through the Match-backed handler", async () => {
const service = makeService(); const service = makeService();
await expect(service.handleOperation({ operation: "put" })).rejects.toMatchObject({ await expect(Effect.runPromise(service.handleOperationEffect({ operation: "put" }))).rejects.toMatchObject({
_tag: "ConfigServiceError", _tag: "ConfigServiceError",
operation: "put", operation: "put",
}); });
await expect(service.handleOperation({ operation: "delete" })).rejects.toMatchObject({ await expect(Effect.runPromise(service.handleOperationEffect({ operation: "delete" }))).rejects.toMatchObject({
_tag: "ConfigServiceError", _tag: "ConfigServiceError",
operation: "delete", operation: "delete",
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "put", operation: "put",
values: [{ type: "prompt", key: "system", value: "hello" }], values: [{ type: "prompt", key: "system", value: "hello" }],
})).resolves.toEqual({ version: 1 }); }))).resolves.toEqual({ version: 1 });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "get", operation: "get",
keys: ["prompt", "system"], keys: ["prompt", "system"],
})).resolves.toEqual({ }))).resolves.toEqual({
version: 1, version: 1,
values: { system: "hello" }, values: { system: "hello" },
}); });
await expect(service.handleOperation({ operation: "list" })).resolves.toEqual({ await expect(Effect.runPromise(service.handleOperationEffect({ operation: "list" }))).resolves.toEqual({
version: 1, version: 1,
directory: ["prompt"], directory: ["prompt"],
}); });
await expect(service.handleOperation({ operation: "config" })).resolves.toEqual({ await expect(Effect.runPromise(service.handleOperationEffect({ operation: "config" }))).resolves.toEqual({
version: 1, version: 1,
config: { prompt: { system: "hello" } }, config: { prompt: { system: "hello" } },
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "getvalues", operation: "getvalues",
type: "prompt", type: "prompt",
})).resolves.toEqual({ }))).resolves.toEqual({
version: 1, version: 1,
values: [{ type: "prompt", key: "system", value: "hello" }], values: [{ type: "prompt", key: "system", value: "hello" }],
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "getvalues-all-ws", operation: "getvalues-all-ws",
type: "prompt", type: "prompt",
})).resolves.toEqual({ }))).resolves.toEqual({
version: 1, version: 1,
values: [{ workspace: "default", type: "prompt", key: "system", value: "hello" }], values: [{ workspace: "default", type: "prompt", key: "system", value: "hello" }],
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "delete", operation: "delete",
keys: ["prompt", "system"], keys: ["prompt", "system"],
})).resolves.toEqual({ version: 2 }); }))).resolves.toEqual({ version: 2 });
}); });
it("pushes config from the stored producer handle", async () => { it("pushes config from the stored producer handle", async () => {
@ -206,10 +209,10 @@ describe("ConfigService operations", () => {
manageProcessSignals: false, manageProcessSignals: false,
pubsub: backend, pubsub: backend,
}); });
const pushProducer = await backend.createProducer<{ const pushProducer = await Effect.runPromise(backend.createProducer<{
readonly version: number; readonly version: number;
readonly config: Record<string, unknown>; readonly config: Record<string, unknown>;
}>({ topic: topics.configPush }); }>({ topic: topics.configPush }));
await Effect.runPromise( await Effect.runPromise(
SynchronizedRef.update(service.state, (state) => ({ SynchronizedRef.update(service.state, (state) => ({
@ -217,11 +220,11 @@ describe("ConfigService operations", () => {
pushProducer, pushProducer,
})), })),
); );
await service.pushConfig(); await Effect.runPromise(service.pushConfigEffect);
await service.handlePut({ await Effect.runPromise(service.handlePutEffect({
operation: "put", operation: "put",
values: [{ type: "prompt", key: "system", value: "hello" }], values: [{ type: "prompt", key: "system", value: "hello" }],
}); }));
expect(backend.sentByTopic.get(topics.configPush)).toEqual([ expect(backend.sentByTopic.get(topics.configPush)).toEqual([
{ version: 0, config: {} }, { version: 0, config: {} },

24
ts/packages/flow/src/__tests__/falkordb-lifecycle.test.ts
View file

@ -17,30 +17,34 @@ class FakeFalkorDBClient implements FalkorDBStoreClient, FalkorDBQueryClient {
connectCount = 0; connectCount = 0;
disconnectCount = 0; disconnectCount = 0;
async connect(): Promise<void> { readonly connect: Effect.Effect<void> = Effect.sync(() => {
this.connectCount += 1; this.connectCount += 1;
} });
async disconnect(): Promise<void> { readonly disconnect: Effect.Effect<void> = Effect.sync(() => {
this.disconnectCount += 1; this.disconnectCount += 1;
} });
} }
class FakeStoreGraph implements FalkorDBStoreGraph { class FakeStoreGraph implements FalkorDBStoreGraph {
readonly queries: string[] = []; readonly queries: string[] = [];
async query<T = unknown>(query: string): Promise<{ readonly data?: Array<T> }> { query<T = unknown>(query: string): Effect.Effect<{ readonly data?: Array<T> }> {
this.queries.push(query); return Effect.sync(() => {
return {}; this.queries.push(query);
return {};
});
} }
} }
class FakeQueryGraph implements FalkorDBQueryGraph { class FakeQueryGraph implements FalkorDBQueryGraph {
readonly queries: string[] = []; readonly queries: string[] = [];
async query<T = unknown>(query: string): Promise<{ readonly data?: Array<T> }> { query<T = unknown>(query: string): Effect.Effect<{ readonly data?: Array<T> }> {
this.queries.push(query); return Effect.sync(() => {
return {}; this.queries.push(query);
return {};
});
} }
} }

116
ts/packages/flow/src/__tests__/flow-manager-service.test.ts
View file

@ -19,29 +19,29 @@ import {FlowManagerError, makeFlowManagerService} from "../flow-manager/service.
class NoopPubSub implements PubSubBackend { class NoopPubSub implements PubSubBackend {
readonly sentByTopic = new Map<string, Array<unknown>>(); readonly sentByTopic = new Map<string, Array<unknown>>();
async createProducer<T>(options: CreateProducerOptions<T>): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>> {
return { return Effect.succeed({
send: async (message) => { send: (message) => Effect.sync(() => {
const sent = this.sentByTopic.get(options.topic) ?? []; const sent = this.sentByTopic.get(options.topic) ?? [];
sent.push(message); sent.push(message);
this.sentByTopic.set(options.topic, sent); this.sentByTopic.set(options.topic, sent);
}, }),
flush: async () => undefined, flush: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async createConsumer<T>(_options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(_options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
return { return Effect.succeed({
receive: async () => null, receive: () => Effect.succeed(null),
acknowledge: async () => undefined, acknowledge: () => Effect.void,
negativeAcknowledge: async () => undefined, negativeAcknowledge: () => Effect.void,
unsubscribe: async () => undefined, unsubscribe: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
class RecordingConfigClient implements RequestResponse<ConfigRequest, ConfigResponse> { class RecordingConfigClient implements RequestResponse<ConfigRequest, ConfigResponse> {
@ -53,25 +53,27 @@ class RecordingConfigClient implements RequestResponse<ConfigRequest, ConfigResp
private readonly legacyFlows: Array<{readonly key: string; readonly value: unknown}> = [], private readonly legacyFlows: Array<{readonly key: string; readonly value: unknown}> = [],
) {} ) {}
async start(): Promise<void> {} readonly start: Effect.Effect<void> = Effect.void;
async stop(): Promise<void> {} readonly stop: Effect.Effect<void> = Effect.void;
async request(request: ConfigRequest): Promise<ConfigResponse> { request(request: ConfigRequest): Effect.Effect<ConfigResponse> {
this.requests.push(request); return Effect.sync(() => {
if (request.operation !== "getvalues") return {}; this.requests.push(request);
if (request.operation !== "getvalues") return {};
if (request.type === "flow-blueprint") { if (request.type === "flow-blueprint") {
return {values: this.blueprints}; return {values: this.blueprints};
} }
if (request.type === "flow") { if (request.type === "flow") {
return {values: this.flows}; return {values: this.flows};
} }
if (request.type === "flows") { if (request.type === "flows") {
return {values: this.legacyFlows}; return {values: this.legacyFlows};
} }
return {values: []}; return {values: []};
});
} }
} }
@ -97,9 +99,9 @@ const seedResponseProducer = async (
backend: NoopPubSub, backend: NoopPubSub,
service: ReturnType<typeof makeFlowManagerService>, service: ReturnType<typeof makeFlowManagerService>,
) => { ) => {
const responseProducer = await backend.createProducer<FlowResponse>({ const responseProducer = await Effect.runPromise(backend.createProducer<FlowResponse>({
topic: topics.flowResponse, topic: topics.flowResponse,
}); }));
await Effect.runPromise( await Effect.runPromise(
SynchronizedRef.update(service.state, (state) => ({ SynchronizedRef.update(service.state, (state) => ({
...state, ...state,
@ -127,43 +129,43 @@ describe("FlowManagerService operations", () => {
const service = makeService(); const service = makeService();
await seedConfigClient(service, configClient); await seedConfigClient(service, configClient);
await expect(service.handleOperation({operation: "list-blueprints"})).resolves.toEqual({ await expect(Effect.runPromise(service.handleOperationEffect({operation: "list-blueprints"}))).resolves.toEqual({
"blueprint-names": ["custom", "default"], "blueprint-names": ["custom", "default"],
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "get-blueprint", operation: "get-blueprint",
"blueprint-name": "custom", "blueprint-name": "custom",
})).resolves.toMatchObject({ }))).resolves.toMatchObject({
"blueprint-definition": "{\"description\":\"Custom\",\"topics\":{\"input\":\"topic.in\"}}", "blueprint-definition": "{\"description\":\"Custom\",\"topics\":{\"input\":\"topic.in\"}}",
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "put-blueprint", operation: "put-blueprint",
"blueprint-name": "added", "blueprint-name": "added",
"blueprint-definition": {description: "Added", topics: {input: "topic.added"}}, "blueprint-definition": {description: "Added", topics: {input: "topic.added"}},
})).resolves.toEqual({}); }))).resolves.toEqual({});
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "delete-blueprint", operation: "delete-blueprint",
"blueprint-name": "custom", "blueprint-name": "custom",
})).resolves.toEqual({}); }))).resolves.toEqual({});
await expect(service.handleOperation({operation: "list-flows"})).resolves.toEqual({ await expect(Effect.runPromise(service.handleOperationEffect({operation: "list-flows"}))).resolves.toEqual({
"flow-ids": ["flow-a"], "flow-ids": ["flow-a"],
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "get-flow", operation: "get-flow",
"flow-id": "flow-a", "flow-id": "flow-a",
})).resolves.toEqual({ }))).resolves.toEqual({
flow: "{\"blueprint-name\":\"custom\",\"description\":\"Alpha\",\"parameters\":{\"limit\":3}}", flow: "{\"blueprint-name\":\"custom\",\"description\":\"Alpha\",\"parameters\":{\"limit\":3}}",
}); });
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "start-flow", operation: "start-flow",
"flow-id": "flow-b", "flow-id": "flow-b",
"blueprint-name": "custom", "blueprint-name": "custom",
})).resolves.toEqual({}); }))).resolves.toEqual({});
await expect(service.handleOperation({ await expect(Effect.runPromise(service.handleOperationEffect({
operation: "stop-flow", operation: "stop-flow",
"flow-id": "flow-a", "flow-id": "flow-a",
})).resolves.toEqual({}); }))).resolves.toEqual({});
await expect(service.handleOperation({operation: "unknown-flow"})).rejects.toMatchObject({ await expect(Effect.runPromise(service.handleOperationEffect({operation: "unknown-flow"}))).rejects.toMatchObject({
_tag: "FlowManagerError", _tag: "FlowManagerError",
operation: "operation", operation: "operation",
message: "Unknown flow operation: unknown-flow", message: "Unknown flow operation: unknown-flow",
@ -180,9 +182,9 @@ describe("FlowManagerService operations", () => {
it("uses tagged errors for invalid flow mutations", async () => { it("uses tagged errors for invalid flow mutations", async () => {
const service = makeService(); const service = makeService();
const startError = await service.handleStartFlow({operation: "start-flow"}) const startError = await Effect.runPromise(service.handleStartFlowEffect({operation: "start-flow"}))
.catch((caught: unknown) => caught); .catch((caught: unknown) => caught);
const stopError = await service.handleStopFlow({operation: "stop-flow"}) const stopError = await Effect.runPromise(service.handleStopFlowEffect({operation: "stop-flow"}))
.catch((caught: unknown) => caught); .catch((caught: unknown) => caught);
expect(startError).toBeInstanceOf(FlowManagerError); expect(startError).toBeInstanceOf(FlowManagerError);
@ -196,12 +198,12 @@ describe("FlowManagerService operations", () => {
const service = makeService(); const service = makeService();
await seedConfigClient(service, configClient); await seedConfigClient(service, configClient);
await service.handleStartFlow({ await Effect.runPromise(service.handleStartFlowEffect({
operation: "start-flow", operation: "start-flow",
"flow-id": "flow-a", "flow-id": "flow-a",
description: "alpha", description: "alpha",
parameters: {limit: 3}, parameters: {limit: 3},
}); }));
let state = await Effect.runPromise(SynchronizedRef.get(service.state)); let state = await Effect.runPromise(SynchronizedRef.get(service.state));
expect(Option.getOrUndefined(HashMap.get(state.flows, "flow-a"))).toMatchObject({ expect(Option.getOrUndefined(HashMap.get(state.flows, "flow-a"))).toMatchObject({
id: "flow-a", id: "flow-a",
@ -211,10 +213,10 @@ describe("FlowManagerService operations", () => {
status: "running", status: "running",
}); });
await service.handleStopFlow({ await Effect.runPromise(service.handleStopFlowEffect({
operation: "stop-flow", operation: "stop-flow",
"flow-id": "flow-a", "flow-id": "flow-a",
}); }));
state = await Effect.runPromise(SynchronizedRef.get(service.state)); state = await Effect.runPromise(SynchronizedRef.get(service.state));
expect(HashMap.has(state.flows, "flow-a")).toBe(false); expect(HashMap.has(state.flows, "flow-a")).toBe(false);
@ -245,7 +247,7 @@ describe("FlowManagerService operations", () => {
const service = makeService(); const service = makeService();
await seedConfigClient(service, configClient); await seedConfigClient(service, configClient);
await service.refreshBlueprintsFromConfig(); await Effect.runPromise(service.refreshBlueprintsFromConfigEffect);
const state = await Effect.runPromise(SynchronizedRef.get(service.state)); const state = await Effect.runPromise(SynchronizedRef.get(service.state));
expect(Option.getOrUndefined(HashMap.get(state.blueprints, "custom"))).toMatchObject({ expect(Option.getOrUndefined(HashMap.get(state.blueprints, "custom"))).toMatchObject({
@ -263,8 +265,8 @@ describe("FlowManagerService operations", () => {
await seedConfigClient(service, configClient); await seedConfigClient(service, configClient);
const results = await Promise.allSettled([ const results = await Promise.allSettled([
service.handleStartFlow({operation: "start-flow", "flow-id": "flow-a"}), Effect.runPromise(service.handleStartFlowEffect({operation: "start-flow", "flow-id": "flow-a"})),
service.handleStartFlow({operation: "start-flow", "flow-id": "flow-a"}), Effect.runPromise(service.handleStartFlowEffect({operation: "start-flow", "flow-id": "flow-a"})),
]); ]);
const state = await Effect.runPromise(SynchronizedRef.get(service.state)); const state = await Effect.runPromise(SynchronizedRef.get(service.state));

134
ts/packages/flow/src/__tests__/gateway-dispatcher.test.ts
View file

@ -18,6 +18,7 @@ import type {
Message, Message,
PubSubBackend, PubSubBackend,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { pubSubError } from "@trustgraph/base";
function createMessage<T>(value: T, properties: Record<string, string> = {}): Message<T> { function createMessage<T>(value: T, properties: Record<string, string> = {}): Message<T> {
return { return {
@ -44,32 +45,38 @@ class TopicConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || this.closed) return message ?? null; const message = this.messages.shift();
if (message !== undefined || this.closed) return Promise.resolve(message ?? null);
return await new Promise((resolve) => { return new Promise((resolve) => {
this.waiters.push(resolve); this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(message: Message<T>): Promise<void> { negativeAcknowledge(message: Message<T>): Effect.Effect<void> {
this.nacked.push(message); return Effect.sync(() => {
this.nacked.push(message);
});
} }
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
this.closeCount += 1; this.closeCount += 1;
} });
} }
class RecordingProducer<T> implements BackendProducer<T> { class RecordingProducer<T> implements BackendProducer<T> {
@ -82,18 +89,23 @@ class RecordingProducer<T> implements BackendProducer<T> {
private readonly onSend: (topic: string, message: T, properties?: Record<string, string>) => void, private readonly onSend: (topic: string, message: T, properties?: Record<string, string>) => void,
) {} ) {}
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.try({
this.onSend(this.topic, message, properties); try: () => {
this.sent.push(properties === undefined ? { message } : { message, properties });
this.onSend(this.topic, message, properties);
},
catch: (error) => pubSubError(`send:${this.topic}`, error),
});
} }
async flush(): Promise<void> { readonly flush: Effect.Effect<void> = Effect.sync(() => {
this.flushCount += 1; this.flushCount += 1;
} });
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
} }
class DispatchBackend implements PubSubBackend { class DispatchBackend implements PubSubBackend {
@ -104,31 +116,35 @@ class DispatchBackend implements PubSubBackend {
readonly consumersByTopic = new Map<string, TopicConsumer<unknown>>(); readonly consumersByTopic = new Map<string, TopicConsumer<unknown>>();
readonly failSendTopics = new Set<string>(); readonly failSendTopics = new Set<string>();
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
this.producerOptions.push(options); return Effect.sync(() => {
let producer = this.producersByTopic.get(options.topic); this.producerOptions.push(options);
if (producer === undefined) { let producer = this.producersByTopic.get(options.topic);
producer = new RecordingProducer<unknown>(options.topic, (topic, message, properties) => { if (producer === undefined) {
this.handleSend(topic, message, properties); producer = new RecordingProducer<unknown>(options.topic, (topic, message, properties) => {
}); this.handleSend(topic, message, properties);
this.producersByTopic.set(options.topic, producer); });
} this.producersByTopic.set(options.topic, producer);
return producer as BackendProducer<T>; }
return producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
this.consumerOptions.push(options); return Effect.sync(() => {
let consumer = this.consumersByTopic.get(options.topic); this.consumerOptions.push(options);
if (consumer === undefined) { let consumer = this.consumersByTopic.get(options.topic);
consumer = new TopicConsumer<unknown>(); if (consumer === undefined) {
this.consumersByTopic.set(options.topic, consumer); consumer = new TopicConsumer<unknown>();
} this.consumersByTopic.set(options.topic, consumer);
return consumer as BackendConsumer<T>; }
return consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closeCount += 1; this.closeCount += 1;
} });
private handleSend(topic: string, message: unknown, properties?: Record<string, string>): void { private handleSend(topic: string, message: unknown, properties?: Record<string, string>): void {
if (this.failSendTopics.has(topic)) { if (this.failSendTopics.has(topic)) {
@ -230,10 +246,10 @@ describe("gateway dispatcher manager", () => {
pubsub: backend, pubsub: backend,
}); });
await manager.start(); await Effect.runPromise(manager.start);
const first = await manager.dispatchGlobalService("config", { operation: "get" }); const first = await Effect.runPromise(manager.dispatchGlobalService("config", { operation: "get" }));
const second = await manager.dispatchGlobalService("config", { operation: "list" }); const second = await Effect.runPromise(manager.dispatchGlobalService("config", { operation: "list" }));
await manager.stop(); await Effect.runPromise(manager.stop);
expect(first).toEqual({ ok: true, echo: { operation: "get" } }); expect(first).toEqual({ ok: true, echo: { operation: "get" } });
expect(second).toEqual({ ok: true, echo: { operation: "list" } }); expect(second).toEqual({ ok: true, echo: { operation: "list" } });
@ -252,12 +268,12 @@ describe("gateway dispatcher manager", () => {
pubsub: backend, pubsub: backend,
}); });
await manager.start(); await Effect.runPromise(manager.start);
const [first, second] = await Promise.all([ const [first, second] = await Effect.runPromise(Effect.all([
manager.dispatchGlobalService("config", { operation: "get" }), manager.dispatchGlobalService("config", { operation: "get" }),
manager.dispatchGlobalService("config", { operation: "list" }), manager.dispatchGlobalService("config", { operation: "list" }),
]); ], { concurrency: "unbounded" }));
await manager.stop(); await Effect.runPromise(manager.stop);
expect(first).toEqual({ ok: true, echo: { operation: "get" } }); expect(first).toEqual({ ok: true, echo: { operation: "get" } });
expect(second).toEqual({ ok: true, echo: { operation: "list" } }); expect(second).toEqual({ ok: true, echo: { operation: "list" } });
@ -274,12 +290,12 @@ describe("gateway dispatcher manager", () => {
}); });
await expect( await expect(
manager.dispatchGlobalService("knowledge", { term: { t: "t" } }), Effect.runPromise(manager.dispatchGlobalService("knowledge", { term: { t: "t" } })),
).rejects.toMatchObject({ ).rejects.toMatchObject({
_tag: "DispatchSerializationError", _tag: "DispatchSerializationError",
operation: "client-term-to-internal", operation: "client-term-to-internal",
}); });
await manager.stop(); await Effect.runPromise(manager.stop);
expect(backend.producerOptions).toHaveLength(0); expect(backend.producerOptions).toHaveLength(0);
expect(backend.consumerOptions).toHaveLength(0); expect(backend.consumerOptions).toHaveLength(0);
@ -296,12 +312,12 @@ describe("gateway dispatcher manager", () => {
}); });
await expect( await expect(
manager.publishToTopic("tg.flow.ingest", { text: "hello" }, "msg-1"), Effect.runPromise(manager.publishToTopic("tg.flow.ingest", { text: "hello" }, "msg-1")),
).rejects.toMatchObject({ ).rejects.toMatchObject({
_tag: "MessagingDeliveryError", _tag: "MessagingDeliveryError",
operation: "send", operation: "send",
}); });
await manager.stop(); await Effect.runPromise(manager.stop);
expect(backend.producersByTopic.get("tg.flow.ingest")?.closeCount).toBe(1); expect(backend.producersByTopic.get("tg.flow.ingest")?.closeCount).toBe(1);
expect(backend.closeCount).toBe(0); expect(backend.closeCount).toBe(0);
@ -316,10 +332,14 @@ describe("gateway dispatcher manager", () => {
}); });
const chunks: Array<{ readonly response: unknown; readonly complete: boolean }> = []; const chunks: Array<{ readonly response: unknown; readonly complete: boolean }> = [];
await manager.dispatchGlobalServiceStreaming("knowledge", { query: "hello" }, async (response, complete) => { await Effect.runPromise(
chunks.push({ response, complete }); manager.dispatchGlobalServiceStreaming("knowledge", { query: "hello" }, (response, complete) =>
}); Effect.sync(() => {
await manager.stop(); chunks.push({ response, complete });
})
),
);
await Effect.runPromise(manager.stop);
expect(chunks).toEqual([ expect(chunks).toEqual([
{ response: { chunk: 1 }, complete: false }, { response: { chunk: 1 }, complete: false },
@ -337,13 +357,13 @@ describe("gateway dispatcher manager", () => {
const chunks: Array<{ readonly response: unknown; readonly complete: boolean }> = []; const chunks: Array<{ readonly response: unknown; readonly complete: boolean }> = [];
await Effect.runPromise( await Effect.runPromise(
manager.dispatchGlobalServiceStreamingEffect("knowledge", { query: "hello" }, (response, complete) => manager.dispatchGlobalServiceStreaming("knowledge", { query: "hello" }, (response, complete) =>
Effect.sync(() => { Effect.sync(() => {
chunks.push({ response, complete }); chunks.push({ response, complete });
}) })
), ),
); );
await manager.stop(); await Effect.runPromise(manager.stop);
expect(chunks).toEqual([ expect(chunks).toEqual([
{ response: { chunk: 1 }, complete: false }, { response: { chunk: 1 }, complete: false },

55
ts/packages/flow/src/__tests__/knowledge-core-service.test.ts
View file

@ -20,29 +20,29 @@ import {makeKnowledgeCoreService} from "../cores/service.js";
class NoopPubSub implements PubSubBackend { class NoopPubSub implements PubSubBackend {
readonly sentByTopic = new Map<string, Array<unknown>>(); readonly sentByTopic = new Map<string, Array<unknown>>();
async createProducer<T>(options: CreateProducerOptions<T>): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>> {
return { return Effect.succeed({
send: async (message) => { send: (message) => Effect.sync(() => {
const sent = this.sentByTopic.get(options.topic) ?? []; const sent = this.sentByTopic.get(options.topic) ?? [];
sent.push(message); sent.push(message);
this.sentByTopic.set(options.topic, sent); this.sentByTopic.set(options.topic, sent);
}, }),
flush: async () => undefined, flush: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async createConsumer<T>(_options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(_options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
return { return Effect.succeed({
receive: async () => null, receive: () => Effect.succeed(null),
acknowledge: async (_message: Message<T>) => undefined, acknowledge: (_message: Message<T>) => Effect.void,
negativeAcknowledge: async (_message: Message<T>) => undefined, negativeAcknowledge: (_message: Message<T>) => Effect.void,
unsubscribe: async () => undefined, unsubscribe: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
const sampleTriple: Triple = { const sampleTriple: Triple = {
@ -63,9 +63,9 @@ const seedResponseProducer = async (
backend: NoopPubSub, backend: NoopPubSub,
service: ReturnType<typeof makeKnowledgeCoreService>, service: ReturnType<typeof makeKnowledgeCoreService>,
) => { ) => {
const responseProducer = await backend.createProducer<KnowledgeResponse>({ const responseProducer = await Effect.runPromise(backend.createProducer<KnowledgeResponse>({
topic: topics.knowledgeResponse, topic: topics.knowledgeResponse,
}); }));
await Effect.runPromise( await Effect.runPromise(
SynchronizedRef.update(service.state, (state) => ({ SynchronizedRef.update(service.state, (state) => ({
...state, ...state,
@ -94,15 +94,15 @@ describe("KnowledgeCoreService operations", () => {
], ],
}; };
await service.putKgCore(request, "put-1"); await Effect.runPromise(service.putKgCoreEffect(request, "put-1"));
const state = await Effect.runPromise(SynchronizedRef.get(service.state)); const state = await Effect.runPromise(SynchronizedRef.get(service.state));
const core = Option.getOrUndefined(HashMap.get(state.kgCores, "alice:core-a")); const core = Option.getOrUndefined(HashMap.get(state.kgCores, "alice:core-a"));
await service.getKgCore({ await Effect.runPromise(service.getKgCoreEffect({
operation: "get-kg-core", operation: "get-kg-core",
user: "alice", user: "alice",
id: "core-a", id: "core-a",
}, "get-1"); }, "get-1"));
await rm(dir, {recursive: true, force: true}); await rm(dir, {recursive: true, force: true});
expect(core?.triples).toEqual([sampleTriple]); expect(core?.triples).toEqual([sampleTriple]);
@ -142,14 +142,14 @@ describe("KnowledgeCoreService operations", () => {
const service = makeService(dir, backend); const service = makeService(dir, backend);
await seedResponseProducer(backend, service); await seedResponseProducer(backend, service);
await Promise.all([ await Effect.runPromise(Effect.all([
service.putKgCore({ service.putKgCoreEffect({
operation: "put-kg-core", operation: "put-kg-core",
user: "alice", user: "alice",
id: "core-b", id: "core-b",
triples: [sampleTriple], triples: [sampleTriple],
}, "put-a"), }, "put-a"),
service.putKgCore({ service.putKgCoreEffect({
operation: "put-kg-core", operation: "put-kg-core",
user: "alice", user: "alice",
id: "core-b", id: "core-b",
@ -161,7 +161,10 @@ describe("KnowledgeCoreService operations", () => {
}, },
], ],
}, "put-b"), }, "put-b"),
]); ], {
concurrency: "unbounded",
discard: true,
}));
const state = await Effect.runPromise(SynchronizedRef.get(service.state)); const state = await Effect.runPromise(SynchronizedRef.get(service.state));
await rm(dir, {recursive: true, force: true}); await rm(dir, {recursive: true, force: true});
@ -183,7 +186,7 @@ describe("KnowledgeCoreService operations", () => {
); );
const service = makeService(dir); const service = makeService(dir);
await service.loadFromDisk(); await Effect.runPromise(service.loadFromDiskEffect);
const state = await Effect.runPromise(SynchronizedRef.get(service.state)); const state = await Effect.runPromise(SynchronizedRef.get(service.state));
await rm(dir, {recursive: true, force: true}); await rm(dir, {recursive: true, force: true});

87
ts/packages/flow/src/__tests__/librarian-service.test.ts
View file

@ -1,6 +1,7 @@
import {mkdtemp, rm} from "node:fs/promises"; import {mkdtemp, rm} from "node:fs/promises";
import {tmpdir} from "node:os"; import {tmpdir} from "node:os";
import {join} from "node:path"; import {join} from "node:path";
import {Effect} from "effect";
import {describe, expect, it} from "vitest"; import {describe, expect, it} from "vitest";
import { import {
type BackendConsumer, type BackendConsumer,
@ -15,25 +16,25 @@ import {
import {makeLibrarianService} from "../librarian/service.js"; import {makeLibrarianService} from "../librarian/service.js";
class NoopPubSub implements PubSubBackend { class NoopPubSub implements PubSubBackend {
async createProducer<T>(_options: CreateProducerOptions<T>): Promise<BackendProducer<T>> { createProducer<T>(_options: CreateProducerOptions<T>): Effect.Effect<BackendProducer<T>> {
return { return Effect.succeed({
send: async () => undefined, send: () => Effect.void,
flush: async () => undefined, flush: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async createConsumer<T>(_options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(_options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
return { return Effect.succeed({
receive: async () => null, receive: () => Effect.succeed(null),
acknowledge: async (_message: Message<T>) => undefined, acknowledge: (_message: Message<T>) => Effect.void,
negativeAcknowledge: async (_message: Message<T>) => undefined, negativeAcknowledge: (_message: Message<T>) => Effect.void,
unsubscribe: async () => undefined, unsubscribe: Effect.void,
close: async () => undefined, close: Effect.void,
}; });
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
const sampleTriple: Triple = { const sampleTriple: Triple = {
@ -66,40 +67,40 @@ describe("LibrarianService schema-backed boundaries", () => {
const service = makeService(dir); const service = makeService(dir);
try { try {
await expect(service.handleLibrarianOperation({ await expect(Effect.runPromise(service.handleLibrarianOperation({
operation: "list-documents", operation: "list-documents",
user: "alice", user: "alice",
})).resolves.toEqual({ }))).resolves.toEqual({
documents: [], documents: [],
"document-metadatas": [], "document-metadatas": [],
}); });
const upload = await service.handleLibrarianOperation({ const upload = await Effect.runPromise(service.handleLibrarianOperation({
operation: "begin-upload", operation: "begin-upload",
documentMetadata: sampleDocument, documentMetadata: sampleDocument,
"document-metadata": sampleDocument, "document-metadata": sampleDocument,
"total-size": 12, "total-size": 12,
"chunk-size": 4, "chunk-size": 4,
}); }));
await expect(service.handleLibrarianOperation({ await expect(Effect.runPromise(service.handleLibrarianOperation({
operation: "get-upload-status", operation: "get-upload-status",
"upload-id": upload["upload-id"], "upload-id": upload["upload-id"],
})).resolves.toMatchObject({ }))).resolves.toMatchObject({
"upload-id": upload["upload-id"], "upload-id": upload["upload-id"],
"upload-state": "in-progress", "upload-state": "in-progress",
"missing-chunks": [0, 1, 2], "missing-chunks": [0, 1, 2],
}); });
await expect(service.handleLibrarianOperation({ await expect(Effect.runPromise(service.handleLibrarianOperation({
operation: "stream-document", operation: "stream-document",
"document-id": "doc-a", "document-id": "doc-a",
})).rejects.toMatchObject({ }))).rejects.toMatchObject({
_tag: "LibrarianServiceError", _tag: "LibrarianServiceError",
operation: "stream-document", operation: "stream-document",
message: "stream-document must be handled as a streaming operation", message: "stream-document must be handled as a streaming operation",
}); });
await expect(service.handleLibrarianOperation(JSON.parse(`{"operation":"unknown-librarian"}`))).rejects.toMatchObject({ await expect(Effect.runPromise(service.handleLibrarianOperation(JSON.parse(`{"operation":"unknown-librarian"}`)))).rejects.toMatchObject({
_tag: "LibrarianServiceError", _tag: "LibrarianServiceError",
operation: "operation", operation: "operation",
message: "Unknown librarian operation: unknown-librarian", message: "Unknown librarian operation: unknown-librarian",
@ -114,14 +115,14 @@ describe("LibrarianService schema-backed boundaries", () => {
const service = makeService(dir); const service = makeService(dir);
try { try {
await expect(service.handleCollectionOperation({ await expect(Effect.runPromise(service.handleCollectionOperation({
operation: "update-collection", operation: "update-collection",
user: "alice", user: "alice",
collection: "docs", collection: "docs",
name: "Docs", name: "Docs",
description: "Documentation", description: "Documentation",
tags: ["reference"], tags: ["reference"],
})).resolves.toEqual({ }))).resolves.toEqual({
collections: [{ collections: [{
user: "alice", user: "alice",
collection: "docs", collection: "docs",
@ -131,10 +132,10 @@ describe("LibrarianService schema-backed boundaries", () => {
}], }],
}); });
await expect(service.handleCollectionOperation({ await expect(Effect.runPromise(service.handleCollectionOperation({
operation: "list-collections", operation: "list-collections",
user: "alice", user: "alice",
})).resolves.toEqual({ }))).resolves.toEqual({
collections: [{ collections: [{
user: "alice", user: "alice",
collection: "docs", collection: "docs",
@ -144,17 +145,17 @@ describe("LibrarianService schema-backed boundaries", () => {
}], }],
}); });
await expect(service.handleCollectionOperation({ await expect(Effect.runPromise(service.handleCollectionOperation({
operation: "delete-collection", operation: "delete-collection",
user: "alice", user: "alice",
collection: "docs", collection: "docs",
})).resolves.toEqual({}); }))).resolves.toEqual({});
await expect(service.handleCollectionOperation({ await expect(Effect.runPromise(service.handleCollectionOperation({
operation: "list-collections", operation: "list-collections",
user: "alice", user: "alice",
})).resolves.toEqual({collections: []}); }))).resolves.toEqual({collections: []});
await expect(service.handleCollectionOperation(JSON.parse(`{"operation":"unknown-collection"}`))).rejects.toMatchObject({ await expect(Effect.runPromise(service.handleCollectionOperation(JSON.parse(`{"operation":"unknown-collection"}`)))).rejects.toMatchObject({
_tag: "LibrarianServiceError", _tag: "LibrarianServiceError",
operation: "collection-operation", operation: "collection-operation",
message: "Unknown collection operation: unknown-collection", message: "Unknown collection operation: unknown-collection",
@ -168,18 +169,18 @@ describe("LibrarianService schema-backed boundaries", () => {
const dir = await mkdtemp(join(tmpdir(), "trustgraph-librarian-service-")); const dir = await mkdtemp(join(tmpdir(), "trustgraph-librarian-service-"));
const service = makeService(dir); const service = makeService(dir);
const response = await service.beginUpload({ const response = await Effect.runPromise(service.beginUpload({
operation: "begin-upload", operation: "begin-upload",
documentMetadata: sampleDocument, documentMetadata: sampleDocument,
"document-metadata": sampleDocument, "document-metadata": sampleDocument,
"total-size": 12, "total-size": 12,
"chunk-size": 4, "chunk-size": 4,
}); }));
const uploadId = response["upload-id"]; const uploadId = response["upload-id"];
const status = await service.getUploadStatus({ const status = await Effect.runPromise(service.getUploadStatus({
operation: "get-upload-status", operation: "get-upload-status",
"upload-id": uploadId, "upload-id": uploadId,
}); }));
await rm(dir, {recursive: true, force: true}); await rm(dir, {recursive: true, force: true});
expect(uploadId).toEqual(expect.any(String)); expect(uploadId).toEqual(expect.any(String));
@ -202,8 +203,8 @@ describe("LibrarianService schema-backed boundaries", () => {
); );
const service = makeService(dir); const service = makeService(dir);
await service.loadFromDisk(); await Effect.runPromise(service.loadFromDisk);
const documents = service.listDocuments({operation: "list-documents", user: "alice"}).documents; const documents = (await Effect.runPromise(service.listDocuments({operation: "list-documents", user: "alice"}))).documents;
await rm(dir, {recursive: true, force: true}); await rm(dir, {recursive: true, force: true});
expect(documents).toEqual([{ expect(documents).toEqual([{
@ -217,15 +218,15 @@ describe("LibrarianService schema-backed boundaries", () => {
const dir = await mkdtemp(join(tmpdir(), "trustgraph-librarian-service-")); const dir = await mkdtemp(join(tmpdir(), "trustgraph-librarian-service-"));
const service = makeService(dir); const service = makeService(dir);
const valid = await service.normaliseDocumentMetadata({ const valid = await Effect.runPromise(service.normaliseDocumentMetadata({
...sampleDocument, ...sampleDocument,
metadata: [sampleTriple], metadata: [sampleTriple],
}); }));
const invalid = await service.normaliseDocumentMetadata({ const invalid = await Effect.runPromise(service.normaliseDocumentMetadata({
...sampleDocument, ...sampleDocument,
id: "doc-b", id: "doc-b",
metadata: [{not: "a triple"}], metadata: [{not: "a triple"}],
}); }));
await rm(dir, {recursive: true, force: true}); await rm(dir, {recursive: true, force: true});
expect(valid.metadata).toEqual([sampleTriple]); expect(valid.metadata).toEqual([sampleTriple]);

70
ts/packages/flow/src/__tests__/prompt-template.test.ts
View file

@ -55,13 +55,15 @@ const waitFor = (condition: () => boolean, label: string) =>
class RecordingProducer<T> implements BackendProducer<T> { class RecordingProducer<T> implements BackendProducer<T> {
readonly sent: Array<{ readonly message: T; readonly properties?: Record<string, string> }> = []; readonly sent: Array<{ readonly message: T; readonly properties?: Record<string, string> }> = [];
async send(message: T, properties?: Record<string, string>): Promise<void> { send(message: T, properties?: Record<string, string>): Effect.Effect<void> {
this.sent.push(properties === undefined ? { message } : { message, properties }); return Effect.sync(() => {
this.sent.push(properties === undefined ? { message } : { message, properties });
});
} }
async flush(): Promise<void> {} readonly flush: Effect.Effect<void> = Effect.void;
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
} }
class PushConsumer<T> implements BackendConsumer<T> { class PushConsumer<T> implements BackendConsumer<T> {
@ -79,30 +81,36 @@ class PushConsumer<T> implements BackendConsumer<T> {
this.messages.push(message); this.messages.push(message);
} }
async receive(): Promise<Message<T> | null> { receive(): Effect.Effect<Message<T> | null> {
const message = this.messages.shift(); return Effect.promise(() => {
if (message !== undefined || this.closed) { const message = this.messages.shift();
return message ?? null; if (message !== undefined || this.closed) {
} return Promise.resolve(message ?? null);
return await new Promise((resolve) => { }
this.waiters.push(resolve); return new Promise<Message<T> | null>((resolve) => {
this.waiters.push(resolve);
});
}); });
} }
async acknowledge(message: Message<T>): Promise<void> { acknowledge(message: Message<T>): Effect.Effect<void> {
this.acknowledged.push(message); return Effect.sync(() => {
this.acknowledged.push(message);
});
} }
async negativeAcknowledge(): Promise<void> {} negativeAcknowledge(): Effect.Effect<void> {
return Effect.void;
}
async unsubscribe(): Promise<void> {} readonly unsubscribe: Effect.Effect<void> = Effect.void;
async close(): Promise<void> { readonly close: Effect.Effect<void> = Effect.sync(() => {
this.closed = true; this.closed = true;
for (const waiter of this.waiters.splice(0)) { for (const waiter of this.waiters.splice(0)) {
waiter(null); waiter(null);
} }
} });
} }
class PromptBackend implements PubSubBackend { class PromptBackend implements PubSubBackend {
@ -110,22 +118,26 @@ class PromptBackend implements PubSubBackend {
readonly consumersByTopic = new Map<string, PushConsumer<unknown>>(); readonly consumersByTopic = new Map<string, PushConsumer<unknown>>();
readonly producersByTopic = new Map<string, RecordingProducer<unknown>>(); readonly producersByTopic = new Map<string, RecordingProducer<unknown>>();
async createProducer<T>(options: CreateProducerOptions): Promise<BackendProducer<T>> { createProducer<T>(options: CreateProducerOptions): Effect.Effect<BackendProducer<T>> {
const producer = new RecordingProducer<unknown>(); return Effect.sync(() => {
this.producersByTopic.set(options.topic, producer); const producer = new RecordingProducer<unknown>();
return producer as BackendProducer<T>; this.producersByTopic.set(options.topic, producer);
return producer as BackendProducer<T>;
});
} }
async createConsumer<T>(options: CreateConsumerOptions): Promise<BackendConsumer<T>> { createConsumer<T>(options: CreateConsumerOptions): Effect.Effect<BackendConsumer<T>> {
if (options.topic === topics.configPush) { return Effect.sync(() => {
return this.configConsumer as unknown as BackendConsumer<T>; if (options.topic === topics.configPush) {
} return this.configConsumer as unknown as BackendConsumer<T>;
const consumer = new PushConsumer<unknown>(); }
this.consumersByTopic.set(options.topic, consumer); const consumer = new PushConsumer<unknown>();
return consumer as BackendConsumer<T>; this.consumersByTopic.set(options.topic, consumer);
return consumer as BackendConsumer<T>;
});
} }
async close(): Promise<void> {} readonly close: Effect.Effect<void> = Effect.void;
pushPromptConfig(): void { pushPromptConfig(): void {
this.configConsumer.push(createMessage({ this.configConsumer.push(createMessage({

55
ts/packages/flow/src/__tests__/qdrant-embeddings.test.ts
View file

@ -33,44 +33,51 @@ class FakeQdrantClient implements QdrantClientLike {
readonly deletedCollections: string[] = []; readonly deletedCollections: string[] = [];
searchResults: ReadonlyArray<QdrantScoredPoint> = []; searchResults: ReadonlyArray<QdrantScoredPoint> = [];
async collectionExists(collectionName: string): Promise<{ readonly exists: boolean }> { collectionExists(collectionName: string): Effect.Effect<{ readonly exists: boolean }> {
this.collectionExistsCalls.push(collectionName); return Effect.sync(() => {
return { exists: this.collections.has(collectionName) }; this.collectionExistsCalls.push(collectionName);
return { exists: this.collections.has(collectionName) };
});
} }
async createCollection( createCollection(
collectionName: string, collectionName: string,
options: { readonly vectors: { readonly size: number; readonly distance: "Cosine" } }, options: { readonly vectors: { readonly size: number; readonly distance: "Cosine" } },
): Promise<void> { ): Effect.Effect<void> {
this.collections.add(collectionName); return Effect.sync(() => {
this.createdCollections.push({ name: collectionName, size: options.vectors.size }); this.collections.add(collectionName);
this.createdCollections.push({ name: collectionName, size: options.vectors.size });
});
} }
async upsert( upsert(
collectionName: string, collectionName: string,
options: { readonly points: ReadonlyArray<FakePoint> }, options: { readonly points: ReadonlyArray<FakePoint> },
): Promise<void> { ): Effect.Effect<void> {
this.upserts.push({ collectionName, points: options.points }); return Effect.sync(() => {
this.upserts.push({ collectionName, points: options.points });
});
} }
async getCollections(): Promise<{ readonly collections: ReadonlyArray<{ readonly name: string }> }> { readonly getCollections: Effect.Effect<{ readonly collections: ReadonlyArray<{ readonly name: string }> }> =
return { collections: Array.from(this.collections, (name) => ({ name })) }; Effect.sync(() => ({ collections: Array.from(this.collections, (name) => ({ name })) }));
deleteCollection(collectionName: string): Effect.Effect<void> {
return Effect.sync(() => {
this.collections.delete(collectionName);
this.deletedCollections.push(collectionName);
});
} }
async deleteCollection(collectionName: string): Promise<void> { search(
this.collections.delete(collectionName);
this.deletedCollections.push(collectionName);
}
async search(
_collectionName: string, _collectionName: string,
_options: { _options: {
readonly vector: ReadonlyArray<number>; readonly vector: ReadonlyArray<number>;
readonly limit: number; readonly limit: number;
readonly with_payload: boolean; readonly with_payload: boolean;
}, },
): Promise<ReadonlyArray<QdrantScoredPoint>> { ): Effect.Effect<ReadonlyArray<QdrantScoredPoint>> {
return this.searchResults; return Effect.sync(() => this.searchResults);
} }
} }
@ -206,22 +213,22 @@ describe("Qdrant embeddings", () => {
}); });
await Effect.runPromise( await Effect.runPromise(
store.storeEffect({ store.store({
user: "alice", user: "alice",
collection: "docs", collection: "docs",
chunks: [{ chunkId: "chunk-a", vector: [1, 2], content: "alpha" }], chunks: [{ chunkId: "chunk-a", vector: [1, 2], content: "alpha" }],
}), }),
); );
await Effect.runPromise( await Effect.runPromise(
store.storeEffect({ store.store({
user: "alice", user: "alice",
collection: "docs", collection: "docs",
chunks: [{ chunkId: "chunk-b", vector: [2, 1], content: "beta" }], chunks: [{ chunkId: "chunk-b", vector: [2, 1], content: "beta" }],
}), }),
); );
await Effect.runPromise(store.deleteCollectionEffect("alice", "docs")); await Effect.runPromise(store.deleteCollection("alice", "docs"));
await Effect.runPromise( await Effect.runPromise(
store.storeEffect({ store.store({
user: "alice", user: "alice",
collection: "docs", collection: "docs",
chunks: [{ chunkId: "chunk-c", vector: [1, 1], content: "gamma" }], chunks: [{ chunkId: "chunk-c", vector: [1, 1], content: "gamma" }],

237
ts/packages/flow/src/__tests__/text-completion-common.test.ts
View file

@ -1,6 +1,5 @@
import { describe, expect, it } from "@effect/vitest"; import { describe, expect, it } from "@effect/vitest";
import type { LlmChunk } from "@trustgraph/base"; import { Context, Effect, Stream } from "effect";
import { Effect, Layer, ManagedRuntime, Stream } from "effect";
import { AiError, LanguageModel, Response } from "effect/unstable/ai"; import { AiError, LanguageModel, Response } from "effect/unstable/ai";
import { import {
llmStreamPart, llmStreamPart,
@ -9,10 +8,9 @@ import {
providerStatusError, providerStatusError,
streamTextCompletionChunks, streamTextCompletionChunks,
textFromContent, textFromContent,
toAsyncGenerator,
} from "../model/text-completion/common.js"; } from "../model/text-completion/common.js";
const languageModelRuntime = ManagedRuntime.make(Layer.empty); const languageModelContext = Context.empty();
const usage = (inputTokens: number, outputTokens: number) => ({ const usage = (inputTokens: number, outputTokens: number) => ({
inputTokens: { inputTokens: {
@ -42,12 +40,6 @@ const aiError = (reason: AiError.AiErrorReason) =>
reason, reason,
}); });
const emptyChunkIterator = (): AsyncIterable<LlmChunk> => ({
[Symbol.asyncIterator]: () => ({
next: () => Promise.resolve({ done: true, value: undefined }),
}),
});
describe("text completion common helpers", () => { describe("text completion common helpers", () => {
it("maps provider rate-limit status fields to tagged retry errors", () => { it("maps provider rate-limit status fields to tagged retry errors", () => {
expect(providerStatusError("test-provider", { status: 429 })).toMatchObject({ expect(providerStatusError("test-provider", { status: 429 })).toMatchObject({
@ -61,19 +53,6 @@ describe("text completion common helpers", () => {
}); });
}); });
it("maps fallback generator throw failures into tagged provider errors", async () => {
const generator = toAsyncGenerator(
emptyChunkIterator(),
(error) => providerRuntimeError("test-provider", error),
);
await expect(generator.throw("provider failed")).rejects.toMatchObject({
_tag: "TextCompletionProviderError",
provider: "test-provider",
message: "provider failed",
});
});
it.effect( it.effect(
"builds streaming chunks from async iterables with final token totals", "builds streaming chunks from async iterables with final token totals",
Effect.fnUntraced(function* () { Effect.fnUntraced(function* () {
@ -117,107 +96,129 @@ describe("text completion common helpers", () => {
expect(textFromContent([{ text: 1 }])).toBe(""); expect(textFromContent([{ text: 1 }])).toBe("");
}); });
it("adapts Effect LanguageModel generateText responses to LlmProvider results", async () => { it.effect(
const provider = makeLanguageModelProvider({ "adapts Effect LanguageModel generateText responses to LlmProvider results",
provider: "FakeLanguageModel", Effect.fnUntraced(function* () {
defaultModel: "fake-model",
defaultTemperature: 0.1,
runtime: languageModelRuntime,
makeLanguageModel: ({ model, temperature }) =>
LanguageModel.make({
generateText: () =>
Effect.succeed([
{ type: "text", text: `model=${model};temperature=${temperature}` },
finishPart(11, 7),
]),
streamText: () => Stream.empty,
}),
});
await expect(provider.generateContent("system", "prompt", "override-model", 0.4)).resolves.toEqual({
text: "model=override-model;temperature=0.4",
inToken: 11,
outToken: 7,
model: "override-model",
});
});
it("adapts Effect LanguageModel stream parts to TrustGraph chunks", async () => {
const provider = makeLanguageModelProvider({
provider: "FakeLanguageModel",
defaultModel: "fake-stream-model",
defaultTemperature: 0,
runtime: languageModelRuntime,
makeLanguageModel: () =>
LanguageModel.make({
generateText: () =>
Effect.succeed([
{ type: "text", text: "unused" },
finishPart(1, 1),
]),
streamText: () =>
Stream.fromArray([
Response.makePart("text-start", { id: "part-1" }),
{ type: "text-delta", id: "part-1", delta: "hel" },
{ type: "text-delta", id: "part-1", delta: "lo" },
finishPart(13, 8),
]),
}),
});
const chunks: Array<LlmChunk> = [];
for await (const chunk of provider.generateContentStream("system", "prompt")) {
chunks.push(chunk);
}
expect(chunks).toEqual([
{
text: "hel",
inToken: null,
outToken: null,
model: "fake-stream-model",
isFinal: false,
},
{
text: "lo",
inToken: null,
outToken: null,
model: "fake-stream-model",
isFinal: false,
},
{
text: "",
inToken: 13,
outToken: 8,
model: "fake-stream-model",
isFinal: true,
},
]);
});
it("maps Effect AI rate and quota failures to TrustGraph retry errors", async () => {
const reasons = [
new AiError.RateLimitError({}),
new AiError.QuotaExhaustedError({}),
];
for (const reason of reasons) {
const provider = makeLanguageModelProvider({ const provider = makeLanguageModelProvider({
provider: "FakeLanguageModel", provider: "FakeLanguageModel",
defaultModel: "fake-model", defaultModel: "fake-model",
defaultTemperature: 0, defaultTemperature: 0.1,
runtime: languageModelRuntime, context: languageModelContext,
makeLanguageModel: () => makeLanguageModel: ({ model, temperature }) =>
LanguageModel.make({ LanguageModel.make({
generateText: () => Effect.fail(aiError(reason)), generateText: () =>
streamText: () => Stream.fail(aiError(reason)), Effect.succeed([
{ type: "text", text: `model=${model};temperature=${temperature}` },
finishPart(11, 7),
]),
streamText: () => Stream.empty,
}), }),
}); });
await expect(provider.generateContent("system", "prompt")).rejects.toMatchObject({ const result = yield* provider.generateContent("system", "prompt", "override-model", 0.4);
_tag: "TooManyRequestsError", expect(result).toEqual({
message: "Rate limit exceeded", text: "model=override-model;temperature=0.4",
inToken: 11,
outToken: 7,
model: "override-model",
}); });
} }),
}); );
it.effect(
"adapts Effect LanguageModel stream parts to TrustGraph chunks",
Effect.fnUntraced(function* () {
const provider = makeLanguageModelProvider({
provider: "FakeLanguageModel",
defaultModel: "fake-stream-model",
defaultTemperature: 0,
context: languageModelContext,
makeLanguageModel: () =>
LanguageModel.make({
generateText: () =>
Effect.succeed([
{ type: "text", text: "unused" },
finishPart(1, 1),
]),
streamText: () =>
Stream.fromArray([
Response.makePart("text-start", { id: "part-1" }),
{ type: "text-delta", id: "part-1", delta: "hel" },
{ type: "text-delta", id: "part-1", delta: "lo" },
finishPart(13, 8),
]),
}),
});
const chunks = yield* Stream.runCollect(
provider.generateContentStream("system", "prompt"),
);
expect(Array.from(chunks)).toEqual([
{
text: "hel",
inToken: null,
outToken: null,
model: "fake-stream-model",
isFinal: false,
},
{
text: "lo",
inToken: null,
outToken: null,
model: "fake-stream-model",
isFinal: false,
},
{
text: "",
inToken: 13,
outToken: 8,
model: "fake-stream-model",
isFinal: true,
},
]);
}),
);
it.effect(
"maps Effect AI rate and quota failures to TrustGraph retry errors",
Effect.fnUntraced(function* () {
const reasons = [
new AiError.RateLimitError({}),
new AiError.QuotaExhaustedError({}),
];
for (const reason of reasons) {
const provider = makeLanguageModelProvider({
provider: "FakeLanguageModel",
defaultModel: "fake-model",
defaultTemperature: 0,
context: languageModelContext,
makeLanguageModel: () =>
LanguageModel.make({
generateText: () => Effect.fail(aiError(reason)),
streamText: () => Stream.fail(aiError(reason)),
}),
});
const generateError = yield* provider.generateContent("system", "prompt").pipe(
Effect.flip,
);
expect(generateError).toMatchObject({
_tag: "TooManyRequestsError",
message: "Rate limit exceeded",
});
const streamError = yield* Stream.runCollect(
provider.generateContentStream("system", "prompt"),
).pipe(
Effect.flip,
);
expect(streamError).toMatchObject({
_tag: "TooManyRequestsError",
message: "Rate limit exceeded",
});
}
}),
);
}); });

2
ts/packages/flow/src/agent/mcp-tool/index.ts
View file

@ -1 +1 @@
export { McpToolService, run, runMain } from "./service.js"; export { McpToolService, program, runMain } from "./service.js";

12
ts/packages/flow/src/agent/mcp-tool/service.ts
View file

@ -31,7 +31,7 @@ import {
type MessagingDeliveryError, type MessagingDeliveryError,
type Spec, type Spec,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Context, Effect, Layer, ManagedRuntime, Ref } from "effect"; import { Context, Effect, Layer, Ref } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
@ -342,9 +342,9 @@ export function makeMcpToolService(config: ProcessorConfig): McpToolService {
provide: (effect) => effect.pipe(Effect.provideService(McpToolRuntime, runtime)), provide: (effect) => effect.pipe(Effect.provideService(McpToolRuntime, runtime)),
}); });
service.registerConfigHandler((pushedConfig, version) => service.registerConfigHandler((pushedConfig, version) =>
Effect.runPromise(onMcpConfig(pushedConfig, version).pipe( onMcpConfig(pushedConfig, version).pipe(
Effect.provideService(McpToolRuntime, runtime), Effect.provideService(McpToolRuntime, runtime),
)), ),
); );
return service; return service;
} }
@ -358,12 +358,6 @@ export const program = makeFlowProcessorProgram<ProcessorConfig, never, McpToolR
layer: () => McpToolRuntimeLive, layer: () => McpToolRuntimeLive,
}); });
const mcpToolRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return mcpToolRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

36
ts/packages/flow/src/agent/react/service.ts
View file

@ -46,7 +46,7 @@ import {
type MessagingDeliveryError, type MessagingDeliveryError,
type Spec, type Spec,
} from "@trustgraph/base"; } from "@trustgraph/base";
import {Context, Effect, Layer, ManagedRuntime, Match, Ref} from "effect"; import {Context, Effect, Layer, Match, Ref} from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as Predicate from "effect/Predicate"; import * as Predicate from "effect/Predicate";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
@ -64,13 +64,6 @@ import type { AgentTool, ToolArg } from "./types.js";
const MAX_ITERATIONS = 10; const MAX_ITERATIONS = 10;
class AgentToolExecutionError extends S.TaggedErrorClass<AgentToolExecutionError>()(
"AgentToolExecutionError",
{
message: S.String,
},
) {}
const AgentResponseProducer = makeProducerSpec<AgentResponse>("agent-response"); const AgentResponseProducer = makeProducerSpec<AgentResponse>("agent-response");
const AgentLlmClient = makeRequestResponseSpec<TextCompletionRequest, TextCompletionResponse>( const AgentLlmClient = makeRequestResponseSpec<TextCompletionRequest, TextCompletionResponse>(
"llm", "llm",
@ -157,7 +150,7 @@ const buildConfiguredTool = Effect.fn("AgentService.buildConfiguredTool")(functi
: "Query the knowledge graph for information about entities and their relationships.", : "Query the knowledge graph for information about entities and their relationships.",
args: [{ name: "question", type: "string", description: "The question to ask" }], args: [{ name: "question", type: "string", description: "The question to ask" }],
config, config,
execute: () => Promise.resolve(""), execute: () => Effect.succeed(""),
}) })
), ),
@ -170,7 +163,7 @@ const buildConfiguredTool = Effect.fn("AgentService.buildConfiguredTool")(functi
: "Search documents for relevant information.", : "Search documents for relevant information.",
args: [{ name: "question", type: "string", description: "The question to search for" }], args: [{ name: "question", type: "string", description: "The question to search for" }],
config, config,
execute: () => Promise.resolve(""), execute: () => Effect.succeed(""),
}) })
), ),
@ -187,7 +180,7 @@ const buildConfiguredTool = Effect.fn("AgentService.buildConfiguredTool")(functi
{ name: "object", type: "string", description: "Object entity (optional)" }, { name: "object", type: "string", description: "Object entity (optional)" },
], ],
config, config,
execute: () => Promise.resolve(""), execute: () => Effect.succeed(""),
}) })
), ),
@ -203,7 +196,7 @@ const buildConfiguredTool = Effect.fn("AgentService.buildConfiguredTool")(functi
description, description,
args, args,
config, config,
execute: () => Promise.resolve(""), execute: () => Effect.succeed(""),
}); });
}), }),
@ -355,12 +348,9 @@ const executeTool = (
tool: AgentTool, tool: AgentTool,
input: string, input: string,
): Effect.Effect<string> => ): Effect.Effect<string> =>
Effect.tryPromise({ tool.execute(input).pipe(
try: () => tool.execute(input), Effect.catch((cause) =>
catch: (cause) => AgentToolExecutionError.make({ message: errorMessage(cause) }), Effect.succeed(`Error executing tool: ${errorMessage(cause)}`),
}).pipe(
Effect.catch((error: AgentToolExecutionError) =>
Effect.succeed(`Error executing tool: ${error.message}`),
), ),
); );
@ -520,9 +510,9 @@ export function makeAgentService(config: ProcessorConfig): AgentService {
provide: (effect) => effect.pipe(Effect.provideService(AgentRuntime, runtime)), provide: (effect) => effect.pipe(Effect.provideService(AgentRuntime, runtime)),
}); });
service.registerConfigHandler((pushedConfig, version) => service.registerConfigHandler((pushedConfig, version) =>
Effect.runPromise(onToolsConfig(pushedConfig, version).pipe( onToolsConfig(pushedConfig, version).pipe(
Effect.provideService(AgentRuntime, runtime), Effect.provideService(AgentRuntime, runtime),
)), ),
); );
Effect.runSync(Effect.log("[AgentService] Service initialized")); Effect.runSync(Effect.log("[AgentService] Service initialized"));
return service; return service;
@ -616,12 +606,6 @@ export const program = makeFlowProcessorProgram<ProcessorConfig, never, AgentRun
layer: () => AgentRuntimeLive, layer: () => AgentRuntimeLive,
}); });
const agentRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return agentRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

34
ts/packages/flow/src/agent/react/tools.ts
View file

@ -24,7 +24,7 @@ import * as O from "effect/Option";
import * as Predicate from "effect/Predicate"; import * as Predicate from "effect/Predicate";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import type { AgentTool, ToolArg } from "./types.js"; import { agentToolError, type AgentTool, type ToolArg } from "./types.js";
const decodeJsonUnknown = S.decodeUnknownOption(S.UnknownFromJsonString); const decodeJsonUnknown = S.decodeUnknownOption(S.UnknownFromJsonString);
const decodeTerm = S.decodeUnknownOption(TermSchema); const decodeTerm = S.decodeUnknownOption(TermSchema);
@ -88,14 +88,16 @@ export function createKnowledgeQueryTool(
description: "The question to ask the knowledge graph", description: "The question to ask the knowledge graph",
}, },
], ],
execute: (input: string): Promise<string> => Effect.runPromise(Effect.gen(function* () { execute: Effect.fn("KnowledgeQuery.execute")(function* (input: string) {
const question = parseQuestion(input); const question = parseQuestion(input);
yield* Effect.log(`[KnowledgeQuery] Executing: "${question.slice(0, 60)}..." collection=${collection}`); yield* Effect.log(`[KnowledgeQuery] Executing: "${question.slice(0, 60)}..." collection=${collection}`);
const request: GraphRagRequest = { const request: GraphRagRequest = {
query: question, query: question,
...(collection !== undefined ? { collection } : {}), ...(collection !== undefined ? { collection } : {}),
}; };
const res = yield* client.request(request); const res = yield* client.request(request).pipe(
Effect.mapError((cause) => agentToolError("knowledge-query", cause)),
);
yield* Effect.log(`[KnowledgeQuery] Response (${res.response?.length ?? 0} chars): ${res.error !== undefined ? `ERROR: ${res.error.message}` : `${res.response?.slice(0, 300)}...`}`); yield* Effect.log(`[KnowledgeQuery] Response (${res.response?.length ?? 0} chars): ${res.error !== undefined ? `ERROR: ${res.error.message}` : `${res.response?.slice(0, 300)}...`}`);
const explainTriples = res.explain_triples; const explainTriples = res.explain_triples;
@ -108,7 +110,7 @@ export function createKnowledgeQueryTool(
if (res.error !== undefined) return `Error: ${res.error.message}`; if (res.error !== undefined) return `Error: ${res.error.message}`;
return res.response; return res.response;
})), }),
}; };
} }
@ -130,16 +132,18 @@ export function createDocumentQueryTool(
description: "The question to search documents for", description: "The question to search documents for",
}, },
], ],
execute: (input: string): Promise<string> => Effect.runPromise(Effect.gen(function* () { execute: Effect.fn("DocumentQuery.execute")(function* (input: string) {
const question = parseQuestion(input); const question = parseQuestion(input);
const request: DocumentRagRequest = { const request: DocumentRagRequest = {
query: question, query: question,
...(collection !== undefined ? { collection } : {}), ...(collection !== undefined ? { collection } : {}),
}; };
const res = yield* client.request(request); const res = yield* client.request(request).pipe(
Effect.mapError((cause) => agentToolError("document-query", cause)),
);
if (res.error !== undefined) return `Error: ${res.error.message}`; if (res.error !== undefined) return `Error: ${res.error.message}`;
return res.response; return res.response;
})), }),
}; };
} }
@ -221,7 +225,7 @@ export function createTriplesQueryTool(
description: "The object entity to search for (optional)", description: "The object entity to search for (optional)",
}, },
], ],
execute: (input: string): Promise<string> => Effect.runPromise(Effect.gen(function* () { execute: Effect.fn("TriplesQuery.execute")(function* (input: string) {
const { s, p, o, limit } = parseTriplesInput(input); const { s, p, o, limit } = parseTriplesInput(input);
const request: TriplesQueryRequest = { const request: TriplesQueryRequest = {
limit: limit ?? 20, limit: limit ?? 20,
@ -230,7 +234,9 @@ export function createTriplesQueryTool(
...(o !== undefined ? { o } : {}), ...(o !== undefined ? { o } : {}),
...(collection !== undefined ? { collection } : {}), ...(collection !== undefined ? { collection } : {}),
}; };
const res = yield* client.request(request); const res = yield* client.request(request).pipe(
Effect.mapError((cause) => agentToolError("triples-query", cause)),
);
if (res.error !== undefined) return `Error: ${res.error.message}`; if (res.error !== undefined) return `Error: ${res.error.message}`;
@ -243,7 +249,7 @@ export function createTriplesQueryTool(
`(${termToString(t.s)}) -[${termToString(t.p)}]-> (${termToString(t.o)})`, `(${termToString(t.s)}) -[${termToString(t.p)}]-> (${termToString(t.o)})`,
); );
return lines.join("\n"); return lines.join("\n");
})), }),
}; };
} }
@ -263,12 +269,14 @@ export function createMcpTool(
name: toolName, name: toolName,
description, description,
args, args,
execute: (input: string): Promise<string> => Effect.runPromise(Effect.gen(function* () { execute: Effect.fn("McpTool.execute")(function* (input: string) {
const res = yield* client.request({ name: toolName, parameters: input }); const res = yield* client.request({ name: toolName, parameters: input }).pipe(
Effect.mapError((cause) => agentToolError("mcp-tool", cause)),
);
if (res.error !== undefined) return `Error: ${res.error.message}`; if (res.error !== undefined) return `Error: ${res.error.message}`;
if (res.text !== undefined) return res.text; if (res.text !== undefined) return res.text;
if (res.object !== undefined) return res.object; if (res.object !== undefined) return res.object;
return "No content"; return "No content";
})), }),
}; };
} }

26
ts/packages/flow/src/agent/react/types.ts
View file

@ -2,6 +2,24 @@
* Types for the ReAct agent service. * Types for the ReAct agent service.
*/ */
import type { Effect } from "effect";
import * as S from "effect/Schema";
import { errorMessage } from "@trustgraph/base";
export class AgentToolError extends S.TaggedErrorClass<AgentToolError>()(
"AgentToolError",
{
message: S.String,
operation: S.String,
},
) {}
export const agentToolError = (operation: string, cause: unknown): AgentToolError =>
AgentToolError.make({
operation,
message: errorMessage(cause),
});
export interface ToolArg { export interface ToolArg {
name: string; name: string;
type: string; type: string;
@ -12,7 +30,7 @@ export interface AgentTool {
name: string; name: string;
description: string; description: string;
args: ToolArg[]; args: ToolArg[];
execute: (input: string) => Promise<string>; execute: (input: string) => Effect.Effect<string, AgentToolError>;
/** Full tool config from config-push (used by tool filtering). */ /** Full tool config from config-push (used by tool filtering). */
config?: Record<string, unknown>; config?: Record<string, unknown>;
} }
@ -30,6 +48,6 @@ export interface ParsedEvent {
content: string; content: string;
} }
export type OnThought = (text: string, isFinal: boolean) => Promise<void>; export type OnThought = (text: string, isFinal: boolean) => Effect.Effect<void, AgentToolError>;
export type OnObservation = (text: string, isFinal: boolean) => Promise<void>; export type OnObservation = (text: string, isFinal: boolean) => Effect.Effect<void, AgentToolError>;
export type OnAnswer = (text: string) => Promise<void>; export type OnAnswer = (text: string) => Effect.Effect<void, AgentToolError>;

12
ts/packages/flow/src/chunking/service.ts
View file

@ -26,14 +26,14 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { recursiveSplit } from "./recursive-splitter.js"; import { recursiveSplit } from "./recursive-splitter.js";
const DEFAULT_CHUNK_SIZE = 2000; const DEFAULT_CHUNK_SIZE = 2000;
const DEFAULT_CHUNK_OVERLAP = 100; const DEFAULT_CHUNK_OVERLAP = 100;
const ChunkSizeParameter = makeParameterSpec("chunk-size", S.Number); const ChunkSizeParameter = makeParameterSpec("chunk-size", S.Finite);
const ChunkOverlapParameter = makeParameterSpec("chunk-overlap", S.Number); const ChunkOverlapParameter = makeParameterSpec("chunk-overlap", S.Finite);
const ChunkOutputProducer = makeProducerSpec<Chunk>("chunk-output"); const ChunkOutputProducer = makeProducerSpec<Chunk>("chunk-output");
const ChunkTriplesProducer = makeProducerSpec<Triples>("chunk-triples"); const ChunkTriplesProducer = makeProducerSpec<Triples>("chunk-triples");
@ -108,12 +108,6 @@ export const program = makeFlowProcessorProgram({
specs: () => makeChunkingSpecs(), specs: () => makeChunkingSpecs(),
}); });
const chunkingRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return chunkingRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

179
ts/packages/flow/src/config/service.ts
View file

@ -5,7 +5,7 @@
*/ */
import {NodeRuntime} from "@effect/platform-node"; import {NodeRuntime} from "@effect/platform-node";
import {Duration, Effect, HashMap, Layer, ManagedRuntime, Match, Option, SynchronizedRef} from "effect"; import {Duration, Effect, HashMap, Match, Option, SynchronizedRef} from "effect";
import * as Predicate from "effect/Predicate"; import * as Predicate from "effect/Predicate";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { import {
@ -16,6 +16,7 @@ import {
makeAsyncProcessor, makeAsyncProcessor,
makeProcessorProgram, makeProcessorProgram,
optionalStringConfig, optionalStringConfig,
processorLifecycleError,
topics, topics,
type AsyncProcessorRuntime, type AsyncProcessorRuntime,
type BackendConsumer, type BackendConsumer,
@ -26,7 +27,7 @@ import {
type Message, type Message,
type ProcessorConfig, type ProcessorConfig,
} from "@trustgraph/base"; } from "@trustgraph/base";
import {readTextFile, writeTextFile} from "../runtime/effect-files.js"; import {readTextFileEffect, writeTextFileEffect} from "../runtime/effect-files.js";
export interface ConfigServiceConfig extends ProcessorConfig { export interface ConfigServiceConfig extends ProcessorConfig {
readonly persistPath?: string; readonly persistPath?: string;
@ -38,7 +39,7 @@ interface ConfigPush {
} }
const ConfigPushSchema = S.Struct({ const ConfigPushSchema = S.Struct({
version: S.Number, version: S.Finite,
config: S.Record(S.String, S.Unknown), config: S.Record(S.String, S.Unknown),
}); });
@ -84,7 +85,7 @@ interface ConfigServiceState {
} }
const PersistedConfigSchema = S.Struct({ const PersistedConfigSchema = S.Struct({
version: S.optionalKey(S.Number), version: S.optionalKey(S.Finite),
data: S.optionalKey(S.Record(S.String, S.Record(S.String, S.Unknown))), data: S.optionalKey(S.Record(S.String, S.Record(S.String, S.Unknown))),
workspaces: S.optionalKey(S.Record(S.String, S.Record(S.String, S.Record(S.String, S.Unknown)))), workspaces: S.optionalKey(S.Record(S.String, S.Record(S.String, S.Record(S.String, S.Unknown)))),
}); });
@ -94,24 +95,17 @@ type PersistedConfig = typeof PersistedConfigSchema.Type;
export interface ConfigService extends AsyncProcessorRuntime<ConfigServiceError> { export interface ConfigService extends AsyncProcessorRuntime<ConfigServiceError> {
readonly state: SynchronizedRef.SynchronizedRef<ConfigServiceState>; readonly state: SynchronizedRef.SynchronizedRef<ConfigServiceState>;
readonly persistPath: string | null; readonly persistPath: string | null;
readonly handleMessage: (msg: Message<ConfigRequest>) => Promise<void>;
readonly handleMessageEffect: (msg: Message<ConfigRequest>) => Effect.Effect<void, ConfigServiceError>; readonly handleMessageEffect: (msg: Message<ConfigRequest>) => Effect.Effect<void, ConfigServiceError>;
readonly handleOperation: (request: ConfigRequest) => Promise<ConfigResponse>;
readonly handleOperationEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, ConfigServiceError>; readonly handleOperationEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, ConfigServiceError>;
readonly handleGet: (request: ConfigRequest) => ConfigResponse; readonly handleGet: (request: ConfigRequest) => ConfigResponse;
readonly handlePut: (request: ConfigRequest) => Promise<ConfigResponse>;
readonly handlePutEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, ConfigServiceError>; readonly handlePutEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, ConfigServiceError>;
readonly handleDelete: (request: ConfigRequest) => Promise<ConfigResponse>;
readonly handleDeleteEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, ConfigServiceError>; readonly handleDeleteEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, ConfigServiceError>;
readonly handleList: (request: ConfigRequest) => ConfigResponse; readonly handleList: (request: ConfigRequest) => ConfigResponse;
readonly handleGetValues: (request: ConfigRequest) => ConfigResponse; readonly handleGetValues: (request: ConfigRequest) => ConfigResponse;
readonly handleGetValuesAllWorkspaces: (request: ConfigRequest) => ConfigResponse; readonly handleGetValuesAllWorkspaces: (request: ConfigRequest) => ConfigResponse;
readonly handleConfigDump: (request: ConfigRequest) => ConfigResponse; readonly handleConfigDump: (request: ConfigRequest) => ConfigResponse;
readonly pushConfig: () => Promise<void>;
readonly pushConfigEffect: Effect.Effect<void, ConfigServiceError>; readonly pushConfigEffect: Effect.Effect<void, ConfigServiceError>;
readonly persist: () => Promise<void>;
readonly persistEffect: Effect.Effect<void>; readonly persistEffect: Effect.Effect<void>;
readonly loadFromDisk: () => Promise<void>;
readonly loadFromDiskEffect: Effect.Effect<void>; readonly loadFromDiskEffect: Effect.Effect<void>;
} }
@ -325,10 +319,9 @@ const persistStateEffect = Effect.fn("ConfigService.persistState")(
Effect.mapError((cause) => configServiceError("persist-encode", cause)), Effect.mapError((cause) => configServiceError("persist-encode", cause)),
); );
yield* Effect.tryPromise({ yield* writeTextFileEffect(persistPath, json).pipe(
try: () => writeTextFile(persistPath, json), Effect.mapError((cause) => configServiceError("persist-write", cause)),
catch: (cause) => configServiceError("persist-write", cause), );
});
}, },
(effect) => (effect) =>
effect.pipe( effect.pipe(
@ -344,24 +337,21 @@ const pushConfigWithStateEffect = Effect.fn("ConfigService.pushConfigWithState")
const pushProducer = state.pushProducer; const pushProducer = state.pushProducer;
if (pushProducer === null) return; if (pushProducer === null) return;
yield* Effect.tryPromise({ yield* pushProducer.send({
try: () => version: state.version,
pushProducer.send({ config: configDumpForState(state),
version: state.version, }).pipe(
config: configDumpForState(state), Effect.mapError((cause) => configServiceError("push-config", cause)),
}), );
catch: (cause) => configServiceError("push-config", cause),
});
yield* Effect.log(`[ConfigService] Pushed configuration version ${state.version}`); yield* Effect.log(`[ConfigService] Pushed configuration version ${state.version}`);
}); });
const readPersistedConfigEffect = Effect.fn("ConfigService.readPersistedConfig")( const readPersistedConfigEffect = Effect.fn("ConfigService.readPersistedConfig")(
function* (persistPath: string) { function* (persistPath: string) {
const raw = yield* Effect.tryPromise({ const raw = yield* readTextFileEffect(persistPath).pipe(
try: () => readTextFile(persistPath), Effect.mapError((cause) => configServiceError("persist-read", cause)),
catch: (cause) => configServiceError("persist-read", cause), );
});
return yield* S.decodeUnknownEffect(PersistedConfigJsonSchema)(raw).pipe( return yield* S.decodeUnknownEffect(PersistedConfigJsonSchema)(raw).pipe(
Effect.mapError((cause) => configServiceError("persist-decode", cause)), Effect.mapError((cause) => configServiceError("persist-decode", cause)),
); );
@ -644,24 +634,21 @@ const closeConfigResourcesEffect = Effect.fn("ConfigService.closeResources")(fun
const consumer = state.consumer; const consumer = state.consumer;
if (consumer !== null) { if (consumer !== null) {
yield* Effect.tryPromise({ yield* consumer.close.pipe(
try: () => consumer.close(), Effect.mapError((cause) => configServiceError("close-consumer", cause)),
catch: (cause) => configServiceError("close-consumer", cause), );
});
} }
const responseProducer = state.responseProducer; const responseProducer = state.responseProducer;
if (responseProducer !== null) { if (responseProducer !== null) {
yield* Effect.tryPromise({ yield* responseProducer.close.pipe(
try: () => responseProducer.close(), Effect.mapError((cause) => configServiceError("close-response-producer", cause)),
catch: (cause) => configServiceError("close-response-producer", cause), );
});
} }
const pushProducer = state.pushProducer; const pushProducer = state.pushProducer;
if (pushProducer !== null) { if (pushProducer !== null) {
yield* Effect.tryPromise({ yield* pushProducer.close.pipe(
try: () => pushProducer.close(), Effect.mapError((cause) => configServiceError("close-push-producer", cause)),
catch: (cause) => configServiceError("close-push-producer", cause), );
});
} }
yield* updateHandles(stateRef, { yield* updateHandles(stateRef, {
@ -680,17 +667,15 @@ const consumeOnceEffect = Effect.fnUntraced(function* (
return yield* configServiceError("consume", "Config consumer not started"); return yield* configServiceError("consume", "Config consumer not started");
} }
const msg = yield* Effect.tryPromise({ const msg = yield* consumer.receive(2000).pipe(
try: () => consumer.receive(2000), Effect.mapError((cause) => configServiceError("consume-receive", cause)),
catch: (cause) => configServiceError("consume-receive", cause), );
});
if (msg === null) return; if (msg === null) return;
yield* service.handleMessageEffect(msg); yield* service.handleMessageEffect(msg);
yield* Effect.tryPromise({ yield* consumer.acknowledge(msg).pipe(
try: () => consumer.acknowledge(msg), Effect.mapError((cause) => configServiceError("consume-acknowledge", cause)),
catch: (cause) => configServiceError("consume-acknowledge", cause), );
});
}); });
const runConfigServiceEffect = Effect.fn("ConfigService.run")(function* ( const runConfigServiceEffect = Effect.fn("ConfigService.run")(function* (
@ -698,35 +683,29 @@ const runConfigServiceEffect = Effect.fn("ConfigService.run")(function* (
) { ) {
yield* service.loadFromDiskEffect; yield* service.loadFromDiskEffect;
const responseProducer = yield* Effect.tryPromise({ const responseProducer = yield* service.pubsub.createProducer<ConfigResponse>({
try: () => topic: topics.configResponse,
service.pubsub.createProducer<ConfigResponse>({ schema: ConfigResponseSchema,
topic: topics.configResponse, }).pipe(
schema: ConfigResponseSchema, Effect.mapError((cause) => configServiceError("response-producer", cause)),
}), );
catch: (cause) => configServiceError("response-producer", cause),
});
yield* updateHandles(service.state, {responseProducer}); yield* updateHandles(service.state, {responseProducer});
const pushProducer = yield* Effect.tryPromise({ const pushProducer = yield* service.pubsub.createProducer<ConfigPush>({
try: () => topic: topics.configPush,
service.pubsub.createProducer<ConfigPush>({ schema: ConfigPushSchema,
topic: topics.configPush, }).pipe(
schema: ConfigPushSchema, Effect.mapError((cause) => configServiceError("push-producer", cause)),
}), );
catch: (cause) => configServiceError("push-producer", cause),
});
yield* updateHandles(service.state, {pushProducer}); yield* updateHandles(service.state, {pushProducer});
const consumer = yield* Effect.tryPromise({ const consumer = yield* service.pubsub.createConsumer<ConfigRequest>({
try: () => topic: topics.configRequest,
service.pubsub.createConsumer<ConfigRequest>({ subscription: `${service.config.id}-config-request`,
topic: topics.configRequest, schema: ConfigRequestSchema,
subscription: `${service.config.id}-config-request`, }).pipe(
schema: ConfigRequestSchema, Effect.mapError((cause) => configServiceError("consumer", cause)),
}), );
catch: (cause) => configServiceError("consumer", cause),
});
const state = yield* updateHandles(service.state, {consumer}); const state = yield* updateHandles(service.state, {consumer});
yield* pushConfigWithStateEffect(state); yield* pushConfigWithStateEffect(state);
@ -762,7 +741,6 @@ export function makeConfigService(config: ConfigServiceConfig): ConfigService {
const base = makeAsyncProcessor<ConfigServiceError>(config, { const base = makeAsyncProcessor<ConfigServiceError>(config, {
runEffect: () => getService.pipe(Effect.flatMap(runConfigServiceEffect)), runEffect: () => getService.pipe(Effect.flatMap(runConfigServiceEffect)),
}); });
const baseStop = base.stop;
const persistPath = config.persistPath ?? null; const persistPath = config.persistPath ?? null;
const handleOperationEffect = Effect.fn("ConfigService.handleOperation")(function* ( const handleOperationEffect = Effect.fn("ConfigService.handleOperation")(function* (
@ -800,10 +778,9 @@ export function makeConfigService(config: ConfigServiceConfig): ConfigService {
if (responseProducer === null) { if (responseProducer === null) {
return yield* configServiceError("respond", "Config response producer not started"); return yield* configServiceError("respond", "Config response producer not started");
} }
yield* Effect.tryPromise({ yield* responseProducer.send(response, {id: requestId}).pipe(
try: () => responseProducer.send(response, {id: requestId}), Effect.mapError((cause) => configServiceError("respond", cause)),
catch: (cause) => configServiceError("respond", cause), );
});
}); });
yield* handleOperationEffect(request).pipe( yield* handleOperationEffect(request).pipe(
@ -830,40 +807,42 @@ export function makeConfigService(config: ConfigServiceConfig): ConfigService {
yield* Effect.log(`[ConfigService] Loaded persisted config (version=${next.version}, workspaces=${HashMap.size(next.store)})`); yield* Effect.log(`[ConfigService] Loaded persisted config (version=${next.version}, workspaces=${HashMap.size(next.store)})`);
}); });
service = Object.assign(base, { const serviceStopEffect = closeConfigResourcesEffect(state).pipe(
Effect.mapError((cause) => processorLifecycleError(config.id, "stop", cause)),
Effect.flatMap(() => base.stop),
);
const serviceBase = Object.create(base, {
stop: {
value: serviceStopEffect,
writable: true,
enumerable: true,
configurable: true,
},
stopEffect: {
value: serviceStopEffect,
writable: true,
enumerable: true,
configurable: true,
},
});
service = Object.assign(serviceBase, {
state, state,
persistPath, persistPath,
handleMessage: (msg: Message<ConfigRequest>) => Effect.runPromise(handleMessageEffect(msg)),
handleMessageEffect, handleMessageEffect,
handleOperation: (request: ConfigRequest) => Effect.runPromise(handleOperationEffect(request)),
handleOperationEffect, handleOperationEffect,
handleGet: (request: ConfigRequest) => handleGetWithState(stateSnapshot(state), request), handleGet: (request: ConfigRequest) => handleGetWithState(stateSnapshot(state), request),
handlePut: (request: ConfigRequest) => Effect.runPromise(handlePutEffect(state, persistPath, request)),
handlePutEffect: (request: ConfigRequest) => handlePutEffect(state, persistPath, request), handlePutEffect: (request: ConfigRequest) => handlePutEffect(state, persistPath, request),
handleDelete: (request: ConfigRequest) => Effect.runPromise(handleDeleteEffect(state, persistPath, request)),
handleDeleteEffect: (request: ConfigRequest) => handleDeleteEffect(state, persistPath, request), handleDeleteEffect: (request: ConfigRequest) => handleDeleteEffect(state, persistPath, request),
handleList: (request: ConfigRequest) => handleListWithState(stateSnapshot(state), request), handleList: (request: ConfigRequest) => handleListWithState(stateSnapshot(state), request),
handleGetValues: (request: ConfigRequest) => handleGetValuesWithState(stateSnapshot(state), request), handleGetValues: (request: ConfigRequest) => handleGetValuesWithState(stateSnapshot(state), request),
handleGetValuesAllWorkspaces: (request: ConfigRequest) => handleGetValuesAllWorkspacesWithState(stateSnapshot(state), request), handleGetValuesAllWorkspaces: (request: ConfigRequest) => handleGetValuesAllWorkspacesWithState(stateSnapshot(state), request),
handleConfigDump: (request: ConfigRequest) => handleConfigDumpWithState(stateSnapshot(state), request), handleConfigDump: (request: ConfigRequest) => handleConfigDumpWithState(stateSnapshot(state), request),
pushConfig: () => Effect.runPromise(SynchronizedRef.get(state).pipe(Effect.flatMap(pushConfigWithStateEffect))),
pushConfigEffect: SynchronizedRef.get(state).pipe(Effect.flatMap(pushConfigWithStateEffect)), pushConfigEffect: SynchronizedRef.get(state).pipe(Effect.flatMap(pushConfigWithStateEffect)),
persist: () => Effect.runPromise(SynchronizedRef.get(state).pipe(Effect.flatMap((current) => persistStateEffect(persistPath, current)))),
persistEffect: SynchronizedRef.get(state).pipe(Effect.flatMap((current) => persistStateEffect(persistPath, current))), persistEffect: SynchronizedRef.get(state).pipe(Effect.flatMap((current) => persistStateEffect(persistPath, current))),
loadFromDisk: () => Effect.runPromise(loadFromDiskEffect()),
loadFromDiskEffect: loadFromDiskEffect(), loadFromDiskEffect: loadFromDiskEffect(),
stop: () => }) as ConfigService;
Effect.runPromise(
closeConfigResourcesEffect(state).pipe(
Effect.flatMap(() =>
Effect.tryPromise({
try: () => baseStop(),
catch: (cause) => configServiceError("stop", cause),
})
),
),
),
});
return service; return service;
} }
@ -887,12 +866,6 @@ export const program = makeProcessorProgram({
make: (config) => makeConfigService(config), make: (config) => makeConfigService(config),
}); });
const configServiceRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return configServiceRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

158
ts/packages/flow/src/cores/service.ts
View file

@ -15,6 +15,7 @@ import {
makeAsyncProcessor, makeAsyncProcessor,
makeProcessorProgram, makeProcessorProgram,
optionalStringConfig, optionalStringConfig,
processorLifecycleError,
topics, topics,
type AsyncProcessorRuntime, type AsyncProcessorRuntime,
type BackendConsumer, type BackendConsumer,
@ -24,17 +25,18 @@ import {
type KnowledgeResponse, type KnowledgeResponse,
type Message, type Message,
type ProcessorConfig, type ProcessorConfig,
type PubSubError,
} from "@trustgraph/base"; } from "@trustgraph/base";
import {Duration, Effect, HashMap, Layer, ManagedRuntime, Match, SynchronizedRef} from "effect"; import {Duration, Effect, HashMap, Match, SynchronizedRef} from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import {ensureDirectory, joinPath, readTextFile, writeTextFile} from "../runtime/effect-files.js"; import {ensureDirectoryEffect, joinPath, readTextFileEffect, writeTextFileEffect} from "../runtime/effect-files.js";
export interface KnowledgeCoreServiceConfig extends ProcessorConfig { export interface KnowledgeCoreServiceConfig extends ProcessorConfig {
readonly dataDir?: string; readonly dataDir?: string;
} }
const NumberArray = S.Array(S.Number).pipe(S.mutable); const NumberArray = S.Array(S.Finite).pipe(S.mutable);
const NumberArrays = S.Array(NumberArray).pipe(S.mutable); const NumberArrays = S.Array(NumberArray).pipe(S.mutable);
const GraphEmbeddingSchema = S.Struct({ const GraphEmbeddingSchema = S.Struct({
@ -98,35 +100,20 @@ export interface KnowledgeCoreService extends AsyncProcessorRuntime<KnowledgeCor
readonly coreKey: (user: string, id: string) => string; readonly coreKey: (user: string, id: string) => string;
readonly graphEmbeddings: (request: KnowledgeRequest) => ReadonlyArray<GraphEmbedding>; readonly graphEmbeddings: (request: KnowledgeRequest) => ReadonlyArray<GraphEmbedding>;
readonly documentEmbeddings: (request: KnowledgeRequest) => DocumentEmbeddingsCore | undefined; readonly documentEmbeddings: (request: KnowledgeRequest) => DocumentEmbeddingsCore | undefined;
readonly handleMessage: (msg: Message<KnowledgeRequest>) => Promise<void>;
readonly handleMessageEffect: (msg: Message<KnowledgeRequest>) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly handleMessageEffect: (msg: Message<KnowledgeRequest>) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly handleOperation: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly handleOperationEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly handleOperationEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly listKgCores: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly listKgCoresEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly listKgCoresEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly getKgCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly getKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly getKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly deleteKgCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly deleteKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly deleteKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly putKgCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly putKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly putKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly loadKgCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly loadKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly loadKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly unloadKgCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly unloadKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly unloadKgCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly listDeCores: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly listDeCoresEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly listDeCoresEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly getDeCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly getDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly getDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly deleteDeCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly deleteDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly deleteDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly putDeCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly putDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly putDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly loadDeCore: (request: KnowledgeRequest, requestId: string) => Promise<void>;
readonly loadDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>; readonly loadDeCoreEffect: (request: KnowledgeRequest, requestId: string) => Effect.Effect<void, KnowledgeCoreServiceError>;
readonly persist: () => Promise<void>;
readonly persistEffect: Effect.Effect<void, never>; readonly persistEffect: Effect.Effect<void, never>;
readonly loadFromDisk: () => Promise<void>;
readonly loadFromDiskEffect: Effect.Effect<void, never>; readonly loadFromDiskEffect: Effect.Effect<void, never>;
} }
@ -204,20 +191,12 @@ const updateHandles = (
responseProducer: handles.responseProducer === undefined ? state.responseProducer : handles.responseProducer, responseProducer: handles.responseProducer === undefined ? state.responseProducer : handles.responseProducer,
})); }));
const tryPromise = <A>(
operation: string,
evaluate: () => Promise<A>,
): Effect.Effect<A, KnowledgeCoreServiceError> =>
Effect.tryPromise({
try: evaluate,
catch: (cause) => knowledgeCoreServiceError(operation, cause),
});
const closeResource = ( const closeResource = (
resource: {readonly close: () => Promise<void>}, resource: {readonly close: Effect.Effect<void, PubSubError>},
operation: string, operation: string,
): Effect.Effect<void> => ): Effect.Effect<void> =>
tryPromise(operation, () => resource.close()).pipe( resource.close.pipe(
Effect.mapError((cause) => knowledgeCoreServiceError(operation, cause)),
Effect.catch((error) => Effect.catch((error) =>
Effect.logError("[KnowledgeCoreService] Failed to close resource", { Effect.logError("[KnowledgeCoreService] Failed to close resource", {
error: error.message, error: error.message,
@ -237,12 +216,16 @@ const sendResponse = Effect.fnUntraced(function* (
return yield* knowledgeCoreServiceError(operation, "Knowledge response producer not started"); return yield* knowledgeCoreServiceError(operation, "Knowledge response producer not started");
} }
yield* tryPromise(operation, () => responseProducer.send(response, {id: requestId})); yield* responseProducer.send(response, {id: requestId}).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError(operation, cause)),
);
}); });
const readPersistedKnowledgeEffect = Effect.fn("KnowledgeCoreService.readPersistedKnowledge")( const readPersistedKnowledgeEffect = Effect.fn("KnowledgeCoreService.readPersistedKnowledge")(
function* (persistPath: string) { function* (persistPath: string) {
const raw = yield* tryPromise("load-read", () => readTextFile(persistPath)); const raw = yield* readTextFileEffect(persistPath).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("load-read", cause)),
);
const current = S.decodeUnknownOption(PersistedKnowledgeSnapshotJsonSchema)(raw); const current = S.decodeUnknownOption(PersistedKnowledgeSnapshotJsonSchema)(raw);
if (O.isSome(current)) { if (O.isSome(current)) {
return { return {
@ -282,7 +265,9 @@ const persistStateEffect = Effect.fn("KnowledgeCoreService.persistState")(
const json = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(snapshot).pipe( const json = yield* S.encodeUnknownEffect(S.UnknownFromJsonString)(snapshot).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("persist-encode", cause)), Effect.mapError((cause) => knowledgeCoreServiceError("persist-encode", cause)),
); );
yield* tryPromise("persist-write", () => writeTextFile(persistPath, json)); yield* writeTextFileEffect(persistPath, json).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("persist-write", cause)),
);
}, },
(effect) => (effect) =>
effect.pipe( effect.pipe(
@ -317,12 +302,16 @@ const closeKnowledgeResourcesEffect = Effect.fn("KnowledgeCoreService.closeResou
const consumer = state.consumer; const consumer = state.consumer;
if (consumer !== null) { if (consumer !== null) {
yield* tryPromise("close-consumer", () => consumer.close()); yield* consumer.close.pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("close-consumer", cause)),
);
} }
const responseProducer = state.responseProducer; const responseProducer = state.responseProducer;
if (responseProducer !== null) { if (responseProducer !== null) {
yield* tryPromise("close-response-producer", () => responseProducer.close()); yield* responseProducer.close.pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("close-response-producer", cause)),
);
} }
yield* updateHandles(stateRef, { yield* updateHandles(stateRef, {
@ -339,33 +328,39 @@ const consumeOnceEffect = Effect.fnUntraced(function* (
return yield* knowledgeCoreServiceError("consume", "Knowledge request consumer not started"); return yield* knowledgeCoreServiceError("consume", "Knowledge request consumer not started");
} }
const msg = yield* tryPromise("consume-receive", () => consumer.receive(2000)); const msg = yield* consumer.receive(2000).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("consume-receive", cause)),
);
if (msg === null) return; if (msg === null) return;
yield* service.handleMessageEffect(msg); yield* service.handleMessageEffect(msg);
yield* tryPromise("consume-acknowledge", () => consumer.acknowledge(msg)); yield* consumer.acknowledge(msg).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("consume-acknowledge", cause)),
);
}); });
const runKnowledgeCoreServiceEffect = Effect.fn("KnowledgeCoreService.run")(function* ( const runKnowledgeCoreServiceEffect = Effect.fn("KnowledgeCoreService.run")(function* (
service: KnowledgeCoreService, service: KnowledgeCoreService,
) { ) {
yield* tryPromise("ensure-directory", () => ensureDirectory(service.dataDir)); yield* ensureDirectoryEffect(service.dataDir).pipe(
Effect.mapError((cause) => knowledgeCoreServiceError("ensure-directory", cause)),
);
yield* service.loadFromDiskEffect; yield* service.loadFromDiskEffect;
const responseProducer = yield* tryPromise("response-producer", () => const responseProducer = yield* service.pubsub.createProducer<KnowledgeResponse>({
service.pubsub.createProducer<KnowledgeResponse>({ topic: topics.knowledgeResponse,
topic: topics.knowledgeResponse, schema: KnowledgeResponseSchema,
schema: KnowledgeResponseSchema, }).pipe(
}), Effect.mapError((cause) => knowledgeCoreServiceError("response-producer", cause)),
); );
yield* updateHandles(service.state, {responseProducer}); yield* updateHandles(service.state, {responseProducer});
const consumer = yield* tryPromise("consumer", () => const consumer = yield* service.pubsub.createConsumer<KnowledgeRequest>({
service.pubsub.createConsumer<KnowledgeRequest>({ topic: topics.knowledgeRequest,
topic: topics.knowledgeRequest, subscription: `${service.config.id}-knowledge-request`,
subscription: `${service.config.id}-knowledge-request`, schema: KnowledgeRequestSchema,
schema: KnowledgeRequestSchema, }).pipe(
}), Effect.mapError((cause) => knowledgeCoreServiceError("consumer", cause)),
); );
yield* updateHandles(service.state, {consumer}); yield* updateHandles(service.state, {consumer});
@ -504,12 +499,11 @@ const loadKgCoreEffect = Effect.fn("loadKgCoreEffect")(function* (
if (core.triples.length > 0) { if (core.triples.length > 0) {
yield* Effect.acquireUseRelease( yield* Effect.acquireUseRelease(
tryPromise("triples-producer", () => service.pubsub.createProducer<unknown>({topic: "tg.flow.triples"}).pipe(
service.pubsub.createProducer<unknown>({topic: "tg.flow.triples"}), Effect.mapError((cause) => knowledgeCoreServiceError("triples-producer", cause)),
), ),
(producer) => (producer) =>
tryPromise("send-triples", () => producer.send({
producer.send({
metadata: { metadata: {
id: coreId, id: coreId,
root: coreId, root: coreId,
@ -517,8 +511,9 @@ const loadKgCoreEffect = Effect.fn("loadKgCoreEffect")(function* (
collection: request.collection ?? "default", collection: request.collection ?? "default",
}, },
triples: core.triples, triples: core.triples,
}), }).pipe(
), Effect.mapError((cause) => knowledgeCoreServiceError("send-triples", cause)),
),
(producer) => closeResource(producer, "close-triples-producer"), (producer) => closeResource(producer, "close-triples-producer"),
); );
} }
@ -637,7 +632,6 @@ export function makeKnowledgeCoreService(config: KnowledgeCoreServiceConfig): Kn
const base = makeAsyncProcessor<KnowledgeCoreServiceError>(config, { const base = makeAsyncProcessor<KnowledgeCoreServiceError>(config, {
runEffect: () => getService.pipe(Effect.flatMap(runKnowledgeCoreServiceEffect)), runEffect: () => getService.pipe(Effect.flatMap(runKnowledgeCoreServiceEffect)),
}); });
const baseStop = base.stop;
const handleOperationEffect = Effect.fn("KnowledgeCoreService.handleOperation")(function* ( const handleOperationEffect = Effect.fn("KnowledgeCoreService.handleOperation")(function* (
request: KnowledgeRequest, request: KnowledgeRequest,
@ -699,54 +693,50 @@ export function makeKnowledgeCoreService(config: KnowledgeCoreServiceConfig): Kn
yield* Effect.log(`[KnowledgeCoreService] Loaded persisted state (kg=${HashMap.size(next.kgCores)}, de=${HashMap.size(next.deCores)})`); yield* Effect.log(`[KnowledgeCoreService] Loaded persisted state (kg=${HashMap.size(next.kgCores)}, de=${HashMap.size(next.deCores)})`);
}); });
service = Object.assign(base, { const serviceStopEffect = closeKnowledgeResourcesEffect(state).pipe(
Effect.mapError((cause) => processorLifecycleError(config.id, "stop", cause)),
Effect.flatMap(() => base.stop),
);
const serviceBase = Object.create(base, {
stop: {
value: serviceStopEffect,
writable: true,
enumerable: true,
configurable: true,
},
stopEffect: {
value: serviceStopEffect,
writable: true,
enumerable: true,
configurable: true,
},
});
service = Object.assign(serviceBase, {
state, state,
dataDir, dataDir,
persistPath, persistPath,
coreKey, coreKey,
graphEmbeddings: graphEmbeddingsFor, graphEmbeddings: graphEmbeddingsFor,
documentEmbeddings: documentEmbeddingsFor, documentEmbeddings: documentEmbeddingsFor,
handleMessage: (msg: Message<KnowledgeRequest>) => Effect.runPromise(handleMessageEffect(msg)),
handleMessageEffect, handleMessageEffect,
handleOperation: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(handleOperationEffect(request, requestId)),
handleOperationEffect, handleOperationEffect,
listKgCores: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(listKgCoresEffect(state, request, requestId)),
listKgCoresEffect: (request: KnowledgeRequest, requestId: string) => listKgCoresEffect(state, request, requestId), listKgCoresEffect: (request: KnowledgeRequest, requestId: string) => listKgCoresEffect(state, request, requestId),
getKgCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(getKgCoreEffect(state, request, requestId)),
getKgCoreEffect: (request: KnowledgeRequest, requestId: string) => getKgCoreEffect(state, request, requestId), getKgCoreEffect: (request: KnowledgeRequest, requestId: string) => getKgCoreEffect(state, request, requestId),
deleteKgCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(deleteKgCoreEffect(state, persistPath, request, requestId)),
deleteKgCoreEffect: (request: KnowledgeRequest, requestId: string) => deleteKgCoreEffect(state, persistPath, request, requestId), deleteKgCoreEffect: (request: KnowledgeRequest, requestId: string) => deleteKgCoreEffect(state, persistPath, request, requestId),
putKgCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(putKgCoreEffect(state, persistPath, request, requestId)),
putKgCoreEffect: (request: KnowledgeRequest, requestId: string) => putKgCoreEffect(state, persistPath, request, requestId), putKgCoreEffect: (request: KnowledgeRequest, requestId: string) => putKgCoreEffect(state, persistPath, request, requestId),
loadKgCore: (request: KnowledgeRequest, requestId: string) =>
Effect.runPromise(getService.pipe(Effect.flatMap((current) => loadKgCoreEffect(state, current, request, requestId)))),
loadKgCoreEffect: (request: KnowledgeRequest, requestId: string) => loadKgCoreEffect: (request: KnowledgeRequest, requestId: string) =>
getService.pipe(Effect.flatMap((current) => loadKgCoreEffect(state, current, request, requestId))), getService.pipe(Effect.flatMap((current) => loadKgCoreEffect(state, current, request, requestId))),
unloadKgCore: (_request: KnowledgeRequest, requestId: string) => Effect.runPromise(sendResponse(state, {}, requestId)),
unloadKgCoreEffect: (_request: KnowledgeRequest, requestId: string) => sendResponse(state, {}, requestId), unloadKgCoreEffect: (_request: KnowledgeRequest, requestId: string) => sendResponse(state, {}, requestId),
listDeCores: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(listDeCoresEffect(state, request, requestId)),
listDeCoresEffect: (request: KnowledgeRequest, requestId: string) => listDeCoresEffect(state, request, requestId), listDeCoresEffect: (request: KnowledgeRequest, requestId: string) => listDeCoresEffect(state, request, requestId),
getDeCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(getDeCoreEffect(state, request, requestId)),
getDeCoreEffect: (request: KnowledgeRequest, requestId: string) => getDeCoreEffect(state, request, requestId), getDeCoreEffect: (request: KnowledgeRequest, requestId: string) => getDeCoreEffect(state, request, requestId),
deleteDeCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(deleteDeCoreEffect(state, persistPath, request, requestId)),
deleteDeCoreEffect: (request: KnowledgeRequest, requestId: string) => deleteDeCoreEffect(state, persistPath, request, requestId), deleteDeCoreEffect: (request: KnowledgeRequest, requestId: string) => deleteDeCoreEffect(state, persistPath, request, requestId),
putDeCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(putDeCoreEffect(state, persistPath, request, requestId)),
putDeCoreEffect: (request: KnowledgeRequest, requestId: string) => putDeCoreEffect(state, persistPath, request, requestId), putDeCoreEffect: (request: KnowledgeRequest, requestId: string) => putDeCoreEffect(state, persistPath, request, requestId),
loadDeCore: (request: KnowledgeRequest, requestId: string) => Effect.runPromise(loadDeCoreEffect(state, request, requestId)),
loadDeCoreEffect: (request: KnowledgeRequest, requestId: string) => loadDeCoreEffect(state, request, requestId), loadDeCoreEffect: (request: KnowledgeRequest, requestId: string) => loadDeCoreEffect(state, request, requestId),
persist: () => Effect.runPromise(SynchronizedRef.get(state).pipe(Effect.flatMap((current) => persistStateEffect(persistPath, current)))),
persistEffect: SynchronizedRef.get(state).pipe(Effect.flatMap((current) => persistStateEffect(persistPath, current))), persistEffect: SynchronizedRef.get(state).pipe(Effect.flatMap((current) => persistStateEffect(persistPath, current))),
loadFromDisk: () => Effect.runPromise(loadFromDiskEffect()),
loadFromDiskEffect: loadFromDiskEffect(), loadFromDiskEffect: loadFromDiskEffect(),
stop: () => }) as KnowledgeCoreService;
Effect.runPromise(
closeKnowledgeResourcesEffect(state).pipe(
Effect.flatMap(() =>
tryPromise("base-stop", () => baseStop())
),
),
),
});
return service; return service;
} }
@ -770,12 +760,6 @@ export const program = makeProcessorProgram({
make: (config) => makeKnowledgeCoreService(config), make: (config) => makeKnowledgeCoreService(config),
}); });
const knowledgeCoreRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return knowledgeCoreRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

10
ts/packages/flow/src/decoding/pdf-decoder.ts
View file

@ -39,7 +39,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Clock, Effect, Layer, ManagedRuntime } from "effect"; import { Clock, Effect } from "effect";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
export class PdfDecoderError extends S.TaggedErrorClass<PdfDecoderError>()( export class PdfDecoderError extends S.TaggedErrorClass<PdfDecoderError>()(
@ -48,7 +48,7 @@ export class PdfDecoderError extends S.TaggedErrorClass<PdfDecoderError>()(
message: S.String, message: S.String,
operation: S.String, operation: S.String,
documentId: S.String, documentId: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -257,12 +257,6 @@ export const program = makeFlowProcessorProgram({
specs: () => makePdfDecoderSpecs(), specs: () => makePdfDecoderSpecs(),
}); });
const pdfDecoderRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return pdfDecoderRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

15
ts/packages/flow/src/embeddings/ollama.ts
View file

@ -5,7 +5,7 @@
*/ */
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { Config, Effect, Layer, ManagedRuntime } from "effect"; import { Config, Effect, Layer } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { import {
@ -25,7 +25,7 @@ export interface OllamaEmbeddingsConfig extends ProcessorConfig {
fetch?: typeof fetch; fetch?: typeof fetch;
} }
const EmbeddingVector = S.Array(S.Number); const EmbeddingVector = S.Array(S.Finite);
const OllamaEmbedResponse = S.Struct({ const OllamaEmbedResponse = S.Struct({
embeddings: S.Array(EmbeddingVector), embeddings: S.Array(EmbeddingVector),
@ -71,9 +71,6 @@ const loadOllamaEmbeddingsConfig = Effect.fn("OllamaEmbeddings.loadConfig")(func
} satisfies ResolvedOllamaEmbeddingsConfig; } satisfies ResolvedOllamaEmbeddingsConfig;
}); });
const responseJson = (response: Response): Promise<unknown> =>
response.json();
const makeOllamaEmbeddingsFromConfig = ({ const makeOllamaEmbeddingsFromConfig = ({
defaultModel, defaultModel,
ollamaHost, ollamaHost,
@ -116,7 +113,7 @@ const makeOllamaEmbeddingsFromConfig = ({
} }
const data = yield* Effect.tryPromise({ const data = yield* Effect.tryPromise({
try: () => responseJson(response), try: () => response.json(),
catch: (error) => ollamaEmbeddingsError("ollama.response-json", error), catch: (error) => ollamaEmbeddingsError("ollama.response-json", error),
}); });
const decoded = yield* S.decodeUnknownEffect(OllamaEmbedResponse)(data).pipe( const decoded = yield* S.decodeUnknownEffect(OllamaEmbedResponse)(data).pipe(
@ -166,12 +163,6 @@ export const program = makeFlowProcessorProgram<OllamaEmbeddingsConfig, Embeddin
layer: (config) => OllamaEmbeddingsLive(config), layer: (config) => OllamaEmbeddingsLive(config),
}); });
const ollamaEmbeddingsRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return ollamaEmbeddingsRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

8
ts/packages/flow/src/extract/knowledge-extract.ts
View file

@ -35,7 +35,7 @@ import {
type Spec, type Spec,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
@ -392,12 +392,6 @@ export const program = makeFlowProcessorProgram({
specs: () => makeKnowledgeExtractSpecs(), specs: () => makeKnowledgeExtractSpecs(),
}); });
const knowledgeExtractRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return knowledgeExtractRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

254
ts/packages/flow/src/flow-manager/service.ts
View file

@ -30,23 +30,45 @@ import {
type FlowRequest, type FlowRequest,
type FlowResponse, type FlowResponse,
errorMessage, errorMessage,
processorLifecycleError,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { makeProcessorProgram } from "@trustgraph/base"; import { makeProcessorProgram } from "@trustgraph/base";
import type { Message } from "@trustgraph/base"; import type { Message } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { Duration, Effect, HashMap, Layer, ManagedRuntime, Match, Option, SynchronizedRef } from "effect"; import { Duration, Effect, HashMap, Match, Option, SynchronizedRef } from "effect";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
// ---------- Internal state types ---------- // ---------- Internal state types ----------
interface FlowInstance { class FlowInstanceRunning extends S.Class<FlowInstanceRunning>("FlowInstanceRunning")({
id: string; id: S.String,
blueprintName: string; blueprintName: S.String,
description: string; description: S.optionalKey(S.String),
parameters: Record<string, unknown>; parameters: S.Record(S.String, S.Unknown),
status: "running" | "stopped"; status: S.tag("running")
}) {}
class FlowInstanceStopped extends S.Class<FlowInstanceStopped>("FlowInstanceStopped")({
id: S.String,
blueprintName: S.String,
description: S.optionalKey(S.String),
parameters: S.Record(S.String, S.Unknown),
status: S.tag("stopped")
}) {
} }
export const FlowInstance = S.Union(
[
FlowInstanceRunning,
FlowInstanceStopped
]
).pipe(
S.toTaggedUnion("status")
);
export type FlowInstance = typeof FlowInstance.Type;
interface Blueprint { interface Blueprint {
description: string; description: string;
topics: Record<string, string>; topics: Record<string, string>;
@ -175,35 +197,21 @@ interface FlowManagerServiceState {
export interface FlowManagerService extends AsyncProcessorRuntime<FlowManagerError> { export interface FlowManagerService extends AsyncProcessorRuntime<FlowManagerError> {
readonly state: SynchronizedRef.SynchronizedRef<FlowManagerServiceState>; readonly state: SynchronizedRef.SynchronizedRef<FlowManagerServiceState>;
readonly handleMessage: (msg: Message<FlowRequest>) => Promise<void>;
readonly handleMessageEffect: (msg: Message<FlowRequest>) => Effect.Effect<void, FlowManagerError>; readonly handleMessageEffect: (msg: Message<FlowRequest>) => Effect.Effect<void, FlowManagerError>;
readonly configRequest: (request: ConfigRequest) => Promise<ConfigResponse>;
readonly configRequestEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, FlowManagerError>; readonly configRequestEffect: (request: ConfigRequest) => Effect.Effect<ConfigResponse, FlowManagerError>;
readonly ensureDefaultBlueprint: () => Promise<void>;
readonly ensureDefaultBlueprintEffect: Effect.Effect<void, FlowManagerError>; readonly ensureDefaultBlueprintEffect: Effect.Effect<void, FlowManagerError>;
readonly refreshBlueprintsFromConfig: () => Promise<void>;
readonly refreshBlueprintsFromConfigEffect: Effect.Effect<void, FlowManagerError>; readonly refreshBlueprintsFromConfigEffect: Effect.Effect<void, FlowManagerError>;
readonly refreshFlowsFromConfig: () => Promise<void>;
readonly refreshFlowsFromConfigEffect: Effect.Effect<void, FlowManagerError>; readonly refreshFlowsFromConfigEffect: Effect.Effect<void, FlowManagerError>;
readonly handleOperation: (request: FlowRequest) => Promise<FlowResponse>;
readonly handleOperationEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handleOperationEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly handleListBlueprints: () => FlowResponse; readonly handleListBlueprints: () => FlowResponse;
readonly handleGetBlueprint: (request: FlowRequest) => Promise<FlowResponse>;
readonly handleGetBlueprintEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handleGetBlueprintEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly handlePutBlueprint: (request: FlowRequest) => Promise<FlowResponse>;
readonly handlePutBlueprintEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handlePutBlueprintEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly handleDeleteBlueprint: (request: FlowRequest) => Promise<FlowResponse>;
readonly handleDeleteBlueprintEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handleDeleteBlueprintEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly handleListFlows: () => FlowResponse; readonly handleListFlows: () => FlowResponse;
readonly handleGetFlow: (request: FlowRequest) => Promise<FlowResponse>;
readonly handleGetFlowEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handleGetFlowEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly handleStartFlow: (request: FlowRequest) => Promise<FlowResponse>;
readonly handleStartFlowEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handleStartFlowEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly handleStopFlow: (request: FlowRequest) => Promise<FlowResponse>;
readonly handleStopFlowEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>; readonly handleStopFlowEffect: (request: FlowRequest) => Effect.Effect<FlowResponse, FlowManagerError>;
readonly pushFlowsConfig: () => Promise<void>;
readonly pushFlowsConfigEffect: Effect.Effect<void>; readonly pushFlowsConfigEffect: Effect.Effect<void>;
readonly deleteFlowConfig: (id: string) => Promise<void>;
readonly deleteFlowConfigEffect: (id: string) => Effect.Effect<void, FlowManagerError>; readonly deleteFlowConfigEffect: (id: string) => Effect.Effect<void, FlowManagerError>;
} }
@ -259,13 +267,13 @@ function blueprintFromConfig(value: unknown): Blueprint | undefined {
function flowFromConfig(id: string, value: unknown): FlowInstance | undefined { function flowFromConfig(id: string, value: unknown): FlowInstance | undefined {
const parsed = parseConfigRecord(value); const parsed = parseConfigRecord(value);
if (parsed === undefined) return undefined; if (parsed === undefined) return undefined;
return { return FlowInstanceRunning.make({
id, id,
blueprintName: optionalString(parsed["blueprint-name"]) ?? optionalString(parsed.blueprintName) ?? "default", blueprintName: optionalString(parsed["blueprint-name"]) ?? optionalString(parsed.blueprintName) ?? "default",
description: optionalString(parsed.description) ?? "", description: optionalString(parsed.description) ?? "",
parameters: isRecord(parsed.parameters) ? parsed.parameters : {}, parameters: isRecord(parsed.parameters) ? parsed.parameters : {},
status: "running", status: "running",
}; });
} }
const updateHandles = ( const updateHandles = (
@ -291,10 +299,9 @@ const configRequestEffect = Effect.fn("FlowManager.configRequest")(function* (
if (configClient === null) { if (configClient === null) {
return yield* flowManagerError("config-request", "Config client not started"); return yield* flowManagerError("config-request", "Config client not started");
} }
return yield* Effect.tryPromise({ return yield* configClient.request(request).pipe(
try: () => configClient.request(request), Effect.mapError((cause) => flowManagerError("config-request", cause)),
catch: (cause) => flowManagerError("config-request", cause), );
});
}); });
const ensureDefaultBlueprintEffect = Effect.fn("FlowManager.ensureDefaultBlueprint")(function* ( const ensureDefaultBlueprintEffect = Effect.fn("FlowManager.ensureDefaultBlueprint")(function* (
@ -571,24 +578,20 @@ const pushFlowsConfigEffect = Effect.fn("FlowManager.pushFlowsConfig")(
} }
} }
yield* Effect.tryPromise({ yield* configClient.request({
try: () => operation: "put",
configClient.request({ keys: ["flows"],
operation: "put", values: flowsConfig,
keys: ["flows"], }).pipe(
values: flowsConfig, Effect.mapError((cause) => flowManagerError("put-flows-config", cause)),
}), );
catch: (cause) => flowManagerError("put-flows-config", cause), yield* configClient.request({
}); operation: "put",
yield* Effect.tryPromise({ keys: ["flow"],
try: () => values: flowRecords,
configClient.request({ }).pipe(
operation: "put", Effect.mapError((cause) => flowManagerError("put-flow-records", cause)),
keys: ["flow"], );
values: flowRecords,
}),
catch: (cause) => flowManagerError("put-flow-records", cause),
});
yield* Effect.log(`[FlowManager] Pushed flows config (${HashMap.size(state.flows)} active flows)`); yield* Effect.log(`[FlowManager] Pushed flows config (${HashMap.size(state.flows)} active flows)`);
}, },
(effect) => (effect) =>
@ -605,22 +608,18 @@ const deleteFlowConfigEffect = Effect.fn("FlowManager.deleteFlowConfig")(functio
) { ) {
const configClient = (yield* SynchronizedRef.get(stateRef)).configClient; const configClient = (yield* SynchronizedRef.get(stateRef)).configClient;
if (configClient === null) return; if (configClient === null) return;
yield* Effect.tryPromise({ yield* configClient.request({
try: () => operation: "delete",
configClient.request({ keys: ["flows", id],
operation: "delete", }).pipe(
keys: ["flows", id], Effect.mapError((cause) => flowManagerError("delete-flows-config", cause)),
}), );
catch: (cause) => flowManagerError("delete-flows-config", cause), yield* configClient.request({
}); operation: "delete",
yield* Effect.tryPromise({ keys: ["flow", id],
try: () => }).pipe(
configClient.request({ Effect.mapError((cause) => flowManagerError("delete-flow-record", cause)),
operation: "delete", );
keys: ["flow", id],
}),
catch: (cause) => flowManagerError("delete-flow-record", cause),
});
}); });
const closeFlowManagerResourcesEffect = Effect.fn("FlowManager.closeResources")(function* ( const closeFlowManagerResourcesEffect = Effect.fn("FlowManager.closeResources")(function* (
@ -630,24 +629,19 @@ const closeFlowManagerResourcesEffect = Effect.fn("FlowManager.closeResources")(
const consumer = state.consumer; const consumer = state.consumer;
if (consumer !== null) { if (consumer !== null) {
yield* Effect.tryPromise({ yield* consumer.close.pipe(
try: () => consumer.close(), Effect.mapError((cause) => flowManagerError("consumer-close", cause)),
catch: (cause) => flowManagerError("consumer-close", cause), );
});
} }
const responseProducer = state.responseProducer; const responseProducer = state.responseProducer;
if (responseProducer !== null) { if (responseProducer !== null) {
yield* Effect.tryPromise({ yield* responseProducer.close.pipe(
try: () => responseProducer.close(), Effect.mapError((cause) => flowManagerError("response-producer-close", cause)),
catch: (cause) => flowManagerError("response-producer-close", cause), );
});
} }
const configClient = state.configClient; const configClient = state.configClient;
if (configClient !== null) { if (configClient !== null) {
yield* Effect.tryPromise({ yield* configClient.stop;
try: () => configClient.stop(),
catch: (cause) => flowManagerError("config-client-stop", cause),
});
} }
yield* updateHandles(stateRef, { yield* updateHandles(stateRef, {
@ -665,17 +659,15 @@ const consumeOnceEffect = Effect.fnUntraced(function* (
return yield* flowManagerError("consume", "Flow request consumer not started"); return yield* flowManagerError("consume", "Flow request consumer not started");
} }
const msg = yield* Effect.tryPromise({ const msg = yield* consumer.receive(2000).pipe(
try: () => consumer.receive(2000), Effect.mapError((cause) => flowManagerError("consume-receive", cause)),
catch: (cause) => flowManagerError("consume-receive", cause), );
});
if (msg === null) return; if (msg === null) return;
yield* service.handleMessageEffect(msg); yield* service.handleMessageEffect(msg);
yield* Effect.tryPromise({ yield* consumer.acknowledge(msg).pipe(
try: () => consumer.acknowledge(msg), Effect.mapError((cause) => flowManagerError("consume-acknowledge", cause)),
catch: (cause) => flowManagerError("consume-acknowledge", cause), );
});
}); });
const runFlowManagerServiceEffect = Effect.fn("FlowManager.runService")(function* ( const runFlowManagerServiceEffect = Effect.fn("FlowManager.runService")(function* (
@ -688,32 +680,27 @@ const runFlowManagerServiceEffect = Effect.fn("FlowManager.runService")(function
subscription: `${service.config.id}-config-client`, subscription: `${service.config.id}-config-client`,
}); });
yield* updateHandles(service.state, { configClient }); yield* updateHandles(service.state, { configClient });
yield* Effect.tryPromise({ yield* configClient.start.pipe(
try: () => configClient.start(), Effect.mapError((cause) => flowManagerError("config-client-start", cause)),
catch: (cause) => flowManagerError("config-client-start", cause), );
});
yield* ensureDefaultBlueprintEffect(service.state); yield* ensureDefaultBlueprintEffect(service.state);
yield* refreshBlueprintsFromConfigEffect(service.state); yield* refreshBlueprintsFromConfigEffect(service.state);
const responseProducer = yield* Effect.tryPromise({ const responseProducer = yield* service.pubsub.createProducer<FlowResponse>({
try: () => topic: topics.flowResponse,
service.pubsub.createProducer<FlowResponse>({ schema: FlowResponseSchema,
topic: topics.flowResponse, }).pipe(
schema: FlowResponseSchema, Effect.mapError((cause) => flowManagerError("response-producer", cause)),
}), );
catch: (cause) => flowManagerError("response-producer", cause),
});
yield* updateHandles(service.state, { responseProducer }); yield* updateHandles(service.state, { responseProducer });
const consumer = yield* Effect.tryPromise({ const consumer = yield* service.pubsub.createConsumer<FlowRequest>({
try: () => topic: topics.flowRequest,
service.pubsub.createConsumer<FlowRequest>({ subscription: `${service.config.id}-flow-request`,
topic: topics.flowRequest, schema: FlowRequestSchema,
subscription: `${service.config.id}-flow-request`, }).pipe(
schema: FlowRequestSchema, Effect.mapError((cause) => flowManagerError("consumer", cause)),
}), );
catch: (cause) => flowManagerError("consumer", cause),
});
yield* updateHandles(service.state, { consumer }); yield* updateHandles(service.state, { consumer });
yield* Effect.log(`[FlowManager] Listening on ${topics.flowRequest}`); yield* Effect.log(`[FlowManager] Listening on ${topics.flowRequest}`);
@ -748,7 +735,6 @@ export function makeFlowManagerService(config: ProcessorConfig): FlowManagerServ
const base = makeAsyncProcessor<FlowManagerError>(config, { const base = makeAsyncProcessor<FlowManagerError>(config, {
runEffect: () => getService.pipe(Effect.flatMap(runFlowManagerServiceEffect)), runEffect: () => getService.pipe(Effect.flatMap(runFlowManagerServiceEffect)),
}); });
const baseStop = base.stop;
const handleOperationEffect = Effect.fn("FlowManager.handleOperation")(function* (request: FlowRequest) { const handleOperationEffect = Effect.fn("FlowManager.handleOperation")(function* (request: FlowRequest) {
const op = optionalString(request.operation); const op = optionalString(request.operation);
@ -784,10 +770,9 @@ export function makeFlowManagerService(config: ProcessorConfig): FlowManagerServ
if (responseProducer === null) { if (responseProducer === null) {
return yield* flowManagerError("respond", "Flow response producer not started"); return yield* flowManagerError("respond", "Flow response producer not started");
} }
yield* Effect.tryPromise({ yield* responseProducer.send(response, { id: requestId }).pipe(
try: () => responseProducer.send(response, { id: requestId }), Effect.mapError((cause) => flowManagerError("respond", cause)),
catch: (cause) => flowManagerError("respond", cause), );
});
}); });
yield* handleOperationEffect(request).pipe( yield* handleOperationEffect(request).pipe(
@ -800,50 +785,45 @@ export function makeFlowManagerService(config: ProcessorConfig): FlowManagerServ
); );
}); });
const flowManagerService: FlowManagerService = Object.assign(base, { const serviceStopEffect = closeFlowManagerResourcesEffect(state).pipe(
Effect.mapError((cause) => processorLifecycleError(config.id, "stop", cause)),
Effect.flatMap(() => base.stop),
);
const serviceBase = Object.create(base, {
stop: {
value: serviceStopEffect,
writable: true,
enumerable: true,
configurable: true,
},
stopEffect: {
value: serviceStopEffect,
writable: true,
enumerable: true,
configurable: true,
},
});
const flowManagerService = Object.assign(serviceBase, {
state, state,
handleMessage: (msg: Message<FlowRequest>) => Effect.runPromise(handleMessageEffect(msg)),
handleMessageEffect, handleMessageEffect,
configRequest: (request: ConfigRequest) => Effect.runPromise(configRequestEffect(state, request)),
configRequestEffect: (request: ConfigRequest) => configRequestEffect(state, request), configRequestEffect: (request: ConfigRequest) => configRequestEffect(state, request),
ensureDefaultBlueprint: () => Effect.runPromise(ensureDefaultBlueprintEffect(state)),
ensureDefaultBlueprintEffect: ensureDefaultBlueprintEffect(state), ensureDefaultBlueprintEffect: ensureDefaultBlueprintEffect(state),
refreshBlueprintsFromConfig: () => Effect.runPromise(refreshBlueprintsFromConfigEffect(state)),
refreshBlueprintsFromConfigEffect: refreshBlueprintsFromConfigEffect(state), refreshBlueprintsFromConfigEffect: refreshBlueprintsFromConfigEffect(state),
refreshFlowsFromConfig: () => Effect.runPromise(refreshFlowsFromConfigEffect(state)),
refreshFlowsFromConfigEffect: refreshFlowsFromConfigEffect(state), refreshFlowsFromConfigEffect: refreshFlowsFromConfigEffect(state),
handleOperation: (request: FlowRequest) => Effect.runPromise(handleOperationEffect(request)),
handleOperationEffect, handleOperationEffect,
handleListBlueprints: () => handleListBlueprintsWithState(state.pipe(stateSnapshot)), handleListBlueprints: () => handleListBlueprintsWithState(state.pipe(stateSnapshot)),
handleGetBlueprint: (request: FlowRequest) => Effect.runPromise(handleGetBlueprintEffect(state, request)),
handleGetBlueprintEffect: (request: FlowRequest) => handleGetBlueprintEffect(state, request), handleGetBlueprintEffect: (request: FlowRequest) => handleGetBlueprintEffect(state, request),
handlePutBlueprint: (request: FlowRequest) => Effect.runPromise(handlePutBlueprintEffect(state, request)),
handlePutBlueprintEffect: (request: FlowRequest) => handlePutBlueprintEffect(state, request), handlePutBlueprintEffect: (request: FlowRequest) => handlePutBlueprintEffect(state, request),
handleDeleteBlueprint: (request: FlowRequest) => Effect.runPromise(handleDeleteBlueprintEffect(state, request)),
handleDeleteBlueprintEffect: (request: FlowRequest) => handleDeleteBlueprintEffect(state, request), handleDeleteBlueprintEffect: (request: FlowRequest) => handleDeleteBlueprintEffect(state, request),
handleListFlows: () => handleListFlowsWithState(state.pipe(stateSnapshot)), handleListFlows: () => handleListFlowsWithState(state.pipe(stateSnapshot)),
handleGetFlow: (request: FlowRequest) => Effect.runPromise(handleGetFlowEffect(state, request)),
handleGetFlowEffect: (request: FlowRequest) => handleGetFlowEffect(state, request), handleGetFlowEffect: (request: FlowRequest) => handleGetFlowEffect(state, request),
handleStartFlow: (request: FlowRequest) => Effect.runPromise(handleStartFlowEffect(state, request)),
handleStartFlowEffect: (request: FlowRequest) => handleStartFlowEffect(state, request), handleStartFlowEffect: (request: FlowRequest) => handleStartFlowEffect(state, request),
handleStopFlow: (request: FlowRequest) => Effect.runPromise(handleStopFlowEffect(state, request)),
handleStopFlowEffect: (request: FlowRequest) => handleStopFlowEffect(state, request), handleStopFlowEffect: (request: FlowRequest) => handleStopFlowEffect(state, request),
pushFlowsConfig: () => Effect.runPromise(pushFlowsConfigEffect(state)),
pushFlowsConfigEffect: pushFlowsConfigEffect(state), pushFlowsConfigEffect: pushFlowsConfigEffect(state),
deleteFlowConfig: (id: string) => Effect.runPromise(deleteFlowConfigEffect(state, id)),
deleteFlowConfigEffect: (id: string) => deleteFlowConfigEffect(state, id), deleteFlowConfigEffect: (id: string) => deleteFlowConfigEffect(state, id),
stop: () => }) as FlowManagerService;
Effect.runPromise(
closeFlowManagerResourcesEffect(state).pipe(
Effect.flatMap(() =>
Effect.tryPromise({
try: () => baseStop(),
catch: (cause) => flowManagerError("base-stop", cause),
})
),
),
),
});
service = flowManagerService; service = flowManagerService;
return flowManagerService; return flowManagerService;
@ -856,12 +836,6 @@ export const program = makeProcessorProgram({
make: (config) => makeFlowManagerService(config), make: (config) => makeFlowManagerService(config),
}); });
const flowManagerRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return flowManagerRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

141
ts/packages/flow/src/gateway/dispatch/manager.ts
View file

@ -31,7 +31,6 @@ import {
type DispatchSerializationError, type DispatchSerializationError,
} from "./serialize.js"; } from "./serialize.js";
export type Responder = (response: unknown, complete: boolean) => Promise<void>;
export type EffectResponder<E = never, R = never> = ( export type EffectResponder<E = never, R = never> = (
response: unknown, response: unknown,
complete: boolean, complete: boolean,
@ -106,18 +105,13 @@ function topicName(name: string): string {
// ---------- Manager ---------- // ---------- Manager ----------
export interface DispatcherManager { export interface DispatcherManager {
readonly start: () => Promise<void>; readonly start: Effect.Effect<void, MessagingLifecycleError>;
readonly stop: () => Promise<void>; readonly stop: Effect.Effect<void, MessagingLifecycleError>;
readonly dispatchGlobalService: ( readonly dispatchGlobalService: (
kind: string, kind: string,
request: Record<string, unknown>, request: Record<string, unknown>,
) => Promise<unknown>; ) => Effect.Effect<unknown, DispatcherStreamError>;
readonly dispatchGlobalServiceStreaming: ( readonly dispatchGlobalServiceStreaming: <E = never, R = never>(
kind: string,
request: Record<string, unknown>,
responder: Responder,
) => Promise<void>;
readonly dispatchGlobalServiceStreamingEffect: <E = never, R = never>(
kind: string, kind: string,
request: Record<string, unknown>, request: Record<string, unknown>,
responder: EffectResponder<E, R>, responder: EffectResponder<E, R>,
@ -126,14 +120,8 @@ export interface DispatcherManager {
flow: string, flow: string,
kind: string, kind: string,
request: Record<string, unknown>, request: Record<string, unknown>,
) => Promise<unknown>; ) => Effect.Effect<unknown, DispatcherStreamError>;
readonly dispatchFlowServiceStreaming: ( readonly dispatchFlowServiceStreaming: <E = never, R = never>(
flow: string,
kind: string,
request: Record<string, unknown>,
responder: Responder,
) => Promise<void>;
readonly dispatchFlowServiceStreamingEffect: <E = never, R = never>(
flow: string, flow: string,
kind: string, kind: string,
request: Record<string, unknown>, request: Record<string, unknown>,
@ -143,7 +131,7 @@ export interface DispatcherManager {
topic: string, topic: string,
message: unknown, message: unknown,
id?: string, id?: string,
) => Promise<void>; ) => Effect.Effect<void, MessagingDeliveryError>;
} }
export const dispatcherManagerFlowServiceNames = (): readonly string[] => [ export const dispatcherManagerFlowServiceNames = (): readonly string[] => [
@ -214,8 +202,6 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
runtime = nextRuntime; runtime = nextRuntime;
}); });
const start = (): Promise<void> => Effect.runPromise(startEffect());
const stopEffect = Effect.fn("DispatcherManager.stop")(function* () { const stopEffect = Effect.fn("DispatcherManager.stop")(function* () {
const current = runtime; const current = runtime;
runtime = null; runtime = null;
@ -225,15 +211,12 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
} }
if (ownsPubSub) { if (ownsPubSub) {
yield* Effect.tryPromise({ yield* pubsub.close.pipe(
try: () => pubsub.close(), Effect.mapError((cause) => messagingLifecycleError("gateway-dispatcher", "close-pubsub", cause)),
catch: (cause) => messagingLifecycleError("gateway-dispatcher", "close-pubsub", cause), );
});
} }
}); });
const stop = (): Promise<void> => Effect.runPromise(stopEffect());
// ---------- Internal helpers ---------- // ---------- Internal helpers ----------
const ensureRuntimeEffect = Effect.fn("DispatcherManager.ensureRuntime")(function* () { const ensureRuntimeEffect = Effect.fn("DispatcherManager.ensureRuntime")(function* () {
@ -303,13 +286,7 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
// ---------- Global service dispatch ---------- // ---------- Global service dispatch ----------
const dispatchGlobalService = ( const dispatchGlobalService = Effect.fn("DispatcherManager.dispatchGlobalService")(function* (
kind: string,
request: Record<string, unknown>,
): Promise<unknown> =>
Effect.runPromise(dispatchGlobalServiceEffect(kind, request));
const dispatchGlobalServiceEffect = Effect.fn("DispatcherManager.dispatchGlobalService")(function* (
kind: string, kind: string,
request: Record<string, unknown>, request: Record<string, unknown>,
) { ) {
@ -321,7 +298,7 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
return yield* translateResponseEffect(kind, response); return yield* translateResponseEffect(kind, response);
}); });
const dispatchGlobalServiceStreamingEffect = Effect.fn("DispatcherManager.dispatchGlobalServiceStreaming")(function* < const dispatchGlobalServiceStreaming = Effect.fn("DispatcherManager.dispatchGlobalServiceStreaming")(function* <
E, E,
R, R,
>( >(
@ -342,34 +319,9 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
}); });
}); });
const dispatchGlobalServiceStreaming = (
kind: string,
request: Record<string, unknown>,
responder: Responder,
): Promise<void> =>
Effect.runPromise(
dispatchGlobalServiceStreamingEffect(kind, request, (response, complete) =>
Effect.tryPromise({
try: () => responder(response, complete),
catch: (error) => messagingDeliveryError(
resolveGlobalTopics(kind).responseTopic,
"stream-responder",
error,
),
})
),
);
// ---------- Flow-scoped service dispatch ---------- // ---------- Flow-scoped service dispatch ----------
const dispatchFlowService = ( const dispatchFlowService = Effect.fn("DispatcherManager.dispatchFlowService")(function* (
flow: string,
kind: string,
request: Record<string, unknown>,
): Promise<unknown> =>
Effect.runPromise(dispatchFlowServiceEffect(flow, kind, request));
const dispatchFlowServiceEffect = Effect.fn("DispatcherManager.dispatchFlowService")(function* (
flow: string, flow: string,
kind: string, kind: string,
request: Record<string, unknown>, request: Record<string, unknown>,
@ -386,7 +338,7 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
return yield* translateResponseEffect(kind, response); return yield* translateResponseEffect(kind, response);
}); });
const dispatchFlowServiceStreamingEffect = Effect.fn("DispatcherManager.dispatchFlowServiceStreaming")(function* < const dispatchFlowServiceStreaming = Effect.fn("DispatcherManager.dispatchFlowServiceStreaming")(function* <
E, E,
R, R,
>( >(
@ -412,65 +364,40 @@ export function makeDispatcherManager(config: GatewayConfig): DispatcherManager
}); });
}); });
const dispatchFlowServiceStreaming = (
flow: string,
kind: string,
request: Record<string, unknown>,
responder: Responder,
): Promise<void> =>
Effect.runPromise(
dispatchFlowServiceStreamingEffect(flow, kind, request, (response, complete) =>
Effect.tryPromise({
try: () => responder(response, complete),
catch: (error) => messagingDeliveryError(
resolveFlowTopics(kind).responseTopic,
"stream-responder",
error,
),
})
),
);
// ---------- Fire-and-forget publish ---------- // ---------- Fire-and-forget publish ----------
/** /**
* Publish a single message to an arbitrary topic (no request/response). * Publish a single message to an arbitrary topic (no request/response).
* Used for injecting documents into the processing pipeline. * Used for injecting documents into the processing pipeline.
*/ */
const publishToTopic = (topic: string, message: unknown, id?: string): Promise<void> => const publishToTopic = (topic: string, message: unknown, id?: string) =>
Effect.runPromise( Effect.acquireUseRelease(
Effect.acquireUseRelease( pubsub.createProducer<unknown>({ topic }).pipe(
Effect.tryPromise({ Effect.mapError((cause) => messagingDeliveryError(topic, "create-producer", cause)),
try: () => pubsub.createProducer<unknown>({ topic }),
catch: (cause) => messagingDeliveryError(topic, "create-producer", cause),
}),
(producer) =>
Effect.gen(function* () {
const timestamp = yield* Clock.currentTimeMillis;
const suffix = yield* Random.nextIntBetween(0, 36 ** 6, { halfOpen: true });
const messageId = id ?? `pub-${timestamp}-${suffix.toString(36).padStart(6, "0")}`;
yield* Effect.tryPromise({
try: () => producer.send(message, { id: messageId }),
catch: (cause) => messagingDeliveryError(topic, "send", cause),
});
}),
(producer) => Effect.tryPromise({
try: () => producer.close(),
catch: (cause) => messagingDeliveryError(topic, "close-producer", cause),
}),
), ),
(producer) =>
Effect.gen(function* () {
const timestamp = yield* Clock.currentTimeMillis;
const suffix = yield* Random.nextIntBetween(0, 36 ** 6, { halfOpen: true });
const messageId = id ?? `pub-${timestamp}-${suffix.toString(36).padStart(6, "0")}`;
yield* producer.send(message, { id: messageId }).pipe(
Effect.mapError((cause) => messagingDeliveryError(topic, "send", cause)),
);
}),
(producer) =>
producer.close.pipe(
Effect.mapError((cause) => messagingDeliveryError(topic, "close-producer", cause)),
),
); );
return { return {
start, start: startEffect(),
stop, stop: stopEffect(),
dispatchGlobalService, dispatchGlobalService,
dispatchGlobalServiceStreaming, dispatchGlobalServiceStreaming,
dispatchGlobalServiceStreamingEffect,
dispatchFlowService, dispatchFlowService,
dispatchFlowServiceStreaming, dispatchFlowServiceStreaming,
dispatchFlowServiceStreamingEffect,
publishToTopic, publishToTopic,
}; };
} }

2
ts/packages/flow/src/gateway/index.ts
View file

@ -1,4 +1,4 @@
export { createGateway, run, type GatewayConfig } from "./server.js"; export { createGateway, program, runMain, type GatewayConfig } from "./server.js";
export { export {
dispatcherManagerFlowServiceNames, dispatcherManagerFlowServiceNames,
dispatcherManagerGlobalServiceNames, dispatcherManagerGlobalServiceNames,

13
ts/packages/flow/src/gateway/rpc-server.ts
View file

@ -40,10 +40,9 @@ const makeGatewayRpcHandlers = (dispatcher: DispatcherManager) =>
TrustGraphRpcs.toLayer(Effect.succeed( TrustGraphRpcs.toLayer(Effect.succeed(
TrustGraphRpcs.of({ TrustGraphRpcs.of({
Dispatch: (payload) => Dispatch: (payload) =>
Effect.tryPromise({ dispatchOne(dispatcher, payload).pipe(
try: () => dispatchOne(dispatcher, payload), Effect.mapError((cause) => DispatchError.make({ message: errorMessage(cause) })),
catch: (cause) => DispatchError.make({ message: errorMessage(cause) }), ),
}),
DispatchStream: Effect.fn("GatewayRpc.DispatchStream")(function* (payload) { DispatchStream: Effect.fn("GatewayRpc.DispatchStream")(function* (payload) {
const queue = yield* Queue.bounded<DispatchStreamChunk, DispatchError | Cause.Done>(16); const queue = yield* Queue.bounded<DispatchStreamChunk, DispatchError | Cause.Done>(16);
yield* Effect.addFinalizer(() => Queue.shutdown(queue)); yield* Effect.addFinalizer(() => Queue.shutdown(queue));
@ -64,7 +63,7 @@ const makeGatewayRpcHandlers = (dispatcher: DispatcherManager) =>
function dispatchOne( function dispatchOne(
dispatcher: DispatcherManager, dispatcher: DispatcherManager,
payload: DispatchPayload, payload: DispatchPayload,
): Promise<unknown> { ): Effect.Effect<unknown, DispatcherStreamError> {
if (payload.scope === "flow") { if (payload.scope === "flow") {
return dispatcher.dispatchFlowService( return dispatcher.dispatchFlowService(
payload.flow ?? "default", payload.flow ?? "default",
@ -81,7 +80,7 @@ function dispatchStreamEffect(
responder: (response: unknown, complete: boolean) => Effect.Effect<void>, responder: (response: unknown, complete: boolean) => Effect.Effect<void>,
): Effect.Effect<void, DispatcherStreamError> { ): Effect.Effect<void, DispatcherStreamError> {
if (payload.scope === "flow") { if (payload.scope === "flow") {
return dispatcher.dispatchFlowServiceStreamingEffect( return dispatcher.dispatchFlowServiceStreaming(
payload.flow ?? "default", payload.flow ?? "default",
payload.service, payload.service,
payload.request, payload.request,
@ -89,7 +88,7 @@ function dispatchStreamEffect(
); );
} }
return dispatcher.dispatchGlobalServiceStreamingEffect( return dispatcher.dispatchGlobalServiceStreaming(
payload.service, payload.service,
payload.request, payload.request,
responder, responder,

489
ts/packages/flow/src/gateway/server.ts
View file

@ -1,19 +1,16 @@
/** @effect-diagnostics nodeBuiltinImport:skip-file effectFnOpportunity:skip-file catchToOrElseSucceed:skip-file */
/** /**
* API Gateway — HTTP + WebSocket server. * API Gateway -- Effect HTTP + RPC server.
*
* Replaces the Python aiohttp gateway with Fastify.
* Uses Effect RPC over WebSocket for streaming client requests.
* *
* Python reference: trustgraph-flow/trustgraph/gateway/service.py * Python reference: trustgraph-flow/trustgraph/gateway/service.py
*/ */
import Fastify, { type FastifyReply } from "fastify"; import { createServer } from "node:http";
import websocketPlugin from "@fastify/websocket"; import { NodeHttpServer, NodeRuntime } from "@effect/platform-node";
import { NodeRuntime } from "@effect/platform-node"; import { Clock, Config, Effect, Exit, Layer, Random, Scope } from "effect";
import { Cause, Clock, Config, Effect, Exit, Layer, ManagedRuntime, Random, Scope } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import { HttpRouter, HttpServerRequest, HttpServerResponse } from "effect/unstable/http";
import * as RpcSerialization from "effect/unstable/rpc/RpcSerialization"; import * as RpcSerialization from "effect/unstable/rpc/RpcSerialization";
import * as EffectSocket from "effect/unstable/socket/Socket";
import { import {
formatPrometheusMetrics, formatPrometheusMetrics,
messagingLifecycleError, messagingLifecycleError,
@ -22,8 +19,8 @@ import {
toTgError, toTgError,
type PubSubBackend, type PubSubBackend,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { makeDispatcherManager } from "./dispatch/manager.js"; import { makeDispatcherManager, type DispatcherManager } from "./dispatch/manager.js";
import { makeGatewayRpcServer } from "./rpc-server.js"; import { makeGatewayRpcServer, type GatewayRpcServer } from "./rpc-server.js";
export interface GatewayConfig { export interface GatewayConfig {
port: number; port: number;
@ -33,231 +30,253 @@ export interface GatewayConfig {
pubsub?: PubSubBackend; pubsub?: PubSubBackend;
} }
export function createGateway(config: GatewayConfig) { const isRecord = (value: unknown): value is Record<string, unknown> =>
const app = Fastify({ logger: true }); typeof value === "object" && value !== null && !Array.isArray(value);
const dispatcher = makeDispatcherManager(config);
const sendDispatchResult = (reply: FastifyReply, result: unknown): unknown => { const json = (body: unknown, status = 200) =>
const err = (result as Record<string, unknown>)?.error as { type?: string; message?: string } | undefined; HttpServerResponse.jsonUnsafe(body, { status });
if (err !== undefined) {
const statusCode = err.type === "not-found" ? 404 : 400;
return reply.code(statusCode).send(result);
}
return result;
};
const sendDispatchError = (reply: FastifyReply, error: unknown): unknown => const badRequest = (message: string) =>
reply.code(500).send({ error: toTgError(error) }); json({ error: { type: "bad-request", message } }, 400);
return Effect.runPromise( const dispatchError = (error: unknown) =>
json({ error: toTgError(error) }, 500);
const dispatchResult = (result: unknown) => {
const err = isRecord(result) && isRecord(result.error)
? result.error as { readonly type?: string; readonly message?: string }
: undefined;
if (err !== undefined) {
return json(result, err.type === "not-found" ? 404 : 400);
}
return json(result);
};
const readJsonRecord = Effect.gen(function* () {
const request = yield* HttpServerRequest.HttpServerRequest;
const body = yield* request.json;
return isRecord(body) ? body : {};
});
const bearerAuthResponse = (config: GatewayConfig) =>
Effect.gen(function* () {
if (config.secret === undefined || config.secret.length === 0) return null;
const request = yield* HttpServerRequest.HttpServerRequest;
const auth = request.headers.authorization;
return auth === `Bearer ${config.secret}`
? null
: json({ error: "Unauthorized" }, 401);
});
type RouteRequirements =
| HttpServerRequest.HttpServerRequest
| HttpRouter.RouteContext;
const withBearerAuth = (
config: GatewayConfig,
handler: Effect.Effect<HttpServerResponse.HttpServerResponse, never, RouteRequirements>,
) =>
Effect.gen(function* () {
const denied = yield* bearerAuthResponse(config);
if (denied !== null) return denied;
return yield* handler;
});
const withDispatchError = <A, E>(
effect: Effect.Effect<A, E>,
operation: string,
): Effect.Effect<HttpServerResponse.HttpServerResponse> =>
effect.pipe(
Effect.mapError((cause) => messagingLifecycleError("gateway", operation, cause)),
Effect.map(dispatchResult),
Effect.catch((error) => Effect.succeed(dispatchError(error))),
);
const workbenchDispatch = (
config: GatewayConfig,
dispatcher: DispatcherManager,
) =>
withBearerAuth(
config,
Effect.gen(function* () { Effect.gen(function* () {
yield* Effect.tryPromise({ const body = yield* readJsonRecord.pipe(
try: () => app.register(websocketPlugin), Effect.catch(() => Effect.succeed<Record<string, unknown>>({})),
catch: (cause) => messagingLifecycleError("gateway", "register-websocket", cause), );
}); const service = typeof body.service === "string" ? body.service : undefined;
const payload = isRecord(body.request) ? body.request : undefined;
if (service === undefined || service.length === 0 || payload === undefined) {
return badRequest("service and request are required");
}
yield* Effect.tryPromise({ const dispatch = body.scope === "flow"
try: () => dispatcher.start(), ? dispatcher.dispatchFlowService(
catch: (cause) => messagingLifecycleError("gateway", "dispatcher-start", cause), typeof body.flow === "string" ? body.flow : "default",
}); service,
payload,
)
: dispatcher.dispatchGlobalService(service, payload);
return yield* withDispatchError(dispatch, "workbench-dispatch");
}),
);
const globalDispatch = (
config: GatewayConfig,
dispatcher: DispatcherManager,
) =>
withBearerAuth(
config,
Effect.gen(function* () {
const params = yield* HttpRouter.params;
const body = yield* readJsonRecord.pipe(
Effect.catch(() => Effect.succeed<Record<string, unknown>>({})),
);
return yield* withDispatchError(
dispatcher.dispatchGlobalService(params.kind ?? "", body),
"global-dispatch",
);
}),
);
const flowDispatch = (
config: GatewayConfig,
dispatcher: DispatcherManager,
) =>
withBearerAuth(
config,
Effect.gen(function* () {
const params = yield* HttpRouter.params;
const body = yield* readJsonRecord.pipe(
Effect.catch(() => Effect.succeed<Record<string, unknown>>({})),
);
return yield* withDispatchError(
dispatcher.dispatchFlowService(params.flow ?? "default", params.kind ?? "", body),
"flow-dispatch",
);
}),
);
const flowLoad = (
config: GatewayConfig,
dispatcher: DispatcherManager,
) =>
withBearerAuth(
config,
Effect.gen(function* () {
const params = yield* HttpRouter.params;
const body = yield* readJsonRecord.pipe(
Effect.catch(() => Effect.succeed<Record<string, unknown>>({})),
);
const documentId = typeof body.documentId === "string" ? body.documentId : undefined;
if (documentId === undefined || documentId.length === 0) {
return badRequest("documentId is required");
}
const user = typeof body.user === "string" ? body.user : "default";
const collection = typeof body.collection === "string" ? body.collection : "default";
const timestamp = yield* Clock.currentTimeMillis;
const suffix = yield* Random.nextIntBetween(0, 36 ** 6, { halfOpen: true });
const metadata = {
id: `load-${timestamp}-${suffix.toString(36).padStart(6, "0")}`,
root: documentId,
user,
collection,
};
yield* dispatcher.publishToTopic("tg.flow.document", { metadata, documentId }).pipe(
Effect.mapError((cause) => messagingLifecycleError("gateway", "publish-load", cause)),
);
return json({ status: "processing", documentId, flow: params.flow ?? "default" });
}).pipe(
Effect.catch((error) => Effect.succeed(dispatchError(error))),
),
);
const rpcRoute = (
config: GatewayConfig,
rpcServer: GatewayRpcServer,
rpcScope: Scope.Scope,
) =>
Effect.gen(function* () {
const request = yield* HttpServerRequest.HttpServerRequest;
const url = new URL(request.url, "http://localhost");
const token = url.searchParams.get("token");
if (config.secret !== undefined && config.secret.length > 0 && token !== config.secret) {
return json({ error: "Unauthorized" }, 401);
}
const socket = yield* request.upgrade;
yield* rpcServer.onSocket(socket, headersFrom(request.headers)).pipe(
Scope.provide(rpcScope),
);
return HttpServerResponse.empty();
}).pipe(
Effect.catch((error) => Effect.succeed(dispatchError(error))),
);
const metricsRoute =
formatPrometheusMetrics.pipe(
Effect.map((body) =>
HttpServerResponse.text(body, {
headers: { "content-type": prometheusContentType },
})
),
);
const gatewayRoutes = (
config: GatewayConfig,
dispatcher: DispatcherManager,
rpcServer: GatewayRpcServer,
rpcScope: Scope.Scope,
) =>
Layer.mergeAll(
HttpRouter.add("POST", "/api/v1/workbench/dispatch", workbenchDispatch(config, dispatcher)),
HttpRouter.add("POST", "/api/v1/:kind", globalDispatch(config, dispatcher)),
HttpRouter.add("POST", "/api/v1/flow/:flow/service/:kind", flowDispatch(config, dispatcher)),
HttpRouter.add("POST", "/api/v1/flow/:flow/load", flowLoad(config, dispatcher)),
HttpRouter.add("GET", "/api/v1/rpc", rpcRoute(config, rpcServer, rpcScope)),
HttpRouter.add("GET", "/api/v1/metrics", metricsRoute),
);
export function createGateway(config: GatewayConfig) {
return Layer.effectDiscard(
Effect.scoped(Effect.gen(function* () {
const dispatcher = makeDispatcherManager(config);
yield* dispatcher.start.pipe(
Effect.mapError((cause) => messagingLifecycleError("gateway", "dispatcher-start", cause)),
);
yield* Effect.addFinalizer(() =>
dispatcher.stop.pipe(
Effect.catch((cause) =>
Effect.logError("[Gateway] Failed to stop dispatcher", {
error: cause.message,
operation: cause.operation,
}),
),
),
);
const rpcScope = yield* Scope.make(); const rpcScope = yield* Scope.make();
yield* Effect.addFinalizer(() => Scope.close(rpcScope, Exit.void));
const rpcServer = yield* makeGatewayRpcServer(dispatcher).pipe( const rpcServer = yield* makeGatewayRpcServer(dispatcher).pipe(
Effect.provideService(RpcSerialization.RpcSerialization, RpcSerialization.ndjson), Effect.provideService(RpcSerialization.RpcSerialization, RpcSerialization.ndjson),
Scope.provide(rpcScope), Scope.provide(rpcScope),
); );
return { rpcScope, rpcServer }; const serverLayer = HttpRouter.serve(
}), gatewayRoutes(config, dispatcher, rpcServer, rpcScope),
).then(({ rpcScope, rpcServer }) => { ).pipe(
// Authentication middleware Layer.provideMerge(NodeHttpServer.layer(createServer, {
app.addHook("onRequest", (request, reply) => { port: config.port,
if (request.url === "/api/v1/metrics") return; host: "0.0.0.0",
if (request.url.startsWith("/api/v1/rpc")) return; // RPC socket auth via query param })),
if (config.secret !== undefined && config.secret.length > 0) {
const auth = request.headers.authorization;
if (auth === undefined || auth !== `Bearer ${config.secret}`) {
reply.code(401).send({ error: "Unauthorized" });
}
}
});
app.post<{
Body: {
scope?: string;
service?: string;
flow?: string;
request?: Record<string, unknown>;
};
}>("/api/v1/workbench/dispatch", (request, reply) => {
const body = request.body;
const service = body.service;
const payload = body.request;
if (service === undefined || service.length === 0 || payload === undefined) {
return reply.code(400).send({
error: { type: "bad-request", message: "service and request are required" },
});
}
return Effect.runPromise(
Effect.tryPromise({
try: () =>
body.scope === "flow"
? dispatcher.dispatchFlowService(body.flow ?? "default", service, payload)
: dispatcher.dispatchGlobalService(service, payload),
catch: (cause) => messagingLifecycleError("gateway", "workbench-dispatch", cause),
}).pipe(
Effect.map((result) => sendDispatchResult(reply, result)),
Effect.catch((error) => Effect.succeed(sendDispatchError(reply, error))),
),
);
});
// REST endpoint: POST /api/v1/:kind (global services)
app.post<{ Params: { kind: string } }>("/api/v1/:kind", (request, reply) => {
const { kind } = request.params;
const body = request.body as Record<string, unknown>;
return Effect.runPromise(
Effect.tryPromise({
try: () => dispatcher.dispatchGlobalService(kind, body),
catch: (cause) => messagingLifecycleError("gateway", "global-dispatch", cause),
}).pipe(
Effect.map((result) => sendDispatchResult(reply, result)),
Effect.catch((error) => Effect.succeed(sendDispatchError(reply, error))),
),
);
});
// REST endpoint: POST /api/v1/flow/:flow/service/:kind (flow-scoped services)
app.post<{ Params: { flow: string; kind: string } }>(
"/api/v1/flow/:flow/service/:kind",
(request, reply) => {
const { flow, kind } = request.params;
const body = request.body as Record<string, unknown>;
return Effect.runPromise(
Effect.tryPromise({
try: () => dispatcher.dispatchFlowService(flow, kind, body),
catch: (cause) => messagingLifecycleError("gateway", "flow-dispatch", cause),
}).pipe(
Effect.map((result) => sendDispatchResult(reply, result)),
Effect.catch((error) => Effect.succeed(sendDispatchError(reply, error))),
),
);
},
); );
// REST endpoint: POST /api/v1/flow/:flow/load (trigger document processing) yield* Effect.log(`[Gateway] Listening on port ${config.port}`);
app.post<{ Params: { flow: string } }>( return yield* Layer.launch(serverLayer);
"/api/v1/flow/:flow/load", })),
(request, reply) => { );
const { flow } = request.params;
const body = request.body as {
documentId?: string;
user?: string;
collection?: string;
};
if (body.documentId === undefined || body.documentId.length === 0) {
return reply.code(400).send({
error: { type: "bad-request", message: "documentId is required" },
});
}
return Effect.runPromise(
Effect.gen(function* () {
const user = body.user ?? "default";
const collection = body.collection ?? "default";
const documentId = body.documentId;
const timestamp = yield* Clock.currentTimeMillis;
const suffix = yield* Random.nextIntBetween(0, 36 ** 6, { halfOpen: true });
// Publish Document message to the decode-input topic
const topic = "tg.flow.document";
const metadata = {
id: `load-${timestamp}-${suffix.toString(36).padStart(6, "0")}`,
root: documentId,
user,
collection,
};
yield* Effect.tryPromise({
try: () => dispatcher.publishToTopic(topic, { metadata, documentId }),
catch: (cause) => messagingLifecycleError("gateway", "publish-load", cause),
});
return { status: "processing", documentId, flow };
}).pipe(
Effect.catch((error) => Effect.succeed(sendDispatchError(reply, error))),
),
);
},
);
// Effect RPC WebSocket endpoint: /api/v1/rpc
app.get("/api/v1/rpc", { websocket: true }, (socket, request) => {
const url = new URL(request.url, `http://${request.headers.host}`);
const token = url.searchParams.get("token");
if (config.secret !== undefined && config.secret.length > 0 && token !== config.secret) {
socket.close(4001, "Unauthorized");
return;
}
const program = Effect.scoped(
Effect.gen(function* () {
const effectSocket = yield* EffectSocket.fromWebSocket(
Effect.succeed(socket as unknown as globalThis.WebSocket),
{ closeCodeIsError: (code) => code !== 1000 },
);
yield* rpcServer.onSocket(effectSocket, headersFrom(request.headers));
}),
);
void Effect.runPromise(
program.pipe(
Scope.provide(rpcScope),
Effect.sandbox,
Effect.catch((cause) =>
Effect.logError("[Gateway] RPC WebSocket error", { error: Cause.pretty(cause) }).pipe(
Effect.flatMap(() =>
Effect.sync(() => {
if (socket.readyState === 1) {
socket.close(1011, "Internal server error");
}
}),
),
)
),
),
);
});
// Metrics endpoint — returns Effect metrics in Prometheus exposition format.
app.get("/api/v1/metrics", (_, reply) => {
reply.header("content-type", prometheusContentType);
return Effect.runPromise(formatPrometheusMetrics);
});
return {
start: () => app.listen({ port: config.port, host: "0.0.0.0" }),
stop: () =>
Effect.runPromise(
Effect.gen(function* () {
yield* Effect.tryPromise({
try: () => app.close(),
catch: (cause) => messagingLifecycleError("gateway", "app-close", cause),
});
yield* Scope.close(rpcScope, Exit.void);
yield* Effect.tryPromise({
try: () => dispatcher.stop(),
catch: (cause) => messagingLifecycleError("gateway", "dispatcher-stop", cause),
});
}),
),
};
});
} }
function headersFrom(headers: Record<string, string | string[] | number | undefined>): ReadonlyArray<[string, string]> { function headersFrom(headers: Record<string, string | string[] | number | undefined>): ReadonlyArray<[string, string]> {
@ -269,10 +288,6 @@ function headersFrom(headers: Record<string, string | string[] | number | undefi
}); });
} }
export function run(): Promise<void> {
return gatewayRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }
@ -290,22 +305,8 @@ export const loadGatewayConfig = Effect.fn("loadGatewayConfig")(function* () {
} satisfies GatewayConfig; } satisfies GatewayConfig;
}); });
export const program = Effect.scoped( export const gatewayProgram = (config: GatewayConfig) => Layer.launch(createGateway(config));
Effect.gen(function* () {
const config = yield* loadGatewayConfig();
const gateway = yield* Effect.promise(() => createGateway(config)).pipe(Effect.orDie);
yield* Effect.addFinalizer(() => Effect.promise(() => gateway.stop()).pipe(Effect.orDie));
yield* Effect.promise(() => gateway.start()).pipe(
Effect.orDie,
Effect.withSpan("trustgraph.gateway.start", {
attributes: {
"trustgraph.gateway.port": config.port,
},
}),
);
yield* Effect.log(`[Gateway] Listening on port ${config.port}`);
return yield* Effect.never;
}),
);
const gatewayRuntime = ManagedRuntime.make(Layer.empty); export const program = loadGatewayConfig().pipe(
Effect.flatMap(gatewayProgram),
);

752
ts/packages/flow/src/librarian/service.ts
View file

File diff suppressed because it is too large Load diff

64
ts/packages/flow/src/model/text-completion/azure-openai.ts
View file

@ -18,9 +18,8 @@ import {
type LlmProvider, type LlmProvider,
type ProcessorConfig, type ProcessorConfig,
type LlmResult, type LlmResult,
type LlmChunk,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime, Stream } from "effect"; import { Effect, Stream } from "effect";
import { import {
llmStreamPart, llmStreamPart,
makeTextCompletionLayer, makeTextCompletionLayer,
@ -28,7 +27,6 @@ import {
providerStatusError, providerStatusError,
requiredString, requiredString,
streamTextCompletionChunks, streamTextCompletionChunks,
toAsyncGenerator,
type TextCompletionConfigError, type TextCompletionConfigError,
type TextCompletionRuntimeError, type TextCompletionRuntimeError,
} from "./common.ts"; } from "./common.ts";
@ -89,7 +87,7 @@ const mapAzureOpenAIError = (error: unknown): TextCompletionRuntimeError =>
const makeAzureOpenAIProviderFromClient = ( const makeAzureOpenAIProviderFromClient = (
resolved: ResolvedAzureOpenAIConfig, resolved: ResolvedAzureOpenAIConfig,
client: AzureOpenAI, client: AzureOpenAI,
): LlmProvider => { ): LlmProvider<TextCompletionRuntimeError> => {
const { const {
defaultModel, defaultModel,
defaultTemperature, defaultTemperature,
@ -102,31 +100,29 @@ const makeAzureOpenAIProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): Promise<LlmResult> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
return Effect.runPromise( return Effect.tryPromise({
Effect.tryPromise({ try: () =>
try: () => client.chat.completions.create({
client.chat.completions.create({
model: modelName,
messages: [
{ role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
max_completion_tokens: maxOutput,
}),
catch: mapAzureOpenAIError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.choices[0].message.content ?? "",
inToken: resp.usage?.prompt_tokens ?? 0,
outToken: resp.usage?.completion_tokens ?? 0,
model: modelName, model: modelName,
})), messages: [
), { role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
max_completion_tokens: maxOutput,
}),
catch: mapAzureOpenAIError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.choices[0].message.content ?? "",
inToken: resp.usage?.prompt_tokens ?? 0,
outToken: resp.usage?.completion_tokens ?? 0,
model: modelName,
})),
); );
}, },
supportsStreaming: () => true, supportsStreaming: () => true,
@ -135,11 +131,11 @@ const makeAzureOpenAIProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): AsyncGenerator<LlmChunk> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
const stream = Stream.fromEffect( return Stream.fromEffect(
Effect.tryPromise({ Effect.tryPromise({
try: () => try: () =>
client.chat.completions.create({ client.chat.completions.create({
@ -169,13 +165,13 @@ const makeAzureOpenAIProviderFromClient = (
}) })
), ),
); );
return toAsyncGenerator(Stream.toAsyncIterable(stream), mapAzureOpenAIError);
}, },
} satisfies LlmProvider; } satisfies LlmProvider<TextCompletionRuntimeError>;
}; };
export function makeAzureOpenAIProvider(config: AzureOpenAIProcessorConfig): LlmProvider { export function makeAzureOpenAIProvider(
config: AzureOpenAIProcessorConfig,
): LlmProvider<TextCompletionRuntimeError> {
return Effect.runSync(makeAzureOpenAIProviderEffect(config)); return Effect.runSync(makeAzureOpenAIProviderEffect(config));
} }
@ -217,12 +213,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => makeTextCompletionLayer(makeAzureOpenAIProviderEffect(config)), layer: (config) => makeTextCompletionLayer(makeAzureOpenAIProviderEffect(config)),
}); });
const azureOpenAITextCompletionRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return azureOpenAITextCompletionRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

29
ts/packages/flow/src/model/text-completion/claude.ts
View file

@ -14,7 +14,7 @@ import {
type LlmProvider, type LlmProvider,
type ProcessorConfig, type ProcessorConfig,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime, Redacted } from "effect"; import { Effect, Layer, Redacted } from "effect";
import { FetchHttpClient } from "effect/unstable/http"; import { FetchHttpClient } from "effect/unstable/http";
import { import {
makeLanguageModelProvider, makeLanguageModelProvider,
@ -55,30 +55,31 @@ const loadClaudeConfig = Effect.fn("loadClaudeConfig")(function* (config: Claude
} satisfies ResolvedClaudeConfig; } satisfies ResolvedClaudeConfig;
}); });
const makeClaudeRuntime = (apiKey: string) => const makeClaudeLayer = (apiKey: string) =>
ManagedRuntime.make( AnthropicClient.layer({
AnthropicClient.layer({ apiKey: Redacted.make(apiKey),
apiKey: Redacted.make(apiKey), }).pipe(
}).pipe( Layer.provide(FetchHttpClient.layer),
Layer.provide(FetchHttpClient.layer),
),
); );
export function makeClaudeProvider(config: ClaudeProcessorConfig): LlmProvider { export function makeClaudeProvider(
return Effect.runSync(makeClaudeProviderEffect(config)); config: ClaudeProcessorConfig,
): LlmProvider<TextCompletionRuntimeError> {
return Effect.runSync(Effect.scoped(makeClaudeProviderEffect(config)));
} }
export const makeClaudeProviderEffect = Effect.fn("makeClaudeProvider")(function* ( export const makeClaudeProviderEffect = Effect.fn("makeClaudeProvider")(function* (
config: ClaudeProcessorConfig, config: ClaudeProcessorConfig,
) { ) {
const resolved = yield* loadClaudeConfig(config); const resolved = yield* loadClaudeConfig(config);
const context = yield* Layer.build(makeClaudeLayer(resolved.apiKey));
yield* Effect.log("[Claude] LLM service initialized"); yield* Effect.log("[Claude] LLM service initialized");
return makeLanguageModelProvider({ return makeLanguageModelProvider({
provider: "Claude", provider: "Claude",
defaultModel: resolved.defaultModel, defaultModel: resolved.defaultModel,
defaultTemperature: resolved.defaultTemperature, defaultTemperature: resolved.defaultTemperature,
runtime: makeClaudeRuntime(resolved.apiKey), context,
makeLanguageModel: ({ model, temperature }) => makeLanguageModel: ({ model, temperature }) =>
AnthropicLanguageModel.make({ AnthropicLanguageModel.make({
model, model,
@ -110,12 +111,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => makeTextCompletionLayer(makeClaudeProviderEffect(config)), layer: (config) => makeTextCompletionLayer(makeClaudeProviderEffect(config)),
}); });
const claudeTextCompletionRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return claudeTextCompletionRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

80
ts/packages/flow/src/model/text-completion/common.ts
View file

@ -7,10 +7,11 @@ import {
type LlmResult, type LlmResult,
type LlmProvider, type LlmProvider,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Config, Effect, Layer, ManagedRuntime, Match, Ref, Result, Stream } from "effect"; import { Config, Context, Effect, Layer, Match, Ref, Result, Stream } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as Predicate from "effect/Predicate"; import * as Predicate from "effect/Predicate";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import type * as Scope from "effect/Scope";
import { AiError, LanguageModel, Prompt, Response } from "effect/unstable/ai"; import { AiError, LanguageModel, Prompt, Response } from "effect/unstable/ai";
export class TextCompletionConfigError extends S.TaggedErrorClass<TextCompletionConfigError>()( export class TextCompletionConfigError extends S.TaggedErrorClass<TextCompletionConfigError>()(
@ -43,15 +44,15 @@ export interface LanguageModelProviderOptions<Requirements> {
readonly provider: string; readonly provider: string;
readonly defaultModel: string; readonly defaultModel: string;
readonly defaultTemperature: number; readonly defaultTemperature: number;
readonly runtime: ManagedRuntime.ManagedRuntime<Requirements, TextCompletionRuntimeError>; readonly context: Context.Context<Requirements>;
readonly makeLanguageModel: ( readonly makeLanguageModel: (
request: LanguageModelProviderRequest, request: LanguageModelProviderRequest,
) => Effect.Effect<LanguageModel.Service, TextCompletionRuntimeError, Requirements>; ) => Effect.Effect<LanguageModel.Service, TextCompletionRuntimeError, Requirements>;
} }
export const makeTextCompletionLayer = <E, R>( export const makeTextCompletionLayer = <ProviderError, E, R>(
provider: Effect.Effect<LlmProvider, E, R>, provider: Effect.Effect<LlmProvider<ProviderError>, E, R>,
): Layer.Layer<Llm, E, R> => ): Layer.Layer<Llm, E, Exclude<R, Scope.Scope>> =>
Layer.effect(Llm)( Layer.effect(Llm)(
provider.pipe( provider.pipe(
Effect.map((resolvedProvider) => Effect.map((resolvedProvider) =>
@ -279,39 +280,25 @@ const languageModelStreamChunk = (
Match.orElse(() => Effect.succeed(Result.fail(undefined))), Match.orElse(() => Effect.succeed(Result.fail(undefined))),
); );
const runLanguageModelStream = <RuntimeRequirements, StreamRequirements extends RuntimeRequirements>(
runtime: ManagedRuntime.ManagedRuntime<RuntimeRequirements, TextCompletionRuntimeError>,
stream: Stream.Stream<LlmChunk, TextCompletionRuntimeError, StreamRequirements>,
): AsyncIterable<LlmChunk> => ({
[Symbol.asyncIterator]: () => {
const iterator = runtime.context().then((context) =>
Stream.toAsyncIterableWith(stream, context)[Symbol.asyncIterator]()
);
return {
next: () => iterator.then((current) => current.next()),
};
},
});
export const makeLanguageModelProvider = <Requirements>( export const makeLanguageModelProvider = <Requirements>(
options: LanguageModelProviderOptions<Requirements>, options: LanguageModelProviderOptions<Requirements>,
): LlmProvider => ({ ): LlmProvider<TextCompletionRuntimeError> => ({
generateContent: (system, prompt, model, temperature) => { generateContent: (system, prompt, model, temperature) => {
const modelName = model ?? options.defaultModel; const modelName = model ?? options.defaultModel;
const temp = temperature ?? options.defaultTemperature; const temp = temperature ?? options.defaultTemperature;
return options.runtime.runPromise( return Effect.gen(function* () {
Effect.gen(function* () { const languageModel = yield* options.makeLanguageModel({
const languageModel = yield* options.makeLanguageModel({ model: modelName,
model: modelName, temperature: temp,
temperature: temp, });
}); const response = yield* languageModel.generateText({
const response = yield* languageModel.generateText({ prompt: languageModelPrompt(system, prompt),
prompt: languageModelPrompt(system, prompt), }).pipe(
}).pipe( Effect.mapError((error) => effectAiProviderError(options.provider, error)),
Effect.mapError((error) => effectAiProviderError(options.provider, error)), );
); return languageModelResult(response, modelName);
return languageModelResult(response, modelName); }).pipe(
}), Effect.provideContext(options.context),
); );
}, },
supportsStreaming: () => true, supportsStreaming: () => true,
@ -333,30 +320,9 @@ export const makeLanguageModelProvider = <Requirements>(
), ),
); );
}), }),
).pipe(
Stream.provideContext(options.context),
); );
return toAsyncGenerator(runLanguageModelStream(options.runtime, stream), (error) => return stream;
effectAiProviderError(options.provider, error)
);
}, },
}); });
export const toAsyncGenerator = (
iterable: AsyncIterable<LlmChunk>,
mapError: (error: unknown) => TextCompletionRuntimeError,
): AsyncGenerator<LlmChunk> => {
const iterator = iterable[Symbol.asyncIterator]();
let generator: AsyncGenerator<LlmChunk>;
generator = {
next: (value?: unknown) => iterator.next(value),
return: (value?: unknown) =>
iterator.return === undefined
? Promise.resolve({ done: true, value })
: iterator.return(value),
throw: (error?: unknown) =>
iterator.throw === undefined
? Promise.reject(mapError(error))
: iterator.throw(error),
[Symbol.asyncIterator]: () => generator,
};
return generator;
};

64
ts/packages/flow/src/model/text-completion/mistral.ts
View file

@ -16,9 +16,8 @@ import {
type LlmProvider, type LlmProvider,
type ProcessorConfig, type ProcessorConfig,
type LlmResult, type LlmResult,
type LlmChunk,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime, Stream } from "effect"; import { Effect, Stream } from "effect";
import { import {
llmStreamPart, llmStreamPart,
makeTextCompletionLayer, makeTextCompletionLayer,
@ -27,7 +26,6 @@ import {
requiredString, requiredString,
streamTextCompletionChunks, streamTextCompletionChunks,
textFromContent, textFromContent,
toAsyncGenerator,
type TextCompletionConfigError, type TextCompletionConfigError,
type TextCompletionRuntimeError, type TextCompletionRuntimeError,
} from "./common.ts"; } from "./common.ts";
@ -71,7 +69,7 @@ const mapMistralError = (error: unknown): TextCompletionRuntimeError =>
const makeMistralProviderFromClient = ( const makeMistralProviderFromClient = (
resolved: ResolvedMistralConfig, resolved: ResolvedMistralConfig,
client: Mistral, client: Mistral,
): LlmProvider => { ): LlmProvider<TextCompletionRuntimeError> => {
const { const {
defaultModel, defaultModel,
defaultTemperature, defaultTemperature,
@ -84,31 +82,29 @@ const makeMistralProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): Promise<LlmResult> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
return Effect.runPromise( return Effect.tryPromise({
Effect.tryPromise({ try: () =>
try: () => client.chat.complete({
client.chat.complete({
model: modelName,
messages: [
{ role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
maxTokens: maxOutput,
}),
catch: mapMistralError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: textFromContent(resp.choices?.[0]?.message?.content),
inToken: resp.usage?.promptTokens ?? 0,
outToken: resp.usage?.completionTokens ?? 0,
model: modelName, model: modelName,
})), messages: [
), { role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
maxTokens: maxOutput,
}),
catch: mapMistralError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: textFromContent(resp.choices?.[0]?.message?.content),
inToken: resp.usage?.promptTokens ?? 0,
outToken: resp.usage?.completionTokens ?? 0,
model: modelName,
})),
); );
}, },
supportsStreaming: () => true, supportsStreaming: () => true,
@ -117,11 +113,11 @@ const makeMistralProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): AsyncGenerator<LlmChunk> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
const stream = Stream.fromEffect( return Stream.fromEffect(
Effect.tryPromise({ Effect.tryPromise({
try: () => try: () =>
client.chat.stream({ client.chat.stream({
@ -149,13 +145,13 @@ const makeMistralProviderFromClient = (
}) })
), ),
); );
return toAsyncGenerator(Stream.toAsyncIterable(stream), mapMistralError);
}, },
} satisfies LlmProvider; } satisfies LlmProvider<TextCompletionRuntimeError>;
}; };
export function makeMistralProvider(config: MistralProcessorConfig): LlmProvider { export function makeMistralProvider(
config: MistralProcessorConfig,
): LlmProvider<TextCompletionRuntimeError> {
return Effect.runSync(makeMistralProviderEffect(config)); return Effect.runSync(makeMistralProviderEffect(config));
} }
@ -192,12 +188,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => makeTextCompletionLayer(makeMistralProviderEffect(config)), layer: (config) => makeTextCompletionLayer(makeMistralProviderEffect(config)),
}); });
const mistralTextCompletionRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return mistralTextCompletionRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

56
ts/packages/flow/src/model/text-completion/ollama.ts
View file

@ -16,16 +16,14 @@ import {
type LlmProvider, type LlmProvider,
type ProcessorConfig, type ProcessorConfig,
type LlmResult, type LlmResult,
type LlmChunk,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime, Stream } from "effect"; import { Effect, Stream } from "effect";
import { import {
llmStreamPart, llmStreamPart,
makeTextCompletionLayer, makeTextCompletionLayer,
optionalStringConfig, optionalStringConfig,
providerRuntimeError, providerRuntimeError,
streamTextCompletionChunks, streamTextCompletionChunks,
toAsyncGenerator,
type TextCompletionConfigError, type TextCompletionConfigError,
type TextCompletionRuntimeError, type TextCompletionRuntimeError,
} from "./common.ts"; } from "./common.ts";
@ -59,7 +57,7 @@ const mapOllamaError = (error: unknown): TextCompletionRuntimeError =>
const makeOllamaProviderFromClient = ( const makeOllamaProviderFromClient = (
resolved: ResolvedOllamaConfig, resolved: ResolvedOllamaConfig,
client: Ollama, client: Ollama,
): LlmProvider => { ): LlmProvider<TextCompletionRuntimeError> => {
const { defaultModel } = resolved; const { defaultModel } = resolved;
return { return {
@ -68,27 +66,25 @@ const makeOllamaProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
_temperature?: number, _temperature?: number,
): Promise<LlmResult> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const fullPrompt = system + "\n\n" + prompt; const fullPrompt = system + "\n\n" + prompt;
return Effect.runPromise( return Effect.tryPromise({
Effect.tryPromise({ try: () =>
try: () => client.generate({
client.generate({
model: modelName,
prompt: fullPrompt,
stream: false,
}),
catch: mapOllamaError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.response,
inToken: resp.prompt_eval_count ?? 0,
outToken: resp.eval_count ?? 0,
model: modelName, model: modelName,
})), prompt: fullPrompt,
), stream: false,
}),
catch: mapOllamaError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.response,
inToken: resp.prompt_eval_count ?? 0,
outToken: resp.eval_count ?? 0,
model: modelName,
})),
); );
}, },
supportsStreaming: () => true, supportsStreaming: () => true,
@ -97,11 +93,11 @@ const makeOllamaProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
_temperature?: number, _temperature?: number,
): AsyncGenerator<LlmChunk> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const fullPrompt = system + "\n\n" + prompt; const fullPrompt = system + "\n\n" + prompt;
const stream = Stream.fromEffect( return Stream.fromEffect(
Effect.tryPromise({ Effect.tryPromise({
try: () => try: () =>
client.generate({ client.generate({
@ -125,13 +121,13 @@ const makeOllamaProviderFromClient = (
}) })
), ),
); );
return toAsyncGenerator(Stream.toAsyncIterable(stream), mapOllamaError);
}, },
} satisfies LlmProvider; } satisfies LlmProvider<TextCompletionRuntimeError>;
}; };
export function makeOllamaProvider(config: OllamaProcessorConfig): LlmProvider { export function makeOllamaProvider(
config: OllamaProcessorConfig,
): LlmProvider<TextCompletionRuntimeError> {
return Effect.runSync(makeOllamaProviderEffect(config)); return Effect.runSync(makeOllamaProviderEffect(config));
} }
@ -170,12 +166,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => makeTextCompletionLayer(makeOllamaProviderEffect(config)), layer: (config) => makeTextCompletionLayer(makeOllamaProviderEffect(config)),
}); });
const ollamaTextCompletionRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return ollamaTextCompletionRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

62
ts/packages/flow/src/model/text-completion/openai-compatible.ts
View file

@ -19,9 +19,8 @@ import {
type LlmProvider, type LlmProvider,
type ProcessorConfig, type ProcessorConfig,
type LlmResult, type LlmResult,
type LlmChunk,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime, Stream } from "effect"; import { Effect, Stream } from "effect";
import { import {
llmStreamPart, llmStreamPart,
makeTextCompletionLayer, makeTextCompletionLayer,
@ -29,7 +28,6 @@ import {
providerStatusError, providerStatusError,
requiredString, requiredString,
streamTextCompletionChunks, streamTextCompletionChunks,
toAsyncGenerator,
type TextCompletionConfigError, type TextCompletionConfigError,
type TextCompletionRuntimeError, type TextCompletionRuntimeError,
} from "./common.ts"; } from "./common.ts";
@ -79,7 +77,7 @@ const mapOpenAICompatibleError = (error: unknown): TextCompletionRuntimeError =>
const makeOpenAICompatibleProviderFromClient = ( const makeOpenAICompatibleProviderFromClient = (
resolved: ResolvedOpenAICompatibleConfig, resolved: ResolvedOpenAICompatibleConfig,
client: OpenAI, client: OpenAI,
): LlmProvider => { ): LlmProvider<TextCompletionRuntimeError> => {
const { const {
defaultModel, defaultModel,
defaultTemperature, defaultTemperature,
@ -92,31 +90,29 @@ const makeOpenAICompatibleProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): Promise<LlmResult> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
return Effect.runPromise( return Effect.tryPromise({
Effect.tryPromise({ try: () =>
try: () => client.chat.completions.create({
client.chat.completions.create({
model: modelName,
messages: [
{ role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
max_tokens: maxOutput,
}),
catch: mapOpenAICompatibleError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.choices[0].message.content ?? "",
inToken: resp.usage?.prompt_tokens ?? 0,
outToken: resp.usage?.completion_tokens ?? 0,
model: modelName, model: modelName,
})), messages: [
), { role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
max_tokens: maxOutput,
}),
catch: mapOpenAICompatibleError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.choices[0].message.content ?? "",
inToken: resp.usage?.prompt_tokens ?? 0,
outToken: resp.usage?.completion_tokens ?? 0,
model: modelName,
})),
); );
}, },
supportsStreaming: () => true, supportsStreaming: () => true,
@ -125,11 +121,11 @@ const makeOpenAICompatibleProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): AsyncGenerator<LlmChunk> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
const stream = Stream.fromEffect( return Stream.fromEffect(
Effect.tryPromise({ Effect.tryPromise({
try: () => try: () =>
client.chat.completions.create({ client.chat.completions.create({
@ -158,15 +154,13 @@ const makeOpenAICompatibleProviderFromClient = (
}) })
), ),
); );
return toAsyncGenerator(Stream.toAsyncIterable(stream), mapOpenAICompatibleError);
}, },
} satisfies LlmProvider; } satisfies LlmProvider<TextCompletionRuntimeError>;
}; };
export function makeOpenAICompatibleProvider( export function makeOpenAICompatibleProvider(
config: OpenAICompatibleProcessorConfig, config: OpenAICompatibleProcessorConfig,
): LlmProvider { ): LlmProvider<TextCompletionRuntimeError> {
return Effect.runSync(makeOpenAICompatibleProviderEffect(config)); return Effect.runSync(makeOpenAICompatibleProviderEffect(config));
} }
@ -203,12 +197,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => makeTextCompletionLayer(makeOpenAICompatibleProviderEffect(config)), layer: (config) => makeTextCompletionLayer(makeOpenAICompatibleProviderEffect(config)),
}); });
const openAICompatibleTextCompletionRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return openAICompatibleTextCompletionRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

64
ts/packages/flow/src/model/text-completion/openai.ts
View file

@ -14,9 +14,8 @@ import {
type LlmProvider, type LlmProvider,
type ProcessorConfig, type ProcessorConfig,
type LlmResult, type LlmResult,
type LlmChunk,
} from "@trustgraph/base"; } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime, Stream } from "effect"; import { Effect, Stream } from "effect";
import { import {
llmStreamPart, llmStreamPart,
makeTextCompletionLayer, makeTextCompletionLayer,
@ -24,7 +23,6 @@ import {
providerStatusError, providerStatusError,
requiredString, requiredString,
streamTextCompletionChunks, streamTextCompletionChunks,
toAsyncGenerator,
type TextCompletionConfigError, type TextCompletionConfigError,
type TextCompletionRuntimeError, type TextCompletionRuntimeError,
} from "./common.ts"; } from "./common.ts";
@ -68,7 +66,7 @@ const mapOpenAIError = (error: unknown): TextCompletionRuntimeError =>
const makeOpenAIProviderFromClient = ( const makeOpenAIProviderFromClient = (
resolved: ResolvedOpenAIConfig, resolved: ResolvedOpenAIConfig,
client: OpenAI, client: OpenAI,
): LlmProvider => { ): LlmProvider<TextCompletionRuntimeError> => {
const { const {
defaultModel, defaultModel,
defaultTemperature, defaultTemperature,
@ -81,31 +79,29 @@ const makeOpenAIProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): Promise<LlmResult> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
return Effect.runPromise( return Effect.tryPromise({
Effect.tryPromise({ try: () =>
try: () => client.chat.completions.create({
client.chat.completions.create({
model: modelName,
messages: [
{ role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
max_completion_tokens: maxOutput,
}),
catch: mapOpenAIError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.choices[0].message.content ?? "",
inToken: resp.usage?.prompt_tokens ?? 0,
outToken: resp.usage?.completion_tokens ?? 0,
model: modelName, model: modelName,
})), messages: [
), { role: "system", content: system },
{ role: "user", content: prompt },
],
temperature: temp,
max_completion_tokens: maxOutput,
}),
catch: mapOpenAIError,
}).pipe(
Effect.map((resp): LlmResult => ({
text: resp.choices[0].message.content ?? "",
inToken: resp.usage?.prompt_tokens ?? 0,
outToken: resp.usage?.completion_tokens ?? 0,
model: modelName,
})),
); );
}, },
supportsStreaming: () => true, supportsStreaming: () => true,
@ -114,11 +110,11 @@ const makeOpenAIProviderFromClient = (
prompt: string, prompt: string,
model?: string, model?: string,
temperature?: number, temperature?: number,
): AsyncGenerator<LlmChunk> => { ) => {
const modelName = model ?? defaultModel; const modelName = model ?? defaultModel;
const temp = temperature ?? defaultTemperature; const temp = temperature ?? defaultTemperature;
const stream = Stream.fromEffect( return Stream.fromEffect(
Effect.tryPromise({ Effect.tryPromise({
try: () => try: () =>
client.chat.completions.create({ client.chat.completions.create({
@ -148,13 +144,13 @@ const makeOpenAIProviderFromClient = (
}) })
), ),
); );
return toAsyncGenerator(Stream.toAsyncIterable(stream), mapOpenAIError);
}, },
} satisfies LlmProvider; } satisfies LlmProvider<TextCompletionRuntimeError>;
}; };
export function makeOpenAIProvider(config: OpenAIProcessorConfig): LlmProvider { export function makeOpenAIProvider(
config: OpenAIProcessorConfig,
): LlmProvider<TextCompletionRuntimeError> {
return Effect.runSync(makeOpenAIProviderEffect(config)); return Effect.runSync(makeOpenAIProviderEffect(config));
} }
@ -195,12 +191,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => makeTextCompletionLayer(makeOpenAIProviderEffect(config)), layer: (config) => makeTextCompletionLayer(makeOpenAIProviderEffect(config)),
}); });
const openAITextCompletionRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return openAITextCompletionRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

12
ts/packages/flow/src/prompt/template.ts
View file

@ -40,7 +40,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import * as MutableHashMap from "effect/MutableHashMap"; import * as MutableHashMap from "effect/MutableHashMap";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
@ -167,9 +167,7 @@ export function makePromptTemplateService(config: PromptTemplateConfig): PromptT
specifications: runtime.specs, specifications: runtime.specs,
}); });
for (const handler of runtime.configHandlers) { for (const handler of runtime.configHandlers) {
service.registerConfigHandler((pushedConfig, version) => service.registerConfigHandler(handler);
Effect.runPromise(handler(pushedConfig, version)),
);
} }
Effect.runSync(Effect.log("[PromptTemplate] Service initialized")); Effect.runSync(Effect.log("[PromptTemplate] Service initialized"));
return service; return service;
@ -199,12 +197,6 @@ export const program = makeFlowProcessorProgram({
configHandlers: (config: PromptTemplateConfig) => promptTemplateRuntime(config).configHandlers, configHandlers: (config: PromptTemplateConfig) => promptTemplateRuntime(config).configHandlers,
}); });
const promptRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return promptRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

77
ts/packages/flow/src/qdrant/client.ts
View file

@ -1,4 +1,7 @@
import { QdrantClient, type QdrantClientParams } from "@qdrant/js-client-rest"; import { QdrantClient, type QdrantClientParams } from "@qdrant/js-client-rest";
import { errorMessage } from "@trustgraph/base";
import { Effect } from "effect";
import * as S from "effect/Schema";
export interface QdrantCollectionStatus { export interface QdrantCollectionStatus {
readonly exists: boolean; readonly exists: boolean;
@ -17,8 +20,21 @@ export interface QdrantScoredPoint {
readonly payload?: unknown; readonly payload?: unknown;
} }
export class QdrantClientError extends S.TaggedErrorClass<QdrantClientError>()("QdrantClientError", {
message: S.String,
operation: S.String,
cause: S.Defect({ includeStack: true }),
}) {}
const qdrantClientError = (operation: string, cause: unknown) =>
QdrantClientError.make({
operation,
message: errorMessage(cause),
cause,
});
export interface QdrantClientLike { export interface QdrantClientLike {
readonly collectionExists: (collectionName: string) => Promise<QdrantCollectionStatus>; readonly collectionExists: (collectionName: string) => Effect.Effect<QdrantCollectionStatus, QdrantClientError>;
readonly createCollection: ( readonly createCollection: (
collectionName: string, collectionName: string,
options: { options: {
@ -27,7 +43,7 @@ export interface QdrantClientLike {
readonly distance: "Cosine"; readonly distance: "Cosine";
}; };
}, },
) => Promise<unknown>; ) => Effect.Effect<void, QdrantClientError>;
readonly upsert: ( readonly upsert: (
collectionName: string, collectionName: string,
options: { options: {
@ -37,9 +53,9 @@ export interface QdrantClientLike {
readonly payload?: Record<string, unknown>; readonly payload?: Record<string, unknown>;
}>; }>;
}, },
) => Promise<unknown>; ) => Effect.Effect<void, QdrantClientError>;
readonly getCollections: () => Promise<QdrantCollections>; readonly getCollections: Effect.Effect<QdrantCollections, QdrantClientError>;
readonly deleteCollection: (collectionName: string) => Promise<unknown>; readonly deleteCollection: (collectionName: string) => Effect.Effect<void, QdrantClientError>;
readonly search: ( readonly search: (
collectionName: string, collectionName: string,
options: { options: {
@ -47,7 +63,7 @@ export interface QdrantClientLike {
readonly limit: number; readonly limit: number;
readonly with_payload: boolean; readonly with_payload: boolean;
}, },
) => Promise<ReadonlyArray<QdrantScoredPoint>>; ) => Effect.Effect<ReadonlyArray<QdrantScoredPoint>, QdrantClientError>;
} }
export type QdrantClientFactory = (params: QdrantClientParams) => QdrantClientLike; export type QdrantClientFactory = (params: QdrantClientParams) => QdrantClientLike;
@ -61,24 +77,41 @@ export const makeQdrantClient = (
} }
const client = new QdrantClient(params); const client = new QdrantClient(params);
const tryQdrantPromise = <A>(operation: string, try_: () => PromiseLike<A>) =>
Effect.tryPromise({
try: try_,
catch: (cause) => qdrantClientError(operation, cause),
});
return { return {
collectionExists: (collectionName) => client.collectionExists(collectionName), collectionExists: (collectionName) =>
createCollection: (collectionName, options) => client.createCollection(collectionName, options), tryQdrantPromise("collection-exists", () => client.collectionExists(collectionName)),
createCollection: (collectionName, options) =>
tryQdrantPromise("create-collection", () => client.createCollection(collectionName, options)).pipe(
Effect.asVoid,
),
upsert: (collectionName, options) => upsert: (collectionName, options) =>
client.upsert(collectionName, { tryQdrantPromise("upsert", () =>
points: options.points.map((point) => ({ client.upsert(collectionName, {
id: point.id, points: options.points.map((point) => ({
vector: Array.from(point.vector), id: point.id,
...(point.payload !== undefined ? { payload: point.payload } : {}), vector: Array.from(point.vector),
})), ...(point.payload !== undefined ? { payload: point.payload } : {}),
}), })),
getCollections: () => client.getCollections(), })
deleteCollection: (collectionName) => client.deleteCollection(collectionName), ).pipe(Effect.asVoid),
getCollections: tryQdrantPromise("get-collections", () => client.getCollections()),
deleteCollection: (collectionName) =>
tryQdrantPromise("delete-collection", () => client.deleteCollection(collectionName)).pipe(
Effect.asVoid,
),
search: (collectionName, options) => search: (collectionName, options) =>
client.search(collectionName, { tryQdrantPromise("search", () =>
vector: Array.from(options.vector), client.search(collectionName, {
limit: options.limit, vector: Array.from(options.vector),
with_payload: options.with_payload, limit: options.limit,
}), with_payload: options.with_payload,
})
),
}; };
}; };

12
ts/packages/flow/src/query/embeddings/qdrant-doc-service.ts
View file

@ -24,7 +24,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import { import {
QdrantDocEmbeddingsQueryLive, QdrantDocEmbeddingsQueryLive,
QdrantDocEmbeddingsQueryService, QdrantDocEmbeddingsQueryService,
@ -111,12 +111,10 @@ const provideQdrantDocEmbeddingsQuery = (processorId: string) =>
}); });
export function makeDocEmbeddingsQueryService(config: ProcessorConfig): DocEmbeddingsQueryService { export function makeDocEmbeddingsQueryService(config: ProcessorConfig): DocEmbeddingsQueryService {
const service = makeFlowProcessor(config, { return makeFlowProcessor(config, {
specifications: makeDocEmbeddingsQuerySpecs(), specifications: makeDocEmbeddingsQuerySpecs(),
provide: provideQdrantDocEmbeddingsQuery(config.id), provide: provideQdrantDocEmbeddingsQuery(config.id),
}); });
void Effect.runPromise(Effect.log("[DocEmbeddingsQuery] Service initialized"));
return service;
} }
export const DocEmbeddingsQueryService = makeDocEmbeddingsQueryService; export const DocEmbeddingsQueryService = makeDocEmbeddingsQueryService;
@ -131,12 +129,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => QdrantDocEmbeddingsQueryLive(config), layer: (config) => QdrantDocEmbeddingsQueryLive(config),
}); });
const docEmbeddingsQueryRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return docEmbeddingsQueryRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

41
ts/packages/flow/src/query/embeddings/qdrant-doc.ts
View file

@ -37,7 +37,7 @@ export class QdrantDocEmbeddingsQueryError extends S.TaggedErrorClass<QdrantDocE
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -73,8 +73,7 @@ const decodeDocPointPayload = (payload: unknown) =>
S.decodeUnknownEffect(DocPointPayloadSchema)(payload).pipe(Effect.option); S.decodeUnknownEffect(DocPointPayloadSchema)(payload).pipe(Effect.option);
export interface QdrantDocEmbeddingsQuery { export interface QdrantDocEmbeddingsQuery {
readonly query: (request: DocEmbeddingsQueryRequest) => Promise<ReadonlyArray<ChunkMatch>>; readonly query: (
readonly queryEffect: (
request: DocEmbeddingsQueryRequest, request: DocEmbeddingsQueryRequest,
) => Effect.Effect<ReadonlyArray<ChunkMatch>, QdrantDocEmbeddingsQueryError>; ) => Effect.Effect<ReadonlyArray<ChunkMatch>, QdrantDocEmbeddingsQueryError>;
} }
@ -95,7 +94,7 @@ const makeQdrantDocEmbeddingsQueryClient = (
const makeQdrantDocEmbeddingsQueryFromClient = ( const makeQdrantDocEmbeddingsQueryFromClient = (
client: QdrantClientLike, client: QdrantClientLike,
): QdrantDocEmbeddingsQueryServiceShape => { ): QdrantDocEmbeddingsQueryServiceShape => {
const queryEffect = Effect.fn("QdrantDocEmbeddingsQuery.query")(function* (request: DocEmbeddingsQueryRequest) { const queryImpl = Effect.fn("QdrantDocEmbeddingsQuery.query")(function* (request: DocEmbeddingsQueryRequest) {
const { vector, user, collection, limit } = request; const { vector, user, collection, limit } = request;
if (vector.length === 0) { if (vector.length === 0) {
@ -106,10 +105,9 @@ const makeQdrantDocEmbeddingsQueryFromClient = (
const collectionName = `d_${user}_${collection}_${dim}`; const collectionName = `d_${user}_${collection}_${dim}`;
// Check if collection exists -- return empty if not // Check if collection exists -- return empty if not
const exists = yield* Effect.tryPromise({ const exists = yield* client.collectionExists(collectionName).pipe(
try: () => client.collectionExists(collectionName), Effect.mapError((cause) => qdrantDocEmbeddingsQueryError("collection-exists", cause)),
catch: (cause) => qdrantDocEmbeddingsQueryError("collection-exists", cause), );
});
if (!exists.exists) { if (!exists.exists) {
yield* Effect.log( yield* Effect.log(
`[QdrantDocQuery] Collection ${collectionName} does not exist, returning empty results`, `[QdrantDocQuery] Collection ${collectionName} does not exist, returning empty results`,
@ -117,15 +115,16 @@ const makeQdrantDocEmbeddingsQueryFromClient = (
return []; return [];
} }
const searchResult = yield* Effect.tryPromise({ const searchResult = yield* client.search(
try: () => collectionName,
client.search(collectionName, { {
vector, vector,
limit, limit,
with_payload: true, with_payload: true,
}), },
catch: (cause) => qdrantDocEmbeddingsQueryError("search", cause), ).pipe(
}); Effect.mapError((cause) => qdrantDocEmbeddingsQueryError("search", cause)),
);
const chunks: ChunkMatch[] = []; const chunks: ChunkMatch[] = [];
for (const point of searchResult) { for (const point of searchResult) {
@ -146,7 +145,7 @@ const makeQdrantDocEmbeddingsQueryFromClient = (
}); });
return { return {
query: queryEffect, query: queryImpl,
}; };
}; };
@ -172,12 +171,8 @@ const withQdrantDocEmbeddingsQuery = <A>(
export function makeQdrantDocEmbeddingsQuery( export function makeQdrantDocEmbeddingsQuery(
config: QdrantDocQueryConfig = {}, config: QdrantDocQueryConfig = {},
): QdrantDocEmbeddingsQuery { ): QdrantDocEmbeddingsQuery {
const queryEffect = (request: DocEmbeddingsQueryRequest) =>
withQdrantDocEmbeddingsQuery(config, (query) => query.query(request));
return { return {
query: (request) => Effect.runPromise(queryEffect(request)), query: (request) => withQdrantDocEmbeddingsQuery(config, (query) => query.query(request)),
queryEffect,
}; };
} }

12
ts/packages/flow/src/query/embeddings/qdrant-graph-service.ts
View file

@ -24,7 +24,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import { import {
QdrantGraphEmbeddingsQueryLive, QdrantGraphEmbeddingsQueryLive,
QdrantGraphEmbeddingsQueryService, QdrantGraphEmbeddingsQueryService,
@ -112,12 +112,10 @@ const provideQdrantGraphEmbeddingsQuery = (processorId: string) =>
}); });
export function makeGraphEmbeddingsQueryService(config: ProcessorConfig): GraphEmbeddingsQueryService { export function makeGraphEmbeddingsQueryService(config: ProcessorConfig): GraphEmbeddingsQueryService {
const service = makeFlowProcessor(config, { return makeFlowProcessor(config, {
specifications: makeGraphEmbeddingsQuerySpecs(), specifications: makeGraphEmbeddingsQuerySpecs(),
provide: provideQdrantGraphEmbeddingsQuery(config.id), provide: provideQdrantGraphEmbeddingsQuery(config.id),
}); });
void Effect.runPromise(Effect.log("[GraphEmbeddingsQuery] Service initialized"));
return service;
} }
export const GraphEmbeddingsQueryService = makeGraphEmbeddingsQueryService; export const GraphEmbeddingsQueryService = makeGraphEmbeddingsQueryService;
@ -132,12 +130,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => QdrantGraphEmbeddingsQueryLive(config), layer: (config) => QdrantGraphEmbeddingsQueryLive(config),
}); });
const graphEmbeddingsQueryRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return graphEmbeddingsQueryRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

41
ts/packages/flow/src/query/embeddings/qdrant-graph.ts
View file

@ -39,7 +39,7 @@ export class QdrantGraphEmbeddingsQueryError extends S.TaggedErrorClass<QdrantGr
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -81,8 +81,7 @@ const decodeGraphPointPayload = (payload: unknown) =>
S.decodeUnknownEffect(GraphPointPayloadSchema)(payload).pipe(Effect.option); S.decodeUnknownEffect(GraphPointPayloadSchema)(payload).pipe(Effect.option);
export interface QdrantGraphEmbeddingsQuery { export interface QdrantGraphEmbeddingsQuery {
readonly query: (request: GraphEmbeddingsQueryRequest) => Promise<ReadonlyArray<EntityMatch>>; readonly query: (
readonly queryEffect: (
request: GraphEmbeddingsQueryRequest, request: GraphEmbeddingsQueryRequest,
) => Effect.Effect<ReadonlyArray<EntityMatch>, QdrantGraphEmbeddingsQueryError>; ) => Effect.Effect<ReadonlyArray<EntityMatch>, QdrantGraphEmbeddingsQueryError>;
} }
@ -104,7 +103,7 @@ const makeQdrantGraphEmbeddingsQueryFromClient = (
client: QdrantClientLike, client: QdrantClientLike,
): QdrantGraphEmbeddingsQueryServiceShape => { ): QdrantGraphEmbeddingsQueryServiceShape => {
const queryEffect = Effect.fn("QdrantGraphEmbeddingsQuery.query")(function* ( const queryImpl = Effect.fn("QdrantGraphEmbeddingsQuery.query")(function* (
request: GraphEmbeddingsQueryRequest, request: GraphEmbeddingsQueryRequest,
) { ) {
const { vector, user, collection, limit } = request; const { vector, user, collection, limit } = request;
@ -117,10 +116,9 @@ const makeQdrantGraphEmbeddingsQueryFromClient = (
const collectionName = `t_${user}_${collection}_${dim}`; const collectionName = `t_${user}_${collection}_${dim}`;
// Check if collection exists -- return empty if not // Check if collection exists -- return empty if not
const exists = yield* Effect.tryPromise({ const exists = yield* client.collectionExists(collectionName).pipe(
try: () => client.collectionExists(collectionName), Effect.mapError((cause) => qdrantGraphEmbeddingsQueryError("collection-exists", cause)),
catch: (cause) => qdrantGraphEmbeddingsQueryError("collection-exists", cause), );
});
if (!exists.exists) { if (!exists.exists) {
yield* Effect.log( yield* Effect.log(
`[QdrantGraphQuery] Collection ${collectionName} does not exist, returning empty results`, `[QdrantGraphQuery] Collection ${collectionName} does not exist, returning empty results`,
@ -130,15 +128,16 @@ const makeQdrantGraphEmbeddingsQueryFromClient = (
// Query 2x the limit so we have a better chance of getting `limit` // Query 2x the limit so we have a better chance of getting `limit`
// unique entities after deduplication (same heuristic as Python impl) // unique entities after deduplication (same heuristic as Python impl)
const searchResult = yield* Effect.tryPromise({ const searchResult = yield* client.search(
try: () => collectionName,
client.search(collectionName, { {
vector, vector,
limit: limit * 2, limit: limit * 2,
with_payload: true, with_payload: true,
}), },
catch: (cause) => qdrantGraphEmbeddingsQueryError("search", cause), ).pipe(
}); Effect.mapError((cause) => qdrantGraphEmbeddingsQueryError("search", cause)),
);
const entitySet = new Set<string>(); const entitySet = new Set<string>();
const entities: EntityMatch[] = []; const entities: EntityMatch[] = [];
@ -168,7 +167,7 @@ const makeQdrantGraphEmbeddingsQueryFromClient = (
}); });
return { return {
query: queryEffect, query: queryImpl,
}; };
}; };
@ -194,12 +193,8 @@ const withQdrantGraphEmbeddingsQuery = <A>(
export function makeQdrantGraphEmbeddingsQuery( export function makeQdrantGraphEmbeddingsQuery(
config: QdrantGraphQueryConfig = {}, config: QdrantGraphQueryConfig = {},
): QdrantGraphEmbeddingsQuery { ): QdrantGraphEmbeddingsQuery {
const queryEffect = (request: GraphEmbeddingsQueryRequest) =>
withQdrantGraphEmbeddingsQuery(config, (query) => query.query(request));
return { return {
query: (request) => Effect.runPromise(queryEffect(request)), query: (request) => withQdrantGraphEmbeddingsQuery(config, (query) => query.query(request)),
queryEffect,
}; };
} }

12
ts/packages/flow/src/query/triples/falkordb-service.ts
View file

@ -24,7 +24,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import { import {
FalkorDBTriplesQueryLive, FalkorDBTriplesQueryLive,
FalkorDBTriplesQueryService, FalkorDBTriplesQueryService,
@ -98,12 +98,10 @@ const provideFalkorDBTriplesQuery = (processorId: string) =>
}); });
export function makeTriplesQueryService(config: ProcessorConfig): TriplesQueryService { export function makeTriplesQueryService(config: ProcessorConfig): TriplesQueryService {
const service = makeFlowProcessor(config, { return makeFlowProcessor(config, {
specifications: makeTriplesQuerySpecs(), specifications: makeTriplesQuerySpecs(),
provide: provideFalkorDBTriplesQuery(config.id), provide: provideFalkorDBTriplesQuery(config.id),
}); });
void Effect.runPromise(Effect.log("[TriplesQuery] Service initialized"));
return service;
} }
export const TriplesQueryService = makeTriplesQueryService; export const TriplesQueryService = makeTriplesQueryService;
@ -118,12 +116,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => FalkorDBTriplesQueryLive(config), layer: (config) => FalkorDBTriplesQueryLive(config),
}); });
const triplesQueryRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return triplesQueryRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

48
ts/packages/flow/src/query/triples/falkordb.ts
View file

@ -13,8 +13,8 @@ import * as Predicate from "effect/Predicate";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
export interface FalkorDBClosableClient { export interface FalkorDBClosableClient {
readonly connect: () => Promise<unknown>; readonly connect: Effect.Effect<void, FalkorDBTriplesQueryError>;
readonly disconnect: () => Promise<unknown>; readonly disconnect: Effect.Effect<void, FalkorDBTriplesQueryError>;
} }
export type FalkorDBQueryOptions = Parameters<Graph["query"]>[1]; export type FalkorDBQueryOptions = Parameters<Graph["query"]>[1];
@ -23,7 +23,7 @@ export interface FalkorDBQueryGraph {
readonly query: <T = unknown>( readonly query: <T = unknown>(
query: string, query: string,
options?: FalkorDBQueryOptions, options?: FalkorDBQueryOptions,
) => Promise<{ readonly data?: Array<T> }>; ) => Effect.Effect<{ readonly data?: Array<T> }, FalkorDBTriplesQueryError>;
} }
export type FalkorDBQueryClientFactory = (url: string) => FalkorDBClosableClient; export type FalkorDBQueryClientFactory = (url: string) => FalkorDBClosableClient;
@ -73,7 +73,7 @@ export interface FalkorDBTriplesQuery {
p?: Term, p?: Term,
o?: Term, o?: Term,
limit?: number, limit?: number,
) => Promise<Triple[]>; ) => Effect.Effect<ReadonlyArray<Triple>, FalkorDBTriplesQueryError>;
} }
export class FalkorDBTriplesQueryError extends S.TaggedErrorClass<FalkorDBTriplesQueryError>()( export class FalkorDBTriplesQueryError extends S.TaggedErrorClass<FalkorDBTriplesQueryError>()(
@ -81,7 +81,7 @@ export class FalkorDBTriplesQueryError extends S.TaggedErrorClass<FalkorDBTriple
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -113,6 +113,12 @@ interface FalkorDBQueryConnection {
readonly graph: FalkorDBQueryGraph; readonly graph: FalkorDBQueryGraph;
} }
const tryFalkorDBPromise = <A>(operation: string, try_: () => PromiseLike<A>) =>
Effect.tryPromise({
try: try_,
catch: (cause) => falkorDBTriplesQueryError(operation, cause),
});
const resolveFalkorDBQueryConfig = Effect.fn("FalkorDBTriplesQuery.resolveConfig")(function* ( const resolveFalkorDBQueryConfig = Effect.fn("FalkorDBTriplesQuery.resolveConfig")(function* (
config: FalkorDBQueryConfig, config: FalkorDBQueryConfig,
) { ) {
@ -149,16 +155,21 @@ const connectFalkorDBTriplesQuery = Effect.fn("FalkorDBTriplesQuery.connect")(fu
const client = clientFactory(url); const client = clientFactory(url);
return { client, graph: graphFactory(client, database) }; return { client, graph: graphFactory(client, database) };
} }
const client = createClient({ url }); const sdkClient = createClient({ url });
return { client, graph: new Graph(client, database) }; const client: FalkorDBClosableClient = {
connect: tryFalkorDBPromise("connect", () => sdkClient.connect()).pipe(Effect.asVoid),
disconnect: tryFalkorDBPromise("disconnect", () => sdkClient.disconnect()).pipe(Effect.asVoid),
};
const sdkGraph = new Graph(sdkClient, database);
const graph: FalkorDBQueryGraph = {
query: (query, options) => tryFalkorDBPromise("graph-query", () => sdkGraph.query(query, options)),
};
return { client, graph };
}, },
catch: (cause) => falkorDBTriplesQueryError("create-client", cause), catch: (cause) => falkorDBTriplesQueryError("create-client", cause),
}); });
yield* Effect.tryPromise({ yield* client.connect.pipe(
try: () => client.connect(),
catch: (cause) => falkorDBTriplesQueryError("connect", cause),
}).pipe(
Effect.tapError((error) => Effect.tapError((error) =>
Effect.logError("[FalkorDBTriplesQuery] Connection failed", { Effect.logError("[FalkorDBTriplesQuery] Connection failed", {
error: error.message, error: error.message,
@ -174,10 +185,7 @@ const connectFalkorDBTriplesQuery = Effect.fn("FalkorDBTriplesQuery.connect")(fu
const disconnectFalkorDBTriplesQuery = ( const disconnectFalkorDBTriplesQuery = (
connection: FalkorDBQueryConnection, connection: FalkorDBQueryConnection,
): Effect.Effect<void> => ): Effect.Effect<void> =>
Effect.tryPromise({ connection.client.disconnect.pipe(
try: () => connection.client.disconnect(),
catch: (cause) => falkorDBTriplesQueryError("disconnect", cause),
}).pipe(
Effect.catch((error) => Effect.catch((error) =>
Effect.logError("[FalkorDBTriplesQuery] Disconnect failed", { Effect.logError("[FalkorDBTriplesQuery] Disconnect failed", {
error: error.message, error: error.message,
@ -201,10 +209,8 @@ const queryRows = (
query: string, query: string,
options?: FalkorDBQueryOptions, options?: FalkorDBQueryOptions,
): Effect.Effect<ReadonlyArray<unknown>, FalkorDBTriplesQueryError> => ): Effect.Effect<ReadonlyArray<unknown>, FalkorDBTriplesQueryError> =>
Effect.tryPromise({ graph.query<unknown>(query, options).pipe(
try: () => graph.query<unknown>(query, options), Effect.mapError((cause) => falkorDBTriplesQueryError(operation, cause)),
catch: (cause) => falkorDBTriplesQueryError(operation, cause),
}).pipe(
Effect.map((result) => result.data ?? []), Effect.map((result) => result.data ?? []),
); );
@ -480,9 +486,7 @@ export function makeFalkorDBTriplesQuery(
): FalkorDBTriplesQuery { ): FalkorDBTriplesQuery {
return { return {
queryTriples: (s, p, o, limit = 100) => queryTriples: (s, p, o, limit = 100) =>
Effect.runPromise( withFalkorDBTriplesQuery(config, (query) => query.queryTriples(s, p, o, limit)),
withFalkorDBTriplesQuery(config, (query) => query.queryTriples(s, p, o, limit)),
).then((triples) => Array.from(triples)),
}; };
} }

8
ts/packages/flow/src/retrieval/document-rag-service.ts
View file

@ -31,7 +31,7 @@ import {
type TextCompletionRequest, type TextCompletionRequest,
type TextCompletionResponse, type TextCompletionResponse,
} from "@trustgraph/base"; } from "@trustgraph/base";
import {Effect, Layer, ManagedRuntime} from "effect"; import {Effect} from "effect";
import { import {
DocumentRagEngine, DocumentRagEngine,
DocumentRagEngineError, DocumentRagEngineError,
@ -139,12 +139,6 @@ export const program = makeFlowProcessorProgram({
layer: () => DocumentRagLive, layer: () => DocumentRagLive,
}); });
const documentRagRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return documentRagRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

12
ts/packages/flow/src/retrieval/document-rag.ts
View file

@ -26,7 +26,10 @@ export interface DocumentRagClients {
prompt: EffectRequestResponse<PromptRequest, PromptResponse>; prompt: EffectRequestResponse<PromptRequest, PromptResponse>;
} }
export type ChunkCallback = (text: string, endOfStream: boolean) => Promise<void>; export type ChunkCallback = (
text: string,
endOfStream: boolean,
) => Effect.Effect<void, DocumentRagEngineError>;
export interface DocumentRagQueryOptions { export interface DocumentRagQueryOptions {
readonly collection?: string; readonly collection?: string;
@ -39,7 +42,7 @@ export class DocumentRagEngineError extends S.TaggedErrorClass<DocumentRagEngine
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -82,14 +85,13 @@ export interface DocumentRag {
readonly query: ( readonly query: (
queryText: string, queryText: string,
options?: DocumentRagQueryOptions, options?: DocumentRagQueryOptions,
) => Promise<string>; ) => Effect.Effect<string, DocumentRagEngineError>;
} }
export function makeDocumentRag(clients: DocumentRagClients): DocumentRag { export function makeDocumentRag(clients: DocumentRagClients): DocumentRag {
const engine = makeDocumentRagEngine(); const engine = makeDocumentRagEngine();
return { return {
query: (queryText, options) => query: (queryText, options) => engine.query(clients, queryText, options),
Effect.runPromise(engine.query(clients, queryText, options)),
}; };
} }

8
ts/packages/flow/src/retrieval/graph-rag-service.ts
View file

@ -33,7 +33,7 @@ import {
type TriplesQueryRequest, type TriplesQueryRequest,
type TriplesQueryResponse, type TriplesQueryResponse,
} from "@trustgraph/base"; } from "@trustgraph/base";
import {Effect, Layer, ManagedRuntime} from "effect"; import {Effect} from "effect";
import { import {
GraphRagEngine, GraphRagEngine,
GraphRagEngineError, GraphRagEngineError,
@ -173,12 +173,6 @@ export const program = makeFlowProcessorProgram({
layer: () => GraphRagLive, layer: () => GraphRagLive,
}); });
const graphRagRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return graphRagRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

21
ts/packages/flow/src/retrieval/graph-rag.ts
View file

@ -42,7 +42,10 @@ export interface GraphRagClients {
prompt: EffectRequestResponse<PromptRequest, PromptResponse>; prompt: EffectRequestResponse<PromptRequest, PromptResponse>;
} }
export type ChunkCallback = (text: string, endOfStream: boolean) => Promise<void>; export type ChunkCallback = (
text: string,
endOfStream: boolean,
) => Effect.Effect<void, GraphRagEngineError>;
export interface GraphRagQueryOptions { export interface GraphRagQueryOptions {
readonly collection?: string; readonly collection?: string;
@ -69,7 +72,7 @@ export class GraphRagEngineError extends S.TaggedErrorClass<GraphRagEngineError>
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -135,7 +138,7 @@ export interface GraphRag {
readonly query: ( readonly query: (
queryText: string, queryText: string,
options?: GraphRagQueryOptions, options?: GraphRagQueryOptions,
) => Promise<GraphRagResult>; ) => Effect.Effect<GraphRagResult, GraphRagEngineError>;
} }
export function makeGraphRag( export function makeGraphRag(
@ -144,8 +147,7 @@ export function makeGraphRag(
): GraphRag { ): GraphRag {
const engine = makeGraphRagEngine(); const engine = makeGraphRagEngine();
return { return {
query: (queryText, options) => query: (queryText, options) => engine.query(clients, queryText, options, config),
Effect.runPromise(engine.query(clients, queryText, options, config)),
}; };
} }
@ -403,10 +405,9 @@ const synthesize = Effect.fn("GraphRagEngine.synthesize")(function* (
return Effect.succeed(resp.endOfStream === true); return Effect.succeed(resp.endOfStream === true);
} }
fullText += resp.response; fullText += resp.response;
return Effect.tryPromise({ return chunkCallback(resp.response, resp.endOfStream === true).pipe(
try: () => chunkCallback(resp.response, resp.endOfStream === true).then(() => resp.endOfStream === true), Effect.as(resp.endOfStream === true),
catch: (cause) => graphRagError("synthesize-stream-callback", cause), );
});
}, },
}, },
); );
@ -427,7 +428,7 @@ const synthesize = Effect.fn("GraphRagEngine.synthesize")(function* (
const ScoredEdge = S.Struct({ const ScoredEdge = S.Struct({
id: S.String, id: S.String,
score: S.Number, score: S.Finite,
}); });
const ScoredEdgesFromJson = S.Array(ScoredEdge).pipe(S.fromJsonString); const ScoredEdgesFromJson = S.Array(ScoredEdge).pipe(S.fromJsonString);
const ScoredEdgeFromJson = ScoredEdge.pipe(S.fromJsonString); const ScoredEdgeFromJson = ScoredEdge.pipe(S.fromJsonString);

61
ts/packages/flow/src/runtime/effect-files.ts
View file

@ -1,10 +1,10 @@
/** @effect-diagnostics strictEffectProvide:skip-file */
import * as BunFileSystem from "@effect/platform-bun/BunFileSystem"; import * as BunFileSystem from "@effect/platform-bun/BunFileSystem";
import { Effect, ManagedRuntime } from "effect"; import { Effect } from "effect";
import * as FileSystem from "effect/FileSystem"; import * as FileSystem from "effect/FileSystem";
import type { PlatformError } from "effect/PlatformError"; import type { PlatformError } from "effect/PlatformError";
const fileSystemRuntime = ManagedRuntime.make(BunFileSystem.layer);
export function joinPath(...segments: string[]): string { export function joinPath(...segments: string[]): string {
const joined = segments const joined = segments
.filter((segment) => segment.length > 0) .filter((segment) => segment.length > 0)
@ -22,52 +22,33 @@ export function dirnamePath(path: string): string {
return normalized.slice(0, index); return normalized.slice(0, index);
} }
export const ensureDirectoryEffect = (path: string): Effect.Effect<void, PlatformError, FileSystem.FileSystem> => const withFileSystem = <A, E>(
Effect.flatMap(FileSystem.FileSystem, (fs) => effect: Effect.Effect<A, E, FileSystem.FileSystem>,
): Effect.Effect<A, E> =>
effect.pipe(Effect.provide(BunFileSystem.layer));
export const ensureDirectoryEffect = (path: string): Effect.Effect<void, PlatformError> =>
withFileSystem(Effect.flatMap(FileSystem.FileSystem, (fs) =>
fs.makeDirectory(path, { recursive: true }) fs.makeDirectory(path, { recursive: true })
); ));
export function ensureDirectory(path: string): Promise<void> { export const readTextFileEffect = (path: string): Effect.Effect<string, PlatformError> =>
return fileSystemRuntime.runPromise(ensureDirectoryEffect(path)); withFileSystem(Effect.flatMap(FileSystem.FileSystem, (fs) => fs.readFileString(path)));
}
export const readTextFileEffect = (path: string): Effect.Effect<string, PlatformError, FileSystem.FileSystem> => export const readBinaryFileEffect = (path: string): Effect.Effect<Uint8Array, PlatformError> =>
Effect.flatMap(FileSystem.FileSystem, (fs) => fs.readFileString(path)); withFileSystem(Effect.flatMap(FileSystem.FileSystem, (fs) => fs.readFile(path)));
export function readTextFile(path: string): Promise<string> {
return fileSystemRuntime.runPromise(readTextFileEffect(path));
}
export const readBinaryFileEffect = (path: string): Effect.Effect<Uint8Array, PlatformError, FileSystem.FileSystem> =>
Effect.flatMap(FileSystem.FileSystem, (fs) => fs.readFile(path));
export function readBinaryFile(path: string): Promise<Uint8Array> {
return fileSystemRuntime.runPromise(readBinaryFileEffect(path));
}
export const writeTextFileEffect = ( export const writeTextFileEffect = (
path: string, path: string,
data: string, data: string,
): Effect.Effect<void, PlatformError, FileSystem.FileSystem> => ): Effect.Effect<void, PlatformError> =>
Effect.flatMap(FileSystem.FileSystem, (fs) => fs.writeFileString(path, data)); withFileSystem(Effect.flatMap(FileSystem.FileSystem, (fs) => fs.writeFileString(path, data)));
export function writeTextFile(path: string, data: string): Promise<void> {
return fileSystemRuntime.runPromise(writeTextFileEffect(path, data));
}
export const writeBinaryFileEffect = ( export const writeBinaryFileEffect = (
path: string, path: string,
data: Uint8Array, data: Uint8Array,
): Effect.Effect<void, PlatformError, FileSystem.FileSystem> => ): Effect.Effect<void, PlatformError> =>
Effect.flatMap(FileSystem.FileSystem, (fs) => fs.writeFile(path, data)); withFileSystem(Effect.flatMap(FileSystem.FileSystem, (fs) => fs.writeFile(path, data)));
export function writeBinaryFile(path: string, data: Uint8Array): Promise<void> { export const removePathEffect = (path: string): Effect.Effect<void, PlatformError> =>
return fileSystemRuntime.runPromise(writeBinaryFileEffect(path, data)); withFileSystem(Effect.flatMap(FileSystem.FileSystem, (fs) => fs.remove(path)));
}
export const removePathEffect = (path: string): Effect.Effect<void, PlatformError, FileSystem.FileSystem> =>
Effect.flatMap(FileSystem.FileSystem, (fs) => fs.remove(path));
export function removePath(path: string): Promise<void> {
return fileSystemRuntime.runPromise(removePathEffect(path));
}

12
ts/packages/flow/src/storage/embeddings/graph-embeddings-service.ts
View file

@ -29,7 +29,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import { import {
QdrantGraphEmbeddingsStoreLive, QdrantGraphEmbeddingsStoreLive,
QdrantGraphEmbeddingsStoreService, QdrantGraphEmbeddingsStoreService,
@ -113,12 +113,10 @@ const provideQdrantGraphEmbeddingsStore = (processorId: string) =>
}); });
export function makeGraphEmbeddingsStoreService(config: ProcessorConfig): GraphEmbeddingsStoreService { export function makeGraphEmbeddingsStoreService(config: ProcessorConfig): GraphEmbeddingsStoreService {
const service = makeFlowProcessor(config, { return makeFlowProcessor(config, {
specifications: makeGraphEmbeddingsStoreSpecs(), specifications: makeGraphEmbeddingsStoreSpecs(),
provide: provideQdrantGraphEmbeddingsStore(config.id), provide: provideQdrantGraphEmbeddingsStore(config.id),
}); });
void Effect.runPromise(Effect.log("[GraphEmbeddingsStore] Service initialized"));
return service;
} }
export const GraphEmbeddingsStoreService = makeGraphEmbeddingsStoreService; export const GraphEmbeddingsStoreService = makeGraphEmbeddingsStoreService;
@ -133,12 +131,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => QdrantGraphEmbeddingsStoreLive(config), layer: (config) => QdrantGraphEmbeddingsStoreLive(config),
}); });
const graphEmbeddingsStoreRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return graphEmbeddingsStoreRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

98
ts/packages/flow/src/storage/embeddings/qdrant-doc.ts
View file

@ -38,7 +38,7 @@ export class QdrantDocEmbeddingsStoreError extends S.TaggedErrorClass<QdrantDocE
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -85,12 +85,10 @@ const randomPointId = Effect.fn("QdrantDocEmbeddings.randomPointId")(function* (
}); });
export interface QdrantDocEmbeddingsStore { export interface QdrantDocEmbeddingsStore {
readonly store: (message: DocEmbeddingsMessage) => Promise<void>; readonly store: (
readonly deleteCollection: (user: string, collection: string) => Promise<void>;
readonly storeEffect: (
message: DocEmbeddingsMessage, message: DocEmbeddingsMessage,
) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>; ) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>;
readonly deleteCollectionEffect: ( readonly deleteCollection: (
user: string, user: string,
collection: string, collection: string,
) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>; ) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>;
@ -133,25 +131,25 @@ const makeQdrantDocEmbeddingsStoreFromClient = (
) { ) {
if (MutableHashSet.has(knownCollections, name)) return; if (MutableHashSet.has(knownCollections, name)) return;
const exists = yield* Effect.tryPromise({ const exists = yield* client.collectionExists(name).pipe(
try: () => client.collectionExists(name), Effect.mapError((cause) => qdrantDocEmbeddingsStoreError("collection-exists", cause)),
catch: (cause) => qdrantDocEmbeddingsStoreError("collection-exists", cause), );
});
if (!exists.exists) { if (!exists.exists) {
yield* Effect.log(`[QdrantDocEmbeddings] Creating collection ${name} (dim=${dim})`); yield* Effect.log(`[QdrantDocEmbeddings] Creating collection ${name} (dim=${dim})`);
yield* Effect.tryPromise({ yield* client.createCollection(
try: () => name,
client.createCollection(name, { {
vectors: { size: dim, distance: "Cosine" }, vectors: { size: dim, distance: "Cosine" },
}), },
catch: (cause) => qdrantDocEmbeddingsStoreError("create-collection", cause), ).pipe(
}); Effect.mapError((cause) => qdrantDocEmbeddingsStoreError("create-collection", cause)),
);
} }
MutableHashSet.add(knownCollections, name); MutableHashSet.add(knownCollections, name);
}); });
const storeEffect = Effect.fn("QdrantDocEmbeddings.store")(function* (message: DocEmbeddingsMessage) { const storeImpl = Effect.fn("QdrantDocEmbeddings.store")(function* (message: DocEmbeddingsMessage) {
for (const chunk of message.chunks) { for (const chunk of message.chunks) {
if (chunk.chunkId.length === 0) continue; if (chunk.chunkId.length === 0) continue;
if (chunk.vector.length === 0) continue; if (chunk.vector.length === 0) continue;
@ -162,37 +160,37 @@ const makeQdrantDocEmbeddingsStoreFromClient = (
yield* ensureCollectionEffect(name, dim); yield* ensureCollectionEffect(name, dim);
const id = yield* randomPointId(); const id = yield* randomPointId();
yield* Effect.tryPromise({ yield* client.upsert(
try: () => name,
client.upsert(name, { {
points: [ points: [
{ {
id, id,
vector: chunk.vector, vector: chunk.vector,
payload: { payload: {
chunk_id: chunk.chunkId, chunk_id: chunk.chunkId,
...(chunk.content !== undefined && chunk.content.length > 0 ...(chunk.content !== undefined && chunk.content.length > 0
? { content: chunk.content } ? { content: chunk.content }
: {}), : {}),
},
}, },
], },
}), ],
catch: (cause) => qdrantDocEmbeddingsStoreError("upsert", cause), },
}); ).pipe(
Effect.mapError((cause) => qdrantDocEmbeddingsStoreError("upsert", cause)),
);
} }
}); });
const deleteCollectionEffect = Effect.fn("QdrantDocEmbeddings.deleteCollection")(function* ( const deleteCollectionImpl = Effect.fn("QdrantDocEmbeddings.deleteCollection")(function* (
user: string, user: string,
collection: string, collection: string,
) { ) {
const prefix = `d_${user}_${collection}_`; const prefix = `d_${user}_${collection}_`;
const allCollections = yield* Effect.tryPromise({ const allCollections = yield* client.getCollections.pipe(
try: () => client.getCollections(), Effect.mapError((cause) => qdrantDocEmbeddingsStoreError("get-collections", cause)),
catch: (cause) => qdrantDocEmbeddingsStoreError("get-collections", cause), );
});
const matching = allCollections.collections.filter((c) => const matching = allCollections.collections.filter((c) =>
c.name.startsWith(prefix), c.name.startsWith(prefix),
); );
@ -203,10 +201,9 @@ const makeQdrantDocEmbeddingsStoreFromClient = (
} }
for (const coll of matching) { for (const coll of matching) {
yield* Effect.tryPromise({ yield* client.deleteCollection(coll.name).pipe(
try: () => client.deleteCollection(coll.name), Effect.mapError((cause) => qdrantDocEmbeddingsStoreError("delete-collection", cause)),
catch: (cause) => qdrantDocEmbeddingsStoreError("delete-collection", cause), );
});
MutableHashSet.remove(knownCollections, coll.name); MutableHashSet.remove(knownCollections, coll.name);
yield* Effect.log(`[QdrantDocEmbeddings] Deleted collection: ${coll.name}`); yield* Effect.log(`[QdrantDocEmbeddings] Deleted collection: ${coll.name}`);
} }
@ -217,8 +214,8 @@ const makeQdrantDocEmbeddingsStoreFromClient = (
}); });
return { return {
store: storeEffect, store: storeImpl,
deleteCollection: deleteCollectionEffect, deleteCollection: deleteCollectionImpl,
}; };
}; };
@ -244,16 +241,9 @@ const withQdrantDocEmbeddingsStore = <A>(
export function makeQdrantDocEmbeddingsStore( export function makeQdrantDocEmbeddingsStore(
config: QdrantDocEmbeddingsConfig = {}, config: QdrantDocEmbeddingsConfig = {},
): QdrantDocEmbeddingsStore { ): QdrantDocEmbeddingsStore {
const storeEffect = (message: DocEmbeddingsMessage) =>
withQdrantDocEmbeddingsStore(config, (store) => store.store(message));
const deleteCollectionEffect = (user: string, collection: string) =>
withQdrantDocEmbeddingsStore(config, (store) => store.deleteCollection(user, collection));
return { return {
store: (message) => Effect.runPromise(storeEffect(message)), store: (message) => withQdrantDocEmbeddingsStore(config, (store) => store.store(message)),
deleteCollection: (user, collection) => deleteCollection: (user, collection) =>
Effect.runPromise(deleteCollectionEffect(user, collection)), withQdrantDocEmbeddingsStore(config, (store) => store.deleteCollection(user, collection)),
storeEffect,
deleteCollectionEffect,
}; };
} }

90
ts/packages/flow/src/storage/embeddings/qdrant-graph.ts
View file

@ -38,7 +38,7 @@ export class QdrantGraphEmbeddingsStoreError extends S.TaggedErrorClass<QdrantGr
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -96,12 +96,10 @@ function getTermValue(term: Term): string | null {
} }
export interface QdrantGraphEmbeddingsStore { export interface QdrantGraphEmbeddingsStore {
readonly store: (message: GraphEmbeddingsMessage) => Promise<void>; readonly store: (
readonly deleteCollection: (user: string, collection: string) => Promise<void>;
readonly storeEffect: (
message: GraphEmbeddingsMessage, message: GraphEmbeddingsMessage,
) => Effect.Effect<void, QdrantGraphEmbeddingsStoreError>; ) => Effect.Effect<void, QdrantGraphEmbeddingsStoreError>;
readonly deleteCollectionEffect: ( readonly deleteCollection: (
user: string, user: string,
collection: string, collection: string,
) => Effect.Effect<void, QdrantGraphEmbeddingsStoreError>; ) => Effect.Effect<void, QdrantGraphEmbeddingsStoreError>;
@ -134,25 +132,25 @@ const makeQdrantGraphEmbeddingsStoreFromClient = (
) { ) {
if (MutableHashSet.has(knownCollections, name)) return; if (MutableHashSet.has(knownCollections, name)) return;
const exists = yield* Effect.tryPromise({ const exists = yield* client.collectionExists(name).pipe(
try: () => client.collectionExists(name), Effect.mapError((cause) => qdrantGraphEmbeddingsStoreError("collection-exists", cause)),
catch: (cause) => qdrantGraphEmbeddingsStoreError("collection-exists", cause), );
});
if (!exists.exists) { if (!exists.exists) {
yield* Effect.log(`[QdrantGraphEmbeddings] Creating collection ${name} (dim=${dim})`); yield* Effect.log(`[QdrantGraphEmbeddings] Creating collection ${name} (dim=${dim})`);
yield* Effect.tryPromise({ yield* client.createCollection(
try: () => name,
client.createCollection(name, { {
vectors: { size: dim, distance: "Cosine" }, vectors: { size: dim, distance: "Cosine" },
}), },
catch: (cause) => qdrantGraphEmbeddingsStoreError("create-collection", cause), ).pipe(
}); Effect.mapError((cause) => qdrantGraphEmbeddingsStoreError("create-collection", cause)),
);
} }
MutableHashSet.add(knownCollections, name); MutableHashSet.add(knownCollections, name);
}); });
const storeEffect = Effect.fn("QdrantGraphEmbeddings.store")(function* (message: GraphEmbeddingsMessage) { const storeImpl = Effect.fn("QdrantGraphEmbeddings.store")(function* (message: GraphEmbeddingsMessage) {
for (const entry of message.entities) { for (const entry of message.entities) {
const entityValue = getTermValue(entry.entity); const entityValue = getTermValue(entry.entity);
if (entityValue === null || entityValue.length === 0) continue; if (entityValue === null || entityValue.length === 0) continue;
@ -169,32 +167,32 @@ const makeQdrantGraphEmbeddingsStoreFromClient = (
} }
const id = yield* randomPointId(); const id = yield* randomPointId();
yield* Effect.tryPromise({ yield* client.upsert(
try: () => name,
client.upsert(name, { {
points: [ points: [
{ {
id, id,
vector: entry.vector, vector: entry.vector,
payload, payload,
}, },
], ],
}), },
catch: (cause) => qdrantGraphEmbeddingsStoreError("upsert", cause), ).pipe(
}); Effect.mapError((cause) => qdrantGraphEmbeddingsStoreError("upsert", cause)),
);
} }
}); });
const deleteCollectionEffect = Effect.fn("QdrantGraphEmbeddings.deleteCollection")(function* ( const deleteCollectionImpl = Effect.fn("QdrantGraphEmbeddings.deleteCollection")(function* (
user: string, user: string,
collection: string, collection: string,
) { ) {
const prefix = `t_${user}_${collection}_`; const prefix = `t_${user}_${collection}_`;
const allCollections = yield* Effect.tryPromise({ const allCollections = yield* client.getCollections.pipe(
try: () => client.getCollections(), Effect.mapError((cause) => qdrantGraphEmbeddingsStoreError("get-collections", cause)),
catch: (cause) => qdrantGraphEmbeddingsStoreError("get-collections", cause), );
});
const matching = allCollections.collections.filter((c) => const matching = allCollections.collections.filter((c) =>
c.name.startsWith(prefix), c.name.startsWith(prefix),
); );
@ -205,10 +203,9 @@ const makeQdrantGraphEmbeddingsStoreFromClient = (
} }
for (const coll of matching) { for (const coll of matching) {
yield* Effect.tryPromise({ yield* client.deleteCollection(coll.name).pipe(
try: () => client.deleteCollection(coll.name), Effect.mapError((cause) => qdrantGraphEmbeddingsStoreError("delete-collection", cause)),
catch: (cause) => qdrantGraphEmbeddingsStoreError("delete-collection", cause), );
});
MutableHashSet.remove(knownCollections, coll.name); MutableHashSet.remove(knownCollections, coll.name);
yield* Effect.log(`[QdrantGraphEmbeddings] Deleted collection: ${coll.name}`); yield* Effect.log(`[QdrantGraphEmbeddings] Deleted collection: ${coll.name}`);
} }
@ -219,8 +216,8 @@ const makeQdrantGraphEmbeddingsStoreFromClient = (
}); });
return { return {
store: storeEffect, store: storeImpl,
deleteCollection: deleteCollectionEffect, deleteCollection: deleteCollectionImpl,
}; };
}; };
@ -246,17 +243,10 @@ const withQdrantGraphEmbeddingsStore = <A>(
export function makeQdrantGraphEmbeddingsStore( export function makeQdrantGraphEmbeddingsStore(
config: QdrantGraphEmbeddingsConfig = {}, config: QdrantGraphEmbeddingsConfig = {},
): QdrantGraphEmbeddingsStore { ): QdrantGraphEmbeddingsStore {
const storeEffect = (message: GraphEmbeddingsMessage) =>
withQdrantGraphEmbeddingsStore(config, (store) => store.store(message));
const deleteCollectionEffect = (user: string, collection: string) =>
withQdrantGraphEmbeddingsStore(config, (store) => store.deleteCollection(user, collection));
return { return {
store: (message) => Effect.runPromise(storeEffect(message)), store: (message) => withQdrantGraphEmbeddingsStore(config, (store) => store.store(message)),
deleteCollection: (user, collection) => deleteCollection: (user, collection) =>
Effect.runPromise(deleteCollectionEffect(user, collection)), withQdrantGraphEmbeddingsStore(config, (store) => store.deleteCollection(user, collection)),
storeEffect,
deleteCollectionEffect,
}; };
} }

12
ts/packages/flow/src/storage/triples/falkordb-service.ts
View file

@ -21,7 +21,7 @@ import {
} from "@trustgraph/base"; } from "@trustgraph/base";
import { NodeRuntime } from "@effect/platform-node"; import { NodeRuntime } from "@effect/platform-node";
import { makeFlowProcessorProgram } from "@trustgraph/base"; import { makeFlowProcessorProgram } from "@trustgraph/base";
import { Effect, Layer, ManagedRuntime } from "effect"; import { Effect } from "effect";
import { import {
FalkorDBTriplesStoreLive, FalkorDBTriplesStoreLive,
FalkorDBTriplesStoreService, FalkorDBTriplesStoreService,
@ -73,12 +73,10 @@ const provideFalkorDBTriplesStore = (processorId: string) =>
}); });
export function makeTriplesStoreService(config: ProcessorConfig): TriplesStoreService { export function makeTriplesStoreService(config: ProcessorConfig): TriplesStoreService {
const service = makeFlowProcessor(config, { return makeFlowProcessor(config, {
specifications: makeTriplesStoreSpecs(), specifications: makeTriplesStoreSpecs(),
provide: provideFalkorDBTriplesStore(config.id), provide: provideFalkorDBTriplesStore(config.id),
}); });
void Effect.runPromise(Effect.log("[TriplesStore] Service initialized"));
return service;
} }
export const TriplesStoreService = makeTriplesStoreService; export const TriplesStoreService = makeTriplesStoreService;
@ -93,12 +91,6 @@ export const program = makeFlowProcessorProgram<
layer: (config) => FalkorDBTriplesStoreLive(config), layer: (config) => FalkorDBTriplesStoreLive(config),
}); });
const triplesStoreRuntime = ManagedRuntime.make(Layer.empty);
export function run(): Promise<void> {
return triplesStoreRuntime.runPromise(program);
}
export function runMain(): void { export function runMain(): void {
NodeRuntime.runMain(program); NodeRuntime.runMain(program);
} }

77
ts/packages/flow/src/storage/triples/falkordb.ts
View file

@ -13,8 +13,8 @@ import { Config, Context, Effect, Layer, Match } from "effect";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
export interface FalkorDBClosableClient { export interface FalkorDBClosableClient {
readonly connect: () => Promise<unknown>; readonly connect: Effect.Effect<void, FalkorDBTriplesStoreError>;
readonly disconnect: () => Promise<unknown>; readonly disconnect: Effect.Effect<void, FalkorDBTriplesStoreError>;
} }
export type FalkorDBStoreQueryOptions = Parameters<Graph["query"]>[1]; export type FalkorDBStoreQueryOptions = Parameters<Graph["query"]>[1];
@ -23,7 +23,7 @@ export interface FalkorDBStoreGraph {
readonly query: <T = unknown>( readonly query: <T = unknown>(
query: string, query: string,
options?: FalkorDBStoreQueryOptions, options?: FalkorDBStoreQueryOptions,
) => Promise<{ readonly data?: Array<T> }>; ) => Effect.Effect<{ readonly data?: Array<T> }, FalkorDBTriplesStoreError>;
} }
export type FalkorDBStoreClientFactory = (url: string) => FalkorDBClosableClient; export type FalkorDBStoreClientFactory = (url: string) => FalkorDBClosableClient;
@ -51,28 +51,39 @@ function getTermValue(term: Term): string {
} }
export interface FalkorDBTriplesStore { export interface FalkorDBTriplesStore {
readonly createNode: (uri: string, user: string, collection: string) => Promise<void>; readonly createNode: (
readonly createLiteral: (value: string, user: string, collection: string) => Promise<void>; uri: string,
user: string,
collection: string,
) => Effect.Effect<void, FalkorDBTriplesStoreError>;
readonly createLiteral: (
value: string,
user: string,
collection: string,
) => Effect.Effect<void, FalkorDBTriplesStoreError>;
readonly relateNode: ( readonly relateNode: (
src: string, src: string,
uri: string, uri: string,
dest: string, dest: string,
user: string, user: string,
collection: string, collection: string,
) => Promise<void>; ) => Effect.Effect<void, FalkorDBTriplesStoreError>;
readonly relateLiteral: ( readonly relateLiteral: (
src: string, src: string,
uri: string, uri: string,
dest: string, dest: string,
user: string, user: string,
collection: string, collection: string,
) => Promise<void>; ) => Effect.Effect<void, FalkorDBTriplesStoreError>;
readonly storeTriples: ( readonly storeTriples: (
triples: Triple[], triples: Triple[],
user?: string, user?: string,
collection?: string, collection?: string,
) => Promise<void>; ) => Effect.Effect<void, FalkorDBTriplesStoreError>;
readonly deleteCollection: (user: string, collection: string) => Promise<void>; readonly deleteCollection: (
user: string,
collection: string,
) => Effect.Effect<void, FalkorDBTriplesStoreError>;
} }
export class FalkorDBTriplesStoreError extends S.TaggedErrorClass<FalkorDBTriplesStoreError>()( export class FalkorDBTriplesStoreError extends S.TaggedErrorClass<FalkorDBTriplesStoreError>()(
@ -80,7 +91,7 @@ export class FalkorDBTriplesStoreError extends S.TaggedErrorClass<FalkorDBTriple
{ {
message: S.String, message: S.String,
operation: S.String, operation: S.String,
cause: S.DefectWithStack, cause: S.Defect({ includeStack: true }),
}, },
) {} ) {}
@ -115,6 +126,12 @@ interface FalkorDBStoreConnection {
readonly graph: FalkorDBStoreGraph; readonly graph: FalkorDBStoreGraph;
} }
const tryFalkorDBPromise = <A>(operation: string, try_: () => PromiseLike<A>) =>
Effect.tryPromise({
try: try_,
catch: (cause) => falkorDBTriplesStoreError(operation, cause),
});
interface FalkorDBTriplesStoreEffectShape { interface FalkorDBTriplesStoreEffectShape {
readonly createNode: ( readonly createNode: (
uri: string, uri: string,
@ -187,16 +204,21 @@ const connectFalkorDBTriplesStore = Effect.fn("FalkorDBTriplesStore.connect")(fu
const client = clientFactory(url); const client = clientFactory(url);
return { client, graph: graphFactory(client, database) }; return { client, graph: graphFactory(client, database) };
} }
const client = createClient({ url }); const sdkClient = createClient({ url });
return { client, graph: new Graph(client, database) }; const client: FalkorDBClosableClient = {
connect: tryFalkorDBPromise("connect", () => sdkClient.connect()).pipe(Effect.asVoid),
disconnect: tryFalkorDBPromise("disconnect", () => sdkClient.disconnect()).pipe(Effect.asVoid),
};
const sdkGraph = new Graph(sdkClient, database);
const graph: FalkorDBStoreGraph = {
query: (query, options) => tryFalkorDBPromise("graph-query", () => sdkGraph.query(query, options)),
};
return { client, graph };
}, },
catch: (cause) => falkorDBTriplesStoreError("create-client", cause), catch: (cause) => falkorDBTriplesStoreError("create-client", cause),
}); });
yield* Effect.tryPromise({ yield* client.connect.pipe(
try: () => client.connect(),
catch: (cause) => falkorDBTriplesStoreError("connect", cause),
}).pipe(
Effect.tapError((error) => Effect.tapError((error) =>
Effect.logError("[FalkorDBTriplesStore] Connection failed", { Effect.logError("[FalkorDBTriplesStore] Connection failed", {
error: error.message, error: error.message,
@ -212,10 +234,7 @@ const connectFalkorDBTriplesStore = Effect.fn("FalkorDBTriplesStore.connect")(fu
const disconnectFalkorDBTriplesStore = ( const disconnectFalkorDBTriplesStore = (
connection: FalkorDBStoreConnection, connection: FalkorDBStoreConnection,
): Effect.Effect<void> => ): Effect.Effect<void> =>
Effect.tryPromise({ connection.client.disconnect.pipe(
try: () => connection.client.disconnect(),
catch: (cause) => falkorDBTriplesStoreError("disconnect", cause),
}).pipe(
Effect.catch((error) => Effect.catch((error) =>
Effect.logError("[FalkorDBTriplesStore] Disconnect failed", { Effect.logError("[FalkorDBTriplesStore] Disconnect failed", {
error: error.message, error: error.message,
@ -239,10 +258,8 @@ const runGraphQuery = (
query: string, query: string,
options?: FalkorDBStoreQueryOptions, options?: FalkorDBStoreQueryOptions,
): Effect.Effect<void, FalkorDBTriplesStoreError> => ): Effect.Effect<void, FalkorDBTriplesStoreError> =>
Effect.tryPromise({ graph.query(query, options).pipe(
try: () => graph.query(query, options), Effect.mapError((cause) => falkorDBTriplesStoreError(operation, cause)),
catch: (cause) => falkorDBTriplesStoreError(operation, cause),
}).pipe(
Effect.asVoid, Effect.asVoid,
); );
@ -390,17 +407,17 @@ const withFalkorDBTriplesStore = <A>(
export function makeFalkorDBTriplesStore(config: FalkorDBConfig = {}): FalkorDBTriplesStore { export function makeFalkorDBTriplesStore(config: FalkorDBConfig = {}): FalkorDBTriplesStore {
return { return {
createNode: (uri, user, collection) => createNode: (uri, user, collection) =>
Effect.runPromise(withFalkorDBTriplesStore(config, (store) => store.createNode(uri, user, collection))), withFalkorDBTriplesStore(config, (store) => store.createNode(uri, user, collection)),
createLiteral: (value, user, collection) => createLiteral: (value, user, collection) =>
Effect.runPromise(withFalkorDBTriplesStore(config, (store) => store.createLiteral(value, user, collection))), withFalkorDBTriplesStore(config, (store) => store.createLiteral(value, user, collection)),
relateNode: (src, uri, dest, user, collection) => relateNode: (src, uri, dest, user, collection) =>
Effect.runPromise(withFalkorDBTriplesStore(config, (store) => store.relateNode(src, uri, dest, user, collection))), withFalkorDBTriplesStore(config, (store) => store.relateNode(src, uri, dest, user, collection)),
relateLiteral: (src, uri, dest, user, collection) => relateLiteral: (src, uri, dest, user, collection) =>
Effect.runPromise(withFalkorDBTriplesStore(config, (store) => store.relateLiteral(src, uri, dest, user, collection))), withFalkorDBTriplesStore(config, (store) => store.relateLiteral(src, uri, dest, user, collection)),
storeTriples: (triples, user = "default", collection = "default") => storeTriples: (triples, user = "default", collection = "default") =>
Effect.runPromise(withFalkorDBTriplesStore(config, (store) => store.storeTriples(triples, user, collection))), withFalkorDBTriplesStore(config, (store) => store.storeTriples(triples, user, collection)),
deleteCollection: (user, collection) => deleteCollection: (user, collection) =>
Effect.runPromise(withFalkorDBTriplesStore(config, (store) => store.deleteCollection(user, collection))), withFalkorDBTriplesStore(config, (store) => store.deleteCollection(user, collection)),
}; };
} }

28
ts/packages/mcp/package.json
View file

@ -13,23 +13,21 @@
"dependencies": { "dependencies": {
"@trustgraph/base": "workspace:*", "@trustgraph/base": "workspace:*",
"@trustgraph/client": "workspace:*", "@trustgraph/client": "workspace:*",
"@effect/platform-node": "4.0.0-beta.75", "@effect/platform-node": "4.0.0-beta.78",
"@effect/platform-node-shared": "4.0.0-beta.75", "@effect/platform-node-shared": "4.0.0-beta.78",
"@effect/ai-anthropic": "4.0.0-beta.75", "@effect/ai-anthropic": "4.0.0-beta.78",
"@effect/ai-openai": "4.0.0-beta.75", "@effect/ai-openai": "4.0.0-beta.78",
"@effect/ai-openrouter": "4.0.0-beta.75", "@effect/ai-openrouter": "4.0.0-beta.78",
"@effect/atom-react": "4.0.0-beta.75", "@effect/atom-react": "4.0.0-beta.78",
"@effect/openapi-generator": "4.0.0-beta.75", "@effect/openapi-generator": "4.0.0-beta.78",
"@effect/opentelemetry": "4.0.0-beta.75", "@effect/opentelemetry": "4.0.0-beta.78",
"@effect/platform-browser": "4.0.0-beta.75", "@effect/platform-browser": "4.0.0-beta.78",
"@effect/platform-bun": "4.0.0-beta.75", "@effect/platform-bun": "4.0.0-beta.78",
"@effect/tsgo": "0.13.0", "@effect/tsgo": "0.14.0",
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78"
"@modelcontextprotocol/sdk": "^1.8.0",
"zod": "^3.23.0"
}, },
"devDependencies": { "devDependencies": {
"@effect/vitest": "4.0.0-beta.75", "@effect/vitest": "4.0.0-beta.78",
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
"typescript": "^5.8.0", "typescript": "^5.8.0",
"vitest": "^4.1.6" "vitest": "^4.1.6"

Some files were not shown because too many files have changed in this diff Show more