Add fakeable Qdrant Effect services

This commit is contained in:
elpresidank 2026-06-02 04:10:03 -05:00
parent d38ce475fd
commit 8287e1cf93
10 changed files with 671 additions and 154 deletions

View file

@ -12,14 +12,15 @@ Verified source roots:
- Effect v4 subtree: `/home/elpresidank/YeeBois/projects/beep-effect2/.repos/effect-v4` - Effect v4 subtree: `/home/elpresidank/YeeBois/projects/beep-effect2/.repos/effect-v4`
- Installed Effect beta used by this workspace: `ts/node_modules/effect` - Installed Effect beta used by this workspace: `ts/node_modules/effect`
Current signal counts from `ts/packages` after the 2026-06-02 FalkorDB scoped Current signal counts from `ts/packages` after the 2026-06-02 Qdrant
client lifecycle slice: config/schema/fakeability slice:
| Signal | Count | | Signal | Count |
| --- | ---: | | --- | ---: |
| `Effect.runPromise` | 165 | | `Effect.runPromise` | 172 |
| `Effect.runPromiseWith` | 0 | | `Effect.runPromiseWith` | 0 |
| `Effect.cached` | 0 | | `Effect.cached` | 0 |
| `Layer.succeed` | 19 |
| `Map<` | 82 | | `Map<` | 82 |
| `WebSocket` | 62 | | `WebSocket` | 62 |
| `new Map` | 60 | | `new Map` | 60 |
@ -105,6 +106,14 @@ Notes:
`Effect.acquireRelease` and disconnect them on scope close. The `Effect.acquireRelease` and disconnect them on scope close. The
`Effect.runPromise` count increased by two because the new lifecycle tests `Effect.runPromise` count increased by two because the new lifecycle tests
run scoped programs at the test boundary. run scoped programs at the test boundary.
- The Qdrant config/schema/fakeability slice removed direct production
`new QdrantClient`, sync config loading, payload casts, and Qdrant
`Layer.succeed` service construction from graph/doc store/query modules.
The installed Qdrant client exposes no public close/disconnect method, so
this remains a fakeable construction and Schema decode slice rather than a
scoped finalizer slice. `Effect.runPromise` increased because the new tests
and legacy service initialization logs run Effects at compatibility
boundaries.
- `Record<string, any>` and `throwLibrarianServiceError` are now clean in - `Record<string, any>` and `throwLibrarianServiceError` are now clean in
`ts/packages`. `ts/packages`.
@ -785,11 +794,6 @@ Notes:
instead of record/string type assertions. instead of record/string type assertions.
- New lifecycle tests use fake clients/graphs to prove connect on acquire - New lifecycle tests use fake clients/graphs to prove connect on acquire
and disconnect on scope close for both triples store and triples query. and disconnect on scope close for both triples store and triples query.
- Remaining:
- Qdrant graph/doc store/query construction still needs the next
config/schema/fakeability cleanup. The installed Qdrant client exposes no
close/disconnect method, so this should not be treated as a lifecycle
finalizer slice.
- Verification: - Verification:
- `bunx --bun vitest run src/__tests__/falkordb-lifecycle.test.ts` - `bunx --bun vitest run src/__tests__/falkordb-lifecycle.test.ts`
- `bun run --cwd ts/packages/flow build` - `bun run --cwd ts/packages/flow build`
@ -799,6 +803,35 @@ Notes:
- `cd ts && bun run test` - `cd ts && bun run test`
- `git diff --check` - `git diff --check`
### 2026-06-02: Qdrant Config, Schema, And Fakeable Construction Slice
- Status: migrated and root-verified.
- Completed:
- Added `ts/packages/flow/src/qdrant/client.ts` as the narrow fakeable
Qdrant surface used by graph/doc embedding store/query modules.
- Graph and document Qdrant store/query constructors now create clients
through `Effect.try`, load Qdrant config in Effect, and map config/client
failures into their existing `S.TaggedErrorClass` errors.
- Graph and document query payload extraction now uses
`Schema.decodeUnknownEffect(...).pipe(Effect.option)` and skips malformed
Qdrant payloads without type assertions.
- Qdrant graph/doc query Live layers and graph store Live layer now use
`Layer.effect` instead of preconstructing services with `Layer.succeed`.
- Legacy graph store/query/doc query processor providers now acquire Qdrant
services with named `Effect.fn` providers and map startup failures to
`ProcessorLifecycleError`.
- The installed Qdrant client still has no public close/disconnect method,
so no `Effect.acquireRelease` finalizer was added for Qdrant.
- Verification:
- `bunx --bun vitest run src/__tests__/qdrant-embeddings.test.ts`
- `bun run --cwd ts/packages/flow build`
- `cd ts && bun run check:tsgo`
- `bun run --cwd ts/packages/flow test`
- `cd ts && bun run check`
- `cd ts && bun run build`
- `cd ts && bun run test`
- `git diff --check`
## Subagent Findings To Preserve ## Subagent Findings To Preserve
- MCP/workbench: - MCP/workbench:
@ -854,36 +887,15 @@ Notes:
- FalkorDB scoped lifecycle is complete for triples query/store. Use the - FalkorDB scoped lifecycle is complete for triples query/store. Use the
fakeable client/graph factory pattern from that slice for future storage fakeable client/graph factory pattern from that slice for future storage
client tests. client tests.
- Qdrant has no close/disconnect surface in the installed client, so treat it - Qdrant config/schema/fakeability is complete for graph/doc embedding
as a config/schema/fakeability slice rather than an `acquireRelease` close store/query modules. Qdrant still has no close/disconnect surface in the
slice. installed client, so do not reopen it as an `acquireRelease` close slice
without new SDK evidence.
- Ollama/OpenAI-compatible/provider surfaces still need config, schema, and - Ollama/OpenAI-compatible/provider surfaces still need config, schema, and
provider-layer audits. provider-layer audits.
## Ranked Findings ## Ranked Findings
### P1: Qdrant Config, Schema, And Fakeable Construction Cleanup
- TrustGraph evidence:
- `ts/packages/flow/src/storage/embeddings/qdrant-graph.ts`
- `ts/packages/flow/src/storage/embeddings/qdrant-doc.ts`
- `ts/packages/flow/src/query/embeddings/qdrant-graph.ts`
- `ts/packages/flow/src/query/embeddings/qdrant-doc.ts`
- Effect primitives:
- `Config`, `ConfigProvider`, `Layer.effect`, `Schema.decodeUnknownEffect`,
`Predicate`, `Option`.
- Rewrite shape:
- Move Qdrant config loading out of sync factory construction and into
Effect config/layer paths.
- Add fakeable Qdrant client construction before behavior changes.
- Decode query payloads through Schema instead of manual payload casts.
- Do not add an `acquireRelease` finalizer unless a concrete close API is
found in the installed Qdrant client.
- Tests:
- Qdrant graph/doc store/query tests with fake clients.
- Config tests with `ConfigProvider.fromUnknown`.
- Schema decode failure tests for malformed payloads.
### P2: Provider Layer And Effect AI Cleanup ### P2: Provider Layer And Effect AI Cleanup
- TrustGraph evidence: - TrustGraph evidence:
@ -936,10 +948,9 @@ Notes:
## Recommended PR Order ## Recommended PR Order
1. Qdrant config/schema/fakeable construction cleanup. 1. Client streaming facade completion normalization.
2. Client streaming facade completion normalization. 2. Provider layer and Effect AI cleanup.
3. Provider layer and Effect AI cleanup. 3. MCP parity/deletion decision and workbench platform polish.
4. MCP parity/deletion decision and workbench platform polish.
## No-Op Rules ## No-Op Rules

View file

@ -0,0 +1,206 @@
import { Effect } from "effect";
import { describe, expect, it } from "vitest";
import {
QdrantDocEmbeddingsQueryLive,
QdrantDocEmbeddingsQueryService,
} from "../query/embeddings/qdrant-doc.js";
import {
QdrantGraphEmbeddingsQueryLive,
QdrantGraphEmbeddingsQueryService,
} from "../query/embeddings/qdrant-graph.js";
import type { QdrantClientLike, QdrantScoredPoint } from "../qdrant/client.js";
import { makeQdrantDocEmbeddingsStore } from "../storage/embeddings/qdrant-doc.js";
import {
QdrantGraphEmbeddingsStoreLive,
QdrantGraphEmbeddingsStoreService,
} from "../storage/embeddings/qdrant-graph.js";
import type { Term } from "@trustgraph/base";
interface FakePoint {
readonly id: string;
readonly vector: ReadonlyArray<number>;
readonly payload?: Record<string, unknown>;
}
class FakeQdrantClient implements QdrantClientLike {
readonly collections = new Set<string>();
readonly createdCollections: Array<{ readonly name: string; readonly size: number }> = [];
readonly upserts: Array<{
readonly collectionName: string;
readonly points: ReadonlyArray<FakePoint>;
}> = [];
readonly deletedCollections: string[] = [];
searchResults: ReadonlyArray<QdrantScoredPoint> = [];
async collectionExists(collectionName: string): Promise<{ readonly exists: boolean }> {
return { exists: this.collections.has(collectionName) };
}
async createCollection(
collectionName: string,
options: { readonly vectors: { readonly size: number; readonly distance: "Cosine" } },
): Promise<void> {
this.collections.add(collectionName);
this.createdCollections.push({ name: collectionName, size: options.vectors.size });
}
async upsert(
collectionName: string,
options: { readonly points: ReadonlyArray<FakePoint> },
): Promise<void> {
this.upserts.push({ collectionName, points: options.points });
}
async getCollections(): Promise<{ readonly collections: ReadonlyArray<{ readonly name: string }> }> {
return { collections: Array.from(this.collections, (name) => ({ name })) };
}
async deleteCollection(collectionName: string): Promise<void> {
this.collections.delete(collectionName);
this.deletedCollections.push(collectionName);
}
async search(
_collectionName: string,
_options: {
readonly vector: ReadonlyArray<number>;
readonly limit: number;
readonly with_payload: boolean;
},
): Promise<ReadonlyArray<QdrantScoredPoint>> {
return this.searchResults;
}
}
describe("Qdrant embeddings", () => {
it("queries graph payloads through Schema and skips malformed points", async () => {
const client = new FakeQdrantClient();
client.collections.add("t_alice_demo_2");
client.searchResults = [
{ score: 0.9, payload: { entity: "https://example.com/entity" } },
{ score: 0.8, payload: { entity: 123 } },
{ score: 0.7, payload: { entity: "" } },
{ score: 0.6, payload: { entity: "plain entity" } },
];
const matches = await Effect.runPromise(
Effect.gen(function* () {
const query = yield* QdrantGraphEmbeddingsQueryService;
return yield* query.query({
vector: [0.1, 0.2],
user: "alice",
collection: "demo",
limit: 10,
});
}).pipe(
Effect.provide(
QdrantGraphEmbeddingsQueryLive({
url: "http://qdrant.test",
clientFactory: () => client,
}),
),
),
);
expect(matches).toEqual([
{
entity: { type: "IRI", iri: "https://example.com/entity" },
score: 0.9,
},
{
entity: { type: "LITERAL", value: "plain entity" },
score: 0.6,
},
]);
});
it("queries document payloads through Schema and skips malformed points", async () => {
const client = new FakeQdrantClient();
client.collections.add("d_alice_docs_2");
client.searchResults = [
{ score: 0.9, payload: { chunk_id: "chunk-a", content: "alpha" } },
{ score: 0.8, payload: { chunk_id: 123, content: "bad" } },
{ score: 0.7, payload: { chunk_id: "" } },
{ score: 0.6, payload: { chunk_id: "chunk-b" } },
];
const matches = await Effect.runPromise(
Effect.gen(function* () {
const query = yield* QdrantDocEmbeddingsQueryService;
return yield* query.query({
vector: [0.1, 0.2],
user: "alice",
collection: "docs",
limit: 10,
});
}).pipe(
Effect.provide(
QdrantDocEmbeddingsQueryLive({
url: "http://qdrant.test",
clientFactory: () => client,
}),
),
),
);
expect(matches).toEqual([
{ chunkId: "chunk-a", score: 0.9, content: "alpha" },
{ chunkId: "chunk-b", score: 0.6 },
]);
});
it("uses an injected graph store client for collection creation and upsert", async () => {
const client = new FakeQdrantClient();
const entity: Term = { type: "IRI", iri: "https://example.com/entity" };
await Effect.runPromise(
Effect.gen(function* () {
const store = yield* QdrantGraphEmbeddingsStoreService;
yield* store.store({
user: "alice",
collection: "graph",
entities: [{ entity, vector: [1, 2, 3], chunkId: "chunk-a" }],
});
}).pipe(
Effect.provide(
QdrantGraphEmbeddingsStoreLive({
url: "http://qdrant.test",
clientFactory: () => client,
}),
),
),
);
expect(client.createdCollections).toEqual([{ name: "t_alice_graph_3", size: 3 }]);
expect(client.upserts).toHaveLength(1);
expect(client.upserts[0]?.collectionName).toBe("t_alice_graph_3");
expect(client.upserts[0]?.points[0]?.payload).toEqual({
entity: "https://example.com/entity",
chunk_id: "chunk-a",
});
});
it("uses an injected document store client for collection creation and upsert", async () => {
const client = new FakeQdrantClient();
const store = makeQdrantDocEmbeddingsStore({
url: "http://qdrant.test",
clientFactory: () => client,
});
await Effect.runPromise(
store.storeEffect({
user: "alice",
collection: "docs",
chunks: [{ chunkId: "chunk-a", vector: [1, 2], content: "alpha" }],
}),
);
expect(client.createdCollections).toEqual([{ name: "d_alice_docs_2", size: 2 }]);
expect(client.upserts).toHaveLength(1);
expect(client.upserts[0]?.collectionName).toBe("d_alice_docs_2");
expect(client.upserts[0]?.points[0]?.payload).toEqual({
chunk_id: "chunk-a",
content: "alpha",
});
});
});

View file

@ -0,0 +1,84 @@
import { QdrantClient, type QdrantClientParams } from "@qdrant/js-client-rest";
export interface QdrantCollectionStatus {
readonly exists: boolean;
}
export interface QdrantCollectionDescription {
readonly name: string;
}
export interface QdrantCollections {
readonly collections: ReadonlyArray<QdrantCollectionDescription>;
}
export interface QdrantScoredPoint {
readonly score: number;
readonly payload?: unknown;
}
export interface QdrantClientLike {
readonly collectionExists: (collectionName: string) => Promise<QdrantCollectionStatus>;
readonly createCollection: (
collectionName: string,
options: {
readonly vectors: {
readonly size: number;
readonly distance: "Cosine";
};
},
) => Promise<unknown>;
readonly upsert: (
collectionName: string,
options: {
readonly points: ReadonlyArray<{
readonly id: string;
readonly vector: ReadonlyArray<number>;
readonly payload?: Record<string, unknown>;
}>;
},
) => Promise<unknown>;
readonly getCollections: () => Promise<QdrantCollections>;
readonly deleteCollection: (collectionName: string) => Promise<unknown>;
readonly search: (
collectionName: string,
options: {
readonly vector: ReadonlyArray<number>;
readonly limit: number;
readonly with_payload: boolean;
},
) => Promise<ReadonlyArray<QdrantScoredPoint>>;
}
export type QdrantClientFactory = (params: QdrantClientParams) => QdrantClientLike;
export const makeQdrantClient = (
factory: QdrantClientFactory | undefined,
params: QdrantClientParams,
): QdrantClientLike => {
if (factory !== undefined) {
return factory(params);
}
const client = new QdrantClient(params);
return {
collectionExists: (collectionName) => client.collectionExists(collectionName),
createCollection: (collectionName, options) => client.createCollection(collectionName, options),
upsert: (collectionName, options) =>
client.upsert(collectionName, {
points: options.points.map((point) => ({
id: point.id,
vector: Array.from(point.vector),
...(point.payload !== undefined ? { payload: point.payload } : {}),
})),
}),
getCollections: () => client.getCollections(),
deleteCollection: (collectionName) => client.deleteCollection(collectionName),
search: (collectionName, options) =>
client.search(collectionName, {
vector: Array.from(options.vector),
limit: options.limit,
with_payload: options.with_payload,
}),
};
};

View file

@ -11,8 +11,10 @@ import {
makeFlowProcessor, makeFlowProcessor,
makeConsumerSpec, makeConsumerSpec,
makeProducerSpec, makeProducerSpec,
processorLifecycleError,
type ProcessorConfig, type ProcessorConfig,
type FlowProcessorRuntime, type FlowProcessorRuntime,
type FlowProcessorStartEffect,
type FlowContext, type FlowContext,
type FlowResourceNotFoundError, type FlowResourceNotFoundError,
type MessagingDeliveryError, type MessagingDeliveryError,
@ -26,8 +28,9 @@ import { Effect, Layer, ManagedRuntime } from "effect";
import { import {
QdrantDocEmbeddingsQueryLive, QdrantDocEmbeddingsQueryLive,
QdrantDocEmbeddingsQueryService, QdrantDocEmbeddingsQueryService,
makeQdrantDocEmbeddingsQueryService, makeQdrantDocEmbeddingsQueryServiceEffect,
type QdrantDocQueryConfig, type QdrantDocQueryConfig,
type QdrantDocEmbeddingsQueryError,
} from "./qdrant-doc.js"; } from "./qdrant-doc.js";
const DocumentEmbeddingsResponseProducer = makeProducerSpec<DocumentEmbeddingsResponse>("document-embeddings-response"); const DocumentEmbeddingsResponseProducer = makeProducerSpec<DocumentEmbeddingsResponse>("document-embeddings-response");
@ -92,25 +95,37 @@ export const makeDocEmbeddingsQuerySpecs = (): ReadonlyArray<Spec<QdrantDocEmbed
export type DocEmbeddingsQueryService = FlowProcessorRuntime<QdrantDocEmbeddingsQueryService>; export type DocEmbeddingsQueryService = FlowProcessorRuntime<QdrantDocEmbeddingsQueryService>;
const provideQdrantDocEmbeddingsQuery = (processorId: string) =>
Effect.fn("DocEmbeddingsQueryService.provideQdrant")(function* (
effect: FlowProcessorStartEffect<QdrantDocEmbeddingsQueryService>,
) {
const query = yield* makeQdrantDocEmbeddingsQueryServiceEffect().pipe(
Effect.mapError((error) => processorLifecycleError(processorId, "qdrant-doc-query-connect", error)),
);
yield* effect.pipe(
Effect.provideService(
QdrantDocEmbeddingsQueryService,
QdrantDocEmbeddingsQueryService.of(query),
),
);
});
export function makeDocEmbeddingsQueryService(config: ProcessorConfig): DocEmbeddingsQueryService { export function makeDocEmbeddingsQueryService(config: ProcessorConfig): DocEmbeddingsQueryService {
const query = makeQdrantDocEmbeddingsQueryService();
const service = makeFlowProcessor(config, { const service = makeFlowProcessor(config, {
specifications: makeDocEmbeddingsQuerySpecs(), specifications: makeDocEmbeddingsQuerySpecs(),
provide: (effect) => provide: provideQdrantDocEmbeddingsQuery(config.id),
effect.pipe(
Effect.provideService(
QdrantDocEmbeddingsQueryService,
QdrantDocEmbeddingsQueryService.of(query),
),
),
}); });
Effect.runSync(Effect.log("[DocEmbeddingsQuery] Service initialized")); void Effect.runPromise(Effect.log("[DocEmbeddingsQuery] Service initialized"));
return service; return service;
} }
export const DocEmbeddingsQueryService = makeDocEmbeddingsQueryService; export const DocEmbeddingsQueryService = makeDocEmbeddingsQueryService;
export const program = makeFlowProcessorProgram<ProcessorConfig & QdrantDocQueryConfig, never, QdrantDocEmbeddingsQueryService>({ export const program = makeFlowProcessorProgram<
ProcessorConfig & QdrantDocQueryConfig,
QdrantDocEmbeddingsQueryError,
QdrantDocEmbeddingsQueryService
>({
id: "doc-embeddings-query", id: "doc-embeddings-query",
specs: () => makeDocEmbeddingsQuerySpecs(), specs: () => makeDocEmbeddingsQuerySpecs(),
layer: (config) => QdrantDocEmbeddingsQueryLive(config), layer: (config) => QdrantDocEmbeddingsQueryLive(config),

View file

@ -7,15 +7,16 @@
* Python reference: trustgraph-flow/trustgraph/query/doc_embeddings/qdrant/service.py * Python reference: trustgraph-flow/trustgraph/query/doc_embeddings/qdrant/service.py
*/ */
import { QdrantClient } from "@qdrant/js-client-rest";
import { errorMessage } from "@trustgraph/base"; import { errorMessage } from "@trustgraph/base";
import { Config, Context, Effect, Layer } from "effect"; import { Config, Context, Effect, Layer } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { makeQdrantClient, type QdrantClientFactory, type QdrantClientLike } from "../../qdrant/client.js";
export interface QdrantDocQueryConfig { export interface QdrantDocQueryConfig {
url?: string; url?: string;
apiKey?: string; apiKey?: string;
clientFactory?: QdrantClientFactory;
} }
export interface ChunkMatch { export interface ChunkMatch {
@ -63,25 +64,37 @@ const loadQdrantDocQueryConfig = Effect.fn("QdrantDocEmbeddingsQuery.loadConfig"
} satisfies ResolvedQdrantDocQueryConfig; } satisfies ResolvedQdrantDocQueryConfig;
}); });
const DocPointPayloadSchema = S.Struct({
chunk_id: S.String,
content: S.optionalKey(S.String),
});
const decodeDocPointPayload = (payload: unknown) =>
S.decodeUnknownEffect(DocPointPayloadSchema)(payload).pipe(Effect.option);
export interface QdrantDocEmbeddingsQuery { export interface QdrantDocEmbeddingsQuery {
readonly query: (request: DocEmbeddingsQueryRequest) => Promise<ChunkMatch[]>; readonly query: (request: DocEmbeddingsQueryRequest) => Promise<ReadonlyArray<ChunkMatch>>;
readonly queryEffect: ( readonly queryEffect: (
request: DocEmbeddingsQueryRequest, request: DocEmbeddingsQueryRequest,
) => Effect.Effect<ReadonlyArray<ChunkMatch>, QdrantDocEmbeddingsQueryError>; ) => Effect.Effect<ReadonlyArray<ChunkMatch>, QdrantDocEmbeddingsQueryError>;
} }
export function makeQdrantDocEmbeddingsQuery( const makeQdrantDocEmbeddingsQueryClient = (
config: QdrantDocQueryConfig = {}, config: QdrantDocQueryConfig,
): QdrantDocEmbeddingsQuery { resolved: ResolvedQdrantDocQueryConfig,
const resolved = Effect.runSync(loadQdrantDocQueryConfig(config)); ) =>
Effect.try({
const client = new QdrantClient({ try: () =>
url: resolved.url, makeQdrantClient(config.clientFactory, {
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}), url: resolved.url,
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}),
}),
catch: (cause) => qdrantDocEmbeddingsQueryError("create-client", cause),
}); });
Effect.runSync(Effect.log("[QdrantDocQuery] Query service initialized")); const makeQdrantDocEmbeddingsQueryFromClient = (
client: QdrantClientLike,
): QdrantDocEmbeddingsQueryServiceShape => {
const queryEffect = Effect.fn("QdrantDocEmbeddingsQuery.query")(function* (request: DocEmbeddingsQueryRequest) { const queryEffect = Effect.fn("QdrantDocEmbeddingsQuery.query")(function* (request: DocEmbeddingsQueryRequest) {
const { vector, user, collection, limit } = request; const { vector, user, collection, limit } = request;
@ -116,20 +129,52 @@ export function makeQdrantDocEmbeddingsQuery(
const chunks: ChunkMatch[] = []; const chunks: ChunkMatch[] = [];
for (const point of searchResult) { for (const point of searchResult) {
const payload = point.payload as Record<string, unknown> | undefined; const payload = yield* decodeDocPointPayload(point.payload);
const chunkId = payload?.chunk_id as string | undefined; if (O.isNone(payload)) continue;
if (chunkId !== undefined && chunkId.length > 0) {
chunks.push({ const chunkId = payload.value.chunk_id;
chunkId, if (chunkId.length === 0) continue;
score: point.score,
...(typeof payload?.content === "string" ? { content: payload.content } : {}), chunks.push({
}); chunkId,
} score: point.score,
...(payload.value.content !== undefined ? { content: payload.value.content } : {}),
});
} }
return chunks; return chunks;
}); });
return {
query: queryEffect,
};
};
export const makeQdrantDocEmbeddingsQueryServiceEffect = Effect.fn(
"makeQdrantDocEmbeddingsQueryServiceEffect",
)(function* (config: QdrantDocQueryConfig = {}) {
const resolved = yield* loadQdrantDocQueryConfig(config).pipe(
Effect.mapError((cause) => qdrantDocEmbeddingsQueryError("load-config", cause)),
);
const client = yield* makeQdrantDocEmbeddingsQueryClient(config, resolved);
yield* Effect.log("[QdrantDocQuery] Query service initialized");
return makeQdrantDocEmbeddingsQueryFromClient(client);
});
const withQdrantDocEmbeddingsQuery = <A>(
config: QdrantDocQueryConfig,
use: (query: QdrantDocEmbeddingsQueryServiceShape) => Effect.Effect<A, QdrantDocEmbeddingsQueryError>,
) =>
makeQdrantDocEmbeddingsQueryServiceEffect(config).pipe(
Effect.flatMap(use),
);
export function makeQdrantDocEmbeddingsQuery(
config: QdrantDocQueryConfig = {},
): QdrantDocEmbeddingsQuery {
const queryEffect = (request: DocEmbeddingsQueryRequest) =>
withQdrantDocEmbeddingsQuery(config, (query) => query.query(request));
return { return {
query: (request) => Effect.runPromise(queryEffect(request)), query: (request) => Effect.runPromise(queryEffect(request)),
queryEffect, queryEffect,
@ -151,17 +196,16 @@ export class QdrantDocEmbeddingsQueryService extends Context.Service<
export const makeQdrantDocEmbeddingsQueryService = ( export const makeQdrantDocEmbeddingsQueryService = (
config: QdrantDocQueryConfig = {}, config: QdrantDocQueryConfig = {},
): QdrantDocEmbeddingsQueryServiceShape => { ): QdrantDocEmbeddingsQueryServiceShape => ({
const query = makeQdrantDocEmbeddingsQuery(config); query: (request) => withQdrantDocEmbeddingsQuery(config, (query) => query.query(request)),
return { });
query: query.queryEffect,
};
};
export const QdrantDocEmbeddingsQueryLive = ( export const QdrantDocEmbeddingsQueryLive = (
config: QdrantDocQueryConfig = {}, config: QdrantDocQueryConfig = {},
): Layer.Layer<QdrantDocEmbeddingsQueryService> => ): Layer.Layer<QdrantDocEmbeddingsQueryService, QdrantDocEmbeddingsQueryError> =>
Layer.succeed( Layer.effect(
QdrantDocEmbeddingsQueryService, QdrantDocEmbeddingsQueryService,
QdrantDocEmbeddingsQueryService.of(makeQdrantDocEmbeddingsQueryService(config)), makeQdrantDocEmbeddingsQueryServiceEffect(config).pipe(
Effect.map((service) => QdrantDocEmbeddingsQueryService.of(service)),
),
); );

View file

@ -11,8 +11,10 @@ import {
makeFlowProcessor, makeFlowProcessor,
makeConsumerSpec, makeConsumerSpec,
makeProducerSpec, makeProducerSpec,
processorLifecycleError,
type ProcessorConfig, type ProcessorConfig,
type FlowProcessorRuntime, type FlowProcessorRuntime,
type FlowProcessorStartEffect,
type FlowContext, type FlowContext,
type FlowResourceNotFoundError, type FlowResourceNotFoundError,
type MessagingDeliveryError, type MessagingDeliveryError,
@ -26,8 +28,9 @@ import { Effect, Layer, ManagedRuntime } from "effect";
import { import {
QdrantGraphEmbeddingsQueryLive, QdrantGraphEmbeddingsQueryLive,
QdrantGraphEmbeddingsQueryService, QdrantGraphEmbeddingsQueryService,
makeQdrantGraphEmbeddingsQueryService, makeQdrantGraphEmbeddingsQueryServiceEffect,
type QdrantGraphQueryConfig, type QdrantGraphQueryConfig,
type QdrantGraphEmbeddingsQueryError,
} from "./qdrant-graph.js"; } from "./qdrant-graph.js";
const GraphEmbeddingsResponseProducer = makeProducerSpec<GraphEmbeddingsResponse>("graph-embeddings-response"); const GraphEmbeddingsResponseProducer = makeProducerSpec<GraphEmbeddingsResponse>("graph-embeddings-response");
@ -93,25 +96,37 @@ export const makeGraphEmbeddingsQuerySpecs = (): ReadonlyArray<Spec<QdrantGraphE
export type GraphEmbeddingsQueryService = FlowProcessorRuntime<QdrantGraphEmbeddingsQueryService>; export type GraphEmbeddingsQueryService = FlowProcessorRuntime<QdrantGraphEmbeddingsQueryService>;
const provideQdrantGraphEmbeddingsQuery = (processorId: string) =>
Effect.fn("GraphEmbeddingsQueryService.provideQdrant")(function* (
effect: FlowProcessorStartEffect<QdrantGraphEmbeddingsQueryService>,
) {
const query = yield* makeQdrantGraphEmbeddingsQueryServiceEffect().pipe(
Effect.mapError((error) => processorLifecycleError(processorId, "qdrant-graph-query-connect", error)),
);
yield* effect.pipe(
Effect.provideService(
QdrantGraphEmbeddingsQueryService,
QdrantGraphEmbeddingsQueryService.of(query),
),
);
});
export function makeGraphEmbeddingsQueryService(config: ProcessorConfig): GraphEmbeddingsQueryService { export function makeGraphEmbeddingsQueryService(config: ProcessorConfig): GraphEmbeddingsQueryService {
const query = makeQdrantGraphEmbeddingsQueryService();
const service = makeFlowProcessor(config, { const service = makeFlowProcessor(config, {
specifications: makeGraphEmbeddingsQuerySpecs(), specifications: makeGraphEmbeddingsQuerySpecs(),
provide: (effect) => provide: provideQdrantGraphEmbeddingsQuery(config.id),
effect.pipe(
Effect.provideService(
QdrantGraphEmbeddingsQueryService,
QdrantGraphEmbeddingsQueryService.of(query),
),
),
}); });
Effect.runSync(Effect.log("[GraphEmbeddingsQuery] Service initialized")); void Effect.runPromise(Effect.log("[GraphEmbeddingsQuery] Service initialized"));
return service; return service;
} }
export const GraphEmbeddingsQueryService = makeGraphEmbeddingsQueryService; export const GraphEmbeddingsQueryService = makeGraphEmbeddingsQueryService;
export const program = makeFlowProcessorProgram<ProcessorConfig & QdrantGraphQueryConfig, never, QdrantGraphEmbeddingsQueryService>({ export const program = makeFlowProcessorProgram<
ProcessorConfig & QdrantGraphQueryConfig,
QdrantGraphEmbeddingsQueryError,
QdrantGraphEmbeddingsQueryService
>({
id: "graph-embeddings-query", id: "graph-embeddings-query",
specs: () => makeGraphEmbeddingsQuerySpecs(), specs: () => makeGraphEmbeddingsQuerySpecs(),
layer: (config) => QdrantGraphEmbeddingsQueryLive(config), layer: (config) => QdrantGraphEmbeddingsQueryLive(config),

View file

@ -10,15 +10,16 @@
* Python reference: trustgraph-flow/trustgraph/query/graph_embeddings/qdrant/service.py * Python reference: trustgraph-flow/trustgraph/query/graph_embeddings/qdrant/service.py
*/ */
import { QdrantClient } from "@qdrant/js-client-rest";
import { errorMessage, type Term } from "@trustgraph/base"; import { errorMessage, type Term } from "@trustgraph/base";
import { Config, Context, Effect, Layer } from "effect"; import { Config, Context, Effect, Layer } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { makeQdrantClient, type QdrantClientFactory, type QdrantClientLike } from "../../qdrant/client.js";
export interface QdrantGraphQueryConfig { export interface QdrantGraphQueryConfig {
url?: string; url?: string;
apiKey?: string; apiKey?: string;
clientFactory?: QdrantClientFactory;
} }
export interface EntityMatch { export interface EntityMatch {
@ -72,24 +73,36 @@ function createTerm(value: string): Term {
return { type: "LITERAL", value }; return { type: "LITERAL", value };
} }
const GraphPointPayloadSchema = S.Struct({
entity: S.String,
});
const decodeGraphPointPayload = (payload: unknown) =>
S.decodeUnknownEffect(GraphPointPayloadSchema)(payload).pipe(Effect.option);
export interface QdrantGraphEmbeddingsQuery { export interface QdrantGraphEmbeddingsQuery {
readonly query: (request: GraphEmbeddingsQueryRequest) => Promise<EntityMatch[]>; readonly query: (request: GraphEmbeddingsQueryRequest) => Promise<ReadonlyArray<EntityMatch>>;
readonly queryEffect: ( readonly queryEffect: (
request: GraphEmbeddingsQueryRequest, request: GraphEmbeddingsQueryRequest,
) => Effect.Effect<ReadonlyArray<EntityMatch>, QdrantGraphEmbeddingsQueryError>; ) => Effect.Effect<ReadonlyArray<EntityMatch>, QdrantGraphEmbeddingsQueryError>;
} }
export function makeQdrantGraphEmbeddingsQuery( const makeQdrantGraphEmbeddingsQueryClient = (
config: QdrantGraphQueryConfig = {}, config: QdrantGraphQueryConfig,
): QdrantGraphEmbeddingsQuery { resolved: ResolvedQdrantGraphQueryConfig,
const resolved = Effect.runSync(loadQdrantGraphQueryConfig(config)); ) =>
Effect.try({
const client = new QdrantClient({ try: () =>
url: resolved.url, makeQdrantClient(config.clientFactory, {
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}), url: resolved.url,
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}),
}),
catch: (cause) => qdrantGraphEmbeddingsQueryError("create-client", cause),
}); });
Effect.runSync(Effect.log("[QdrantGraphQuery] Query service initialized")); const makeQdrantGraphEmbeddingsQueryFromClient = (
client: QdrantClientLike,
): QdrantGraphEmbeddingsQueryServiceShape => {
const queryEffect = Effect.fn("QdrantGraphEmbeddingsQuery.query")(function* ( const queryEffect = Effect.fn("QdrantGraphEmbeddingsQuery.query")(function* (
request: GraphEmbeddingsQueryRequest, request: GraphEmbeddingsQueryRequest,
@ -131,8 +144,10 @@ export function makeQdrantGraphEmbeddingsQuery(
const entities: EntityMatch[] = []; const entities: EntityMatch[] = [];
for (const point of searchResult) { for (const point of searchResult) {
const payload = point.payload as Record<string, unknown> | undefined; const payload = yield* decodeGraphPointPayload(point.payload);
const entityValue = payload?.entity as string | undefined; if (O.isNone(payload)) continue;
const entityValue = payload.value.entity;
if (entityValue === undefined || entityValue.length === 0) continue; if (entityValue === undefined || entityValue.length === 0) continue;
// Deduplicate by entity value, keeping the highest score (results are // Deduplicate by entity value, keeping the highest score (results are
@ -152,6 +167,36 @@ export function makeQdrantGraphEmbeddingsQuery(
return entities; return entities;
}); });
return {
query: queryEffect,
};
};
export const makeQdrantGraphEmbeddingsQueryServiceEffect = Effect.fn(
"makeQdrantGraphEmbeddingsQueryServiceEffect",
)(function* (config: QdrantGraphQueryConfig = {}) {
const resolved = yield* loadQdrantGraphQueryConfig(config).pipe(
Effect.mapError((cause) => qdrantGraphEmbeddingsQueryError("load-config", cause)),
);
const client = yield* makeQdrantGraphEmbeddingsQueryClient(config, resolved);
yield* Effect.log("[QdrantGraphQuery] Query service initialized");
return makeQdrantGraphEmbeddingsQueryFromClient(client);
});
const withQdrantGraphEmbeddingsQuery = <A>(
config: QdrantGraphQueryConfig,
use: (query: QdrantGraphEmbeddingsQueryServiceShape) => Effect.Effect<A, QdrantGraphEmbeddingsQueryError>,
) =>
makeQdrantGraphEmbeddingsQueryServiceEffect(config).pipe(
Effect.flatMap(use),
);
export function makeQdrantGraphEmbeddingsQuery(
config: QdrantGraphQueryConfig = {},
): QdrantGraphEmbeddingsQuery {
const queryEffect = (request: GraphEmbeddingsQueryRequest) =>
withQdrantGraphEmbeddingsQuery(config, (query) => query.query(request));
return { return {
query: (request) => Effect.runPromise(queryEffect(request)), query: (request) => Effect.runPromise(queryEffect(request)),
queryEffect, queryEffect,
@ -173,17 +218,16 @@ export class QdrantGraphEmbeddingsQueryService extends Context.Service<
export const makeQdrantGraphEmbeddingsQueryService = ( export const makeQdrantGraphEmbeddingsQueryService = (
config: QdrantGraphQueryConfig = {}, config: QdrantGraphQueryConfig = {},
): QdrantGraphEmbeddingsQueryServiceShape => { ): QdrantGraphEmbeddingsQueryServiceShape => ({
const query = makeQdrantGraphEmbeddingsQuery(config); query: (request) => withQdrantGraphEmbeddingsQuery(config, (query) => query.query(request)),
return { });
query: query.queryEffect,
};
};
export const QdrantGraphEmbeddingsQueryLive = ( export const QdrantGraphEmbeddingsQueryLive = (
config: QdrantGraphQueryConfig = {}, config: QdrantGraphQueryConfig = {},
): Layer.Layer<QdrantGraphEmbeddingsQueryService> => ): Layer.Layer<QdrantGraphEmbeddingsQueryService, QdrantGraphEmbeddingsQueryError> =>
Layer.succeed( Layer.effect(
QdrantGraphEmbeddingsQueryService, QdrantGraphEmbeddingsQueryService,
QdrantGraphEmbeddingsQueryService.of(makeQdrantGraphEmbeddingsQueryService(config)), makeQdrantGraphEmbeddingsQueryServiceEffect(config).pipe(
Effect.map((service) => QdrantGraphEmbeddingsQueryService.of(service)),
),
); );

View file

@ -13,8 +13,10 @@ import {
makeFlowProcessor, makeFlowProcessor,
makeConsumerSpec, makeConsumerSpec,
makeRequestResponseSpec, makeRequestResponseSpec,
processorLifecycleError,
type ProcessorConfig, type ProcessorConfig,
type FlowProcessorRuntime, type FlowProcessorRuntime,
type FlowProcessorStartEffect,
type FlowContext, type FlowContext,
type FlowResourceNotFoundError, type FlowResourceNotFoundError,
type MessagingDeliveryError, type MessagingDeliveryError,
@ -30,7 +32,7 @@ import { Effect, Layer, ManagedRuntime } from "effect";
import { import {
QdrantGraphEmbeddingsStoreLive, QdrantGraphEmbeddingsStoreLive,
QdrantGraphEmbeddingsStoreService, QdrantGraphEmbeddingsStoreService,
makeQdrantGraphEmbeddingsStoreService, makeQdrantGraphEmbeddingsStoreServiceEffect,
type QdrantGraphEmbeddingsConfig, type QdrantGraphEmbeddingsConfig,
type QdrantGraphEmbeddingsStoreError, type QdrantGraphEmbeddingsStoreError,
} from "./qdrant-graph.js"; } from "./qdrant-graph.js";
@ -93,19 +95,27 @@ export const makeGraphEmbeddingsStoreSpecs = (): ReadonlyArray<Spec<GraphEmbeddi
export type GraphEmbeddingsStoreService = FlowProcessorRuntime<GraphEmbeddingsStoreRequirements>; export type GraphEmbeddingsStoreService = FlowProcessorRuntime<GraphEmbeddingsStoreRequirements>;
const provideQdrantGraphEmbeddingsStore = (processorId: string) =>
Effect.fn("GraphEmbeddingsStoreService.provideQdrant")(function* (
effect: FlowProcessorStartEffect<GraphEmbeddingsStoreRequirements>,
) {
const store = yield* makeQdrantGraphEmbeddingsStoreServiceEffect().pipe(
Effect.mapError((error) => processorLifecycleError(processorId, "qdrant-graph-store-connect", error)),
);
yield* effect.pipe(
Effect.provideService(
QdrantGraphEmbeddingsStoreService,
QdrantGraphEmbeddingsStoreService.of(store),
),
);
});
export function makeGraphEmbeddingsStoreService(config: ProcessorConfig): GraphEmbeddingsStoreService { export function makeGraphEmbeddingsStoreService(config: ProcessorConfig): GraphEmbeddingsStoreService {
const store = makeQdrantGraphEmbeddingsStoreService();
const service = makeFlowProcessor(config, { const service = makeFlowProcessor(config, {
specifications: makeGraphEmbeddingsStoreSpecs(), specifications: makeGraphEmbeddingsStoreSpecs(),
provide: (effect) => provide: provideQdrantGraphEmbeddingsStore(config.id),
effect.pipe(
Effect.provideService(
QdrantGraphEmbeddingsStoreService,
QdrantGraphEmbeddingsStoreService.of(store),
),
),
}); });
Effect.runSync(Effect.log("[GraphEmbeddingsStore] Service initialized")); void Effect.runPromise(Effect.log("[GraphEmbeddingsStore] Service initialized"));
return service; return service;
} }
@ -113,7 +123,7 @@ export const GraphEmbeddingsStoreService = makeGraphEmbeddingsStoreService;
export const program = makeFlowProcessorProgram< export const program = makeFlowProcessorProgram<
ProcessorConfig & QdrantGraphEmbeddingsConfig, ProcessorConfig & QdrantGraphEmbeddingsConfig,
never, QdrantGraphEmbeddingsStoreError,
GraphEmbeddingsStoreRequirements GraphEmbeddingsStoreRequirements
>({ >({
id: "graph-embeddings-store", id: "graph-embeddings-store",

View file

@ -8,15 +8,16 @@
* Python reference: trustgraph-flow/trustgraph/storage/doc_embeddings/qdrant/write.py * Python reference: trustgraph-flow/trustgraph/storage/doc_embeddings/qdrant/write.py
*/ */
import { QdrantClient } from "@qdrant/js-client-rest";
import { errorMessage } from "@trustgraph/base"; import { errorMessage } from "@trustgraph/base";
import { Config, Effect, Random } from "effect"; import { Config, Effect, Random } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { makeQdrantClient, type QdrantClientFactory, type QdrantClientLike } from "../../qdrant/client.js";
export interface QdrantDocEmbeddingsConfig { export interface QdrantDocEmbeddingsConfig {
url?: string; url?: string;
apiKey?: string; apiKey?: string;
clientFactory?: QdrantClientFactory;
} }
export interface DocEmbeddingChunk { export interface DocEmbeddingChunk {
@ -94,18 +95,33 @@ export interface QdrantDocEmbeddingsStore {
) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>; ) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>;
} }
export function makeQdrantDocEmbeddingsStore( const makeQdrantDocEmbeddingsClient = (
config: QdrantDocEmbeddingsConfig = {}, config: QdrantDocEmbeddingsConfig,
): QdrantDocEmbeddingsStore { resolved: ResolvedQdrantDocEmbeddingsConfig,
const resolved = Effect.runSync(loadQdrantDocEmbeddingsConfig(config)); ) =>
Effect.try({
const client = new QdrantClient({ try: () =>
url: resolved.url, makeQdrantClient(config.clientFactory, {
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}), url: resolved.url,
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}),
}),
catch: (cause) => qdrantDocEmbeddingsStoreError("create-client", cause),
}); });
const knownCollections = new Set<string>();
Effect.runSync(Effect.log("[QdrantDocEmbeddings] Store initialized")); interface QdrantDocEmbeddingsStoreEffectShape {
readonly store: (
message: DocEmbeddingsMessage,
) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>;
readonly deleteCollection: (
user: string,
collection: string,
) => Effect.Effect<void, QdrantDocEmbeddingsStoreError>;
}
const makeQdrantDocEmbeddingsStoreFromClient = (
client: QdrantClientLike,
): QdrantDocEmbeddingsStoreEffectShape => {
const knownCollections = new Set<string>();
const collectionName = (user: string, collection: string, dim: number): string => const collectionName = (user: string, collection: string, dim: number): string =>
`d_${user}_${collection}_${dim}`; `d_${user}_${collection}_${dim}`;
@ -199,6 +215,39 @@ export function makeQdrantDocEmbeddingsStore(
); );
}); });
return {
store: storeEffect,
deleteCollection: deleteCollectionEffect,
};
};
const makeQdrantDocEmbeddingsStoreEffect = Effect.fn("makeQdrantDocEmbeddingsStoreEffect")(function* (
config: QdrantDocEmbeddingsConfig = {},
) {
const resolved = yield* loadQdrantDocEmbeddingsConfig(config).pipe(
Effect.mapError((cause) => qdrantDocEmbeddingsStoreError("load-config", cause)),
);
const client = yield* makeQdrantDocEmbeddingsClient(config, resolved);
yield* Effect.log("[QdrantDocEmbeddings] Store initialized");
return makeQdrantDocEmbeddingsStoreFromClient(client);
});
const withQdrantDocEmbeddingsStore = <A>(
config: QdrantDocEmbeddingsConfig,
use: (store: QdrantDocEmbeddingsStoreEffectShape) => Effect.Effect<A, QdrantDocEmbeddingsStoreError>,
) =>
makeQdrantDocEmbeddingsStoreEffect(config).pipe(
Effect.flatMap(use),
);
export function makeQdrantDocEmbeddingsStore(
config: QdrantDocEmbeddingsConfig = {},
): QdrantDocEmbeddingsStore {
const storeEffect = (message: DocEmbeddingsMessage) =>
withQdrantDocEmbeddingsStore(config, (store) => store.store(message));
const deleteCollectionEffect = (user: string, collection: string) =>
withQdrantDocEmbeddingsStore(config, (store) => store.deleteCollection(user, collection));
return { return {
store: (message) => Effect.runPromise(storeEffect(message)), store: (message) => Effect.runPromise(storeEffect(message)),
deleteCollection: (user, collection) => deleteCollection: (user, collection) =>

View file

@ -8,15 +8,16 @@
* Python reference: trustgraph-flow/trustgraph/storage/graph_embeddings/qdrant/write.py * Python reference: trustgraph-flow/trustgraph/storage/graph_embeddings/qdrant/write.py
*/ */
import { QdrantClient } from "@qdrant/js-client-rest";
import { errorMessage, type Term } from "@trustgraph/base"; import { errorMessage, type Term } from "@trustgraph/base";
import { Config, Context, Effect, Layer, Random } from "effect"; import { Config, Context, Effect, Layer, Random } from "effect";
import * as O from "effect/Option"; import * as O from "effect/Option";
import * as S from "effect/Schema"; import * as S from "effect/Schema";
import { makeQdrantClient, type QdrantClientFactory, type QdrantClientLike } from "../../qdrant/client.js";
export interface QdrantGraphEmbeddingsConfig { export interface QdrantGraphEmbeddingsConfig {
url?: string; url?: string;
apiKey?: string; apiKey?: string;
clientFactory?: QdrantClientFactory;
} }
export interface GraphEmbeddingEntity { export interface GraphEmbeddingEntity {
@ -107,18 +108,23 @@ export interface QdrantGraphEmbeddingsStore {
) => Effect.Effect<void, QdrantGraphEmbeddingsStoreError>; ) => Effect.Effect<void, QdrantGraphEmbeddingsStoreError>;
} }
export function makeQdrantGraphEmbeddingsStore( const makeQdrantGraphEmbeddingsClient = (
config: QdrantGraphEmbeddingsConfig = {}, config: QdrantGraphEmbeddingsConfig,
): QdrantGraphEmbeddingsStore { resolved: ResolvedQdrantGraphEmbeddingsConfig,
const resolved = Effect.runSync(loadQdrantGraphEmbeddingsConfig(config)); ) =>
Effect.try({
const client = new QdrantClient({ try: () =>
url: resolved.url, makeQdrantClient(config.clientFactory, {
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}), url: resolved.url,
...(resolved.apiKey !== undefined ? { apiKey: resolved.apiKey } : {}),
}),
catch: (cause) => qdrantGraphEmbeddingsStoreError("create-client", cause),
}); });
const knownCollections = new Set<string>();
Effect.runSync(Effect.log("[QdrantGraphEmbeddings] Store initialized")); const makeQdrantGraphEmbeddingsStoreFromClient = (
client: QdrantClientLike,
): QdrantGraphEmbeddingsStoreServiceShape => {
const knownCollections = new Set<string>();
const collectionName = (user: string, collection: string, dim: number): string => const collectionName = (user: string, collection: string, dim: number): string =>
`t_${user}_${collection}_${dim}`; `t_${user}_${collection}_${dim}`;
@ -213,6 +219,39 @@ export function makeQdrantGraphEmbeddingsStore(
); );
}); });
return {
store: storeEffect,
deleteCollection: deleteCollectionEffect,
};
};
export const makeQdrantGraphEmbeddingsStoreServiceEffect = Effect.fn(
"makeQdrantGraphEmbeddingsStoreServiceEffect",
)(function* (config: QdrantGraphEmbeddingsConfig = {}) {
const resolved = yield* loadQdrantGraphEmbeddingsConfig(config).pipe(
Effect.mapError((cause) => qdrantGraphEmbeddingsStoreError("load-config", cause)),
);
const client = yield* makeQdrantGraphEmbeddingsClient(config, resolved);
yield* Effect.log("[QdrantGraphEmbeddings] Store initialized");
return makeQdrantGraphEmbeddingsStoreFromClient(client);
});
const withQdrantGraphEmbeddingsStore = <A>(
config: QdrantGraphEmbeddingsConfig,
use: (store: QdrantGraphEmbeddingsStoreServiceShape) => Effect.Effect<A, QdrantGraphEmbeddingsStoreError>,
) =>
makeQdrantGraphEmbeddingsStoreServiceEffect(config).pipe(
Effect.flatMap(use),
);
export function makeQdrantGraphEmbeddingsStore(
config: QdrantGraphEmbeddingsConfig = {},
): QdrantGraphEmbeddingsStore {
const storeEffect = (message: GraphEmbeddingsMessage) =>
withQdrantGraphEmbeddingsStore(config, (store) => store.store(message));
const deleteCollectionEffect = (user: string, collection: string) =>
withQdrantGraphEmbeddingsStore(config, (store) => store.deleteCollection(user, collection));
return { return {
store: (message) => Effect.runPromise(storeEffect(message)), store: (message) => Effect.runPromise(storeEffect(message)),
deleteCollection: (user, collection) => deleteCollection: (user, collection) =>
@ -241,18 +280,18 @@ export class QdrantGraphEmbeddingsStoreService extends Context.Service<
export const makeQdrantGraphEmbeddingsStoreService = ( export const makeQdrantGraphEmbeddingsStoreService = (
config: QdrantGraphEmbeddingsConfig = {}, config: QdrantGraphEmbeddingsConfig = {},
): QdrantGraphEmbeddingsStoreServiceShape => { ): QdrantGraphEmbeddingsStoreServiceShape => ({
const store = makeQdrantGraphEmbeddingsStore(config); store: (message) => withQdrantGraphEmbeddingsStore(config, (store) => store.store(message)),
return { deleteCollection: (user, collection) =>
store: store.storeEffect, withQdrantGraphEmbeddingsStore(config, (store) => store.deleteCollection(user, collection)),
deleteCollection: store.deleteCollectionEffect, });
};
};
export const QdrantGraphEmbeddingsStoreLive = ( export const QdrantGraphEmbeddingsStoreLive = (
config: QdrantGraphEmbeddingsConfig = {}, config: QdrantGraphEmbeddingsConfig = {},
): Layer.Layer<QdrantGraphEmbeddingsStoreService> => ): Layer.Layer<QdrantGraphEmbeddingsStoreService, QdrantGraphEmbeddingsStoreError> =>
Layer.succeed( Layer.effect(
QdrantGraphEmbeddingsStoreService, QdrantGraphEmbeddingsStoreService,
QdrantGraphEmbeddingsStoreService.of(makeQdrantGraphEmbeddingsStoreService(config)), makeQdrantGraphEmbeddingsStoreServiceEffect(config).pipe(
Effect.map((service) => QdrantGraphEmbeddingsStoreService.of(service)),
),
); );