Commit graph

50 commits

Author SHA1 Message Date
elpresidank
f7f29c4df9 Handle gateway RPC socket causes in Effect 2026-06-02 04:42:32 -05:00
elpresidank
32fc7ea32d Share text completion stream helpers 2026-06-02 04:33:48 -05:00
elpresidank
9d3f745fb0 Make Ollama embeddings layer effectful 2026-06-02 04:24:14 -05:00
elpresidank
8287e1cf93 Add fakeable Qdrant Effect services 2026-06-02 04:10:03 -05:00
elpresidank
d38ce475fd Scope FalkorDB triples clients 2026-06-02 03:55:39 -05:00
elpresidank
ce5838db1d Add Effect-native gateway streaming path 2026-06-02 03:43:21 -05:00
elpresidank
44110c5bb4 Add typed flow spec accessors 2026-06-02 03:23:23 -05:00
elpresidank
abb6f3aed0 Add schema-backed parameter spec accessors 2026-06-02 03:10:43 -05:00
elpresidank
b51dc33786 Narrow provider status errors with Predicate 2026-06-02 03:00:52 -05:00
elpresidank
b3aaa9a4af Simplify text completion generator boundary 2026-06-02 02:58:19 -05:00
elpresidank
bfb5494552 Remove provider stream sentinel assertions 2026-06-02 02:55:06 -05:00
elpresidank
74955d6041 Use native runtimes for flow entrypoints 2026-06-02 02:34:03 -05:00
elpresidank
710656be26 Move librarian state into synchronized refs 2026-06-02 01:59:51 -05:00
elpresidank
5f783832e9 Type librarian service runtime loop 2026-06-02 01:47:15 -05:00
elpresidank
e6384e65b9 Replace librarian throw helpers with Effect failures 2026-06-02 01:39:04 -05:00
elpresidank
459347ad12 Model librarian upload schema boundaries 2026-06-02 01:34:46 -05:00
elpresidank
3809a38c46 Migrate flow manager to ref-backed Effect state 2026-06-02 01:26:40 -05:00
elpresidank
ba64fc5add Normalize flow service runtime entrypoints 2026-06-02 01:13:40 -05:00
elpresidank
0da0df81c4 Migrate knowledge core service to ref-backed Effect state 2026-06-02 01:07:12 -05:00
elpresidank
5979d38b99 Remove RAG requestor Promise bridges 2026-06-02 00:54:47 -05:00
elpresidank
88db18fbda Migrate config service to ref-backed Effect runtime 2026-06-02 00:40:44 -05:00
elpresidank
b4ee2b691f Migrate strict Effect runtime surfaces 2026-06-02 00:22:04 -05:00
elpresidank
f6878d4dd7 Enforce strict Effect tsgo migrations 2026-06-01 23:19:54 -05:00
elpresidank
64fb23e7d0 Make gateway dispatcher requestors Effect-scoped 2026-06-01 22:17:50 -05:00
elpresidank
b58e51bf75 Update Effect packages to beta 75 2026-06-01 21:16:38 -05:00
elpresidank
dca2786828 Remove native classes from TS runtime 2026-06-01 20:26:47 -05:00
elpresidank
952daf325d Stabilize TS workbench QA and RPC timeouts 2026-06-01 17:23:34 -05:00
elpresidank
3515106670 Advance TS port Effect workbench 2026-06-01 16:22:25 -05:00
elpresidank
5776171d27 saving 2026-05-30 09:56:23 -05:00
elpresidank
ffd97375a8 saving 2026-05-12 08:06:58 -05:00
elpresidank
6ac5446a76 feat(mcp-tool): wire McpToolService into deploy stack
Three pieces, all required for an end-to-end MCP tool call:

* McpToolService used generic spec names "request"/"response" instead of
  "mcp-tool-request"/"mcp-tool-response", so RequestResponseSpec's
  flow-config topic lookup never matched and consumers bound to literal
  subjects nobody else publishes to.

* Add entrypoints/mcp-tool.mjs (mirrors agent/librarian entrypoints) so
  the service can be launched in the prebuilt trustgraph-ts image.

* Add a `mcp-tool` service block to deploy/docker-compose.yml.

With these three fixes plus a `mcp-tool-request`/`mcp-tool-response`
entry in each flow's topics map, the agent ReAct loop can now invoke
remote MCP tools (verified end-to-end against Brave Search and FireCrawl).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:16:37 -05:00
elpresidank
ee45cb4850 feat: fix RAG pipelines, Beep Graph branding, PWA, and ambient glow UI
Pipeline fixes:
- Fix agent getting empty response from graph-rag by combining answer +
  explain data in single message (RequestResponse returns first msg)
- Fix Doc RAG pipeline: add content field to Qdrant doc payload, seed 10
  document chunks, fix type mismatches across base/flow/client
- Forward explainability events from agent's KnowledgeQuery to client
- Add "agent" to TERM_BEARING_RESPONSE_SERVICES for triple translation
- Fix embeddings env var (OLLAMA_URL), user/collection threading, edge
  scoring threshold, and various protocol mismatches

Branding:
- Rename TrustGraph → Beep Graph (title, sidebar, settings, about)
- Custom lambda + ThugLife pixel glasses SVG logo component
- Forest green color palette (brand-50 through brand-900)
- SVG favicon + PNG icons (16/32/180/192/512)
- PWA manifest with service worker for offline shell caching
- Splash screen with animated logo pulse on app load
- Ambient glow background with drifting green radial blobs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:19:10 -05:00
elpresidank
b854b56558 feat: MCP tool client infrastructure for agent extensibility
Add the full MCP tool pipeline enabling agents to invoke external tools
(like Brave Search) via MCP servers:

- Add ToolRequest/ToolResponse types and mcp-tool topics to @trustgraph/base
- Create McpToolService (FlowProcessor) that connects to external MCP servers
  via @modelcontextprotocol/sdk StreamableHTTP transport
- Add createMcpTool() to wire MCP tools into the agent's ReAct loop
- Implement config-driven tool registration in AgentService with backward-
  compatible fallback to hardcoded tools
- Add tool filtering by group and state (port of Python tool_filter.py)
- Register mcp-tool in gateway dispatcher and export from @trustgraph/flow
- Fix flow restart race condition: skip restart when flow definitions unchanged
- Update seed config with MCP server config and tool definitions
- Add run scripts for MCP tool service and Brave Search MCP server

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 05:45:46 -05:00
elpresidank
f2b376abef fix: FalkorDB result parsing, embeddings routing, triples query response, graph visualization
- Fix FalkorDB triples query: client v5 returns objects not arrays, use named field access
- Fix embeddings service: align spec names to "embeddings-request"/"embeddings-response"
- Fix client triplesQuery: read `triples` field instead of `response` from backend
- Fix graph page crash: guard against non-array triples, accept literals as entity nodes
- Add seed:demo script for AI industry knowledge graph (254 triples, 64 entities)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 04:59:36 -05:00
elpresidank
580ee319a3 fix: prevent dispatcher race condition via promise-based lazy init
Store the initialization Promise in the requestors map synchronously
before yielding, so concurrent callers for the same key await the same
instance — prevents orphaned RequestResponse objects and duplicate NATS
subscriptions. Mirrors upstream fix 8f18ba02.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 12:11:11 -05:00
elpresidank
5e3929a883 fix: comprehensive QA audit — light mode, accessibility, error handling, code quality
- Fix light mode: theme-aware graph node labels, remove prose-invert for
  theme-safe markdown, add brand/semantic color overrides for light backgrounds
- Add 404 catch-all route redirecting unknown paths to /chat
- FalkorDB: add .catch() to connectPromise, add ensureConnected() to all
  store methods (createLiteral, relateNode, relateLiteral, deleteCollection)
- Accessibility: dialog role/aria-modal, toast aria-live, dismiss/zoom/search
  button aria-labels, close panel aria-label
- Lazy-load ForceGraph2D (splits 189KB into separate chunk, main bundle -26%)
- Cap conversation localStorage at 200 messages to prevent quota overflow
- Fix pnpm test: add --passWithNoTests to cli/mcp packages
- Add upload error notification instead of silent catch
- Remove unused class-variance-authority dep and dead tabs.tsx component
- Add @types/node to flow package devDependencies
- Remove stale FIXME comment in messages.ts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:15:59 -05:00
elpresidank
3a80872482 fix: comprehensive QA — resolve 13 bugs, add UX improvements across all services
Client SDK: add .catch() to graphRagStreaming/documentRagStreaming (silent timeout),
null-guard JSON.parse in getPrompts/getSystemPrompt/getPrompt.

Backend: implement "getvalues" config operation for token costs, null-check
createTerm() in FalkorDB triples query, add knowledge-cores service entrypoint
and Docker entry, return proper HTTP 400/404 for gateway error responses.

Workbench: cancel button + elapsed timer for chat, clear agent spinner on error,
flow dialog inline validation, responsive header wrapping, knowledge cores
loading timeout, sidebar/page naming consistency, theme toggle indicator.

Infrastructure: enable Grafana Explore for viewers, add gateway Prometheus
scrape target, fix RAG pipeline dashboard layout (6 panels visible),
filter Service Health to configured targets only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 05:20:10 -05:00
elpresidank
72870a7e2e feat: add unit tests, Docker polish, and workbench UX improvements
Unit tests: Consumer class (7), recursive-splitter (10), parseJsonResponse (11) — 28 total.
Docker: add 5 commented LLM provider services, dev compose override, .env.example.
Workbench: chat persistence, error boundary, disconnect banner, prompts error handling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 03:51:29 -05:00
elpresidank
c7eefee607 feat: add Docker entrypoints, LLM providers, pipeline hardening, workbench pages
Phase 9 — four parallel workstreams:

- Stream A: 14 Docker entrypoints for containerized deployment
- Stream B: Pipeline hardening — robust JSON parsing, LLM retry logic,
  consumer negative-ack, FalkorDB test import fix
- Stream C: Azure OpenAI, OpenAI-compatible, and Mistral LLM providers
- Stream D: Workbench Prompts, Token Cost, Knowledge Cores pages +
  Settings feature switches

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 03:22:55 -05:00
elpresidank
50fb311d2d feat: real PDF pipeline test — end-to-end knowledge extraction working
Add full pipeline test that generates a real PDF, processes it through
the entire pipeline, and verifies knowledge lands in FalkorDB:

- Create test PDF generator using pdf-lib (2-page doc about Acme Corp)
- Add testFullPipeline() to integration tests with store verification
- Fix FalkorDB client connect() — createClient returns unconnected client
  in both TriplesStore and TriplesQuery classes

Results: PDF decoded (2 pages) → chunked (2 chunks) → extracted
(4 relationships) → 16 triples stored in FalkorDB including:
  alice-johnson → is-a-senior-engineer → acme-corporation
  cloudsync → uses-aws-for-hosting → amazon-web-services
  provenance: pages → prov:wasDerivedFrom → source document

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 02:19:12 -05:00
elpresidank
5bc7a1b6fc fix: resolve FlowProcessor topic collisions, librarian timeout, tests
Two bugs found during end-to-end testing:

1. FlowProcessor never restarted flows when config changed — it only
   started them once. Stale NATS JetStream data from previous sessions
   caused services to bind to wrong topics. Fix: stop and restart flows
   on every config push that includes flow definitions.

2. Gateway publishToTopic sent messages without an id property. Pipeline
   FlowProcessor handlers check properties.id and silently return if
   missing. Fix: auto-generate a message id when publishing to topics.

Both fixes validated: 13/13 integration tests passing, PDF decoder
correctly receives and processes document messages through the pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 01:53:55 -05:00
elpresidank
c545213224 feat: add query/retrieval FlowProcessor services and missing runner scripts
Wire up the query and retrieval side of the pipeline so the agent can
answer questions from stored knowledge:

- Triples query service (FalkorDB) — all SPO pattern queries via NATS
- Graph embeddings query service (Qdrant) — entity vector similarity
- Document embeddings query service (Qdrant) — chunk vector similarity
- Graph RAG service — full concept→entity→traverse→score→synthesize pipeline
- Document RAG service — embed→find chunks→synthesize pipeline
- Runner scripts for chunker, extractor, embeddings (missing from Phase 5)
- Add DocumentEmbeddingsRequest/Response schema types
- Add RAG prompt templates (extract-concepts, edge-scoring, synthesize)
- Add graph/doc embeddings query topics to seed config + flow manager
- Add all pipeline/query/retrieval services to docker-compose
- 8 new runner scripts, 8 new pnpm script aliases

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 01:05:54 -05:00
elpresidank
8f7008822a feat: add document pipeline — PDF decoder, Ollama LLM, storage services
Add end-to-end document processing pipeline:
- PDF decoder service (pdfjs-dist) extracts text per page from librarian docs
- Ollama native LLM service for local model inference
- FalkorDB triples store FlowProcessor consumer
- Qdrant graph embeddings store FlowProcessor consumer
- Fix spec name collisions in chunker/extractor (input→chunk-input, etc.)
- Gateway /load endpoint to trigger document processing
- Align flow manager blueprint and seed config with full pipeline topics
- Add runner scripts and test coverage for document load

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 23:47:43 -05:00
elpresidank
25d4227cb5 fix: resolve FlowProcessor topic collisions, librarian timeout, tests
Fix critical bug where all FlowProcessor services shared the same spec
names ("request"/"response"), causing them to steal each other's NATS
topics. Now each service uses unique spec names matching the flow config
topic keys (e.g., "text-completion-request", "prompt-request",
"agent-request").

Fix librarian NATS consumer timeout (500ms → 2000ms, below NATS minimum).

Update seed-config and test-pipeline with correct flow topic mappings.
Add prompt template runner script.

Smoke test results: 11/11 passing (config CRUD, WebSocket, LLM,
librarian CRUD). Agent routing verified via manual curl test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:02:10 -05:00
elpresidank
7db5a1023e feat: add flow manager, config seeding, and expanded integration tests
Flow Management Service:
- FlowManagerService (AsyncProcessor) handling list/get/start/stop flows
  and list/get blueprints via kebab-case wire format
- Default blueprint with all service topic mappings
- Pushes flow config to config service on start/stop

Config Seeding:
- seed-config.ts script pushes prompt templates (extract-relationships,
  extract-definitions, document-prompt, kg-prompt) and default flow
  definition via gateway REST API

Integration Tests:
- Librarian CRUD: add-document, list-documents, get-content, delete
- Agent query: verifies routing through gateway to agent service
- Skip flags: SKIP_LIBRARIAN=1, SKIP_AGENT=1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:37:03 -05:00
elpresidank
f09ef4de45 feat: add document pipeline, ReAct agent, and knowledge core services
Document Pipeline (Team A):
- LibrarianService: document storage with filesystem backend, metadata
  persistence, child document hierarchy, collection management
- ChunkingService: recursive character text splitter with configurable
  chunk size/overlap, FlowProcessor pattern
- KnowledgeExtractService: combined relationship + definition extraction
  using prompt service and LLM, emits RDF triples and entity contexts
- KnowledgeCoreService: knowledge core CRUD with streaming export and
  flow-based loading

ReAct Agent (Team B):
- StreamingReActParser: state machine for parsing LLM output into
  Thought/Action/ActionInput/FinalAnswer sections
- Three MVP tools: KnowledgeQuery (GraphRAG), DocumentQuery (DocRAG),
  TriplesQuery with RequestResponse clients
- AgentService FlowProcessor with ReAct loop, tool execution, and
  streaming chunk responses (thought/observation/answer)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:19:37 -05:00
elpresidank
5ed3f0e2d8 feat: add schema foundation for document pipeline, agent, and deployment
Add missing topics (librarian, knowledge, collection-management, flow),
pipeline message types (TextDocument, Chunk, Triples, EntityContexts),
service message types (Librarian, Knowledge, Collection, Flow CRUD),
and update AgentResponse for streaming chunk format.

Add RequestResponseSpec enabling flow-scoped request/response calls
(needed by knowledge extraction and agent services). Add requestor
registry to Flow class with proper lifecycle management.

Add end_of_dialog to gateway's isComplete() check for agent streaming.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:11:29 -05:00
elpresidank
0042f9259c fix: linter cleanup on flow service implementations
Minor fixes from linter: readonly modifiers, unused parameter prefixes,
type narrowing in graph-rag BFS traversal and edge scoring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 22:52:40 -05:00
elpresidank
b6536eca38 init 2026-04-05 22:44:45 -05:00
elpresidank
e26caa0b12 saving 2026-04-05 21:09:33 -05:00