Commit graph

48 commits

Author SHA1 Message Date
elpresidank
459347ad12 Model librarian upload schema boundaries 2026-06-02 01:34:46 -05:00
elpresidank
3809a38c46 Migrate flow manager to ref-backed Effect state 2026-06-02 01:26:40 -05:00
elpresidank
ba64fc5add Normalize flow service runtime entrypoints 2026-06-02 01:13:40 -05:00
elpresidank
0da0df81c4 Migrate knowledge core service to ref-backed Effect state 2026-06-02 01:07:12 -05:00
elpresidank
5979d38b99 Remove RAG requestor Promise bridges 2026-06-02 00:54:47 -05:00
elpresidank
88db18fbda Migrate config service to ref-backed Effect runtime 2026-06-02 00:40:44 -05:00
elpresidank
b4ee2b691f Migrate strict Effect runtime surfaces 2026-06-02 00:22:04 -05:00
elpresidank
f6878d4dd7 Enforce strict Effect tsgo migrations 2026-06-01 23:19:54 -05:00
elpresidank
64fb23e7d0 Make gateway dispatcher requestors Effect-scoped 2026-06-01 22:17:50 -05:00
elpresidank
a0d2575273 Migrate request-response facade to Effect runtime 2026-06-01 22:11:03 -05:00
elpresidank
b58e51bf75 Update Effect packages to beta 75 2026-06-01 21:16:38 -05:00
elpresidank
dca2786828 Remove native classes from TS runtime 2026-06-01 20:26:47 -05:00
elpresidank
952daf325d Stabilize TS workbench QA and RPC timeouts 2026-06-01 17:23:34 -05:00
elpresidank
3515106670 Advance TS port Effect workbench 2026-06-01 16:22:25 -05:00
elpresidank
5776171d27 saving 2026-05-30 09:56:23 -05:00
elpresidank
ffd97375a8 saving 2026-05-12 08:06:58 -05:00
elpresidank
a20dd1999c saving 2026-05-11 19:44:40 -05:00
elpresidank
6ac5446a76 feat(mcp-tool): wire McpToolService into deploy stack
Three pieces, all required for an end-to-end MCP tool call:

* McpToolService used generic spec names "request"/"response" instead of
  "mcp-tool-request"/"mcp-tool-response", so RequestResponseSpec's
  flow-config topic lookup never matched and consumers bound to literal
  subjects nobody else publishes to.

* Add entrypoints/mcp-tool.mjs (mirrors agent/librarian entrypoints) so
  the service can be launched in the prebuilt trustgraph-ts image.

* Add a `mcp-tool` service block to deploy/docker-compose.yml.

With these three fixes plus a `mcp-tool-request`/`mcp-tool-response`
entry in each flow's topics map, the agent ReAct loop can now invoke
remote MCP tools (verified end-to-end against Brave Search and FireCrawl).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:16:37 -05:00
elpresidank
4c356cd24c fix(client): use correct put/delete config wire shape
ConfigApi.putConfig and deleteConfig (and the duplicate in FlowsApi) sent
a flat values:[{type,key,value}] array and a keys:{type,key} object —
neither matches the ConfigService schema, which requires keys:[namespace,
...innerKeys] and values:Record<string,unknown>. Every save in the
workbench /mcp-tools page returned `Put requires at least one key
(namespace)`.

putConfig now groups items by type (namespace) and issues one put per
group; deleteConfig sends keys:[type, key].

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:16:28 -05:00
elpresidank
ee45cb4850 feat: fix RAG pipelines, Beep Graph branding, PWA, and ambient glow UI
Pipeline fixes:
- Fix agent getting empty response from graph-rag by combining answer +
  explain data in single message (RequestResponse returns first msg)
- Fix Doc RAG pipeline: add content field to Qdrant doc payload, seed 10
  document chunks, fix type mismatches across base/flow/client
- Forward explainability events from agent's KnowledgeQuery to client
- Add "agent" to TERM_BEARING_RESPONSE_SERVICES for triple translation
- Fix embeddings env var (OLLAMA_URL), user/collection threading, edge
  scoring threshold, and various protocol mismatches

Branding:
- Rename TrustGraph → Beep Graph (title, sidebar, settings, about)
- Custom lambda + ThugLife pixel glasses SVG logo component
- Forest green color palette (brand-50 through brand-900)
- SVG favicon + PNG icons (16/32/180/192/512)
- PWA manifest with service worker for offline shell caching
- Splash screen with animated logo pulse on app load
- Ambient glow background with drifting green radial blobs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:19:10 -05:00
elpresidank
87f6e5eb05 feat: chat message actions, explainability graphs, and graph query filters
Add chat UX improvements: message actions toolbar (copy/delete/regenerate)
on hover, inline explainability subgraph visualization from RAG/agent
queries, and token metadata for all chat modes. Enhance graph page with
SPO query filters, configurable triple limit, and type legend overlay.
Extract shared graph utilities for reuse across components.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 02:55:46 -05:00
elpresidank
d5dd15be72 feat: MCP Tools management UI with QA accessibility fixes
Add dedicated /mcp-tools page for managing MCP servers and tools from the
workbench. Includes CRUD dialogs, config API integration, and feature flag
gating via mcpTools switch. QA pass also fixes accessibility across existing
pages: aria-expanded on chat phase blocks, tabpanel tabindex on prompts,
toggle contrast ratio (WCAG 2.1 SC 1.4.11) on settings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 00:59:20 -05:00
elpresidank
338adf8668 fix: global focus-visible rings and light-mode border contrast
- Add global focus-visible outline for buttons, switches, selects, and
  inputs so all interactive elements show a visible brand-500 ring on
  keyboard focus (not just NavLinks and dialog close)
- Darken light-mode --color-border from #e4e4e7 to #d4d4d8 so input
  borders, dividers, and mode selector outlines are visible on white

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 23:28:38 -05:00
elpresidank
d097b790ff fix: comprehensive a11y and contrast QA pass across workbench
Automated QA loop (6 parallel browser agents, 2 rounds) found and fixed
15 accessibility, contrast, and responsive issues across all 8 pages:

- WCAG contrast: light-mode warning (#854d0e), error (#b91c1c), toggle
  off-state (surface-400), connection badge (fg-muted)
- ARIA: mode selector group+pressed, tab pattern ids+labelledby, nav
  and aside labels, dialog focus-return, alert roles on banners
- Responsive: library header flex-wrap, search/button aria-labels
- Focus: NavLink visible ring, dialog close button ring

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 23:26:28 -05:00
elpresidank
77a5fa5044 fix: QA regression pass — graph sizing, focus trap, contrast, accessibility
- Fix graph canvas using window dimensions instead of container by using
  ResizeObserver ref callback (attaches when conditionally-rendered
  container mounts)
- Fix dialog focus trap escaping to background — filter hidden/disabled
  elements from focusable selector
- Fix sidebar connection status and disconnection banner contrast — use
  semantic text-warning/bg-warning instead of amber-400 (1.65:1 → 4.5:1+)
- Add aria-label to chat textarea, htmlFor/id pairs to flows dialog inputs
- Add ARIA tab pattern to prompts page (role=tablist/tab/tabpanel,
  aria-selected, aria-controls)
- Fix prompts heading hierarchy (H1→H2 instead of H1→H3)
- Add flex-wrap to flows page header, fix badge contrast across pages
- Fix service-call race condition with early returns instead of console.log

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 07:48:01 -05:00
elpresidank
b854b56558 feat: MCP tool client infrastructure for agent extensibility
Add the full MCP tool pipeline enabling agents to invoke external tools
(like Brave Search) via MCP servers:

- Add ToolRequest/ToolResponse types and mcp-tool topics to @trustgraph/base
- Create McpToolService (FlowProcessor) that connects to external MCP servers
  via @modelcontextprotocol/sdk StreamableHTTP transport
- Add createMcpTool() to wire MCP tools into the agent's ReAct loop
- Implement config-driven tool registration in AgentService with backward-
  compatible fallback to hardcoded tools
- Add tool filtering by group and state (port of Python tool_filter.py)
- Register mcp-tool in gateway dispatcher and export from @trustgraph/flow
- Fix flow restart race condition: skip restart when flow definitions unchanged
- Update seed config with MCP server config and tool definitions
- Add run scripts for MCP tool service and Brave Search MCP server

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 05:45:46 -05:00
elpresidank
f2b376abef fix: FalkorDB result parsing, embeddings routing, triples query response, graph visualization
- Fix FalkorDB triples query: client v5 returns objects not arrays, use named field access
- Fix embeddings service: align spec names to "embeddings-request"/"embeddings-response"
- Fix client triplesQuery: read `triples` field instead of `response` from backend
- Fix graph page crash: guard against non-array triples, accept literals as entity nodes
- Add seed:demo script for AI industry knowledge graph (254 triples, 64 entities)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 04:59:36 -05:00
elpresidank
580ee319a3 fix: prevent dispatcher race condition via promise-based lazy init
Store the initialization Promise in the requestors map synchronously
before yielding, so concurrent callers for the same key await the same
instance — prevents orphaned RequestResponse objects and duplicate NATS
subscriptions. Mirrors upstream fix 8f18ba02.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 12:11:11 -05:00
elpresidank
5e3929a883 fix: comprehensive QA audit — light mode, accessibility, error handling, code quality
- Fix light mode: theme-aware graph node labels, remove prose-invert for
  theme-safe markdown, add brand/semantic color overrides for light backgrounds
- Add 404 catch-all route redirecting unknown paths to /chat
- FalkorDB: add .catch() to connectPromise, add ensureConnected() to all
  store methods (createLiteral, relateNode, relateLiteral, deleteCollection)
- Accessibility: dialog role/aria-modal, toast aria-live, dismiss/zoom/search
  button aria-labels, close panel aria-label
- Lazy-load ForceGraph2D (splits 189KB into separate chunk, main bundle -26%)
- Cap conversation localStorage at 200 messages to prevent quota overflow
- Fix pnpm test: add --passWithNoTests to cli/mcp packages
- Add upload error notification instead of silent catch
- Remove unused class-variance-authority dep and dead tabs.tsx component
- Add @types/node to flow package devDependencies
- Remove stale FIXME comment in messages.ts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:15:59 -05:00
elpresidank
9ef9ef854f fix: iterative QA pass — resolve remaining bugs, UX and accessibility improvements
Three QA iterations to convergence (zero issues remaining):

Workbench UI:
- Connection badge: amber "Connected (no auth)" for unauthenticated state
- Theme persistence: restore script in index.html + localStorage sync
- Settings About section: add bottom padding so content isn't clipped
- Clear messages: cancel in-flight requests when clearing chat
- Feature switch labels: proper casing + acronym handling (MCP, LLM)
- Token Cost badge: hidden during loading state
- ARIA: role="switch", aria-checked on toggles, aria-labels on buttons
- ConfigApi: null-safe chaining for getPrompts/getSystemPrompt

Grafana dashboards:
- Auto-refresh 30s on all 3 dashboards
- Panel heights reduced to fit viewport without scrolling
- Anonymous role upgraded to Editor for Explore access

Infrastructure:
- Nginx: DNS resolver with variable-based upstream (prevents crash loop)
- Workbench port set to 3002 in .env

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 06:33:22 -05:00
elpresidank
3a80872482 fix: comprehensive QA — resolve 13 bugs, add UX improvements across all services
Client SDK: add .catch() to graphRagStreaming/documentRagStreaming (silent timeout),
null-guard JSON.parse in getPrompts/getSystemPrompt/getPrompt.

Backend: implement "getvalues" config operation for token costs, null-check
createTerm() in FalkorDB triples query, add knowledge-cores service entrypoint
and Docker entry, return proper HTTP 400/404 for gateway error responses.

Workbench: cancel button + elapsed timer for chat, clear agent spinner on error,
flow dialog inline validation, responsive header wrapping, knowledge cores
loading timeout, sidebar/page naming consistency, theme toggle indicator.

Infrastructure: enable Grafana Explore for viewers, add gateway Prometheus
scrape target, fix RAG pipeline dashboard layout (6 panels visible),
filter Service Health to configured targets only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 05:20:10 -05:00
elpresidank
72870a7e2e feat: add unit tests, Docker polish, and workbench UX improvements
Unit tests: Consumer class (7), recursive-splitter (10), parseJsonResponse (11) — 28 total.
Docker: add 5 commented LLM provider services, dev compose override, .env.example.
Workbench: chat persistence, error boundary, disconnect banner, prompts error handling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 03:51:29 -05:00
elpresidank
c7eefee607 feat: add Docker entrypoints, LLM providers, pipeline hardening, workbench pages
Phase 9 — four parallel workstreams:

- Stream A: 14 Docker entrypoints for containerized deployment
- Stream B: Pipeline hardening — robust JSON parsing, LLM retry logic,
  consumer negative-ack, FalkorDB test import fix
- Stream C: Azure OpenAI, OpenAI-compatible, and Mistral LLM providers
- Stream D: Workbench Prompts, Token Cost, Knowledge Cores pages +
  Settings feature switches

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 03:22:55 -05:00
elpresidank
50fb311d2d feat: real PDF pipeline test — end-to-end knowledge extraction working
Add full pipeline test that generates a real PDF, processes it through
the entire pipeline, and verifies knowledge lands in FalkorDB:

- Create test PDF generator using pdf-lib (2-page doc about Acme Corp)
- Add testFullPipeline() to integration tests with store verification
- Fix FalkorDB client connect() — createClient returns unconnected client
  in both TriplesStore and TriplesQuery classes

Results: PDF decoded (2 pages) → chunked (2 chunks) → extracted
(4 relationships) → 16 triples stored in FalkorDB including:
  alice-johnson → is-a-senior-engineer → acme-corporation
  cloudsync → uses-aws-for-hosting → amazon-web-services
  provenance: pages → prov:wasDerivedFrom → source document

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 02:19:12 -05:00
elpresidank
5bc7a1b6fc fix: resolve FlowProcessor topic collisions, librarian timeout, tests
Two bugs found during end-to-end testing:

1. FlowProcessor never restarted flows when config changed — it only
   started them once. Stale NATS JetStream data from previous sessions
   caused services to bind to wrong topics. Fix: stop and restart flows
   on every config push that includes flow definitions.

2. Gateway publishToTopic sent messages without an id property. Pipeline
   FlowProcessor handlers check properties.id and silently return if
   missing. Fix: auto-generate a message id when publishing to topics.

Both fixes validated: 13/13 integration tests passing, PDF decoder
correctly receives and processes document messages through the pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 01:53:55 -05:00
elpresidank
c545213224 feat: add query/retrieval FlowProcessor services and missing runner scripts
Wire up the query and retrieval side of the pipeline so the agent can
answer questions from stored knowledge:

- Triples query service (FalkorDB) — all SPO pattern queries via NATS
- Graph embeddings query service (Qdrant) — entity vector similarity
- Document embeddings query service (Qdrant) — chunk vector similarity
- Graph RAG service — full concept→entity→traverse→score→synthesize pipeline
- Document RAG service — embed→find chunks→synthesize pipeline
- Runner scripts for chunker, extractor, embeddings (missing from Phase 5)
- Add DocumentEmbeddingsRequest/Response schema types
- Add RAG prompt templates (extract-concepts, edge-scoring, synthesize)
- Add graph/doc embeddings query topics to seed config + flow manager
- Add all pipeline/query/retrieval services to docker-compose
- 8 new runner scripts, 8 new pnpm script aliases

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 01:05:54 -05:00
elpresidank
8f7008822a feat: add document pipeline — PDF decoder, Ollama LLM, storage services
Add end-to-end document processing pipeline:
- PDF decoder service (pdfjs-dist) extracts text per page from librarian docs
- Ollama native LLM service for local model inference
- FalkorDB triples store FlowProcessor consumer
- Qdrant graph embeddings store FlowProcessor consumer
- Fix spec name collisions in chunker/extractor (input→chunk-input, etc.)
- Gateway /load endpoint to trigger document processing
- Align flow manager blueprint and seed config with full pipeline topics
- Add runner scripts and test coverage for document load

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 23:47:43 -05:00
elpresidank
8f9de7604e fix: make abstract class constructors protected
Marks FlowProcessor and EmbeddingsService constructors as protected
since these classes should only be instantiated via subclasses.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 21:52:00 -05:00
elpresidank
25d4227cb5 fix: resolve FlowProcessor topic collisions, librarian timeout, tests
Fix critical bug where all FlowProcessor services shared the same spec
names ("request"/"response"), causing them to steal each other's NATS
topics. Now each service uses unique spec names matching the flow config
topic keys (e.g., "text-completion-request", "prompt-request",
"agent-request").

Fix librarian NATS consumer timeout (500ms → 2000ms, below NATS minimum).

Update seed-config and test-pipeline with correct flow topic mappings.
Add prompt template runner script.

Smoke test results: 11/11 passing (config CRUD, WebSocket, LLM,
librarian CRUD). Agent routing verified via manual curl test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:02:10 -05:00
elpresidank
515fc0c264 fix: Docker build fixes, add agent/librarian/flow-manager to compose
Fix Containerfiles:
- Move tsconfig.json to workspace config layer for early availability
- Add missing workspace package.json entries for pnpm lockfile resolution

Docker Compose:
- Move Grafana from port 3000 to 3030 (avoid conflicts)
- Add agent, librarian, and flow-manager app services
- Add librarian-data volume for document persistence

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:41:01 -05:00
elpresidank
7db5a1023e feat: add flow manager, config seeding, and expanded integration tests
Flow Management Service:
- FlowManagerService (AsyncProcessor) handling list/get/start/stop flows
  and list/get blueprints via kebab-case wire format
- Default blueprint with all service topic mappings
- Pushes flow config to config service on start/stop

Config Seeding:
- seed-config.ts script pushes prompt templates (extract-relationships,
  extract-definitions, document-prompt, kg-prompt) and default flow
  definition via gateway REST API

Integration Tests:
- Librarian CRUD: add-document, list-documents, get-content, delete
- Agent query: verifies routing through gateway to agent service
- Skip flags: SKIP_LIBRARIAN=1, SKIP_AGENT=1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:37:03 -05:00
elpresidank
d1f24cf759 feat: add Docker deployment with Containerfile, entrypoints, and nginx
Multi-stage Containerfile for all Node.js services (single image,
different CMD per docker-compose service). ESM entrypoints for gateway,
config, text-completion, prompt, embeddings, agent, and librarian.

Workbench gets a separate Containerfile (nginx:alpine) with SPA routing
and API/WebSocket proxy to gateway.

Docker Compose updated with 6 app services (gateway, config-service,
text-completion, prompt, embeddings, workbench) using shared
trustgraph-ts:local image.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:21:00 -05:00
elpresidank
f09ef4de45 feat: add document pipeline, ReAct agent, and knowledge core services
Document Pipeline (Team A):
- LibrarianService: document storage with filesystem backend, metadata
  persistence, child document hierarchy, collection management
- ChunkingService: recursive character text splitter with configurable
  chunk size/overlap, FlowProcessor pattern
- KnowledgeExtractService: combined relationship + definition extraction
  using prompt service and LLM, emits RDF triples and entity contexts
- KnowledgeCoreService: knowledge core CRUD with streaming export and
  flow-based loading

ReAct Agent (Team B):
- StreamingReActParser: state machine for parsing LLM output into
  Thought/Action/ActionInput/FinalAnswer sections
- Three MVP tools: KnowledgeQuery (GraphRAG), DocumentQuery (DocRAG),
  TriplesQuery with RequestResponse clients
- AgentService FlowProcessor with ReAct loop, tool execution, and
  streaming chunk responses (thought/observation/answer)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:19:37 -05:00
elpresidank
5ed3f0e2d8 feat: add schema foundation for document pipeline, agent, and deployment
Add missing topics (librarian, knowledge, collection-management, flow),
pipeline message types (TextDocument, Chunk, Triples, EntityContexts),
service message types (Librarian, Knowledge, Collection, Flow CRUD),
and update AgentResponse for streaming chunk format.

Add RequestResponseSpec enabling flow-scoped request/response calls
(needed by knowledge extraction and agent services). Add requestor
registry to Flow class with proper lifecycle management.

Add end_of_dialog to gateway's isComplete() check for agent streaming.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:11:29 -05:00
elpresidank
28747e1a92 fix: NATS pipeline bugs, add integration tests and service runners
Fix three critical bugs preventing the NATS message pipeline from working:

- FlowProcessor now subscribes to config-push topic (was missing entirely),
  using DeliverPolicy.All to replay config on service restart
- NATS streams use wildcard subjects (tg.flow.>) instead of per-topic
  narrow filters that caused 503 errors on publish
- Subscriber dispatch loop has exponential backoff on errors to prevent
  tight error loops

Add service runner scripts (gateway, config, LLM) and a 7-test
integration suite that verifies config CRUD, WebSocket round-trip,
and full LLM text-completion through the NATS pipeline.

Fix Docker Compose infra: pin Tempo to v2.6.1, remove deprecated Loki
config fields, add user:0 for volume permissions, remap conflicting
ports (FalkorDB 6380, OTLP 4327/4328).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 23:41:39 -05:00
elpresidank
0042f9259c fix: linter cleanup on flow service implementations
Minor fixes from linter: readonly modifiers, unused parameter prefixes,
type narrowing in graph-rag BFS traversal and edge scoring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 22:52:40 -05:00
elpresidank
b6536eca38 init 2026-04-05 22:44:45 -05:00
elpresidank
e26caa0b12 saving 2026-04-05 21:09:33 -05:00