feat: add document pipeline — PDF decoder, Ollama LLM, storage services

Add end-to-end document processing pipeline:
- PDF decoder service (pdfjs-dist) extracts text per page from librarian docs
- Ollama native LLM service for local model inference
- FalkorDB triples store FlowProcessor consumer
- Qdrant graph embeddings store FlowProcessor consumer
- Fix spec name collisions in chunker/extractor (input→chunk-input, etc.)
- Gateway /load endpoint to trigger document processing
- Align flow manager blueprint and seed config with full pipeline topics
- Add runner scripts and test coverage for document load

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
elpresidank 2026-04-06 23:47:43 -05:00
parent 8f9de7604e
commit 8f7008822a
20 changed files with 894 additions and 37 deletions

View file

@ -17,7 +17,11 @@
"agent": "tsx scripts/run-agent.ts",
"librarian": "tsx scripts/run-librarian.ts",
"knowledge": "tsx scripts/run-knowledge.ts",
"flow-manager": "tsx scripts/run-flow-manager.ts"
"flow-manager": "tsx scripts/run-flow-manager.ts",
"llm:ollama": "tsx scripts/run-ollama.ts",
"pdf-decoder": "tsx scripts/run-pdf-decoder.ts",
"triples-store": "tsx scripts/run-triples-store.ts",
"graph-embeddings-store": "tsx scripts/run-graph-embeddings-store.ts"
},
"devDependencies": {
"nats": "^2.29.0",