apunkt/vestige
1
0
Fork 0
mirror of https://github.com/samvallad33/vestige.git synced 2026-04-25 00:36:22 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

fix: dedup on ingest, Intel Mac CI, npm versions, remove dead TS package

Browse source 
- Route ingest tool through smart_ingest (Prediction Error Gating) to
  prevent duplicate memories when content is similar to existing entries
- Fix Intel Mac release build: use macos-13 runner for x86_64-apple-darwin
  (macos-latest is now ARM64, causing silent cross-compile failures)
- Sync npm package version to 1.1.2 (was 1.0.0 in package.json, 1.1.0
  in postinstall.js BINARY_VERSION)
- Add vestige-restore to npm makeExecutable list
- Remove abandoned packages/core/ TypeScript package (pre-Rust implementation
  referencing FSRS-5, chromadb, ollama — 32K lines of dead code)
- Sync workspace Cargo.toml version to 1.1.2

Closes #5

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Sam Valladares 2026-02-12 02:57:03 -06:00
parent 709c06c2fa
commit a680fa7d2f
49 changed files with 76 additions and 32094 deletions
Download patch file Download diff file Expand all files Collapse all files

35
packages/core/.gitignore vendored
View file

@ -1,35 +0,0 @@
# Dependencies
node_modules/
# Build output
dist/
# Database (user data)
*.db
*.db-wal
*.db-shm
# Environment
.env
.env.local
# IDE
.vscode/
.idea/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Logs
*.log
npm-debug.log*
# Test coverage
coverage/
# Temporary files
*.tmp
*.temp

186
packages/core/README.md
View file

@ -1,186 +0,0 @@
# Vestige
[![npm version](https://img.shields.io/npm/v/vestige-mcp.svg)](https://www.npmjs.com/package/vestige-mcp)
[![MCP Compatible](https://img.shields.io/badge/MCP-Compatible-blue.svg)](https://modelcontextprotocol.io)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
**Git Blame for AI Thoughts** - Memory that decays, strengthens, and discovers connections like the human mind.
![Vestige Demo](./docs/assets/hero-demo.gif)
## Why Vestige?
| Feature | Vestige | Mem0 | Zep | Letta |
|---------|--------|------|-----|-------|
| FSRS-5 spaced repetition | Yes | No | No | No |
| Dual-strength memory | Yes | No | No | No |
| Sentiment-weighted retention | Yes | No | Yes | No |
| Local-first (no cloud) | Yes | No | No | No |
| Git context capture | Yes | No | No | No |
| Semantic connections | Yes | Limited | Yes | Yes |
| Free & open source | Yes | Freemium | Freemium | Yes |
## Quickstart
```bash
# Install
npx vestige-mcp init
# Add to Claude Desktop config
# ~/.config/claude/claude_desktop_config.json (Mac/Linux)
# %APPDATA%\Claude\claude_desktop_config.json (Windows)
{
"mcpServers": {
"vestige": {
"command": "npx",
"args": ["vestige-mcp"]
}
}
}
# Restart Claude Desktop - done!
```
## Key Concepts
### Cognitive Science Foundation
Vestige implements proven memory science:
- **FSRS-5**: State-of-the-art spaced repetition algorithm (powers Anki's 100M+ users)
- **Dual-Strength Memory**: Separate storage and retrieval strength (Bjork & Bjork, 1992)
- **Ebbinghaus Decay**: Memories fade naturally without reinforcement using `R = e^(-t/S)`
- **Sentiment Weighting**: Emotional memories decay slower via AFINN-165 lexicon analysis
### Developer Features
- **Git-Blame for Thoughts**: Every memory captures git branch, commit hash, and changed files
- **REM Cycle**: Background connection discovery between unrelated memories
- **Shadow Self**: Queue unsolved problems for future inspiration when new knowledge arrives
## MCP Tools
| Tool | Description |
|------|-------------|
| `ingest` | Store knowledge with metadata (source, people, tags, git context) |
| `recall` | Search memories by query with relevance ranking |
| `get_knowledge` | Retrieve specific memory by ID |
| `get_related` | Find connected nodes via graph traversal |
| `mark_reviewed` | Reinforce a memory (triggers spaced repetition) |
| `remember_person` | Add/update person in your network |
| `get_person` | Retrieve person details and relationship health |
| `daily_brief` | Get summary of memory state and review queue |
| `health_check` | Check database health with recommendations |
| `backup` | Create timestamped database backup |
## MCP Resources
| Resource | URI | Description |
|----------|-----|-------------|
| Recent memories | `memory://knowledge/recent` | Last 20 stored memories |
| Decaying memories | `memory://knowledge/decaying` | Memories below 50% retention |
| People network | `memory://people/network` | Your relationship graph |
| System context | `memory://context` | Active window, git branch, clipboard |
## CLI Commands
```bash
# Memory
vestige stats # Quick overview
vestige recall "query" # Search memories
vestige review # Show due for review
# Ingestion
vestige eat <url|path> # Ingest documentation
# REM Cycle
vestige dream # Discover connections
vestige dream --dry-run # Preview only
# Shadow Self
vestige problem "desc" # Log unsolved problem
vestige problems # List open problems
vestige solve <id> "fix" # Mark solved
# Context
vestige context # Show current context
vestige watch # Start context daemon
# Maintenance
vestige backup # Create backup
vestige optimize # Vacuum and reindex
vestige decay # Apply memory decay
```
## Configuration
Create `~/.vestige/config.json`:
```json
{
"fsrs": {
"desiredRetention": 0.9,
"maxStability": 365
},
"rem": {
"enabled": true,
"maxAnalyze": 50,
"minStrength": 0.3
},
"decay": {
"sentimentBoost": 2.0
}
}
```
### Database Locations
| File | Path |
|------|------|
| Main database | `~/.vestige/vestige.db` |
| Shadow Self | `~/.vestige/shadow.db` |
| Backups | `~/.vestige/backups/` |
| Context | `~/.vestige/context.json` |
## How It Works
### Memory Decay
```
Retention = e^(-days/stability)
New memory: S=1.0 -> 37% after 1 day
Reviewed once: S=2.5 -> 67% after 1 day
Reviewed 3x: S=15.6 -> 94% after 1 day
Emotional: S x 1.85 boost
```
### REM Cycle Connections
The REM cycle discovers hidden relationships:
| Connection Type | Trigger | Strength |
|----------------|---------|----------|
| `entity_shared` | Same people mentioned | 0.5 + (count * 0.2) |
| `concept_overlap` | 2+ shared concepts | 0.4 + (count * 0.15) |
| `keyword_similarity` | Jaccard > 15% | similarity * 2 |
| `temporal_proximity` | Same day + overlap | 0.3 |
## Documentation
- [API Reference](./docs/api.md) - Full TypeScript API documentation
- [Configuration](./docs/configuration.md) - All config options
- [Architecture](./docs/architecture.md) - System design and data flow
- [Cognitive Science](./docs/cognitive-science.md) - The research behind Vestige
## Contributing
See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.
## License
MIT - see [LICENSE](./LICENSE)
---
**Vestige**: The only AI memory system built on 130 years of cognitive science research.

6126
packages/core/package-lock.json generated
View file

File diff suppressed because it is too large Load diff

74
packages/core/package.json
View file

@ -1,74 +0,0 @@
{
"name": "@vestige/core",
"version": "0.3.0",
"description": "Cognitive memory for AI - FSRS-5, dual-strength, sleep consolidation",
"type": "module",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
},
"./fsrs": {
"types": "./dist/core/fsrs.d.ts",
"import": "./dist/core/fsrs.js"
},
"./database": {
"types": "./dist/core/database.d.ts",
"import": "./dist/core/database.js"
}
},
"bin": {
"vestige": "./dist/cli.js"
},
"scripts": {
"build": "tsup",
"dev": "tsup --watch",
"start": "node dist/index.js",
"inspect": "npx @anthropic-ai/mcp-inspector node dist/index.js",
"test": "rstest",
"lint": "eslint src --ext .ts",
"typecheck": "tsc --noEmit"
},
"keywords": [
"mcp",
"memory",
"cognitive-science",
"fsrs",
"spaced-repetition",
"knowledge-management",
"second-brain",
"ai",
"claude"
],
"author": "samvallad33",
"license": "MIT",
"dependencies": {
"@modelcontextprotocol/sdk": "^1.0.0",
"better-sqlite3": "^11.0.0",
"chokidar": "^3.6.0",
"chromadb": "^1.9.0",
"date-fns": "^3.6.0",
"glob": "^10.4.0",
"gray-matter": "^4.0.3",
"marked": "^12.0.0",
"nanoid": "^5.0.7",
"natural": "^6.12.0",
"node-cron": "^3.0.3",
"ollama": "^0.5.0",
"p-limit": "^6.0.0",
"zod": "^3.23.0"
},
"devDependencies": {
"@rstest/core": "^0.8.0",
"@types/better-sqlite3": "^7.6.10",
"@types/node": "^20.14.0",
"@types/node-cron": "^3.0.11",
"tsup": "^8.1.0",
"typescript": "^5.4.5"
},
"engines": {
"node": ">=20.0.0"
}
}

3920
packages/core/pnpm-lock.yaml generated
View file

File diff suppressed because it is too large Load diff

10
packages/core/rstest.config.ts
View file

@ -1,10 +0,0 @@
import { defineConfig } from '@rstest/core';
export default defineConfig({
testMatch: ['**/*.test.ts'],
setupFiles: ['./src/__tests__/setup.ts'],
coverage: {
include: ['src/**/*.ts'],
exclude: ['src/__tests__/**', 'src/**/*.d.ts'],
},
});

1035
packages/core/src/__tests__/core/dual-strength.test.ts
View file

File diff suppressed because it is too large Load diff

1031
packages/core/src/__tests__/core/fsrs.test.ts
View file

File diff suppressed because it is too large Load diff

476
packages/core/src/__tests__/database.test.ts
View file

@ -1,476 +0,0 @@
import { describe, it, expect, beforeEach, afterEach } from '@rstest/core';
import Database from 'better-sqlite3';
import { nanoid } from 'nanoid';
import {
createTestDatabase,
createTestNode,
createTestPerson,
createTestEdge,
cleanupTestDatabase,
generateTestId,
} from './setup.js';
describe('VestigeDatabase', () => {
let db: Database.Database;
beforeEach(() => {
db = createTestDatabase();
});
afterEach(() => {
cleanupTestDatabase(db);
});
describe('Schema Setup', () => {
it('should create all required tables', () => {
const tables = db.prepare(
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
).all() as { name: string }[];
const tableNames = tables.map(t => t.name);
expect(tableNames).toContain('knowledge_nodes');
expect(tableNames).toContain('knowledge_fts');
expect(tableNames).toContain('people');
expect(tableNames).toContain('interactions');
expect(tableNames).toContain('graph_edges');
expect(tableNames).toContain('sources');
expect(tableNames).toContain('embeddings');
expect(tableNames).toContain('vestige_metadata');
});
it('should create required indexes', () => {
const indexes = db.prepare(
"SELECT name FROM sqlite_master WHERE type='index' AND name NOT LIKE 'sqlite_%'"
).all() as { name: string }[];
const indexNames = indexes.map(i => i.name);
expect(indexNames).toContain('idx_nodes_created_at');
expect(indexNames).toContain('idx_nodes_last_accessed');
expect(indexNames).toContain('idx_nodes_retention');
expect(indexNames).toContain('idx_people_name');
expect(indexNames).toContain('idx_edges_from');
expect(indexNames).toContain('idx_edges_to');
});
});
describe('insertNode', () => {
it('should create a new knowledge node', () => {
const id = nanoid();
const now = new Date().toISOString();
const nodeData = createTestNode({
content: 'Test knowledge content',
tags: ['test', 'knowledge'],
});
const stmt = db.prepare(`
INSERT INTO knowledge_nodes (
id, content, summary,
created_at, updated_at, last_accessed_at, access_count,
retention_strength, stability_factor, sentiment_intensity,
source_type, source_platform,
confidence, people, concepts, events, tags
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
nodeData.content,
null,
now,
now,
now,
0,
1.0,
1.0,
0,
nodeData.sourceType,
nodeData.sourcePlatform,
0.8,
JSON.stringify(nodeData.people),
JSON.stringify(nodeData.concepts),
JSON.stringify(nodeData.events),
JSON.stringify(nodeData.tags)
);
const result = db.prepare('SELECT * FROM knowledge_nodes WHERE id = ?').get(id) as Record<string, unknown>;
expect(result).toBeDefined();
expect(result['content']).toBe('Test knowledge content');
expect(JSON.parse(result['tags'] as string)).toContain('test');
expect(JSON.parse(result['tags'] as string)).toContain('knowledge');
});
it('should store retention and stability factors', () => {
const id = nanoid();
const now = new Date().toISOString();
const nodeData = createTestNode();
const stmt = db.prepare(`
INSERT INTO knowledge_nodes (
id, content,
created_at, updated_at, last_accessed_at,
retention_strength, stability_factor, sentiment_intensity,
storage_strength, retrieval_strength,
source_type, source_platform,
confidence, people, concepts, events, tags
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
nodeData.content,
now,
now,
now,
0.85,
2.5,
0.7,
1.5,
0.9,
nodeData.sourceType,
nodeData.sourcePlatform,
0.8,
'[]',
'[]',
'[]',
'[]'
);
const result = db.prepare('SELECT * FROM knowledge_nodes WHERE id = ?').get(id) as Record<string, unknown>;
expect(result['retention_strength']).toBe(0.85);
expect(result['stability_factor']).toBe(2.5);
expect(result['sentiment_intensity']).toBe(0.7);
expect(result['storage_strength']).toBe(1.5);
expect(result['retrieval_strength']).toBe(0.9);
});
});
describe('searchNodes', () => {
beforeEach(() => {
// Insert test nodes for searching
const nodes = [
{ id: generateTestId(), content: 'TypeScript is a typed superset of JavaScript' },
{ id: generateTestId(), content: 'React is a JavaScript library for building user interfaces' },
{ id: generateTestId(), content: 'Python is a versatile programming language' },
];
const stmt = db.prepare(`
INSERT INTO knowledge_nodes (
id, content, created_at, updated_at, last_accessed_at,
source_type, source_platform, confidence, people, concepts, events, tags
) VALUES (?, ?, datetime('now'), datetime('now'), datetime('now'), 'manual', 'manual', 0.8, '[]', '[]', '[]', '[]')
`);
for (const node of nodes) {
stmt.run(node.id, node.content);
}
});
it('should find nodes by keyword using FTS', () => {
const results = db.prepare(`
SELECT kn.* FROM knowledge_nodes kn
JOIN knowledge_fts fts ON kn.id = fts.id
WHERE knowledge_fts MATCH ?
ORDER BY rank
`).all('JavaScript') as Record<string, unknown>[];
expect(results.length).toBe(2);
expect(results.some(r => (r['content'] as string).includes('TypeScript'))).toBe(true);
expect(results.some(r => (r['content'] as string).includes('React'))).toBe(true);
});
it('should not find unrelated content', () => {
const results = db.prepare(`
SELECT kn.* FROM knowledge_nodes kn
JOIN knowledge_fts fts ON kn.id = fts.id
WHERE knowledge_fts MATCH ?
`).all('Rust') as Record<string, unknown>[];
expect(results.length).toBe(0);
});
it('should find partial matches', () => {
const results = db.prepare(`
SELECT kn.* FROM knowledge_nodes kn
JOIN knowledge_fts fts ON kn.id = fts.id
WHERE knowledge_fts MATCH ?
`).all('programming') as Record<string, unknown>[];
expect(results.length).toBe(1);
expect((results[0]['content'] as string)).toContain('Python');
});
});
describe('People Operations', () => {
it('should insert a person', () => {
const id = nanoid();
const now = new Date().toISOString();
const personData = createTestPerson({
name: 'John Doe',
relationshipType: 'friend',
organization: 'Acme Inc',
});
const stmt = db.prepare(`
INSERT INTO people (
id, name, aliases, relationship_type, organization,
contact_frequency, shared_topics, shared_projects, relationship_health,
social_links, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
personData.name,
JSON.stringify(personData.aliases),
personData.relationshipType,
personData.organization,
personData.contactFrequency,
JSON.stringify(personData.sharedTopics),
JSON.stringify(personData.sharedProjects),
personData.relationshipHealth,
JSON.stringify(personData.socialLinks),
now,
now
);
const result = db.prepare('SELECT * FROM people WHERE id = ?').get(id) as Record<string, unknown>;
expect(result).toBeDefined();
expect(result['name']).toBe('John Doe');
expect(result['relationship_type']).toBe('friend');
expect(result['organization']).toBe('Acme Inc');
});
it('should find person by name', () => {
const id = nanoid();
const now = new Date().toISOString();
db.prepare(`
INSERT INTO people (id, name, aliases, social_links, shared_topics, shared_projects, created_at, updated_at)
VALUES (?, ?, '[]', '{}', '[]', '[]', ?, ?)
`).run(id, 'Jane Smith', now, now);
const result = db.prepare('SELECT * FROM people WHERE name = ?').get('Jane Smith') as Record<string, unknown>;
expect(result).toBeDefined();
expect(result['id']).toBe(id);
});
it('should find person by alias', () => {
const id = nanoid();
const now = new Date().toISOString();
db.prepare(`
INSERT INTO people (id, name, aliases, social_links, shared_topics, shared_projects, created_at, updated_at)
VALUES (?, ?, ?, '{}', '[]', '[]', ?, ?)
`).run(id, 'Robert Johnson', JSON.stringify(['Bob', 'Bobby']), now, now);
const result = db.prepare(`
SELECT * FROM people WHERE name = ? OR aliases LIKE ?
`).get('Bob', '%"Bob"%') as Record<string, unknown>;
expect(result).toBeDefined();
expect(result['name']).toBe('Robert Johnson');
});
});
describe('Graph Edges', () => {
let nodeId1: string;
let nodeId2: string;
beforeEach(() => {
nodeId1 = nanoid();
nodeId2 = nanoid();
const now = new Date().toISOString();
// Create two nodes
const stmt = db.prepare(`
INSERT INTO knowledge_nodes (
id, content, created_at, updated_at, last_accessed_at,
source_type, source_platform, confidence, people, concepts, events, tags
) VALUES (?, ?, ?, ?, ?, 'manual', 'manual', 0.8, '[]', '[]', '[]', '[]')
`);
stmt.run(nodeId1, 'Node 1 content', now, now, now);
stmt.run(nodeId2, 'Node 2 content', now, now, now);
});
it('should create an edge between nodes', () => {
const edgeId = nanoid();
const now = new Date().toISOString();
const edgeData = createTestEdge(nodeId1, nodeId2, {
edgeType: 'relates_to',
weight: 0.8,
});
db.prepare(`
INSERT INTO graph_edges (id, from_id, to_id, edge_type, weight, metadata, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)
`).run(edgeId, edgeData.fromId, edgeData.toId, edgeData.edgeType, edgeData.weight, '{}', now);
const result = db.prepare('SELECT * FROM graph_edges WHERE id = ?').get(edgeId) as Record<string, unknown>;
expect(result).toBeDefined();
expect(result['from_id']).toBe(nodeId1);
expect(result['to_id']).toBe(nodeId2);
expect(result['edge_type']).toBe('relates_to');
expect(result['weight']).toBe(0.8);
});
it('should find related nodes', () => {
const edgeId = nanoid();
const now = new Date().toISOString();
db.prepare(`
INSERT INTO graph_edges (id, from_id, to_id, edge_type, weight, metadata, created_at)
VALUES (?, ?, ?, 'relates_to', 0.5, '{}', ?)
`).run(edgeId, nodeId1, nodeId2, now);
const results = db.prepare(`
SELECT DISTINCT
CASE WHEN from_id = ? THEN to_id ELSE from_id END as related_id
FROM graph_edges
WHERE from_id = ? OR to_id = ?
`).all(nodeId1, nodeId1, nodeId1) as { related_id: string }[];
expect(results.length).toBe(1);
expect(results[0].related_id).toBe(nodeId2);
});
it('should enforce unique constraint on from_id, to_id, edge_type', () => {
const now = new Date().toISOString();
db.prepare(`
INSERT INTO graph_edges (id, from_id, to_id, edge_type, weight, metadata, created_at)
VALUES (?, ?, ?, 'relates_to', 0.5, '{}', ?)
`).run(nanoid(), nodeId1, nodeId2, now);
// Attempting to insert duplicate should fail
expect(() => {
db.prepare(`
INSERT INTO graph_edges (id, from_id, to_id, edge_type, weight, metadata, created_at)
VALUES (?, ?, ?, 'relates_to', 0.7, '{}', ?)
`).run(nanoid(), nodeId1, nodeId2, now);
}).toThrow();
});
});
describe('Decay Simulation', () => {
it('should be able to update retention strength', () => {
const id = nanoid();
const now = new Date().toISOString();
// Insert a node with initial retention
db.prepare(`
INSERT INTO knowledge_nodes (
id, content, created_at, updated_at, last_accessed_at,
retention_strength, stability_factor,
source_type, source_platform, confidence, people, concepts, events, tags
) VALUES (?, 'Test content', ?, ?, ?, 1.0, 1.0, 'manual', 'manual', 0.8, '[]', '[]', '[]', '[]')
`).run(id, now, now, now);
// Simulate decay
const newRetention = 0.75;
db.prepare(`
UPDATE knowledge_nodes SET retention_strength = ? WHERE id = ?
`).run(newRetention, id);
const result = db.prepare('SELECT retention_strength FROM knowledge_nodes WHERE id = ?').get(id) as { retention_strength: number };
expect(result.retention_strength).toBe(0.75);
});
it('should track review count', () => {
const id = nanoid();
const now = new Date().toISOString();
db.prepare(`
INSERT INTO knowledge_nodes (
id, content, created_at, updated_at, last_accessed_at,
review_count, source_type, source_platform, confidence, people, concepts, events, tags
) VALUES (?, 'Test content', ?, ?, ?, 0, 'manual', 'manual', 0.8, '[]', '[]', '[]', '[]')
`).run(id, now, now, now);
// Simulate review
db.prepare(`
UPDATE knowledge_nodes
SET review_count = review_count + 1,
retention_strength = 1.0,
last_accessed_at = ?
WHERE id = ?
`).run(new Date().toISOString(), id);
const result = db.prepare('SELECT review_count, retention_strength FROM knowledge_nodes WHERE id = ?').get(id) as { review_count: number; retention_strength: number };
expect(result.review_count).toBe(1);
expect(result.retention_strength).toBe(1.0);
});
});
describe('Statistics', () => {
it('should count nodes correctly', () => {
const now = new Date().toISOString();
// Insert 3 nodes
for (let i = 0; i < 3; i++) {
db.prepare(`
INSERT INTO knowledge_nodes (
id, content, created_at, updated_at, last_accessed_at,
source_type, source_platform, confidence, people, concepts, events, tags
) VALUES (?, ?, ?, ?, ?, 'manual', 'manual', 0.8, '[]', '[]', '[]', '[]')
`).run(nanoid(), `Node ${i}`, now, now, now);
}
const result = db.prepare('SELECT COUNT(*) as count FROM knowledge_nodes').get() as { count: number };
expect(result.count).toBe(3);
});
it('should count people correctly', () => {
const now = new Date().toISOString();
// Insert 2 people
for (let i = 0; i < 2; i++) {
db.prepare(`
INSERT INTO people (id, name, aliases, social_links, shared_topics, shared_projects, created_at, updated_at)
VALUES (?, ?, '[]', '{}', '[]', '[]', ?, ?)
`).run(nanoid(), `Person ${i}`, now, now);
}
const result = db.prepare('SELECT COUNT(*) as count FROM people').get() as { count: number };
expect(result.count).toBe(2);
});
it('should count edges correctly', () => {
const now = new Date().toISOString();
// Create nodes first
const nodeIds = [nanoid(), nanoid(), nanoid()];
for (const id of nodeIds) {
db.prepare(`
INSERT INTO knowledge_nodes (
id, content, created_at, updated_at, last_accessed_at,
source_type, source_platform, confidence, people, concepts, events, tags
) VALUES (?, 'Content', ?, ?, ?, 'manual', 'manual', 0.8, '[]', '[]', '[]', '[]')
`).run(id, now, now, now);
}
// Insert 2 edges
db.prepare(`
INSERT INTO graph_edges (id, from_id, to_id, edge_type, weight, metadata, created_at)
VALUES (?, ?, ?, 'relates_to', 0.5, '{}', ?)
`).run(nanoid(), nodeIds[0], nodeIds[1], now);
db.prepare(`
INSERT INTO graph_edges (id, from_id, to_id, edge_type, weight, metadata, created_at)
VALUES (?, ?, ?, 'supports', 0.7, '{}', ?)
`).run(nanoid(), nodeIds[1], nodeIds[2], now);
const result = db.prepare('SELECT COUNT(*) as count FROM graph_edges').get() as { count: number };
expect(result.count).toBe(2);
});
});
});

560
packages/core/src/__tests__/fsrs.test.ts
View file

@ -1,560 +0,0 @@
import { describe, it, expect } from '@rstest/core';
import {
FSRSScheduler,
Grade,
FSRS_CONSTANTS,
initialDifficulty,
initialStability,
retrievability,
nextDifficulty,
nextRecallStability,
nextForgetStability,
nextInterval,
applySentimentBoost,
serializeFSRSState,
deserializeFSRSState,
optimalReviewTime,
isReviewDue,
type FSRSState,
type ReviewGrade,
} from '../core/fsrs.js';
describe('FSRS-5 Algorithm', () => {
describe('initialDifficulty', () => {
it('should return higher difficulty for Again grade', () => {
const dAgain = initialDifficulty(Grade.Again);
const dEasy = initialDifficulty(Grade.Easy);
expect(dAgain).toBeGreaterThan(dEasy);
});
it('should clamp difficulty between 1 and 10', () => {
const grades: ReviewGrade[] = [Grade.Again, Grade.Hard, Grade.Good, Grade.Easy];
for (const grade of grades) {
const d = initialDifficulty(grade);
expect(d).toBeGreaterThanOrEqual(FSRS_CONSTANTS.MIN_DIFFICULTY);
expect(d).toBeLessThanOrEqual(FSRS_CONSTANTS.MAX_DIFFICULTY);
}
});
it('should return difficulty in order: Again > Hard > Good > Easy', () => {
const dAgain = initialDifficulty(Grade.Again);
const dHard = initialDifficulty(Grade.Hard);
const dGood = initialDifficulty(Grade.Good);
const dEasy = initialDifficulty(Grade.Easy);
expect(dAgain).toBeGreaterThan(dHard);
expect(dHard).toBeGreaterThan(dGood);
expect(dGood).toBeGreaterThan(dEasy);
});
});
describe('initialStability', () => {
it('should return positive stability for all grades', () => {
const grades: ReviewGrade[] = [Grade.Again, Grade.Hard, Grade.Good, Grade.Easy];
for (const grade of grades) {
const s = initialStability(grade);
expect(s).toBeGreaterThan(0);
}
});
it('should return higher stability for easier grades', () => {
const sAgain = initialStability(Grade.Again);
const sEasy = initialStability(Grade.Easy);
expect(sEasy).toBeGreaterThan(sAgain);
});
it('should ensure minimum stability', () => {
const grades: ReviewGrade[] = [Grade.Again, Grade.Hard, Grade.Good, Grade.Easy];
for (const grade of grades) {
const s = initialStability(grade);
expect(s).toBeGreaterThanOrEqual(FSRS_CONSTANTS.MIN_STABILITY);
}
});
});
describe('retrievability', () => {
it('should return 1.0 when elapsed days is 0', () => {
const r = retrievability(10, 0);
expect(r).toBeCloseTo(1.0, 3);
});
it('should decay over time', () => {
const stability = 10;
const r0 = retrievability(stability, 0);
const r5 = retrievability(stability, 5);
const r30 = retrievability(stability, 30);
expect(r0).toBeGreaterThan(r5);
expect(r5).toBeGreaterThan(r30);
});
it('should decay slower with higher stability', () => {
const elapsedDays = 10;
const rLowStability = retrievability(5, elapsedDays);
const rHighStability = retrievability(50, elapsedDays);
expect(rHighStability).toBeGreaterThan(rLowStability);
});
it('should return 0 when stability is 0 or negative', () => {
expect(retrievability(0, 5)).toBe(0);
expect(retrievability(-1, 5)).toBe(0);
});
it('should return value between 0 and 1', () => {
const testCases = [
{ stability: 1, days: 100 },
{ stability: 100, days: 1 },
{ stability: 10, days: 10 },
];
for (const { stability, days } of testCases) {
const r = retrievability(stability, days);
expect(r).toBeGreaterThanOrEqual(0);
expect(r).toBeLessThanOrEqual(1);
}
});
});
describe('nextDifficulty', () => {
it('should increase difficulty for Again grade', () => {
const currentD = 5;
const newD = nextDifficulty(currentD, Grade.Again);
expect(newD).toBeGreaterThan(currentD);
});
it('should decrease difficulty for Easy grade', () => {
const currentD = 5;
const newD = nextDifficulty(currentD, Grade.Easy);
expect(newD).toBeLessThan(currentD);
});
it('should keep difficulty within bounds', () => {
// Test at extremes
const lowD = nextDifficulty(FSRS_CONSTANTS.MIN_DIFFICULTY, Grade.Easy);
const highD = nextDifficulty(FSRS_CONSTANTS.MAX_DIFFICULTY, Grade.Again);
expect(lowD).toBeGreaterThanOrEqual(FSRS_CONSTANTS.MIN_DIFFICULTY);
expect(highD).toBeLessThanOrEqual(FSRS_CONSTANTS.MAX_DIFFICULTY);
});
});
describe('nextRecallStability', () => {
it('should increase stability after successful recall', () => {
const currentS = 10;
const difficulty = 5;
const r = 0.9;
const newS = nextRecallStability(currentS, difficulty, r, Grade.Good);
expect(newS).toBeGreaterThan(currentS);
});
it('should give bigger boost for Easy grade', () => {
const currentS = 10;
const difficulty = 5;
const r = 0.9;
const sGood = nextRecallStability(currentS, difficulty, r, Grade.Good);
const sEasy = nextRecallStability(currentS, difficulty, r, Grade.Easy);
expect(sEasy).toBeGreaterThan(sGood);
});
it('should apply hard penalty for Hard grade', () => {
const currentS = 10;
const difficulty = 5;
const r = 0.9;
const sGood = nextRecallStability(currentS, difficulty, r, Grade.Good);
const sHard = nextRecallStability(currentS, difficulty, r, Grade.Hard);
expect(sHard).toBeLessThan(sGood);
});
it('should use forget stability for Again grade', () => {
const currentS = 10;
const difficulty = 5;
const r = 0.9;
const sAgain = nextRecallStability(currentS, difficulty, r, Grade.Again);
// Should call nextForgetStability internally, resulting in lower stability
expect(sAgain).toBeLessThan(currentS);
});
});
describe('nextForgetStability', () => {
it('should return lower stability than current', () => {
const currentS = 10;
const difficulty = 5;
const r = 0.3;
const newS = nextForgetStability(difficulty, currentS, r);
expect(newS).toBeLessThan(currentS);
});
it('should return positive stability', () => {
const newS = nextForgetStability(5, 10, 0.5);
expect(newS).toBeGreaterThan(0);
});
it('should keep stability within bounds', () => {
const newS = nextForgetStability(10, 100, 0.1);
expect(newS).toBeGreaterThanOrEqual(FSRS_CONSTANTS.MIN_STABILITY);
expect(newS).toBeLessThanOrEqual(FSRS_CONSTANTS.MAX_STABILITY);
});
});
describe('nextInterval', () => {
it('should return 0 for 0 or negative stability', () => {
expect(nextInterval(0, 0.9)).toBe(0);
expect(nextInterval(-1, 0.9)).toBe(0);
});
it('should return longer intervals for higher stability', () => {
const iLow = nextInterval(5, 0.9);
const iHigh = nextInterval(50, 0.9);
expect(iHigh).toBeGreaterThan(iLow);
});
it('should return shorter intervals for higher desired retention', () => {
const stability = 10;
const i90 = nextInterval(stability, 0.9);
const i95 = nextInterval(stability, 0.95);
expect(i90).toBeGreaterThan(i95);
});
it('should return 0 for 100% retention', () => {
expect(nextInterval(10, 1.0)).toBe(0);
});
it('should return max interval for 0% retention', () => {
expect(nextInterval(10, 0)).toBe(FSRS_CONSTANTS.MAX_STABILITY);
});
});
describe('applySentimentBoost', () => {
it('should not boost stability for neutral sentiment (0)', () => {
const stability = 10;
const boosted = applySentimentBoost(stability, 0, 2.0);
expect(boosted).toBe(stability);
});
it('should apply max boost for max sentiment (1)', () => {
const stability = 10;
const maxBoost = 2.0;
const boosted = applySentimentBoost(stability, 1, maxBoost);
expect(boosted).toBe(stability * maxBoost);
});
it('should apply proportional boost for intermediate sentiment', () => {
const stability = 10;
const maxBoost = 2.0;
const sentiment = 0.5;
const boosted = applySentimentBoost(stability, sentiment, maxBoost);
// Expected: stability * (1 + (maxBoost - 1) * sentiment) = 10 * 1.5 = 15
expect(boosted).toBe(15);
});
it('should clamp sentiment and maxBoost values', () => {
const stability = 10;
// Sentiment should be clamped to 0-1
const boosted1 = applySentimentBoost(stability, -0.5, 2.0);
expect(boosted1).toBe(stability); // Clamped to 0
// maxBoost should be clamped to 1-3
const boosted2 = applySentimentBoost(stability, 1, 5.0);
expect(boosted2).toBe(stability * 3); // Clamped to 3
});
});
});
describe('FSRSScheduler', () => {
describe('constructor', () => {
it('should create scheduler with default config', () => {
const scheduler = new FSRSScheduler();
const config = scheduler.getConfig();
expect(config.desiredRetention).toBe(0.9);
expect(config.maximumInterval).toBe(36500);
expect(config.enableSentimentBoost).toBe(true);
expect(config.maxSentimentBoost).toBe(2);
});
it('should accept custom config', () => {
const scheduler = new FSRSScheduler({
desiredRetention: 0.85,
maximumInterval: 365,
enableSentimentBoost: false,
maxSentimentBoost: 1.5,
});
const config = scheduler.getConfig();
expect(config.desiredRetention).toBe(0.85);
expect(config.maximumInterval).toBe(365);
expect(config.enableSentimentBoost).toBe(false);
expect(config.maxSentimentBoost).toBe(1.5);
});
});
describe('newCard', () => {
it('should create new card with initial state', () => {
const scheduler = new FSRSScheduler();
const state = scheduler.newCard();
expect(state.state).toBe('New');
expect(state.reps).toBe(0);
expect(state.lapses).toBe(0);
expect(state.difficulty).toBeGreaterThanOrEqual(FSRS_CONSTANTS.MIN_DIFFICULTY);
expect(state.difficulty).toBeLessThanOrEqual(FSRS_CONSTANTS.MAX_DIFFICULTY);
expect(state.stability).toBeGreaterThan(0);
expect(state.scheduledDays).toBe(0);
});
});
describe('review', () => {
it('should handle new item review', () => {
const scheduler = new FSRSScheduler();
const state = scheduler.newCard();
const result = scheduler.review(state, Grade.Good, 0);
expect(result.state.stability).toBeGreaterThan(0);
expect(result.state.reps).toBe(1);
expect(result.state.state).not.toBe('New');
expect(result.interval).toBeGreaterThanOrEqual(0);
expect(result.isLapse).toBe(false);
});
it('should handle Again grade as lapse for reviewed cards', () => {
const scheduler = new FSRSScheduler();
let state = scheduler.newCard();
// First review to move out of New state
const result1 = scheduler.review(state, Grade.Good, 0);
state = result1.state;
// Second review with Again (lapse)
const result2 = scheduler.review(state, Grade.Again, 1);
expect(result2.isLapse).toBe(true);
expect(result2.state.lapses).toBe(1);
expect(result2.state.state).toBe('Relearning');
});
it('should apply sentiment boost when enabled', () => {
const scheduler = new FSRSScheduler({ enableSentimentBoost: true, maxSentimentBoost: 2 });
const state = scheduler.newCard();
const resultNoBoost = scheduler.review(state, Grade.Good, 0, 0);
const resultWithBoost = scheduler.review(state, Grade.Good, 0, 1);
expect(resultWithBoost.state.stability).toBeGreaterThan(resultNoBoost.state.stability);
});
it('should not apply sentiment boost when disabled', () => {
const scheduler = new FSRSScheduler({ enableSentimentBoost: false });
const state = scheduler.newCard();
const resultNoBoost = scheduler.review(state, Grade.Good, 0, 0);
const resultWithBoost = scheduler.review(state, Grade.Good, 0, 1);
// Stability should be the same since boost is disabled
expect(resultWithBoost.state.stability).toBe(resultNoBoost.state.stability);
});
it('should respect maximum interval', () => {
const maxInterval = 30;
const scheduler = new FSRSScheduler({ maximumInterval: maxInterval });
const state = scheduler.newCard();
// Review multiple times to build up stability
let currentState = state;
for (let i = 0; i < 10; i++) {
const result = scheduler.review(currentState, Grade.Easy, 0);
expect(result.interval).toBeLessThanOrEqual(maxInterval);
currentState = result.state;
}
});
});
describe('getRetrievability', () => {
it('should return 1.0 for just-reviewed card', () => {
const scheduler = new FSRSScheduler();
const state = scheduler.newCard();
state.lastReview = new Date();
const r = scheduler.getRetrievability(state, 0);
expect(r).toBeCloseTo(1.0, 3);
});
it('should return lower value after time passes', () => {
const scheduler = new FSRSScheduler();
const state = scheduler.newCard();
const r0 = scheduler.getRetrievability(state, 0);
const r10 = scheduler.getRetrievability(state, 10);
expect(r0).toBeGreaterThan(r10);
});
});
describe('previewReviews', () => {
it('should return results for all grades', () => {
const scheduler = new FSRSScheduler();
const state = scheduler.newCard();
const preview = scheduler.previewReviews(state, 0);
expect(preview.again).toBeDefined();
expect(preview.hard).toBeDefined();
expect(preview.good).toBeDefined();
expect(preview.easy).toBeDefined();
});
it('should show increasing intervals from again to easy', () => {
const scheduler = new FSRSScheduler();
let state = scheduler.newCard();
// First review to establish some stability
const result = scheduler.review(state, Grade.Good, 0);
state = result.state;
const preview = scheduler.previewReviews(state, 1);
// Generally, easy should have longest interval, again shortest
expect(preview.easy.interval).toBeGreaterThanOrEqual(preview.good.interval);
expect(preview.good.interval).toBeGreaterThanOrEqual(preview.hard.interval);
});
});
});
describe('FSRS Utility Functions', () => {
describe('serializeFSRSState / deserializeFSRSState', () => {
it('should serialize and deserialize state correctly', () => {
const scheduler = new FSRSScheduler();
const state = scheduler.newCard();
const serialized = serializeFSRSState(state);
const deserialized = deserializeFSRSState(serialized);
expect(deserialized.difficulty).toBe(state.difficulty);
expect(deserialized.stability).toBe(state.stability);
expect(deserialized.state).toBe(state.state);
expect(deserialized.reps).toBe(state.reps);
expect(deserialized.lapses).toBe(state.lapses);
expect(deserialized.scheduledDays).toBe(state.scheduledDays);
});
it('should preserve lastReview date', () => {
const state: FSRSState = {
difficulty: 5,
stability: 10,
state: 'Review',
reps: 5,
lapses: 1,
lastReview: new Date('2024-01-15T12:00:00Z'),
scheduledDays: 7,
};
const serialized = serializeFSRSState(state);
const deserialized = deserializeFSRSState(serialized);
expect(deserialized.lastReview.toISOString()).toBe(state.lastReview.toISOString());
});
});
describe('optimalReviewTime', () => {
it('should return interval based on stability', () => {
const state: FSRSState = {
difficulty: 5,
stability: 10,
state: 'Review',
reps: 3,
lapses: 0,
lastReview: new Date(),
scheduledDays: 7,
};
const interval = optimalReviewTime(state, 0.9);
expect(interval).toBeGreaterThan(0);
});
it('should return shorter interval for higher retention target', () => {
const state: FSRSState = {
difficulty: 5,
stability: 10,
state: 'Review',
reps: 3,
lapses: 0,
lastReview: new Date(),
scheduledDays: 7,
};
const i90 = optimalReviewTime(state, 0.9);
const i95 = optimalReviewTime(state, 0.95);
expect(i90).toBeGreaterThan(i95);
});
});
describe('isReviewDue', () => {
it('should return false for just-created card', () => {
const state: FSRSState = {
difficulty: 5,
stability: 10,
state: 'Review',
reps: 3,
lapses: 0,
lastReview: new Date(),
scheduledDays: 7,
};
expect(isReviewDue(state)).toBe(false);
});
it('should return true when scheduled days have passed', () => {
const pastDate = new Date();
pastDate.setDate(pastDate.getDate() - 10);
const state: FSRSState = {
difficulty: 5,
stability: 10,
state: 'Review',
reps: 3,
lapses: 0,
lastReview: pastDate,
scheduledDays: 7,
};
expect(isReviewDue(state)).toBe(true);
});
it('should use retention threshold when provided', () => {
const pastDate = new Date();
pastDate.setDate(pastDate.getDate() - 5);
const state: FSRSState = {
difficulty: 5,
stability: 10,
state: 'Review',
reps: 3,
lapses: 0,
lastReview: pastDate,
scheduledDays: 30, // Not due by scheduledDays
};
// Check with high retention threshold (should be due)
const isDueHighThreshold = isReviewDue(state, 0.95);
// Check with low retention threshold (might not be due)
const isDueLowThreshold = isReviewDue(state, 0.5);
// With higher threshold, more likely to be due
expect(isDueHighThreshold || !isDueLowThreshold).toBe(true);
});
});
});

1333
packages/core/src/__tests__/integration/mcp-tools.test.ts
View file

File diff suppressed because it is too large Load diff

300
packages/core/src/__tests__/setup.ts
View file

@ -1,300 +0,0 @@
import Database from 'better-sqlite3';
import type { KnowledgeNodeInput, PersonNode, GraphEdge } from '../core/types.js';
/**
* Create an in-memory database for testing
*/
export function createTestDatabase(): Database.Database {
const db = new Database(':memory:');
db.pragma('journal_mode = WAL');
db.pragma('foreign_keys = ON');
// Initialize tables (from database.ts initializeSchema)
db.exec(`
CREATE TABLE IF NOT EXISTS knowledge_nodes (
id TEXT PRIMARY KEY,
content TEXT NOT NULL,
summary TEXT,
-- Temporal metadata
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
last_accessed_at TEXT NOT NULL,
access_count INTEGER DEFAULT 0,
-- Decay modeling (SM-2 inspired spaced repetition)
retention_strength REAL DEFAULT 1.0,
stability_factor REAL DEFAULT 1.0,
sentiment_intensity REAL DEFAULT 0,
next_review_date TEXT,
review_count INTEGER DEFAULT 0,
-- Dual-Strength Memory Model (Bjork & Bjork, 1992)
storage_strength REAL DEFAULT 1.0,
retrieval_strength REAL DEFAULT 1.0,
-- Provenance
source_type TEXT NOT NULL,
source_platform TEXT NOT NULL,
source_id TEXT,
source_url TEXT,
source_chain TEXT DEFAULT '[]',
git_context TEXT,
-- Confidence
confidence REAL DEFAULT 0.8,
is_contradicted INTEGER DEFAULT 0,
contradiction_ids TEXT DEFAULT '[]',
-- Extracted entities (JSON arrays)
people TEXT DEFAULT '[]',
concepts TEXT DEFAULT '[]',
events TEXT DEFAULT '[]',
tags TEXT DEFAULT '[]'
);
CREATE INDEX IF NOT EXISTS idx_nodes_created_at ON knowledge_nodes(created_at);
CREATE INDEX IF NOT EXISTS idx_nodes_last_accessed ON knowledge_nodes(last_accessed_at);
CREATE INDEX IF NOT EXISTS idx_nodes_retention ON knowledge_nodes(retention_strength);
CREATE INDEX IF NOT EXISTS idx_nodes_source_type ON knowledge_nodes(source_type);
CREATE INDEX IF NOT EXISTS idx_nodes_source_platform ON knowledge_nodes(source_platform);
`);
// Full-text search for content
db.exec(`
CREATE VIRTUAL TABLE IF NOT EXISTS knowledge_fts USING fts5(
id,
content,
summary,
tags,
content='knowledge_nodes',
content_rowid='rowid'
);
-- Triggers to keep FTS in sync
CREATE TRIGGER IF NOT EXISTS knowledge_ai AFTER INSERT ON knowledge_nodes BEGIN
INSERT INTO knowledge_fts(rowid, id, content, summary, tags)
VALUES (NEW.rowid, NEW.id, NEW.content, NEW.summary, NEW.tags);
END;
CREATE TRIGGER IF NOT EXISTS knowledge_ad AFTER DELETE ON knowledge_nodes BEGIN
INSERT INTO knowledge_fts(knowledge_fts, rowid, id, content, summary, tags)
VALUES ('delete', OLD.rowid, OLD.id, OLD.content, OLD.summary, OLD.tags);
END;
CREATE TRIGGER IF NOT EXISTS knowledge_au AFTER UPDATE ON knowledge_nodes BEGIN
INSERT INTO knowledge_fts(knowledge_fts, rowid, id, content, summary, tags)
VALUES ('delete', OLD.rowid, OLD.id, OLD.content, OLD.summary, OLD.tags);
INSERT INTO knowledge_fts(rowid, id, content, summary, tags)
VALUES (NEW.rowid, NEW.id, NEW.content, NEW.summary, NEW.tags);
END;
`);
// People table
db.exec(`
CREATE TABLE IF NOT EXISTS people (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
aliases TEXT DEFAULT '[]',
-- Relationship context
how_we_met TEXT,
relationship_type TEXT,
organization TEXT,
role TEXT,
location TEXT,
-- Contact info
email TEXT,
phone TEXT,
social_links TEXT DEFAULT '{}',
-- Communication patterns
last_contact_at TEXT,
contact_frequency REAL DEFAULT 0,
preferred_channel TEXT,
-- Shared context
shared_topics TEXT DEFAULT '[]',
shared_projects TEXT DEFAULT '[]',
-- Meta
notes TEXT,
relationship_health REAL DEFAULT 0.5,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_people_name ON people(name);
CREATE INDEX IF NOT EXISTS idx_people_last_contact ON people(last_contact_at);
`);
// Interactions table
db.exec(`
CREATE TABLE IF NOT EXISTS interactions (
id TEXT PRIMARY KEY,
person_id TEXT NOT NULL,
type TEXT NOT NULL,
date TEXT NOT NULL,
summary TEXT NOT NULL,
topics TEXT DEFAULT '[]',
sentiment REAL,
action_items TEXT DEFAULT '[]',
source_node_id TEXT,
FOREIGN KEY (person_id) REFERENCES people(id) ON DELETE CASCADE,
FOREIGN KEY (source_node_id) REFERENCES knowledge_nodes(id) ON DELETE SET NULL
);
CREATE INDEX IF NOT EXISTS idx_interactions_person ON interactions(person_id);
CREATE INDEX IF NOT EXISTS idx_interactions_date ON interactions(date);
`);
// Graph edges table
db.exec(`
CREATE TABLE IF NOT EXISTS graph_edges (
id TEXT PRIMARY KEY,
from_id TEXT NOT NULL,
to_id TEXT NOT NULL,
edge_type TEXT NOT NULL,
weight REAL DEFAULT 0.5,
metadata TEXT DEFAULT '{}',
created_at TEXT NOT NULL,
UNIQUE(from_id, to_id, edge_type)
);
CREATE INDEX IF NOT EXISTS idx_edges_from ON graph_edges(from_id);
CREATE INDEX IF NOT EXISTS idx_edges_to ON graph_edges(to_id);
CREATE INDEX IF NOT EXISTS idx_edges_type ON graph_edges(edge_type);
`);
// Sources table
db.exec(`
CREATE TABLE IF NOT EXISTS sources (
id TEXT PRIMARY KEY,
type TEXT NOT NULL,
platform TEXT NOT NULL,
original_id TEXT,
url TEXT,
file_path TEXT,
title TEXT,
author TEXT,
publication_date TEXT,
ingested_at TEXT NOT NULL,
last_synced_at TEXT NOT NULL,
content_hash TEXT,
node_count INTEGER DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_sources_platform ON sources(platform);
CREATE INDEX IF NOT EXISTS idx_sources_file_path ON sources(file_path);
`);
// Embeddings reference table
db.exec(`
CREATE TABLE IF NOT EXISTS embeddings (
node_id TEXT PRIMARY KEY,
chroma_id TEXT NOT NULL,
model TEXT NOT NULL,
created_at TEXT NOT NULL,
FOREIGN KEY (node_id) REFERENCES knowledge_nodes(id) ON DELETE CASCADE
);
`);
// Metadata table
db.exec(`
CREATE TABLE IF NOT EXISTS vestige_metadata (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at TEXT NOT NULL
);
`);
return db;
}
/**
* Create test fixtures for knowledge nodes
*/
export function createTestNode(overrides: Partial<Omit<KnowledgeNodeInput, 'id'>> = {}): Omit<KnowledgeNodeInput, 'id'> {
return {
content: 'Test content for knowledge node',
sourceType: 'manual',
sourcePlatform: 'manual',
tags: [],
people: [],
concepts: [],
events: [],
...overrides,
};
}
/**
* Create test fixtures for people
*/
export function createTestPerson(overrides: Partial<Omit<PersonNode, 'id' | 'createdAt' | 'updatedAt'>> = {}): Omit<PersonNode, 'id' | 'createdAt' | 'updatedAt'> {
return {
name: 'Test Person',
relationshipType: 'colleague',
aliases: [],
socialLinks: {},
contactFrequency: 0,
sharedTopics: [],
sharedProjects: [],
relationshipHealth: 0.5,
...overrides,
};
}
/**
* Create test fixtures for graph edges
*/
export function createTestEdge(fromId: string, toId: string, overrides: Partial<Omit<GraphEdge, 'id' | 'createdAt'>> = {}): Omit<GraphEdge, 'id' | 'createdAt'> {
return {
fromId,
toId,
edgeType: 'relates_to',
weight: 0.5,
metadata: {},
...overrides,
};
}
/**
* Clean up test database
*/
export function cleanupTestDatabase(db: Database.Database): void {
try {
db.close();
} catch {
// Ignore close errors
}
}
/**
* Wait for a specified amount of time (useful for async tests)
*/
export function wait(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}
/**
* Generate a unique test ID
*/
export function generateTestId(): string {
return `test-${Date.now()}-${Math.random().toString(36).substring(2, 9)}`;
}
/**
* Create a mock timestamp for consistent testing
*/
export function mockTimestamp(daysAgo: number = 0): Date {
const date = new Date();
date.setDate(date.getDate() - daysAgo);
return date;
}

1464
packages/core/src/cli.ts
View file

File diff suppressed because it is too large Load diff

489
packages/core/src/core/config.ts
View file

@ -1,489 +0,0 @@
/**
* Configuration Management for Vestige MCP
*
* Provides centralized configuration with:
* - Zod schema validation
* - File-based configuration (~/.vestige/config.json)
* - Environment variable overrides
* - Type-safe accessors for all config sections
*
* Configuration priority (highest to lowest):
* 1. Environment variables
* 2. Config file
* 3. Default values
*/
import { z } from 'zod';
import path from 'path';
import os from 'os';
import fs from 'fs';
// ============================================================================
// CONFIGURATION SCHEMA
// ============================================================================
/**
* Database configuration schema
*/
const DatabaseConfigSchema = z.object({
/** Path to the SQLite database file */
path: z.string().default(path.join(os.homedir(), '.vestige', 'vestige.db')),
/** Directory for database backups */
backupDir: z.string().default(path.join(os.homedir(), '.vestige', 'backups')),
/** SQLite busy timeout in milliseconds */
busyTimeout: z.number().default(5000),
/** SQLite cache size in pages (negative = KB) */
cacheSize: z.number().default(64000),
/** Maximum number of backup files to retain */
maxBackups: z.number().default(5),
}).default({});
/**
* FSRS (Free Spaced Repetition Scheduler) algorithm configuration
* Named with 'Config' prefix to avoid collision with FSRSConfigSchema in fsrs.ts
*/
const ConfigFSRSSchema = z.object({
/** Target retention rate (0.7 to 0.99) */
desiredRetention: z.number().min(0.7).max(0.99).default(0.9),
/** Custom FSRS-5 weights (19 values). If not provided, uses defaults. */
weights: z.array(z.number()).length(19).optional(),
/** Enable personalized scheduling based on review history */
enablePersonalization: z.boolean().default(false),
}).default({});
/**
* Dual-strength memory model configuration
* Based on the distinction between storage strength and retrieval strength
*/
const MemoryConfigSchema = z.object({
/** Storage strength boost on passive access (read) */
storageBoostOnAccess: z.number().default(0.05),
/** Storage strength boost on active review */
storageBoostOnReview: z.number().default(0.1),
/** Half-life for retrieval strength decay in days */
retrievalDecayHalfLife: z.number().default(7),
/** Minimum retention strength before memory is considered weak */
minRetentionStrength: z.number().default(0.1),
}).default({});
/**
* Sentiment analysis configuration for emotional memory weighting
*/
const SentimentConfigSchema = z.object({
/** Stability multiplier for highly emotional memories */
stabilityBoost: z.number().default(2.0),
/** Minimum boost applied to any memory */
minBoost: z.number().default(1.0),
}).default({});
/**
* REM (Rapid Eye Movement) cycle configuration
* Handles memory consolidation and connection discovery
*/
const REMConfigSchema = z.object({
/** Enable REM cycle processing */
enabled: z.boolean().default(true),
/** Maximum number of memories to analyze per cycle */
maxAnalyze: z.number().default(50),
/** Minimum connection strength to create an edge */
minConnectionStrength: z.number().default(0.3),
/** Half-life for temporal proximity weighting in days */
temporalHalfLifeDays: z.number().default(7),
/** Decay factor for spreading activation (0-1) */
spreadingActivationDecay: z.number().default(0.8),
}).default({});
/**
* Memory consolidation configuration
* Controls the background process that strengthens important memories
*/
const ConsolidationConfigSchema = z.object({
/** Enable automatic consolidation */
enabled: z.boolean().default(true),
/** Hour of day to run consolidation (0-23) */
scheduleHour: z.number().min(0).max(23).default(3),
/** Window in hours for short-term memory processing */
shortTermWindowHours: z.number().default(24),
/** Minimum importance score for consolidation */
importanceThreshold: z.number().default(0.5),
/** Threshold below which memories may be pruned */
pruneThreshold: z.number().default(0.2),
}).default({});
/**
* Embeddings service configuration
*/
const EmbeddingsConfigSchema = z.object({
/** Embedding provider to use */
provider: z.enum(['ollama', 'fallback']).default('ollama'),
/** Ollama API host URL */
ollamaHost: z.string().default('http://localhost:11434'),
/** Embedding model name */
model: z.string().default('nomic-embed-text'),
/** Maximum text length to embed (characters) */
maxTextLength: z.number().default(8000),
}).default({});
/**
* Vector store configuration for semantic search
*/
const VectorStoreConfigSchema = z.object({
/** Vector store provider */
provider: z.enum(['chromadb', 'sqlite']).default('chromadb'),
/** ChromaDB host URL */
chromaHost: z.string().default('http://localhost:8000'),
/** Name of the embeddings collection */
collectionName: z.string().default('vestige_embeddings'),
}).default({});
/**
* Cache configuration
*/
const CacheConfigSchema = z.object({
/** Enable caching */
enabled: z.boolean().default(true),
/** Maximum number of items in cache */
maxSize: z.number().default(10000),
/** Default time-to-live in milliseconds */
defaultTTLMs: z.number().default(5 * 60 * 1000),
}).default({});
/**
* Logging configuration
*/
const LoggingConfigSchema = z.object({
/** Minimum log level */
level: z.enum(['debug', 'info', 'warn', 'error']).default('info'),
/** Use structured JSON logging */
structured: z.boolean().default(true),
}).default({});
/**
* Input/output limits configuration
*/
const LimitsConfigSchema = z.object({
/** Maximum content length in characters */
maxContentLength: z.number().default(1_000_000),
/** Maximum name/title length in characters */
maxNameLength: z.number().default(500),
/** Maximum query length in characters */
maxQueryLength: z.number().default(10_000),
/** Maximum number of tags per item */
maxTagsCount: z.number().default(100),
/** Maximum items per batch operation */
maxBatchSize: z.number().default(1000),
/** Default pagination limit */
paginationDefault: z.number().default(50),
/** Maximum pagination limit */
paginationMax: z.number().default(500),
}).default({});
/**
* Main configuration schema combining all sections
*/
const ConfigSchema = z.object({
database: DatabaseConfigSchema,
fsrs: ConfigFSRSSchema,
memory: MemoryConfigSchema,
sentiment: SentimentConfigSchema,
rem: REMConfigSchema,
consolidation: ConsolidationConfigSchema,
embeddings: EmbeddingsConfigSchema,
vectorStore: VectorStoreConfigSchema,
cache: CacheConfigSchema,
logging: LoggingConfigSchema,
limits: LimitsConfigSchema,
});
/**
* Inferred TypeScript type from the Zod schema
*/
export type VestigeConfig = z.infer<typeof ConfigSchema>;
// ============================================================================
// CONFIGURATION LOADING
// ============================================================================
/**
* Singleton configuration instance
*/
let config: VestigeConfig | null = null;
/**
* Partial configuration type for environment overrides
*/
interface PartialVestigeConfig {
database?: {
path?: string;
backupDir?: string;
};
logging?: {
level?: string;
};
embeddings?: {
ollamaHost?: string;
model?: string;
};
vectorStore?: {
chromaHost?: string;
};
fsrs?: {
desiredRetention?: number;
};
rem?: {
enabled?: boolean;
};
consolidation?: {
enabled?: boolean;
};
}
/**
* Load environment variable overrides
* Environment variables take precedence over file configuration
*/
function loadEnvConfig(): PartialVestigeConfig {
const env: PartialVestigeConfig = {};
// Database configuration
const dbPath = process.env['VESTIGE_DB_PATH'];
const backupDir = process.env['VESTIGE_BACKUP_DIR'];
if (dbPath || backupDir) {
env.database = {};
if (dbPath) env.database.path = dbPath;
if (backupDir) env.database.backupDir = backupDir;
}
// Logging configuration
const logLevel = process.env['VESTIGE_LOG_LEVEL'];
if (logLevel) {
env.logging = { level: logLevel };
}
// Embeddings configuration
const ollamaHost = process.env['OLLAMA_HOST'];
const embeddingModel = process.env['VESTIGE_EMBEDDING_MODEL'];
if (ollamaHost || embeddingModel) {
env.embeddings = {};
if (ollamaHost) env.embeddings.ollamaHost = ollamaHost;
if (embeddingModel) env.embeddings.model = embeddingModel;
}
// Vector store configuration
const chromaHost = process.env['CHROMA_HOST'];
if (chromaHost) {
env.vectorStore = { chromaHost };
}
// FSRS configuration
const desiredRetention = process.env['VESTIGE_DESIRED_RETENTION'];
if (desiredRetention) {
const retention = parseFloat(desiredRetention);
if (!isNaN(retention)) {
env.fsrs = { desiredRetention: retention };
}
}
// REM configuration
const remEnabled = process.env['VESTIGE_REM_ENABLED'];
if (remEnabled) {
const enabled = remEnabled.toLowerCase() === 'true';
env.rem = { enabled };
}
// Consolidation configuration
const consolidationEnabled = process.env['VESTIGE_CONSOLIDATION_ENABLED'];
if (consolidationEnabled) {
const enabled = consolidationEnabled.toLowerCase() === 'true';
env.consolidation = { enabled };
}
return env;
}
/**
* Deep merge two objects, with source taking precedence
*/
function deepMerge<T extends Record<string, unknown>>(target: T, source: Partial<T>): T {
const result = { ...target };
for (const key of Object.keys(source) as (keyof T)[]) {
const sourceValue = source[key];
const targetValue = result[key];
if (
sourceValue !== undefined &&
typeof sourceValue === 'object' &&
sourceValue !== null &&
!Array.isArray(sourceValue) &&
typeof targetValue === 'object' &&
targetValue !== null &&
!Array.isArray(targetValue)
) {
result[key] = deepMerge(
targetValue as Record<string, unknown>,
sourceValue as Record<string, unknown>
) as T[keyof T];
} else if (sourceValue !== undefined) {
result[key] = sourceValue as T[keyof T];
}
}
return result;
}
/**
* Load configuration from file and environment variables
*
* @param customPath - Optional custom path to config file
* @returns Validated configuration object
*/
export function loadConfig(customPath?: string): VestigeConfig {
if (config) return config;
const configPath = customPath || path.join(os.homedir(), '.vestige', 'config.json');
let fileConfig: Record<string, unknown> = {};
// Load from file if it exists
if (fs.existsSync(configPath)) {
try {
const content = fs.readFileSync(configPath, 'utf-8');
fileConfig = JSON.parse(content) as Record<string, unknown>;
} catch (error) {
console.warn(`Failed to load config from ${configPath}:`, error);
}
}
// Load environment variable overrides
const envConfig = loadEnvConfig();
// Merge configs: file config first, then env overrides
const mergedConfig = deepMerge(fileConfig, envConfig as Record<string, unknown>);
// Validate and parse with Zod (applies defaults)
config = ConfigSchema.parse(mergedConfig);
return config;
}
/**
* Get the current configuration, loading it if necessary
*
* @returns The current configuration object
*/
export function getConfig(): VestigeConfig {
if (!config) {
return loadConfig();
}
return config;
}
/**
* Reset the configuration singleton (useful for testing)
*/
export function resetConfig(): void {
config = null;
}
// ============================================================================
// CONFIGURATION ACCESSORS
// ============================================================================
/**
* Get database configuration
*/
export const getDatabaseConfig = () => getConfig().database;
/**
* Get FSRS algorithm configuration
*/
export const getFSRSConfig = () => getConfig().fsrs;
/**
* Get memory model configuration
*/
export const getMemoryConfig = () => getConfig().memory;
/**
* Get sentiment analysis configuration
*/
export const getSentimentConfig = () => getConfig().sentiment;
/**
* Get REM cycle configuration
*/
export const getREMConfig = () => getConfig().rem;
/**
* Get consolidation configuration
*/
export const getConsolidationConfig = () => getConfig().consolidation;
/**
* Get embeddings service configuration
*/
export const getEmbeddingsConfig = () => getConfig().embeddings;
/**
* Get vector store configuration
*/
export const getVectorStoreConfig = () => getConfig().vectorStore;
/**
* Get cache configuration
*/
export const getCacheConfig = () => getConfig().cache;
/**
* Get logging configuration
*/
export const getLoggingConfig = () => getConfig().logging;
/**
* Get limits configuration
*/
export const getLimitsConfig = () => getConfig().limits;
// ============================================================================
// CONFIGURATION VALIDATION
// ============================================================================
/**
* Validate an unknown config object against the schema
*
* @param configObj - Unknown object to validate
* @returns Validated configuration object
* @throws ZodError if validation fails
*/
export function validateConfig(configObj: unknown): VestigeConfig {
return ConfigSchema.parse(configObj);
}
/**
* Get the Zod schema for configuration validation
*
* @returns The Zod configuration schema
*/
export function getConfigSchema() {
return ConfigSchema;
}
// ============================================================================
// EXPORTS
// ============================================================================
// Export individual schemas for external use
export {
ConfigSchema,
DatabaseConfigSchema,
ConfigFSRSSchema,
MemoryConfigSchema,
SentimentConfigSchema,
REMConfigSchema,
ConsolidationConfigSchema,
EmbeddingsConfigSchema,
VectorStoreConfigSchema,
CacheConfigSchema,
LoggingConfigSchema,
LimitsConfigSchema,
};

409
packages/core/src/core/consolidation.ts
View file

@ -1,409 +0,0 @@
/**
* Sleep Consolidation Simulation
*
* "The brain that consolidates while you sleep."
*
* This module simulates how the human brain consolidates memories during sleep.
* Based on cognitive science research on memory consolidation, it implements:
*
* KEY FEATURES:
* 1. Short-term Memory Processing - Identifies recent memories for consolidation
* 2. Importance-based Promotion - Promotes significant memories to long-term storage
* 3. REM Cycle Integration - Discovers new connections via semantic analysis
* 4. Synaptic Homeostasis - Prunes weak connections to prevent memory overload
* 5. Decay Application - Applies natural memory decay based on forgetting curve
*
* COGNITIVE SCIENCE BASIS:
* - Active Systems Consolidation: Hippocampus replays memories during sleep
* - Synaptic Homeostasis Hypothesis: Weak connections are pruned during sleep
* - Emotional Memory Enhancement: Emotional memories are preferentially consolidated
* - Spreading Activation: Related memories are co-activated and strengthened
*/
import { VestigeDatabase } from './database.js';
import { runREMCycle } from './rem-cycle.js';
import type { KnowledgeNode } from './types.js';
import { logger } from '../utils/logger.js';
// ============================================================================
// TYPES
// ============================================================================
export interface ConsolidationResult {
/** Number of short-term memories processed */
shortTermProcessed: number;
/** Number of memories promoted to long-term storage */
promotedToLongTerm: number;
/** Number of new connections discovered via REM cycle */
connectionsDiscovered: number;
/** Number of weak edges pruned (synaptic homeostasis) */
edgesPruned: number;
/** Number of memories that had decay applied */
decayApplied: number;
/** Duration of consolidation cycle in milliseconds */
duration: number;
}
export interface ConsolidationOptions {
/** Hours to look back for short-term memories. Default: 24 */
shortTermWindowHours?: number;
/** Minimum importance score to promote to long-term. Default: 0.5 */
importanceThreshold?: number;
/** Edge weight below which connections are pruned. Default: 0.2 */
pruneThreshold?: number;
/** Maximum number of memories to analyze in REM cycle. Default: 100 */
maxAnalyze?: number;
}
// ============================================================================
// CONSTANTS
// ============================================================================
/** Default short-term memory window (24 hours) */
const DEFAULT_SHORT_TERM_WINDOW_HOURS = 24;
/** Default importance threshold for long-term promotion */
const DEFAULT_IMPORTANCE_THRESHOLD = 0.5;
/** Default edge weight threshold for pruning */
const DEFAULT_PRUNE_THRESHOLD = 0.2;
/** Default max memories to analyze */
const DEFAULT_MAX_ANALYZE = 100;
/** Weight factors for importance calculation */
const EMOTION_WEIGHT = 0.4;
const ACCESS_WEIGHT = 0.3;
const CONNECTION_WEIGHT = 0.3;
/** Maximum values for normalization */
const MAX_ACCESSES_FOR_IMPORTANCE = 5;
const MAX_CONNECTIONS_FOR_IMPORTANCE = 5;
// ============================================================================
// HELPER FUNCTIONS
// ============================================================================
/**
* Get memories created within the short-term window
* These are candidates for consolidation processing
*/
async function getShortTermMemories(
db: VestigeDatabase,
windowHours: number
): Promise<KnowledgeNode[]> {
const windowStart = new Date(Date.now() - windowHours * 60 * 60 * 1000);
const recentNodes = db.getRecentNodes({ limit: 500 }).items;
return recentNodes.filter(node => node.createdAt >= windowStart);
}
/**
* Calculate importance score for a memory
*
* Importance = f(emotion, access_count, connection_count)
*
* The formula weights three factors:
* - Emotional intensity (40%): Emotionally charged memories are more important
* - Access count (30%): Frequently accessed memories are more important
* - Connection count (30%): Well-connected memories are more important
*
* @returns Importance score from 0 to 1
*/
function calculateImportance(db: VestigeDatabase, memory: KnowledgeNode): number {
// Get connection count for this memory
const connections = db.getRelatedNodes(memory.id, 1).length;
// Get emotional intensity (0 to 1)
const emotion = memory.sentimentIntensity || 0;
// Get access count
const accesses = memory.accessCount;
// Weighted importance formula
// Each component is normalized to 0-1 range
const emotionScore = emotion * EMOTION_WEIGHT;
const accessScore =
(Math.min(MAX_ACCESSES_FOR_IMPORTANCE, accesses) / MAX_ACCESSES_FOR_IMPORTANCE) *
ACCESS_WEIGHT;
const connectionScore =
(Math.min(MAX_CONNECTIONS_FOR_IMPORTANCE, connections) / MAX_CONNECTIONS_FOR_IMPORTANCE) *
CONNECTION_WEIGHT;
const importanceScore = emotionScore + accessScore + connectionScore;
return importanceScore;
}
/**
* Promote a memory to long-term storage
*
* This boosts the storage strength proportional to importance.
* Based on the Dual-Strength Memory Model (Bjork & Bjork, 1992),
* storage strength represents how well the memory is encoded.
*
* Boost factor ranges from 1x (importance=0) to 3x (importance=1)
*/
async function promoteToLongTerm(
db: VestigeDatabase,
nodeId: string,
importance: number
): Promise<void> {
// Calculate boost factor: 1x to 3x based on importance
const boost = 1 + importance * 2;
// Access the internal database connection
// Note: This uses internal access pattern for direct SQL operations
const internalDb = (db as unknown as { db: { prepare: (sql: string) => { run: (...args: unknown[]) => void } } }).db;
internalDb
.prepare(
`
UPDATE knowledge_nodes
SET storage_strength = storage_strength * ?,
stability_factor = stability_factor * ?
WHERE id = ?
`
)
.run(boost, boost, nodeId);
}
/**
* Prune weak connections discovered by REM cycle
*
* This implements synaptic homeostasis - the brain's process of
* removing weak synaptic connections during sleep to:
* 1. Prevent memory overload
* 2. Improve signal-to-noise ratio
* 3. Conserve metabolic resources
*
* Only auto-discovered connections (from REM cycle) are pruned.
* User-created connections are preserved regardless of weight.
*/
async function pruneWeakConnections(
db: VestigeDatabase,
threshold: number
): Promise<number> {
// Access the internal database connection
const internalDb = (db as unknown as { db: { prepare: (sql: string) => { run: (...args: unknown[]) => { changes: number } } } }).db;
// Remove edges below threshold that were auto-discovered by REM cycle
const result = internalDb
.prepare(
`
DELETE FROM graph_edges
WHERE weight < ?
AND json_extract(metadata, '$.discoveredBy') = 'rem_cycle'
`
)
.run(threshold);
return result.changes;
}
// ============================================================================
// MAIN CONSOLIDATION FUNCTION
// ============================================================================
/**
* Run Sleep Consolidation Simulation
*
* Based on cognitive science research on memory consolidation:
*
* PHASE 1: Identify short-term memories
* - Collect memories created within the specified window
* - These represent the "inbox" of memories to process
*
* PHASE 2: Calculate importance and promote
* - Score each memory based on emotion, access, connections
* - Memories above threshold are "promoted" (strengthened)
* - This simulates hippocampal replay during sleep
*
* PHASE 3: Run REM cycle for connection discovery
* - Analyze memories for semantic similarity
* - Discover new connections between related memories
* - Apply spreading activation for transitive connections
*
* PHASE 4: Prune weak connections (synaptic homeostasis)
* - Remove auto-discovered edges below weight threshold
* - Preserves signal-to-noise ratio in memory network
*
* PHASE 5: Apply decay to all memories
* - Apply Ebbinghaus forgetting curve
* - Emotional memories decay slower
* - Well-encoded memories (high storage strength) decay slower
*
* @param db - VestigeDatabase instance
* @param options - Consolidation configuration options
* @returns Results of the consolidation cycle
*/
export async function runConsolidation(
db: VestigeDatabase,
options: ConsolidationOptions = {}
): Promise<ConsolidationResult> {
const startTime = Date.now();
const {
shortTermWindowHours = DEFAULT_SHORT_TERM_WINDOW_HOURS,
importanceThreshold = DEFAULT_IMPORTANCE_THRESHOLD,
pruneThreshold = DEFAULT_PRUNE_THRESHOLD,
maxAnalyze = DEFAULT_MAX_ANALYZE,
} = options;
const result: ConsolidationResult = {
shortTermProcessed: 0,
promotedToLongTerm: 0,
connectionsDiscovered: 0,
edgesPruned: 0,
decayApplied: 0,
duration: 0,
};
logger.info('Starting consolidation cycle', {
shortTermWindowHours,
importanceThreshold,
pruneThreshold,
maxAnalyze,
});
// PHASE 1: Identify short-term memories
// These are memories created within the window that need processing
const shortTermMemories = await getShortTermMemories(db, shortTermWindowHours);
result.shortTermProcessed = shortTermMemories.length;
logger.debug('Phase 1: Identified short-term memories', {
count: shortTermMemories.length,
});
// PHASE 2: Calculate importance and promote to long-term
// This simulates the hippocampal replay that occurs during sleep
for (const memory of shortTermMemories) {
const importance = calculateImportance(db, memory);
if (importance >= importanceThreshold) {
await promoteToLongTerm(db, memory.id, importance);
result.promotedToLongTerm++;
}
}
logger.debug('Phase 2: Promoted memories to long-term storage', {
promoted: result.promotedToLongTerm,
threshold: importanceThreshold,
});
// PHASE 3: Run REM cycle for connection discovery
// This discovers semantic connections between memories
const remResult = await runREMCycle(db, { maxAnalyze });
result.connectionsDiscovered = remResult.connectionsCreated;
logger.debug('Phase 3: REM cycle complete', {
connectionsDiscovered: remResult.connectionsDiscovered,
connectionsCreated: remResult.connectionsCreated,
spreadingActivationEdges: remResult.spreadingActivationEdges,
});
// PHASE 4: Prune weak connections (synaptic homeostasis)
// Remove auto-discovered connections that are below the threshold
result.edgesPruned = await pruneWeakConnections(db, pruneThreshold);
logger.debug('Phase 4: Pruned weak connections', {
edgesPruned: result.edgesPruned,
threshold: pruneThreshold,
});
// PHASE 5: Apply decay to all memories
// Uses Ebbinghaus forgetting curve with emotional weighting
result.decayApplied = db.applyDecay();
logger.debug('Phase 5: Applied memory decay', {
memoriesAffected: result.decayApplied,
});
result.duration = Date.now() - startTime;
logger.info('Consolidation cycle complete', { ...result });
return result;
}
// ============================================================================
// SCHEDULING HELPER
// ============================================================================
/**
* Get recommended next consolidation time
*
* Returns the next occurrence of 3 AM local time.
* This is based on research showing that:
* - Deep sleep (when consolidation occurs) typically happens 3-4 AM
* - System resources are usually free at this time
* - Users are unlikely to be actively using the system
*
* @returns Date object representing the next recommended consolidation time
*/
export function getNextConsolidationTime(): Date {
const now = new Date();
const next = new Date(now);
// Schedule for 3 AM next day
next.setDate(next.getDate() + 1);
next.setHours(3, 0, 0, 0);
return next;
}
/**
* Preview consolidation results without making changes
*
* Useful for understanding what would happen during consolidation
* without actually modifying the database.
*
* Note: This still runs the analysis phases but skips the
* actual modification phases.
*/
export async function previewConsolidation(
db: VestigeDatabase,
options: ConsolidationOptions = {}
): Promise<{
shortTermCount: number;
wouldPromote: number;
potentialConnections: number;
weakEdgeCount: number;
}> {
const {
shortTermWindowHours = DEFAULT_SHORT_TERM_WINDOW_HOURS,
importanceThreshold = DEFAULT_IMPORTANCE_THRESHOLD,
pruneThreshold = DEFAULT_PRUNE_THRESHOLD,
maxAnalyze = DEFAULT_MAX_ANALYZE,
} = options;
// Get short-term memories
const shortTermMemories = await getShortTermMemories(db, shortTermWindowHours);
// Count how many would be promoted
let wouldPromote = 0;
for (const memory of shortTermMemories) {
const importance = calculateImportance(db, memory);
if (importance >= importanceThreshold) {
wouldPromote++;
}
}
// Preview REM cycle (dry run)
const remPreview = await runREMCycle(db, { maxAnalyze, dryRun: true });
// Count weak edges that would be pruned
const internalDb = (db as unknown as { db: { prepare: (sql: string) => { get: (...args: unknown[]) => { count: number } } } }).db;
const weakEdgeResult = internalDb
.prepare(
`
SELECT COUNT(*) as count FROM graph_edges
WHERE weight < ?
AND json_extract(metadata, '$.discoveredBy') = 'rem_cycle'
`
)
.get(pruneThreshold) as { count: number };
return {
shortTermCount: shortTermMemories.length,
wouldPromote,
potentialConnections: remPreview.connectionsDiscovered,
weakEdgeCount: weakEdgeResult.count,
};
}

270
packages/core/src/core/context-watcher.ts
View file

@ -1,270 +0,0 @@
/**
* Ghost in the Shell - Context Watcher
*
* Watches the active window and clipboard to provide contextual awareness.
* Vestige sees what you see.
*
* Features:
* - Active window title detection (macOS via AppleScript)
* - Clipboard monitoring
* - Context file for MCP injection
*/
import { execSync } from 'child_process';
import fs from 'fs';
import path from 'path';
import os from 'os';
// ============================================================================
// TYPES
// ============================================================================
export interface SystemContext {
timestamp: string;
activeWindow: {
app: string;
title: string;
} | null;
clipboard: string | null;
workingDirectory: string;
gitBranch: string | null;
recentFiles: string[];
}
// ============================================================================
// CONTEXT FILE LOCATION
// ============================================================================
const CONTEXT_FILE = path.join(os.homedir(), '.vestige', 'context.json');
// ============================================================================
// PLATFORM-SPECIFIC IMPLEMENTATIONS
// ============================================================================
/**
* Get active window info on macOS using AppleScript
*/
function getActiveWindowMac(): { app: string; title: string } | null {
try {
// Get frontmost app name
const appScript = `
tell application "System Events"
set frontApp to first application process whose frontmost is true
return name of frontApp
end tell
`;
const app = execSync(`osascript -e '${appScript}'`, {
encoding: 'utf-8',
stdio: ['pipe', 'pipe', 'pipe'],
}).trim();
// Get window title
const titleScript = `
tell application "System Events"
tell (first application process whose frontmost is true)
if (count of windows) > 0 then
return name of front window
else
return ""
end if
end tell
end tell
`;
const title = execSync(`osascript -e '${titleScript}'`, {
encoding: 'utf-8',
stdio: ['pipe', 'pipe', 'pipe'],
}).trim();
return { app, title };
} catch {
return null;
}
}
/**
* Get clipboard content on macOS
*/
function getClipboardMac(): string | null {
try {
const content = execSync('pbpaste', {
encoding: 'utf-8',
stdio: ['pipe', 'pipe', 'pipe'],
maxBuffer: 1024 * 100, // 100KB max
});
// Truncate long clipboard content
if (content.length > 2000) {
return content.slice(0, 2000) + '\n... [truncated]';
}
return content || null;
} catch {
return null;
}
}
/**
* Get current git branch
*/
function getGitBranch(): string | null {
try {
return execSync('git rev-parse --abbrev-ref HEAD', {
encoding: 'utf-8',
stdio: ['pipe', 'pipe', 'pipe'],
}).trim();
} catch {
return null;
}
}
/**
* Get recently modified files in current directory
*/
function getRecentFiles(): string[] {
try {
// Get files modified in last hour, sorted by time
const result = execSync(
'find . -type f -mmin -60 -not -path "*/node_modules/*" -not -path "*/.git/*" -not -path "*/dist/*" 2>/dev/null | head -10',
{
encoding: 'utf-8',
stdio: ['pipe', 'pipe', 'pipe'],
}
);
return result
.split('\n')
.map(f => f.trim())
.filter(Boolean)
.slice(0, 10);
} catch {
return [];
}
}
// ============================================================================
// CONTEXT CAPTURE
// ============================================================================
/**
* Capture current system context
*/
export function captureContext(): SystemContext {
const platform = process.platform;
let activeWindow: { app: string; title: string } | null = null;
let clipboard: string | null = null;
if (platform === 'darwin') {
activeWindow = getActiveWindowMac();
clipboard = getClipboardMac();
}
// TODO: Add Windows and Linux support
return {
timestamp: new Date().toISOString(),
activeWindow,
clipboard,
workingDirectory: process.cwd(),
gitBranch: getGitBranch(),
recentFiles: getRecentFiles(),
};
}
/**
* Format context for injection into Claude prompts
*/
export function formatContextForInjection(context: SystemContext): string {
const parts: string[] = [];
if (context.activeWindow) {
parts.push(`Active: ${context.activeWindow.app} - ${context.activeWindow.title}`);
}
if (context.gitBranch) {
parts.push(`Git: ${context.gitBranch}`);
}
if (context.recentFiles.length > 0) {
parts.push(`Recent: ${context.recentFiles.slice(0, 3).join(', ')}`);
}
if (context.clipboard && context.clipboard.length < 500) {
parts.push(`Clipboard: "${context.clipboard.slice(0, 200)}${context.clipboard.length > 200 ? '...' : ''}"`);
}
return parts.join(' | ');
}
/**
* Save context to file for external consumption
*/
export function saveContext(context: SystemContext): void {
try {
const dir = path.dirname(CONTEXT_FILE);
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir, { recursive: true });
}
fs.writeFileSync(CONTEXT_FILE, JSON.stringify(context, null, 2));
} catch {
// Ignore file write errors
}
}
/**
* Read saved context from file
*/
export function readSavedContext(): SystemContext | null {
try {
if (fs.existsSync(CONTEXT_FILE)) {
const content = fs.readFileSync(CONTEXT_FILE, 'utf-8');
return JSON.parse(content) as SystemContext;
}
} catch {
// Ignore read errors
}
return null;
}
// ============================================================================
// WATCHER DAEMON
// ============================================================================
let watcherInterval: NodeJS.Timeout | null = null;
/**
* Start the context watcher daemon
* Updates context every N seconds
*/
export function startContextWatcher(intervalMs: number = 5000): void {
if (watcherInterval) {
console.log('Context watcher already running');
return;
}
console.log(`Starting context watcher (interval: ${intervalMs}ms)`);
// Capture immediately
const context = captureContext();
saveContext(context);
// Then update periodically
watcherInterval = setInterval(() => {
const ctx = captureContext();
saveContext(ctx);
}, intervalMs);
}
/**
* Stop the context watcher daemon
*/
export function stopContextWatcher(): void {
if (watcherInterval) {
clearInterval(watcherInterval);
watcherInterval = null;
console.log('Context watcher stopped');
}
}
/**
* Check if watcher is running
*/
export function isWatcherRunning(): boolean {
return watcherInterval !== null;
}

1975
packages/core/src/core/database.ts
View file

File diff suppressed because it is too large Load diff

788
packages/core/src/core/embeddings.ts
View file

@ -1,788 +0,0 @@
/**
* Embeddings Service - Semantic Understanding for Vestige
*
* Provides vector embeddings for knowledge nodes using Ollama.
* Embeddings enable semantic similarity search and connection discovery.
*
* Features:
* - Ollama integration with nomic-embed-text model (768-dim, fast, high quality)
* - Graceful fallback to TF-IDF when Ollama unavailable
* - Availability caching to reduce connection overhead
* - Batch embedding support for efficiency
* - Utility functions for similarity search
*/
import { Ollama } from 'ollama';
// ============================================================================
// CONFIGURATION
// ============================================================================
/**
* Ollama API endpoint. Defaults to local installation.
*/
const OLLAMA_HOST = process.env['OLLAMA_HOST'] || 'http://localhost:11434';
/**
* Embedding model to use. nomic-embed-text provides:
* - 768 dimensions
* - Fast inference
* - High quality embeddings for semantic search
* - 8192 token context window
*/
const EMBEDDING_MODEL = process.env['VESTIGE_EMBEDDING_MODEL'] || 'nomic-embed-text';
/**
* Maximum characters to embed. nomic-embed-text supports ~8192 tokens,
* but we truncate to 8000 chars for safety margin.
*/
const MAX_TEXT_LENGTH = 8000;
/**
* Cache duration for availability check (5 minutes in ms)
*/
const AVAILABILITY_CACHE_TTL = 5 * 60 * 1000;
/**
* Default request timeout in milliseconds
*/
const DEFAULT_TIMEOUT = 30000;
// ============================================================================
// INTERFACES
// ============================================================================
/**
* Service interface for generating and comparing text embeddings.
* Provides semantic similarity capabilities for knowledge retrieval.
*/
export interface EmbeddingService {
/**
* Generate an embedding vector for the given text.
* @param text - The text to embed
* @returns A promise resolving to a numeric vector
*/
generateEmbedding(text: string): Promise<number[]>;
/**
* Generate embeddings for multiple texts in a single batch.
* More efficient than calling generateEmbedding multiple times.
* @param texts - Array of texts to embed
* @returns A promise resolving to an array of embedding vectors
*/
batchEmbeddings(texts: string[]): Promise<number[][]>;
/**
* Calculate similarity between two embedding vectors.
* @param embA - First embedding vector
* @param embB - Second embedding vector
* @returns Similarity score between 0 and 1
*/
getSimilarity(embA: number[], embB: number[]): number;
/**
* Check if the embedding service is available and ready.
* @returns A promise resolving to true if the service is available
*/
isAvailable(): Promise<boolean>;
}
/**
* Configuration options for embedding services.
*/
export interface EmbeddingServiceConfig {
/** Ollama host URL (default: http://localhost:11434) */
host?: string;
/** Embedding model to use (default: nomic-embed-text) */
model?: string;
/** Request timeout in milliseconds (default: 30000) */
timeout?: number;
}
/**
* Result from embedding generation with metadata.
*/
export interface EmbeddingResult {
embedding: number[];
model: string;
dimension: number;
}
// ============================================================================
// COSINE SIMILARITY
// ============================================================================
/**
* Calculate cosine similarity between two vectors.
* Returns a value between -1 and 1, where:
* - 1 means identical direction
* - 0 means orthogonal (unrelated)
* - -1 means opposite direction
*
* @param a - First vector
* @param b - Second vector
* @returns Cosine similarity score
* @throws Error if vectors have different lengths or are empty
*/
export function cosineSimilarity(a: number[], b: number[]): number {
if (a.length === 0 || b.length === 0) {
throw new Error('Cannot compute cosine similarity of empty vectors');
}
if (a.length !== b.length) {
throw new Error(
`Vector dimension mismatch: ${a.length} vs ${b.length}`
);
}
let dotProduct = 0;
let normA = 0;
let normB = 0;
for (let i = 0; i < a.length; i++) {
const aVal = a[i]!;
const bVal = b[i]!;
dotProduct += aVal * bVal;
normA += aVal * aVal;
normB += bVal * bVal;
}
const magnitude = Math.sqrt(normA) * Math.sqrt(normB);
if (magnitude === 0) {
return 0;
}
return dotProduct / magnitude;
}
/**
* Normalize cosine similarity from [-1, 1] to [0, 1] range.
* Useful when you need a percentage-like similarity score.
*
* @param similarity - Cosine similarity value
* @returns Normalized similarity between 0 and 1
*/
export function normalizedSimilarity(similarity: number): number {
return (similarity + 1) / 2;
}
/**
* Calculate Euclidean distance between two vectors.
*
* @param a - First vector
* @param b - Second vector
* @returns Euclidean distance (lower = more similar)
*/
export function euclideanDistance(a: number[], b: number[]): number {
if (a.length !== b.length) {
throw new Error(`Vector dimension mismatch: ${a.length} vs ${b.length}`);
}
let sum = 0;
for (let i = 0; i < a.length; i++) {
const diff = a[i]! - b[i]!;
sum += diff * diff;
}
return Math.sqrt(sum);
}
// ============================================================================
// OLLAMA EMBEDDING SERVICE
// ============================================================================
/**
* Production embedding service using Ollama with nomic-embed-text model.
* Provides high-quality semantic embeddings for knowledge retrieval.
*
* Features:
* - Automatic text truncation for long inputs
* - Availability caching to reduce connection overhead
* - Graceful error handling with informative messages
* - Batch embedding support for efficiency
*/
export class OllamaEmbeddingService implements EmbeddingService {
private client: Ollama;
private availabilityCache: { available: boolean; timestamp: number } | null = null;
private readonly model: string;
private readonly timeout: number;
constructor(config: EmbeddingServiceConfig = {}) {
const {
host = OLLAMA_HOST,
model = EMBEDDING_MODEL,
timeout = DEFAULT_TIMEOUT,
} = config;
this.client = new Ollama({ host });
this.model = model;
this.timeout = timeout;
}
/**
* Check if Ollama is running and the embedding model is available.
* Results are cached for 5 minutes to reduce overhead.
*/
async isAvailable(): Promise<boolean> {
// Check cache first
if (
this.availabilityCache &&
Date.now() - this.availabilityCache.timestamp < AVAILABILITY_CACHE_TTL
) {
return this.availabilityCache.available;
}
try {
// Try to list models to verify connection with timeout
const response = await Promise.race([
this.client.list(),
new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error('Timeout')), this.timeout)
),
]);
const modelNames = response.models.map((m) => m.name);
// Check if our model is available (handle both "model" and "model:latest" formats)
const modelBase = this.model.split(':')[0];
const available = modelNames.some(
(name) => name === this.model ||
name.startsWith(`${this.model}:`) ||
name.split(':')[0] === modelBase
);
if (!available) {
console.warn(
`Ollama is running but model '${this.model}' not found. ` +
`Available models: ${modelNames.join(', ') || 'none'}. ` +
`Run 'ollama pull ${this.model}' to install.`
);
}
this.availabilityCache = { available, timestamp: Date.now() };
return available;
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.warn(`Ollama not available: ${message}`);
this.availabilityCache = { available: false, timestamp: Date.now() };
return false;
}
}
/**
* Truncate text to fit within the model's context window.
*/
private truncateText(text: string): string {
if (text.length <= MAX_TEXT_LENGTH) {
return text;
}
console.warn(
`Text truncated from ${text.length} to ${MAX_TEXT_LENGTH} characters`
);
return text.slice(0, MAX_TEXT_LENGTH);
}
/**
* Generate an embedding for the given text.
*/
async generateEmbedding(text: string): Promise<number[]> {
if (!text || text.trim().length === 0) {
throw new Error('Cannot generate embedding for empty text');
}
const truncatedText = this.truncateText(text.trim());
try {
const response = await Promise.race([
this.client.embed({
model: this.model,
input: truncatedText,
}),
new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error('Embedding timeout')), this.timeout)
),
]);
// Response contains array of embeddings, we want the first one
if (!response.embeddings || response.embeddings.length === 0) {
throw new Error('No embeddings returned from Ollama');
}
const embedding = response.embeddings[0];
if (!embedding) {
throw new Error('No embedding returned from Ollama');
}
return embedding;
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
throw new Error(`Failed to generate embedding: ${message}`);
}
}
/**
* Generate embeddings for multiple texts in a batch.
* More efficient than individual calls for bulk operations.
*/
async batchEmbeddings(texts: string[]): Promise<number[][]> {
if (texts.length === 0) {
return [];
}
// Filter and truncate texts
const validTexts = texts
.filter((t) => t && t.trim().length > 0)
.map((t) => this.truncateText(t.trim()));
if (validTexts.length === 0) {
return [];
}
try {
const response = await Promise.race([
this.client.embed({
model: this.model,
input: validTexts,
}),
new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error('Batch embedding timeout')), this.timeout * 2)
),
]);
if (!response.embeddings || response.embeddings.length !== validTexts.length) {
throw new Error(
`Expected ${validTexts.length} embeddings, got ${response.embeddings?.length ?? 0}`
);
}
return response.embeddings;
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
throw new Error(`Failed to generate batch embeddings: ${message}`);
}
}
/**
* Calculate similarity between two embedding vectors using cosine similarity.
*/
getSimilarity(embA: number[], embB: number[]): number {
return cosineSimilarity(embA, embB);
}
/**
* Get the model being used.
*/
getModel(): string {
return this.model;
}
/**
* Clear the availability cache, forcing a fresh check on next call.
*/
clearCache(): void {
this.availabilityCache = null;
}
}
// ============================================================================
// FALLBACK EMBEDDING SERVICE
// ============================================================================
/**
* Default vocabulary size for fallback TF-IDF style embeddings.
*/
const DEFAULT_VOCAB_SIZE = 512;
/**
* Fallback embedding service using TF-IDF style word frequency vectors.
* Used when Ollama is not available. Provides basic keyword-based
* similarity that works offline with no dependencies.
*
* Limitations compared to Ollama:
* - No semantic understanding (only keyword matching)
* - Fixed vocabulary may miss domain-specific terms
* - Lower quality similarity scores
*/
export class FallbackEmbeddingService implements EmbeddingService {
private readonly dimensions: number;
private readonly vocabulary: Map<string, number>;
private documentFrequency: Map<string, number>;
private documentCount: number;
constructor(vocabSize: number = DEFAULT_VOCAB_SIZE) {
this.dimensions = vocabSize;
this.vocabulary = new Map();
this.documentFrequency = new Map();
this.documentCount = 0;
}
/**
* Fallback service is always available (runs locally with no dependencies).
*/
async isAvailable(): Promise<boolean> {
return true;
}
/**
* Tokenize text into normalized words.
*/
private tokenize(text: string): string[] {
return text
.toLowerCase()
.replace(/[^\w\s]/g, ' ')
.split(/\s+/)
.filter((word) => word.length > 2 && word.length < 30);
}
/**
* Get or assign a vocabulary index for a word.
* Uses hash-based assignment for consistent but bounded vocabulary.
*/
private getWordIndex(word: string): number {
if (this.vocabulary.has(word)) {
return this.vocabulary.get(word)!;
}
// Simple hash function for consistent word-to-index mapping
let hash = 0;
for (let i = 0; i < word.length; i++) {
const char = word.charCodeAt(i);
hash = ((hash << 5) - hash + char) | 0;
}
const index = Math.abs(hash) % this.dimensions;
this.vocabulary.set(word, index);
return index;
}
/**
* Generate a TF-IDF style embedding vector.
* Uses term frequency weighted by inverse document frequency approximation.
*/
async generateEmbedding(text: string): Promise<number[]> {
if (!text || text.trim().length === 0) {
throw new Error('Cannot generate embedding for empty text');
}
const tokens = this.tokenize(text);
if (tokens.length === 0) {
// Return zero vector for text with no valid tokens
return new Array(this.dimensions).fill(0);
}
// Calculate term frequency
const termFreq = new Map<string, number>();
for (const token of tokens) {
termFreq.set(token, (termFreq.get(token) || 0) + 1);
}
// Update document frequency for IDF
this.documentCount++;
const seenWords = new Set<string>();
for (const token of tokens) {
if (!seenWords.has(token)) {
this.documentFrequency.set(
token,
(this.documentFrequency.get(token) || 0) + 1
);
seenWords.add(token);
}
}
// Build embedding vector
const embedding = new Array(this.dimensions).fill(0);
const maxFreq = Math.max(...termFreq.values());
for (const [word, freq] of termFreq) {
const index = this.getWordIndex(word);
// TF: normalized term frequency (prevents bias towards long documents)
const tf = freq / maxFreq;
// IDF: inverse document frequency (common words get lower weight)
const df = this.documentFrequency.get(word) || 1;
const idf = Math.log((this.documentCount + 1) / (df + 1)) + 1;
// TF-IDF score (may have collisions, add to handle)
embedding[index] += tf * idf;
}
// L2 normalize the vector
const norm = Math.sqrt(embedding.reduce((sum, val) => sum + val * val, 0));
if (norm > 0) {
for (let i = 0; i < embedding.length; i++) {
embedding[i] /= norm;
}
}
return embedding;
}
/**
* Generate embeddings for multiple texts.
*/
async batchEmbeddings(texts: string[]): Promise<number[][]> {
const embeddings: number[][] = [];
for (const text of texts) {
if (text && text.trim().length > 0) {
embeddings.push(await this.generateEmbedding(text));
}
}
return embeddings;
}
/**
* Calculate similarity between two embedding vectors.
*/
getSimilarity(embA: number[], embB: number[]): number {
return cosineSimilarity(embA, embB);
}
/**
* Reset the document frequency statistics.
* Useful when starting fresh with a new corpus.
*/
reset(): void {
this.vocabulary.clear();
this.documentFrequency.clear();
this.documentCount = 0;
}
/**
* Get the dimensionality of embeddings.
*/
getDimensions(): number {
return this.dimensions;
}
}
// ============================================================================
// EMBEDDING CACHE
// ============================================================================
/**
* Simple in-memory cache for embeddings.
* Reduces redundant API calls during REM cycles.
*/
export class EmbeddingCache {
private cache: Map<string, { embedding: number[]; timestamp: number }> = new Map();
private maxSize: number;
private ttlMs: number;
constructor(maxSize: number = 1000, ttlMinutes: number = 60) {
this.maxSize = maxSize;
this.ttlMs = ttlMinutes * 60 * 1000;
}
/**
* Get a cached embedding by node ID.
*/
get(nodeId: string): number[] | null {
const entry = this.cache.get(nodeId);
if (!entry) return null;
// Check if expired
if (Date.now() - entry.timestamp > this.ttlMs) {
this.cache.delete(nodeId);
return null;
}
return entry.embedding;
}
/**
* Cache an embedding for a node ID.
*/
set(nodeId: string, embedding: number[]): void {
// Evict oldest if at capacity
if (this.cache.size >= this.maxSize) {
const oldestKey = this.cache.keys().next().value;
if (oldestKey) {
this.cache.delete(oldestKey);
}
}
this.cache.set(nodeId, {
embedding,
timestamp: Date.now(),
});
}
/**
* Check if a node ID has a cached embedding.
*/
has(nodeId: string): boolean {
return this.get(nodeId) !== null;
}
/**
* Clear all cached embeddings.
*/
clear(): void {
this.cache.clear();
}
/**
* Get the number of cached embeddings.
*/
size(): number {
return this.cache.size;
}
}
// ============================================================================
// FACTORY FUNCTIONS
// ============================================================================
let defaultService: EmbeddingService | null = null;
/**
* Get the default embedding service (singleton).
* Uses cached instance for efficiency.
*/
export function getEmbeddingService(config?: EmbeddingServiceConfig): EmbeddingService {
if (!defaultService) {
defaultService = new OllamaEmbeddingService(config);
}
return defaultService;
}
/**
* Create an embedding service with automatic fallback.
*
* Attempts to use Ollama with nomic-embed-text for high-quality semantic
* embeddings. Falls back to TF-IDF based keyword similarity if Ollama
* is not available.
*
* @param config - Optional configuration for the Ollama service
* @returns A promise resolving to an EmbeddingService instance
*
* @example
* ```typescript
* const embeddings = await createEmbeddingService();
*
* const vec1 = await embeddings.generateEmbedding("TypeScript is great");
* const vec2 = await embeddings.generateEmbedding("JavaScript is popular");
*
* const similarity = embeddings.getSimilarity(vec1, vec2);
* console.log(`Similarity: ${similarity}`);
* ```
*/
export async function createEmbeddingService(
config?: EmbeddingServiceConfig
): Promise<EmbeddingService> {
const ollama = new OllamaEmbeddingService(config);
if (await ollama.isAvailable()) {
console.log(`Using Ollama embedding service with model: ${config?.model || EMBEDDING_MODEL}`);
return ollama;
}
console.warn(
'Ollama not available, using fallback keyword similarity. ' +
'For better results, install Ollama and run: ollama pull nomic-embed-text'
);
return new FallbackEmbeddingService();
}
// ============================================================================
// UTILITY FUNCTIONS
// ============================================================================
/**
* Find the top K most similar items to a query embedding.
*
* @param queryEmbedding - The embedding to search for
* @param candidates - Array of items with embeddings
* @param k - Number of results to return
* @returns Top K items sorted by similarity (highest first)
*
* @example
* ```typescript
* const results = findTopK(queryVec, documents, 10);
* results.forEach(({ item, similarity }) => {
* console.log(`${item.title}: ${similarity.toFixed(3)}`);
* });
* ```
*/
export function findTopK<T extends { embedding: number[] }>(
queryEmbedding: number[],
candidates: T[],
k: number
): Array<T & { similarity: number }> {
const scored = candidates.map((item) => ({
...item,
similarity: cosineSimilarity(queryEmbedding, item.embedding),
}));
scored.sort((a, b) => b.similarity - a.similarity);
return scored.slice(0, k);
}
/**
* Filter items by minimum similarity threshold.
*
* @param queryEmbedding - The embedding to search for
* @param candidates - Array of items with embeddings
* @param minSimilarity - Minimum similarity score (-1 to 1)
* @returns Items with similarity >= minSimilarity, sorted by similarity
*
* @example
* ```typescript
* const relevant = filterBySimilarity(queryVec, documents, 0.7);
* console.log(`Found ${relevant.length} relevant documents`);
* ```
*/
export function filterBySimilarity<T extends { embedding: number[] }>(
queryEmbedding: number[],
candidates: T[],
minSimilarity: number
): Array<T & { similarity: number }> {
const scored = candidates
.map((item) => ({
...item,
similarity: cosineSimilarity(queryEmbedding, item.embedding),
}))
.filter((item) => item.similarity >= minSimilarity);
scored.sort((a, b) => b.similarity - a.similarity);
return scored;
}
/**
* Compute average embedding from multiple vectors.
* Useful for combining multiple documents into a single representation.
*
* @param embeddings - Array of embedding vectors
* @returns Average embedding vector
*/
export function averageEmbedding(embeddings: number[][]): number[] {
if (embeddings.length === 0) {
throw new Error('Cannot compute average of empty embedding array');
}
const firstEmbedding = embeddings[0];
if (!firstEmbedding) {
throw new Error('Cannot compute average of empty embedding array');
}
const dimensions = firstEmbedding.length;
const result = new Array<number>(dimensions).fill(0);
for (const embedding of embeddings) {
if (embedding.length !== dimensions) {
throw new Error('All embeddings must have the same dimensions');
}
for (let i = 0; i < dimensions; i++) {
result[i]! += embedding[i]!;
}
}
for (let i = 0; i < dimensions; i++) {
result[i]! /= embeddings.length;
}
return result;
}

462
packages/core/src/core/errors.ts
View file

@ -1,462 +0,0 @@
/**
* Vestige Error Types
*
* A comprehensive hierarchy of errors for proper error handling and reporting.
* Includes type guards, utilities, and a Result type for functional error handling.
*/
// =============================================================================
// Error Sanitization
// =============================================================================
/**
* Sanitize error messages to prevent information leakage
*/
export function sanitizeErrorMessage(message: string): string {
let sanitized = message;
// Remove file paths
sanitized = sanitized.replace(/\/[^\s]+/g, '[PATH]');
// Remove SQL keywords
sanitized = sanitized.replace(/SELECT|INSERT|UPDATE|DELETE|DROP|CREATE|ALTER/gi, '[SQL]');
// Redact credentials
sanitized = sanitized.replace(
/\b(password|secret|key|token|auth)\s*[=:]\s*\S+/gi,
'[REDACTED]'
);
return sanitized;
}
// =============================================================================
// Base Error Class
// =============================================================================
/**
* Base error class for all Vestige errors
*/
export class VestigeError extends Error {
constructor(
message: string,
public readonly code: string,
public readonly statusCode: number = 500,
public readonly details?: Record<string, unknown>
) {
super(message);
this.name = 'VestigeError';
Error.captureStackTrace(this, this.constructor);
}
toJSON(): {
name: string;
code: string;
message: string;
statusCode: number;
details?: Record<string, unknown>;
} {
const result: {
name: string;
code: string;
message: string;
statusCode: number;
details?: Record<string, unknown>;
} = {
name: this.name,
code: this.code,
message: this.message,
statusCode: this.statusCode,
};
if (this.details !== undefined) {
result.details = this.details;
}
return result;
}
}
// =============================================================================
// Specific Error Types
// =============================================================================
/**
* Validation errors (400)
*/
export class ValidationError extends VestigeError {
constructor(message: string, details?: Record<string, unknown>) {
super(message, 'VALIDATION_ERROR', 400, details);
this.name = 'ValidationError';
}
}
/**
* Resource not found (404)
*/
export class NotFoundError extends VestigeError {
constructor(resource: string, id?: string) {
super(
id ? `${resource} not found: ${id}` : `${resource} not found`,
'NOT_FOUND',
404,
{ resource, id }
);
this.name = 'NotFoundError';
}
}
/**
* Conflict errors (409)
*/
export class ConflictError extends VestigeError {
constructor(message: string, details?: Record<string, unknown>) {
super(message, 'CONFLICT', 409, details);
this.name = 'ConflictError';
}
}
/**
* Database operation errors (500)
*/
export class DatabaseError extends VestigeError {
constructor(message: string, cause?: unknown) {
super(sanitizeErrorMessage(message), 'DATABASE_ERROR', 500, {
cause: String(cause),
});
this.name = 'DatabaseError';
}
}
/**
* Security-related errors (403)
*/
export class SecurityError extends VestigeError {
constructor(message: string, details?: Record<string, unknown>) {
super(message, 'SECURITY_ERROR', 403, details);
this.name = 'SecurityError';
}
}
/**
* Configuration errors (500)
*/
export class ConfigurationError extends VestigeError {
constructor(message: string, details?: Record<string, unknown>) {
super(message, 'CONFIGURATION_ERROR', 500, details);
this.name = 'ConfigurationError';
}
}
/**
* Timeout errors (408)
*/
export class TimeoutError extends VestigeError {
constructor(operation: string, timeoutMs: number) {
super(`Operation timed out: ${operation}`, 'TIMEOUT', 408, {
operation,
timeoutMs,
});
this.name = 'TimeoutError';
}
}
/**
* Embedding service errors
*/
export class EmbeddingError extends VestigeError {
constructor(message: string, cause?: unknown) {
super(message, 'EMBEDDING_ERROR', 500, { cause: String(cause) });
this.name = 'EmbeddingError';
}
}
/**
* Concurrency/locking errors (409)
*/
export class ConcurrencyError extends VestigeError {
constructor(message: string = 'Operation failed due to concurrent access') {
super(message, 'CONCURRENCY_ERROR', 409);
this.name = 'ConcurrencyError';
}
}
/**
* Rate limit errors (429)
*/
export class RateLimitError extends VestigeError {
constructor(message: string, retryAfterMs?: number) {
super(message, 'RATE_LIMIT', 429, { retryAfterMs });
this.name = 'RateLimitError';
}
}
/**
* Authentication errors (401)
*/
export class AuthenticationError extends VestigeError {
constructor(message: string = 'Authentication required') {
super(message, 'AUTHENTICATION_ERROR', 401);
this.name = 'AuthenticationError';
}
}
// =============================================================================
// Error Handling Utilities
// =============================================================================
/**
* Type guard for VestigeError
*/
export function isVestigeError(error: unknown): error is VestigeError {
return error instanceof VestigeError;
}
/**
* Convert unknown error to VestigeError
*/
export function toVestigeError(error: unknown): VestigeError {
if (isVestigeError(error)) {
return error;
}
if (error instanceof Error) {
return new VestigeError(
sanitizeErrorMessage(error.message),
'UNKNOWN_ERROR',
500,
{ originalName: error.name }
);
}
if (typeof error === 'string') {
return new VestigeError(sanitizeErrorMessage(error), 'UNKNOWN_ERROR', 500);
}
return new VestigeError('An unknown error occurred', 'UNKNOWN_ERROR', 500, {
errorType: typeof error,
});
}
/**
* Wrap function to catch and transform errors
*/
export function wrapError<T extends (...args: unknown[]) => Promise<unknown>>(
fn: T,
errorTransform?: (error: unknown) => VestigeError
): T {
const wrapped = async (...args: Parameters<T>): Promise<ReturnType<T>> => {
try {
return (await fn(...args)) as ReturnType<T>;
} catch (error) {
if (errorTransform) {
throw errorTransform(error);
}
throw toVestigeError(error);
}
};
return wrapped as T;
}
/**
* Execute a function with error transformation
*/
export async function withErrorHandling<T>(
fn: () => Promise<T>,
errorTransform?: (error: unknown) => VestigeError
): Promise<T> {
try {
return await fn();
} catch (error) {
if (errorTransform) {
throw errorTransform(error);
}
throw toVestigeError(error);
}
}
/**
* Retry a function with exponential backoff
*/
export async function withRetry<T>(
fn: () => Promise<T>,
options: {
maxRetries?: number;
baseDelayMs?: number;
maxDelayMs?: number;
shouldRetry?: (error: unknown) => boolean;
} = {}
): Promise<T> {
const {
maxRetries = 3,
baseDelayMs = 100,
maxDelayMs = 5000,
shouldRetry = () => true,
} = options;
let lastError: unknown;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error;
if (attempt === maxRetries || !shouldRetry(error)) {
throw toVestigeError(error);
}
const delay = Math.min(baseDelayMs * Math.pow(2, attempt), maxDelayMs);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
throw toVestigeError(lastError);
}
// =============================================================================
// Result Type (Optional Pattern)
// =============================================================================
/**
* Result type for functional error handling
*/
export type Result<T, E = VestigeError> =
| { success: true; data: T }
| { success: false; error: E };
/**
* Create a success result
*/
export function ok<T>(data: T): Result<T, never> {
return { success: true, data };
}
/**
* Create an error result
*/
export function err<E = VestigeError>(error: E): Result<never, E> {
return { success: false, error };
}
/**
* Check if result is success
*/
export function isOk<T, E>(result: Result<T, E>): result is { success: true; data: T } {
return result.success;
}
/**
* Check if result is error
*/
export function isErr<T, E>(result: Result<T, E>): result is { success: false; error: E } {
return !result.success;
}
/**
* Unwrap a result, throwing if it's an error
*/
export function unwrap<T, E>(result: Result<T, E>): T {
if (result.success) {
return result.data;
}
throw (result as { success: false; error: E }).error;
}
/**
* Unwrap a result with a default value
*/
export function unwrapOr<T, E>(result: Result<T, E>, defaultValue: T): T {
if (result.success) {
return result.data;
}
return defaultValue;
}
/**
* Map over a successful result
*/
export function mapResult<T, U, E>(
result: Result<T, E>,
fn: (data: T) => U
): Result<U, E> {
if (result.success) {
return ok(fn(result.data));
}
return result as { success: false; error: E };
}
/**
* Map over an error result
*/
export function mapError<T, E, F>(
result: Result<T, E>,
fn: (error: E) => F
): Result<T, F> {
if (!result.success) {
return err(fn((result as { success: false; error: E }).error));
}
return result as { success: true; data: T };
}
/**
* Execute a function and return a Result
*/
export async function tryCatch<T>(
fn: () => Promise<T>
): Promise<Result<T, VestigeError>> {
try {
const data = await fn();
return ok(data);
} catch (error) {
return err(toVestigeError(error));
}
}
/**
* Execute a synchronous function and return a Result
*/
export function tryCatchSync<T>(fn: () => T): Result<T, VestigeError> {
try {
const data = fn();
return ok(data);
} catch (error) {
return err(toVestigeError(error));
}
}
// =============================================================================
// Error Assertion Helpers
// =============================================================================
/**
* Assert a condition, throwing ValidationError if false
*/
export function assertValid(
condition: boolean,
message: string,
details?: Record<string, unknown>
): asserts condition {
if (!condition) {
throw new ValidationError(message, details);
}
}
/**
* Assert a value is not null or undefined
*/
export function assertDefined<T>(
value: T | null | undefined,
resource: string,
id?: string
): asserts value is T {
if (value === null || value === undefined) {
throw new NotFoundError(resource, id);
}
}
/**
* Assert a value exists, returning it if so
*/
export function requireDefined<T>(
value: T | null | undefined,
resource: string,
id?: string
): T {
assertDefined(value, resource, id);
return value;
}

815
packages/core/src/core/fsrs.ts
View file

@ -1,815 +0,0 @@
/**
* FSRS-5 (Free Spaced Repetition Scheduler) Algorithm Implementation
*
* Based on the FSRS-5 algorithm by Jarrett Ye
* Paper: https://github.com/open-spaced-repetition/fsrs4anki/wiki/The-Algorithm
*
* This is a production-ready implementation with full TypeScript types,
* sentiment integration for emotional memory boosting, and comprehensive
* error handling.
*/
import { z } from 'zod';
// ============================================================================
// FSRS-5 CONSTANTS
// ============================================================================
/**
* FSRS-5 default weights (w0 to w18)
*
* These weights are optimized from millions of Anki review records.
* They control:
* - w0-w3: Initial stability for each grade (Again, Hard, Good, Easy)
* - w4-w5: Initial difficulty calculation
* - w6-w7: Short-term stability calculation
* - w8-w10: Stability increase factors after successful recall
* - w11-w14: Difficulty update parameters
* - w15-w16: Forgetting curve (stability after lapse)
* - w17-w18: Short-term scheduling parameters
*/
export const FSRS_WEIGHTS: readonly [
number, number, number, number, number,
number, number, number, number, number,
number, number, number, number, number,
number, number, number, number
] = [
0.40255, 1.18385, 3.173, 15.69105, 7.1949,
0.5345, 1.4604, 0.0046, 1.54575, 0.1192,
1.01925, 1.9395, 0.11, 0.29605, 2.2698,
0.2315, 2.9898, 0.51655, 0.6621
] as const;
/**
* FSRS algorithm constants
*/
export const FSRS_CONSTANTS = {
/** Maximum difficulty value */
MAX_DIFFICULTY: 10,
/** Minimum difficulty value */
MIN_DIFFICULTY: 1,
/** Minimum stability in days */
MIN_STABILITY: 0.1,
/** Maximum stability in days (approx 100 years) */
MAX_STABILITY: 36500,
/** Default desired retention rate */
DEFAULT_RETENTION: 0.9,
/** Factor for converting retrievability to interval */
DECAY_FACTOR: 0.9,
/** Small epsilon for numerical stability */
EPSILON: 1e-10,
} as const;
// ============================================================================
// TYPES & SCHEMAS
// ============================================================================
/**
* Review grades in FSRS
* - Again (1): Complete failure to recall
* - Hard (2): Recalled with significant difficulty
* - Good (3): Recalled with moderate effort
* - Easy (4): Recalled effortlessly
*/
export const ReviewGradeSchema = z.union([
z.literal(1),
z.literal(2),
z.literal(3),
z.literal(4),
]);
export type ReviewGrade = z.infer<typeof ReviewGradeSchema>;
/**
* Named constants for review grades
*/
export const Grade = {
Again: 1,
Hard: 2,
Good: 3,
Easy: 4,
} as const satisfies Record<string, ReviewGrade>;
/**
* Learning states for FSRS cards
* - New: Never reviewed
* - Learning: In initial learning phase
* - Review: In long-term review phase
* - Relearning: Lapsed and relearning
*/
export const LearningStateSchema = z.enum([
'New',
'Learning',
'Review',
'Relearning',
]);
export type LearningState = z.infer<typeof LearningStateSchema>;
/**
* FSRS card state - represents the memory state of a single item
*/
export const FSRSStateSchema = z.object({
/** Current difficulty (1-10, higher = harder) */
difficulty: z.number().min(FSRS_CONSTANTS.MIN_DIFFICULTY).max(FSRS_CONSTANTS.MAX_DIFFICULTY),
/** Current stability in days (higher = more stable memory) */
stability: z.number().min(FSRS_CONSTANTS.MIN_STABILITY).max(FSRS_CONSTANTS.MAX_STABILITY),
/** Current learning state */
state: LearningStateSchema,
/** Number of times reviewed */
reps: z.number().int().min(0),
/** Number of lapses (times "Again" was pressed in Review state) */
lapses: z.number().int().min(0),
/** Timestamp of last review */
lastReview: z.date(),
/** Scheduled next review date */
scheduledDays: z.number().min(0),
});
export type FSRSState = z.infer<typeof FSRSStateSchema>;
/**
* Input type for FSRSState (for creating new states)
*/
export type FSRSStateInput = z.input<typeof FSRSStateSchema>;
/**
* Result of a review operation
*/
export const ReviewResultSchema = z.object({
/** Updated FSRS state */
state: FSRSStateSchema,
/** Calculated retrievability at time of review */
retrievability: z.number().min(0).max(1),
/** Next review interval in days */
interval: z.number().min(0),
/** Whether this was a lapse */
isLapse: z.boolean(),
});
export type ReviewResult = z.infer<typeof ReviewResultSchema>;
/**
* Type for the 19-element FSRS weights tuple
*/
export type FSRSWeightsTuple = readonly [
number, number, number, number, number,
number, number, number, number, number,
number, number, number, number, number,
number, number, number, number
];
/**
* Zod schema for FSRS weights
*/
const FSRSWeightsSchema = z.array(z.number()).length(19);
/**
* Configuration for FSRS scheduler
*/
export const FSRSConfigSchema = z.object({
/** Desired retention rate (0.7-0.99) */
desiredRetention: z.number().min(0.7).max(0.99).default(0.9),
/** Maximum interval in days */
maximumInterval: z.number().min(1).max(36500).default(36500),
/** Custom weights (must be exactly 19 values) */
weights: FSRSWeightsSchema.optional(),
/** Enable sentiment boost for emotional memories */
enableSentimentBoost: z.boolean().default(true),
/** Maximum sentiment boost multiplier (1.0-3.0) */
maxSentimentBoost: z.number().min(1).max(3).default(2),
});
/**
* Configuration type for FSRS scheduler
*/
export interface FSRSConfig {
/** Desired retention rate (0.7-0.99) */
desiredRetention?: number;
/** Maximum interval in days */
maximumInterval?: number;
/** Custom weights (must be exactly 19 values) */
weights?: readonly number[];
/** Enable sentiment boost for emotional memories */
enableSentimentBoost?: boolean;
/** Maximum sentiment boost multiplier (1.0-3.0) */
maxSentimentBoost?: number;
}
/**
* Resolved (required) configuration type
*/
export interface ResolvedFSRSConfig {
desiredRetention: number;
maximumInterval: number;
weights: readonly number[] | undefined;
enableSentimentBoost: boolean;
maxSentimentBoost: number;
}
// ============================================================================
// CORE FSRS-5 FUNCTIONS
// ============================================================================
/**
* Calculate initial difficulty for a new card based on first rating
*
* Formula: D(G) = w[4] - e^(w[5]*(G-1)) + 1
*
* @param grade - First review grade (1-4)
* @param weights - FSRS weights array
* @returns Initial difficulty (1-10)
*/
export function initialDifficulty(
grade: ReviewGrade,
weights: readonly number[] = FSRS_WEIGHTS
): number {
const w4 = weights[4] ?? FSRS_WEIGHTS[4];
const w5 = weights[5] ?? FSRS_WEIGHTS[5];
// D(G) = w[4] - e^(w[5]*(G-1)) + 1
const d = w4 - Math.exp(w5 * (grade - 1)) + 1;
// Clamp to valid range
return clamp(d, FSRS_CONSTANTS.MIN_DIFFICULTY, FSRS_CONSTANTS.MAX_DIFFICULTY);
}
/**
* Calculate initial stability for a new card based on first rating
*
* Formula: S(G) = w[G-1] (direct lookup from weights 0-3)
*
* Note: FSRS-5 uses the first 4 weights as initial stability values
* for grades 1-4 respectively.
*
* @param grade - First review grade (1-4)
* @param weights - FSRS weights array
* @returns Initial stability in days
*/
export function initialStability(
grade: ReviewGrade,
weights: readonly number[] = FSRS_WEIGHTS
): number {
// FSRS-5: S0(G) = w[G-1]
// Grade is 1-4, so index is 0-3, which is always valid for FSRS_WEIGHTS
const index = grade - 1;
const s = weights[index] ?? FSRS_WEIGHTS[index] ?? FSRS_WEIGHTS[0];
// Ensure minimum stability
return Math.max(FSRS_CONSTANTS.MIN_STABILITY, s);
}
/**
* Calculate retrievability (probability of recall) based on stability and elapsed time
*
* Formula: R = e^(-t/S) where FSRS uses (1 + t/(9*S))^(-1)
*
* This is the power forgetting curve used in FSRS-5.
*
* @param stability - Current stability in days
* @param elapsedDays - Days since last review
* @returns Retrievability (0-1)
*/
export function retrievability(stability: number, elapsedDays: number): number {
if (stability <= 0) {
return 0;
}
if (elapsedDays <= 0) {
return 1;
}
// FSRS-5 power forgetting curve: R = (1 + t/(9*S))^(-1)
// This is equivalent to the power law of forgetting
const factor = 9 * stability;
const r = Math.pow(1 + elapsedDays / factor, -1);
return clamp(r, 0, 1);
}
/**
* Calculate next difficulty after a review
*
* Formula: D' = w[7] * D + (1 - w[7]) * mean_reversion(D, G)
* where mean_reversion uses a linear combination with the initial difficulty
*
* FSRS-5 mean reversion formula:
* D' = D - w[6] * (G - 3)
* Then: D' = w[7] * D0 + (1 - w[7]) * D'
*
* @param currentD - Current difficulty (1-10)
* @param grade - Review grade (1-4)
* @param weights - FSRS weights array
* @returns New difficulty (1-10)
*/
export function nextDifficulty(
currentD: number,
grade: ReviewGrade,
weights: readonly number[] = FSRS_WEIGHTS
): number {
const w6 = weights[6] ?? FSRS_WEIGHTS[6];
const w7 = weights[7] ?? FSRS_WEIGHTS[7];
// Initial difficulty for mean reversion (what D would be for a "Good" rating)
const d0 = initialDifficulty(Grade.Good, weights);
// Delta based on grade deviation from "Good" (3)
// Negative grade (Again=1, Hard=2) increases difficulty
// Positive grade (Easy=4) decreases difficulty
const delta = -w6 * (grade - 3);
// Apply delta to current difficulty
let newD = currentD + delta;
// Mean reversion: blend towards initial difficulty
newD = w7 * d0 + (1 - w7) * newD;
return clamp(newD, FSRS_CONSTANTS.MIN_DIFFICULTY, FSRS_CONSTANTS.MAX_DIFFICULTY);
}
/**
* Calculate next stability after a successful recall
*
* FSRS-5 recall stability formula:
* S'(r) = S * (e^(w[8]) * (11 - D) * S^(-w[9]) * (e^(w[10]*(1-R)) - 1) * hardPenalty * easyBonus + 1)
*
* This is the full FSRS-5 stability increase formula that accounts for:
* - Current stability (S)
* - Difficulty (D)
* - Retrievability at time of review (R)
* - Hard penalty for grade 2
* - Easy bonus for grade 4
*
* @param currentS - Current stability in days
* @param difficulty - Current difficulty (1-10)
* @param retrievabilityR - Retrievability at time of review (0-1)
* @param grade - Review grade (2, 3, or 4 - not 1, which is a lapse)
* @param weights - FSRS weights array
* @returns New stability in days
*/
export function nextRecallStability(
currentS: number,
difficulty: number,
retrievabilityR: number,
grade: ReviewGrade,
weights: readonly number[] = FSRS_WEIGHTS
): number {
if (grade === Grade.Again) {
// Lapse - use forget stability instead
return nextForgetStability(difficulty, currentS, retrievabilityR, weights);
}
const w8 = weights[8] ?? FSRS_WEIGHTS[8];
const w9 = weights[9] ?? FSRS_WEIGHTS[9];
const w10 = weights[10] ?? FSRS_WEIGHTS[10];
const w15 = weights[15] ?? FSRS_WEIGHTS[15];
const w16 = weights[16] ?? FSRS_WEIGHTS[16];
// Hard penalty (grade = 2)
const hardPenalty = grade === Grade.Hard ? w15 : 1;
// Easy bonus (grade = 4)
const easyBonus = grade === Grade.Easy ? w16 : 1;
// FSRS-5 recall stability formula
// S'(r) = S * (e^(w8) * (11 - D) * S^(-w9) * (e^(w10*(1-R)) - 1) * hardPenalty * easyBonus + 1)
const factor =
Math.exp(w8) *
(11 - difficulty) *
Math.pow(currentS, -w9) *
(Math.exp(w10 * (1 - retrievabilityR)) - 1) *
hardPenalty *
easyBonus +
1;
const newS = currentS * factor;
return clamp(newS, FSRS_CONSTANTS.MIN_STABILITY, FSRS_CONSTANTS.MAX_STABILITY);
}
/**
* Calculate stability after a lapse (forgotten/Again rating)
*
* FSRS-5 forget stability formula:
* S'(f) = w[11] * D^(-w[12]) * ((S+1)^w[13] - 1) * e^(w[14]*(1-R))
*
* This calculates the new stability after forgetting, which is typically
* much lower than the previous stability but not zero (some memory trace remains).
*
* @param difficulty - Current difficulty (1-10)
* @param currentS - Current stability before lapse
* @param retrievabilityR - Retrievability at time of review
* @param weights - FSRS weights array
* @returns New stability after lapse in days
*/
export function nextForgetStability(
difficulty: number,
currentS: number,
retrievabilityR: number = 0.5,
weights: readonly number[] = FSRS_WEIGHTS
): number {
const w11 = weights[11] ?? FSRS_WEIGHTS[11];
const w12 = weights[12] ?? FSRS_WEIGHTS[12];
const w13 = weights[13] ?? FSRS_WEIGHTS[13];
const w14 = weights[14] ?? FSRS_WEIGHTS[14];
// S'(f) = w11 * D^(-w12) * ((S+1)^w13 - 1) * e^(w14*(1-R))
const newS =
w11 *
Math.pow(difficulty, -w12) *
(Math.pow(currentS + 1, w13) - 1) *
Math.exp(w14 * (1 - retrievabilityR));
return clamp(newS, FSRS_CONSTANTS.MIN_STABILITY, FSRS_CONSTANTS.MAX_STABILITY);
}
/**
* Calculate next review interval based on stability and desired retention
*
* Formula: I = S * ln(R) / ln(0.9) where we solve for t when R = desired_retention
* Using the power forgetting curve: I = 9 * S * (1/R - 1)
*
* @param stability - Current stability in days
* @param desiredRetention - Target retention rate (default 0.9)
* @returns Interval in days until next review
*/
export function nextInterval(
stability: number,
desiredRetention: number = FSRS_CONSTANTS.DEFAULT_RETENTION
): number {
if (stability <= 0) {
return 0;
}
if (desiredRetention >= 1) {
return 0; // If we want 100% retention, review immediately
}
if (desiredRetention <= 0) {
return FSRS_CONSTANTS.MAX_STABILITY; // If we don't care about retention
}
// Solve for t in: R = (1 + t/(9*S))^(-1)
// t = 9 * S * (R^(-1) - 1)
const interval = 9 * stability * (Math.pow(desiredRetention, -1) - 1);
return Math.max(0, Math.round(interval));
}
/**
* Apply sentiment boost to stability
*
* Emotional memories are encoded more strongly and decay more slowly.
* This function applies a multiplier to stability based on sentiment intensity.
*
* @param stability - Base stability in days
* @param sentimentIntensity - Sentiment intensity (0-1, where 1 = highly emotional)
* @param maxBoost - Maximum boost multiplier (default 2.0)
* @returns Boosted stability in days
*/
export function applySentimentBoost(
stability: number,
sentimentIntensity: number,
maxBoost: number = 2.0
): number {
// Validate inputs
const clampedSentiment = clamp(sentimentIntensity, 0, 1);
const clampedMaxBoost = clamp(maxBoost, 1, 3);
// Linear interpolation: boost = 1 + (maxBoost - 1) * sentimentIntensity
const boost = 1 + (clampedMaxBoost - 1) * clampedSentiment;
return stability * boost;
}
// ============================================================================
// FSRS SCHEDULER CLASS
// ============================================================================
/**
* FSRSScheduler - Main class for FSRS-5 spaced repetition scheduling
*
* Usage:
* ```typescript
* const scheduler = new FSRSScheduler();
*
* // Create initial state for a new card
* const state = scheduler.newCard();
*
* // Process a review
* const result = scheduler.review(state, Grade.Good, 1);
*
* // Get the next review date
* const nextReview = new Date();
* nextReview.setDate(nextReview.getDate() + result.interval);
* ```
*/
export class FSRSScheduler {
private readonly config: ResolvedFSRSConfig;
private readonly weights: readonly number[];
/**
* Create a new FSRS scheduler
*
* @param config - Optional configuration overrides
*/
constructor(config: FSRSConfig = {}) {
const parsed = FSRSConfigSchema.parse({
desiredRetention: config.desiredRetention ?? 0.9,
maximumInterval: config.maximumInterval ?? 36500,
weights: config.weights ? [...config.weights] : undefined,
enableSentimentBoost: config.enableSentimentBoost ?? true,
maxSentimentBoost: config.maxSentimentBoost ?? 2,
});
// Extract weights as a readonly number array (or undefined)
const parsedWeights: readonly number[] | undefined = parsed.weights
? [...parsed.weights]
: undefined;
this.config = {
desiredRetention: parsed.desiredRetention ?? 0.9,
maximumInterval: parsed.maximumInterval ?? 36500,
weights: parsedWeights,
enableSentimentBoost: parsed.enableSentimentBoost ?? true,
maxSentimentBoost: parsed.maxSentimentBoost ?? 2,
};
this.weights = this.config.weights ?? FSRS_WEIGHTS;
}
/**
* Create initial state for a new card
*
* @returns Initial FSRS state
*/
newCard(): FSRSState {
return {
difficulty: initialDifficulty(Grade.Good, this.weights),
stability: initialStability(Grade.Good, this.weights),
state: 'New',
reps: 0,
lapses: 0,
lastReview: new Date(),
scheduledDays: 0,
};
}
/**
* Process a review and calculate next state
*
* @param currentState - Current FSRS state
* @param grade - Review grade (1-4)
* @param elapsedDays - Days since last review (0 for first review)
* @param sentimentBoost - Optional sentiment intensity for emotional memories (0-1)
* @returns Review result with updated state and next interval
*/
review(
currentState: FSRSState,
grade: ReviewGrade,
elapsedDays: number = 0,
sentimentBoost?: number
): ReviewResult {
// Validate grade
const validatedGrade = ReviewGradeSchema.parse(grade);
// Calculate retrievability at time of review
const r = currentState.state === 'New'
? 1
: retrievability(currentState.stability, Math.max(0, elapsedDays));
let newState: FSRSState;
let isLapse = false;
if (currentState.state === 'New') {
// First review - initialize based on grade
newState = this.handleFirstReview(currentState, validatedGrade);
} else if (validatedGrade === Grade.Again) {
// Lapse - memory failed
isLapse = currentState.state === 'Review' || currentState.state === 'Relearning';
newState = this.handleLapse(currentState, r);
} else {
// Successful recall
newState = this.handleRecall(currentState, validatedGrade, r);
}
// Apply sentiment boost if enabled and provided
if (
this.config.enableSentimentBoost &&
sentimentBoost !== undefined &&
sentimentBoost > 0
) {
newState.stability = applySentimentBoost(
newState.stability,
sentimentBoost,
this.config.maxSentimentBoost
);
}
// Calculate next interval
const interval = Math.min(
nextInterval(newState.stability, this.config.desiredRetention),
this.config.maximumInterval
);
newState.scheduledDays = interval;
newState.lastReview = new Date();
return {
state: newState,
retrievability: r,
interval,
isLapse,
};
}
/**
* Handle first review of a new card
*/
private handleFirstReview(currentState: FSRSState, grade: ReviewGrade): FSRSState {
const d = initialDifficulty(grade, this.weights);
const s = initialStability(grade, this.weights);
return {
...currentState,
difficulty: d,
stability: s,
state: grade === Grade.Again ? 'Learning' : grade === Grade.Hard ? 'Learning' : 'Review',
reps: 1,
lapses: grade === Grade.Again ? 1 : 0,
};
}
/**
* Handle a lapse (Again rating)
*/
private handleLapse(currentState: FSRSState, retrievabilityR: number): FSRSState {
const newS = nextForgetStability(
currentState.difficulty,
currentState.stability,
retrievabilityR,
this.weights
);
// Difficulty increases on lapse
const newD = nextDifficulty(currentState.difficulty, Grade.Again, this.weights);
return {
...currentState,
difficulty: newD,
stability: newS,
state: 'Relearning',
reps: currentState.reps + 1,
lapses: currentState.lapses + 1,
};
}
/**
* Handle a successful recall (Hard, Good, or Easy)
*/
private handleRecall(
currentState: FSRSState,
grade: ReviewGrade,
retrievabilityR: number
): FSRSState {
const newS = nextRecallStability(
currentState.stability,
currentState.difficulty,
retrievabilityR,
grade,
this.weights
);
const newD = nextDifficulty(currentState.difficulty, grade, this.weights);
return {
...currentState,
difficulty: newD,
stability: newS,
state: 'Review',
reps: currentState.reps + 1,
};
}
/**
* Get the current retrievability for a state
*
* @param state - FSRS state
* @param elapsedDays - Days since last review (optional, calculated from lastReview if not provided)
* @returns Current retrievability (0-1)
*/
getRetrievability(state: FSRSState, elapsedDays?: number): number {
const days = elapsedDays ?? this.daysSinceReview(state.lastReview);
return retrievability(state.stability, days);
}
/**
* Preview all possible review outcomes without modifying state
*
* @param state - Current FSRS state
* @param elapsedDays - Days since last review
* @returns Object with results for each grade
*/
previewReviews(
state: FSRSState,
elapsedDays: number = 0
): Record<'again' | 'hard' | 'good' | 'easy', ReviewResult> {
return {
again: this.review({ ...state }, Grade.Again, elapsedDays),
hard: this.review({ ...state }, Grade.Hard, elapsedDays),
good: this.review({ ...state }, Grade.Good, elapsedDays),
easy: this.review({ ...state }, Grade.Easy, elapsedDays),
};
}
/**
* Calculate days since a review date
*/
private daysSinceReview(lastReview: Date): number {
const now = new Date();
const diffMs = now.getTime() - lastReview.getTime();
return Math.max(0, diffMs / (1000 * 60 * 60 * 24));
}
/**
* Get scheduler configuration
*/
getConfig(): Readonly<ResolvedFSRSConfig> {
return { ...this.config };
}
/**
* Get scheduler weights
*/
getWeights(): readonly number[] {
return [...this.weights];
}
}
// ============================================================================
// UTILITY FUNCTIONS
// ============================================================================
/**
* Clamp a value between min and max
*/
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, value));
}
/**
* Convert FSRSState to a JSON-serializable format
*/
export function serializeFSRSState(state: FSRSState): string {
return JSON.stringify({
...state,
lastReview: state.lastReview.toISOString(),
});
}
/**
* Parse a serialized FSRSState from JSON
*/
export function deserializeFSRSState(json: string): FSRSState {
const parsed = JSON.parse(json) as Record<string, unknown>;
return FSRSStateSchema.parse({
...parsed,
lastReview: new Date(parsed['lastReview'] as string),
});
}
/**
* Calculate optimal review time based on forgetting index
*
* @param state - Current FSRS state
* @param targetRetention - Target retention rate at review time (default 0.9)
* @returns Days until optimal review
*/
export function optimalReviewTime(
state: FSRSState,
targetRetention: number = FSRS_CONSTANTS.DEFAULT_RETENTION
): number {
return nextInterval(state.stability, targetRetention);
}
/**
* Determine if a review is due
*
* @param state - Current FSRS state
* @param currentRetention - Optional minimum retention threshold (default: use scheduledDays)
* @returns True if review is due
*/
export function isReviewDue(state: FSRSState, currentRetention?: number): boolean {
const daysSinceReview =
(new Date().getTime() - state.lastReview.getTime()) / (1000 * 60 * 60 * 24);
if (currentRetention !== undefined) {
const r = retrievability(state.stability, daysSinceReview);
return r < currentRetention;
}
return daysSinceReview >= state.scheduledDays;
}
// ============================================================================
// EXPORTS
// ============================================================================
export default FSRSScheduler;

12
packages/core/src/core/index.ts
View file

@ -1,12 +0,0 @@
export * from './config.js';
export * from './types.js';
export * from './errors.js';
export * from './database.js';
export * from './context-watcher.js';
export * from './rem-cycle.js';
export * from './consolidation.js';
export * from './shadow-self.js';
export * from './security.js';
export * from './vector-store.js';
export * from './fsrs.js';
export * from './embeddings.js';

721
packages/core/src/core/rem-cycle.ts
View file

@ -1,721 +0,0 @@
/**
* REM Cycle - Nocturnal Optimization with Semantic Understanding
*
* "The brain that dreams while you sleep."
*
* This module discovers connections between unconnected knowledge nodes
* by analyzing semantic similarity, shared concepts, keyword overlap,
* emotional resonance, and spreading activation patterns.
*
* KEY FEATURES:
* 1. Semantic Similarity - Uses embeddings for deep understanding
* 2. Emotional Weighting - Emotionally charged memories create stronger connections
* 3. Spreading Activation - Discovers transitive relationships (A->B->C implies A~C)
* 4. Reconsolidation - Accessing memories strengthens their connections
* 5. Exponential Temporal Proximity - Time-based connection strength decay
*/
import { VestigeDatabase } from './database.js';
import type { KnowledgeNode } from './types.js';
import natural from 'natural';
import {
createEmbeddingService,
type EmbeddingService,
EmbeddingCache,
cosineSimilarity,
} from './embeddings.js';
// ============================================================================
// TYPES
// ============================================================================
type ConnectionType =
| 'concept_overlap'
| 'keyword_similarity'
| 'entity_shared'
| 'temporal_proximity'
| 'semantic_similarity'
| 'spreading_activation';
interface DiscoveredConnection {
nodeA: KnowledgeNode;
nodeB: KnowledgeNode;
reason: string;
strength: number; // 0-1
connectionType: ConnectionType;
}
interface REMCycleResult {
nodesAnalyzed: number;
connectionsDiscovered: number;
connectionsCreated: number;
spreadingActivationEdges: number;
reconsolidatedNodes: number;
duration: number;
semanticEnabled: boolean;
discoveries: Array<{
nodeA: string;
nodeB: string;
reason: string;
type: ConnectionType;
}>;
}
interface REMCycleOptions {
maxAnalyze?: number;
minStrength?: number;
dryRun?: boolean;
/** Enable semantic similarity analysis (requires Ollama) */
enableSemantic?: boolean;
/** Run spreading activation to discover transitive connections */
enableSpreadingActivation?: boolean;
/** Maximum depth for spreading activation */
spreadingActivationDepth?: number;
/** Node IDs that were recently accessed (for reconsolidation) */
recentlyAccessedIds?: string[];
}
// ============================================================================
// CONSTANTS
// ============================================================================
/** Temporal half-life in days for exponential proximity decay */
const TEMPORAL_HALF_LIFE_DAYS = 7;
/** Semantic similarity thresholds */
const SEMANTIC_STRONG_THRESHOLD = 0.7;
const SEMANTIC_MODERATE_THRESHOLD = 0.5;
/** Weight decay for spreading activation (per hop) */
const SPREADING_ACTIVATION_DECAY = 0.8;
/** Reconsolidation strength boost (5%) */
const RECONSOLIDATION_BOOST = 0.05;
// ============================================================================
// SIMILARITY ANALYSIS
// ============================================================================
const tokenizer = new natural.WordTokenizer();
/**
* Extract keywords from content using TF-IDF
*/
function extractKeywords(content: string): string[] {
const tokens = tokenizer.tokenize(content.toLowerCase()) || [];
// Filter out common stop words and short tokens
const stopWords = new Set([
'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for',
'of', 'with', 'by', 'from', 'as', 'is', 'was', 'are', 'were', 'been',
'be', 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could',
'should', 'may', 'might', 'must', 'shall', 'can', 'need', 'dare', 'ought',
'used', 'it', 'its', 'this', 'that', 'these', 'those', 'i', 'you', 'he',
'she', 'we', 'they', 'what', 'which', 'who', 'whom', 'whose', 'where',
'when', 'why', 'how', 'all', 'each', 'every', 'both', 'few', 'more',
'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own',
'same', 'so', 'than', 'too', 'very', 'just', 'also', 'now', 'here',
]);
return tokens.filter(token =>
token.length > 3 &&
!stopWords.has(token) &&
!/^\d+$/.test(token) // Filter pure numbers
);
}
/**
* Calculate Jaccard similarity between two keyword sets
*/
function jaccardSimilarity(setA: Set<string>, setB: Set<string>): number {
const intersection = new Set([...setA].filter(x => setB.has(x)));
const union = new Set([...setA, ...setB]);
if (union.size === 0) return 0;
return intersection.size / union.size;
}
/**
* Find shared concepts between two nodes
*/
function findSharedConcepts(nodeA: KnowledgeNode, nodeB: KnowledgeNode): string[] {
const conceptsA = new Set([...nodeA.concepts, ...nodeA.tags]);
const conceptsB = new Set([...nodeB.concepts, ...nodeB.tags]);
return [...conceptsA].filter(c => conceptsB.has(c));
}
/**
* Find shared people between two nodes
*/
function findSharedPeople(nodeA: KnowledgeNode, nodeB: KnowledgeNode): string[] {
const peopleA = new Set(nodeA.people);
const peopleB = new Set(nodeB.people);
return [...peopleA].filter(p => peopleB.has(p));
}
/**
* Calculate exponential temporal proximity weight
* Uses half-life decay instead of binary same-day check
*/
function calculateTemporalProximity(nodeA: KnowledgeNode, nodeB: KnowledgeNode): number {
const msPerDay = 24 * 60 * 60 * 1000;
const diffMs = Math.abs(nodeA.createdAt.getTime() - nodeB.createdAt.getTime());
const daysBetween = diffMs / msPerDay;
// Exponential decay: weight = e^(-t/half_life)
// At t=0: weight = 1.0
// At t=half_life: weight = 0.5
// At t=2*half_life: weight = 0.25
return Math.exp(-daysBetween / TEMPORAL_HALF_LIFE_DAYS);
}
/**
* Calculate emotional resonance between two nodes
* Returns a boost multiplier (1.0 to 1.5) based on combined emotional intensity
*/
function calculateEmotionalBoost(nodeA: KnowledgeNode, nodeB: KnowledgeNode): number {
const emotionalA = nodeA.sentimentIntensity || 0;
const emotionalB = nodeB.sentimentIntensity || 0;
// Average emotional intensity
const emotionalResonance = (emotionalA + emotionalB) / 2;
// Up to 1.5x boost for highly emotional content
return 1 + (emotionalResonance * 0.5);
}
// ============================================================================
// SEMANTIC ANALYSIS
// ============================================================================
/**
* Analyze semantic connection between two nodes using embeddings
*/
async function analyzeSemanticConnection(
nodeA: KnowledgeNode,
nodeB: KnowledgeNode,
embeddingService: EmbeddingService,
cache: EmbeddingCache
): Promise<DiscoveredConnection | null> {
try {
// Get or generate embeddings
let embeddingA = cache.get(nodeA.id);
let embeddingB = cache.get(nodeB.id);
// Generate missing embeddings
if (!embeddingA) {
embeddingA = await embeddingService.generateEmbedding(nodeA.content);
cache.set(nodeA.id, embeddingA);
}
if (!embeddingB) {
embeddingB = await embeddingService.generateEmbedding(nodeB.content);
cache.set(nodeB.id, embeddingB);
}
// Calculate cosine similarity
const similarity = cosineSimilarity(embeddingA, embeddingB);
// Apply emotional boost
const emotionalBoost = calculateEmotionalBoost(nodeA, nodeB);
const boostedSimilarity = Math.min(1, similarity * emotionalBoost);
// Strong semantic connection
if (similarity >= SEMANTIC_STRONG_THRESHOLD) {
return {
nodeA,
nodeB,
reason: `Strong semantic similarity (${(similarity * 100).toFixed(0)}%)`,
strength: Math.min(1, boostedSimilarity + 0.2), // Boost for strong connections
connectionType: 'semantic_similarity',
};
}
// Moderate semantic connection
if (similarity >= SEMANTIC_MODERATE_THRESHOLD) {
return {
nodeA,
nodeB,
reason: `Moderate semantic similarity (${(similarity * 100).toFixed(0)}%)`,
strength: boostedSimilarity,
connectionType: 'semantic_similarity',
};
}
return null;
} catch {
// If embedding fails, return null to fall back to traditional analysis
return null;
}
}
// ============================================================================
// TRADITIONAL ANALYSIS (FALLBACK)
// ============================================================================
/**
* Analyze potential connection between two nodes using traditional methods
* Used as fallback when embeddings are unavailable
*/
function analyzeTraditionalConnection(
nodeA: KnowledgeNode,
nodeB: KnowledgeNode
): DiscoveredConnection | null {
// Extract keywords from both nodes
const keywordsA = new Set(extractKeywords(nodeA.content));
const keywordsB = new Set(extractKeywords(nodeB.content));
// Calculate keyword similarity
const keywordSim = jaccardSimilarity(keywordsA, keywordsB);
// Find shared concepts/tags
const sharedConcepts = findSharedConcepts(nodeA, nodeB);
// Find shared people
const sharedPeople = findSharedPeople(nodeA, nodeB);
// Calculate temporal proximity weight
const temporalWeight = calculateTemporalProximity(nodeA, nodeB);
// Calculate emotional boost
const emotionalBoost = calculateEmotionalBoost(nodeA, nodeB);
// Determine if there's a meaningful connection
// Priority: shared entities > concept overlap > keyword similarity > temporal
if (sharedPeople.length > 0) {
const baseStrength = Math.min(1, 0.5 + sharedPeople.length * 0.2);
return {
nodeA,
nodeB,
reason: `Shared people: ${sharedPeople.join(', ')}`,
strength: Math.min(1, baseStrength * emotionalBoost),
connectionType: 'entity_shared',
};
}
if (sharedConcepts.length >= 2) {
const baseStrength = Math.min(1, 0.4 + sharedConcepts.length * 0.15);
return {
nodeA,
nodeB,
reason: `Shared concepts: ${sharedConcepts.slice(0, 3).join(', ')}`,
strength: Math.min(1, baseStrength * emotionalBoost),
connectionType: 'concept_overlap',
};
}
if (keywordSim > 0.15) {
// Find the actual overlapping keywords
const overlap = [...keywordsA].filter(k => keywordsB.has(k)).slice(0, 5);
const baseStrength = Math.min(1, keywordSim * 2);
return {
nodeA,
nodeB,
reason: `Keyword overlap (${(keywordSim * 100).toFixed(0)}%): ${overlap.join(', ')}`,
strength: Math.min(1, baseStrength * emotionalBoost),
connectionType: 'keyword_similarity',
};
}
// Temporal proximity with related content
if (temporalWeight > 0.5 && (sharedConcepts.length > 0 || keywordSim > 0.05)) {
const baseStrength = 0.3 + (temporalWeight - 0.5) * 0.4; // Scale 0.3-0.5
return {
nodeA,
nodeB,
reason: `Created ${Math.round((1 - temporalWeight) * TEMPORAL_HALF_LIFE_DAYS * 2)} days apart with related content`,
strength: Math.min(1, baseStrength * emotionalBoost),
connectionType: 'temporal_proximity',
};
}
return null;
}
// ============================================================================
// SPREADING ACTIVATION
// ============================================================================
interface SpreadingActivationResult {
edgesCreated: number;
paths: Array<{
from: string;
via: string;
to: string;
weight: number;
}>;
}
/**
* Apply spreading activation to discover transitive connections
* If A -> B and B -> C exist, creates A -> C with decayed weight
*/
function applySpreadingActivation(
db: VestigeDatabase,
maxDepth: number = 2,
minWeight: number = 0.2
): SpreadingActivationResult {
const result: SpreadingActivationResult = {
edgesCreated: 0,
paths: [],
};
// Get all existing edges
const edges = db['db'].prepare(`
SELECT from_id, to_id, weight FROM graph_edges
WHERE edge_type = 'similar_to'
`).all() as { from_id: string; to_id: string; weight: number }[];
// Build adjacency map (bidirectional)
const adjacency = new Map<string, Map<string, number>>();
for (const edge of edges) {
// Forward direction
if (!adjacency.has(edge.from_id)) {
adjacency.set(edge.from_id, new Map());
}
adjacency.get(edge.from_id)!.set(edge.to_id, edge.weight);
// Reverse direction (treat as undirected)
if (!adjacency.has(edge.to_id)) {
adjacency.set(edge.to_id, new Map());
}
adjacency.get(edge.to_id)!.set(edge.from_id, edge.weight);
}
// Find existing direct connections (to avoid duplicates)
const existingConnections = new Set<string>();
for (const edge of edges) {
existingConnections.add(`${edge.from_id}-${edge.to_id}`);
existingConnections.add(`${edge.to_id}-${edge.from_id}`);
}
// For each node, find 2-hop paths
const newConnections: Array<{
from: string;
to: string;
via: string;
weight: number;
}> = [];
for (const [nodeA, neighborsA] of adjacency) {
for (const [nodeB, weightAB] of neighborsA) {
const neighborsB = adjacency.get(nodeB);
if (!neighborsB) continue;
for (const [nodeC, weightBC] of neighborsB) {
// Skip if A == C or if direct connection already exists
if (nodeA === nodeC) continue;
const connectionKey = `${nodeA}-${nodeC}`;
const reverseKey = `${nodeC}-${nodeA}`;
if (existingConnections.has(connectionKey) || existingConnections.has(reverseKey)) {
continue;
}
// Calculate transitive weight with decay
const transitiveWeight = weightAB * weightBC * SPREADING_ACTIVATION_DECAY;
if (transitiveWeight >= minWeight) {
newConnections.push({
from: nodeA,
to: nodeC,
via: nodeB,
weight: transitiveWeight,
});
// Mark as existing to avoid duplicates
existingConnections.add(connectionKey);
existingConnections.add(reverseKey);
}
}
}
}
// Create the new edges
for (const conn of newConnections) {
try {
db.insertEdge({
fromId: conn.from,
toId: conn.to,
edgeType: 'similar_to',
weight: conn.weight,
metadata: {
discoveredBy: 'spreading_activation',
viaNode: conn.via,
connectionType: 'spreading_activation',
},
createdAt: new Date(),
});
result.edgesCreated++;
result.paths.push(conn);
} catch {
// Edge might already exist, skip
}
}
return result;
}
// ============================================================================
// RECONSOLIDATION
// ============================================================================
/**
* Strengthen connections for recently accessed nodes
* Implements memory reconsolidation - accessing memories makes them stronger
*/
function reconsolidateConnections(db: VestigeDatabase, nodeId: string): number {
let strengthened = 0;
try {
// Get all edges involving this node
const edges = db['db'].prepare(`
SELECT id, weight FROM graph_edges
WHERE from_id = ? OR to_id = ?
`).all(nodeId, nodeId) as { id: string; weight: number }[];
// Strengthen each edge by RECONSOLIDATION_BOOST (5%)
const updateStmt = db['db'].prepare(`
UPDATE graph_edges
SET weight = MIN(1.0, weight * ?)
WHERE id = ?
`);
for (const edge of edges) {
const newWeight = Math.min(1.0, edge.weight * (1 + RECONSOLIDATION_BOOST));
if (newWeight > edge.weight) {
updateStmt.run(newWeight, edge.id);
strengthened++;
}
}
} catch {
// Reconsolidation is optional, don't fail the cycle
}
return strengthened;
}
// ============================================================================
// REM CYCLE MAIN LOGIC
// ============================================================================
/**
* Get nodes that have few or no connections
*/
function getDisconnectedNodes(db: VestigeDatabase, maxEdges: number = 1): KnowledgeNode[] {
// Get all nodes
const result = db.getRecentNodes({ limit: 500 });
const allNodes = result.items;
// Filter to nodes with few connections
const disconnected: KnowledgeNode[] = [];
for (const node of allNodes) {
const related = db.getRelatedNodes(node.id, 1);
if (related.length <= maxEdges) {
disconnected.push(node);
}
}
return disconnected;
}
/**
* Run one REM cycle - discover and create connections
*
* The cycle performs these steps:
* 1. Reconsolidate recently accessed nodes (strengthen existing connections)
* 2. Find disconnected nodes
* 3. Try semantic similarity first (if enabled and available)
* 4. Fall back to traditional analysis (Jaccard, shared concepts, etc.)
* 5. Apply emotional weighting to all connections
* 6. Run spreading activation to find transitive connections
*/
export async function runREMCycle(
db: VestigeDatabase,
options: REMCycleOptions = {}
): Promise<REMCycleResult> {
const startTime = Date.now();
const {
maxAnalyze = 50,
minStrength = 0.3,
dryRun = false,
enableSemantic = true,
enableSpreadingActivation = true,
spreadingActivationDepth = 2,
recentlyAccessedIds = [],
} = options;
const result: REMCycleResult = {
nodesAnalyzed: 0,
connectionsDiscovered: 0,
connectionsCreated: 0,
spreadingActivationEdges: 0,
reconsolidatedNodes: 0,
duration: 0,
semanticEnabled: false,
discoveries: [],
};
// Step 1: Reconsolidate recently accessed nodes
if (!dryRun && recentlyAccessedIds.length > 0) {
for (const nodeId of recentlyAccessedIds) {
const strengthened = reconsolidateConnections(db, nodeId);
if (strengthened > 0) {
result.reconsolidatedNodes++;
}
}
}
// Step 2: Initialize embedding service if semantic analysis is enabled
let embeddingService: EmbeddingService | null = null;
let embeddingCache: EmbeddingCache | null = null;
if (enableSemantic) {
try {
embeddingService = await createEmbeddingService();
const isAvailable = await embeddingService.isAvailable();
result.semanticEnabled = isAvailable;
if (isAvailable) {
embeddingCache = new EmbeddingCache(500, 30); // 500 entries, 30 min TTL
}
} catch {
// Semantic analysis not available, continue without it
result.semanticEnabled = false;
}
}
// Step 3: Get disconnected nodes
const disconnected = getDisconnectedNodes(db, 2);
if (disconnected.length < 2) {
result.duration = Date.now() - startTime;
return result;
}
// Limit analysis
const toAnalyze = disconnected.slice(0, maxAnalyze);
result.nodesAnalyzed = toAnalyze.length;
// Step 4: Compare pairs
const discoveries: DiscoveredConnection[] = [];
const analyzed = new Set<string>();
for (let i = 0; i < toAnalyze.length; i++) {
for (let j = i + 1; j < toAnalyze.length; j++) {
const nodeA = toAnalyze[i];
const nodeB = toAnalyze[j];
if (!nodeA || !nodeB) continue;
// Skip if already have an edge
const pairKey = [nodeA.id, nodeB.id].sort().join('-');
if (analyzed.has(pairKey)) continue;
analyzed.add(pairKey);
let connection: DiscoveredConnection | null = null;
// Try semantic similarity first if available
if (result.semanticEnabled && embeddingService && embeddingCache) {
connection = await analyzeSemanticConnection(
nodeA,
nodeB,
embeddingService,
embeddingCache
);
}
// Fall back to traditional analysis if no semantic connection found
if (!connection) {
connection = analyzeTraditionalConnection(nodeA, nodeB);
}
if (connection && connection.strength >= minStrength) {
discoveries.push(connection);
}
}
}
result.connectionsDiscovered = discoveries.length;
// Step 5: Create edges for discovered connections
if (!dryRun) {
for (const discovery of discoveries) {
try {
db.insertEdge({
fromId: discovery.nodeA.id,
toId: discovery.nodeB.id,
edgeType: 'similar_to',
weight: discovery.strength,
metadata: {
discoveredBy: 'rem_cycle',
reason: discovery.reason,
connectionType: discovery.connectionType,
},
createdAt: new Date(),
});
result.connectionsCreated++;
result.discoveries.push({
nodeA: discovery.nodeA.content.slice(0, 50),
nodeB: discovery.nodeB.content.slice(0, 50),
reason: discovery.reason,
type: discovery.connectionType,
});
} catch {
// Edge might already exist
}
}
// Step 6: Apply spreading activation
if (enableSpreadingActivation) {
const spreadingResult = applySpreadingActivation(db, spreadingActivationDepth, minStrength);
result.spreadingActivationEdges = spreadingResult.edgesCreated;
// Add spreading activation discoveries to results
for (const path of spreadingResult.paths) {
result.discoveries.push({
nodeA: path.from.slice(0, 20),
nodeB: path.to.slice(0, 20),
reason: `Transitive via ${path.via.slice(0, 20)} (${(path.weight * 100).toFixed(0)}%)`,
type: 'spreading_activation',
});
}
}
} else {
// Dry run - just record discoveries
for (const discovery of discoveries) {
result.discoveries.push({
nodeA: discovery.nodeA.content.slice(0, 50),
nodeB: discovery.nodeB.content.slice(0, 50),
reason: discovery.reason,
type: discovery.connectionType,
});
}
}
result.duration = Date.now() - startTime;
return result;
}
/**
* Get a summary of potential discoveries without creating edges
*/
export async function previewREMCycle(db: VestigeDatabase): Promise<REMCycleResult> {
return runREMCycle(db, { dryRun: true, maxAnalyze: 100 });
}
/**
* Trigger reconsolidation for a specific node
* Call this when a node is accessed to strengthen its connections
*/
export function triggerReconsolidation(db: VestigeDatabase, nodeId: string): number {
return reconsolidateConnections(db, nodeId);
}

1013
packages/core/src/core/security.ts
View file

File diff suppressed because it is too large Load diff

403
packages/core/src/core/shadow-self.ts
View file

@ -1,403 +0,0 @@
/**
* The Shadow Self - Unsolved Problems Queue
*
* "Your subconscious that keeps working while you're not looking."
*
* When you say "I don't know how to fix this," Vestige logs it.
* The Shadow periodically re-attacks these problems with new context.
*
* This turns Vestige from a passive memory into an active problem-solver.
*/
import Database from 'better-sqlite3';
import { nanoid } from 'nanoid';
import path from 'path';
import fs from 'fs';
import os from 'os';
// ============================================================================
// TYPES
// ============================================================================
export interface UnsolvedProblem {
id: string;
description: string;
context: string; // Original context when problem was logged
tags: string[];
status: 'open' | 'investigating' | 'solved' | 'abandoned';
priority: number; // 1-5, higher = more urgent
attempts: number; // How many times Shadow has tried to solve it
lastAttemptAt: Date | null;
createdAt: Date;
updatedAt: Date;
solution: string | null; // If solved, what was the solution?
relatedNodeIds: string[]; // Knowledge nodes that might help
}
export interface ShadowInsight {
problemId: string;
insight: string;
source: 'keyword_match' | 'new_knowledge' | 'pattern_recognition';
confidence: number;
relatedNodeIds: string[];
createdAt: Date;
}
// ============================================================================
// DATABASE SETUP
// ============================================================================
const SHADOW_DB_PATH = path.join(os.homedir(), '.vestige', 'shadow.db');
function initializeShadowDb(): Database.Database {
const dir = path.dirname(SHADOW_DB_PATH);
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir, { recursive: true });
}
const db = new Database(SHADOW_DB_PATH);
db.pragma('journal_mode = WAL');
db.pragma('busy_timeout = 5000');
// Unsolved problems table
db.exec(`
CREATE TABLE IF NOT EXISTS unsolved_problems (
id TEXT PRIMARY KEY,
description TEXT NOT NULL,
context TEXT,
tags TEXT DEFAULT '[]',
status TEXT DEFAULT 'open',
priority INTEGER DEFAULT 3,
attempts INTEGER DEFAULT 0,
last_attempt_at TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
solution TEXT,
related_node_ids TEXT DEFAULT '[]'
);
CREATE INDEX IF NOT EXISTS idx_problems_status ON unsolved_problems(status);
CREATE INDEX IF NOT EXISTS idx_problems_priority ON unsolved_problems(priority);
`);
// Insights discovered by Shadow
db.exec(`
CREATE TABLE IF NOT EXISTS shadow_insights (
id TEXT PRIMARY KEY,
problem_id TEXT NOT NULL,
insight TEXT NOT NULL,
source TEXT NOT NULL,
confidence REAL DEFAULT 0.5,
related_node_ids TEXT DEFAULT '[]',
created_at TEXT NOT NULL,
FOREIGN KEY (problem_id) REFERENCES unsolved_problems(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_insights_problem ON shadow_insights(problem_id);
`);
return db;
}
// ============================================================================
// SHADOW SELF CLASS
// ============================================================================
export class ShadowSelf {
private db: Database.Database;
constructor() {
this.db = initializeShadowDb();
}
/**
* Log a new unsolved problem
*/
logProblem(description: string, options: {
context?: string;
tags?: string[];
priority?: number;
} = {}): UnsolvedProblem {
const id = nanoid();
const now = new Date().toISOString();
const stmt = this.db.prepare(`
INSERT INTO unsolved_problems (
id, description, context, tags, status, priority,
attempts, last_attempt_at, created_at, updated_at,
solution, related_node_ids
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
description,
options.context || '',
JSON.stringify(options.tags || []),
'open',
options.priority || 3,
0,
null,
now,
now,
null,
'[]'
);
return this.getProblem(id)!;
}
/**
* Get a specific problem
*/
getProblem(id: string): UnsolvedProblem | null {
const stmt = this.db.prepare('SELECT * FROM unsolved_problems WHERE id = ?');
const row = stmt.get(id) as Record<string, unknown> | undefined;
if (!row) return null;
return this.rowToProblem(row);
}
/**
* Get all open problems
*/
getOpenProblems(): UnsolvedProblem[] {
const stmt = this.db.prepare(`
SELECT * FROM unsolved_problems
WHERE status IN ('open', 'investigating')
ORDER BY priority DESC, created_at ASC
`);
const rows = stmt.all() as Record<string, unknown>[];
return rows.map(row => this.rowToProblem(row));
}
/**
* Update problem status
*/
updateStatus(id: string, status: UnsolvedProblem['status'], solution?: string): void {
const now = new Date().toISOString();
const stmt = this.db.prepare(`
UPDATE unsolved_problems
SET status = ?, solution = ?, updated_at = ?
WHERE id = ?
`);
stmt.run(status, solution || null, now, id);
}
/**
* Mark problem as solved
*/
markSolved(id: string, solution: string): void {
this.updateStatus(id, 'solved', solution);
}
/**
* Add insight to a problem
*/
addInsight(problemId: string, insight: string, options: {
source?: ShadowInsight['source'];
confidence?: number;
relatedNodeIds?: string[];
} = {}): ShadowInsight {
const id = nanoid();
const now = new Date().toISOString();
const stmt = this.db.prepare(`
INSERT INTO shadow_insights (
id, problem_id, insight, source, confidence, related_node_ids, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
problemId,
insight,
options.source || 'keyword_match',
options.confidence || 0.5,
JSON.stringify(options.relatedNodeIds || []),
now
);
// Update problem attempt count
this.db.prepare(`
UPDATE unsolved_problems
SET attempts = attempts + 1,
last_attempt_at = ?,
status = 'investigating',
updated_at = ?
WHERE id = ?
`).run(now, now, problemId);
return {
id,
problemId,
insight,
source: options.source || 'keyword_match',
confidence: options.confidence || 0.5,
relatedNodeIds: options.relatedNodeIds || [],
createdAt: new Date(now),
};
}
/**
* Get insights for a problem
*/
getInsights(problemId: string): ShadowInsight[] {
const stmt = this.db.prepare(`
SELECT * FROM shadow_insights
WHERE problem_id = ?
ORDER BY created_at DESC
`);
const rows = stmt.all(problemId) as Record<string, unknown>[];
return rows.map(row => ({
id: row['id'] as string,
problemId: row['problem_id'] as string,
insight: row['insight'] as string,
source: row['source'] as ShadowInsight['source'],
confidence: row['confidence'] as number,
relatedNodeIds: JSON.parse(row['related_node_ids'] as string || '[]'),
createdAt: new Date(row['created_at'] as string),
}));
}
/**
* Get problems that haven't been worked on recently
*/
getStaleProblems(hoursSinceLastAttempt: number = 24): UnsolvedProblem[] {
const cutoff = new Date(Date.now() - hoursSinceLastAttempt * 60 * 60 * 1000);
const stmt = this.db.prepare(`
SELECT * FROM unsolved_problems
WHERE status IN ('open', 'investigating')
AND (last_attempt_at IS NULL OR last_attempt_at < ?)
ORDER BY priority DESC
`);
const rows = stmt.all(cutoff.toISOString()) as Record<string, unknown>[];
return rows.map(row => this.rowToProblem(row));
}
/**
* Get statistics
*/
getStats(): {
total: number;
open: number;
investigating: number;
solved: number;
abandoned: number;
totalInsights: number;
} {
const statusCounts = this.db.prepare(`
SELECT status, COUNT(*) as count FROM unsolved_problems GROUP BY status
`).all() as { status: string; count: number }[];
const insightCount = this.db.prepare(`
SELECT COUNT(*) as count FROM shadow_insights
`).get() as { count: number };
const stats = {
total: 0,
open: 0,
investigating: 0,
solved: 0,
abandoned: 0,
totalInsights: insightCount.count,
};
for (const { status, count } of statusCounts) {
stats.total += count;
if (status === 'open') stats.open = count;
if (status === 'investigating') stats.investigating = count;
if (status === 'solved') stats.solved = count;
if (status === 'abandoned') stats.abandoned = count;
}
return stats;
}
private rowToProblem(row: Record<string, unknown>): UnsolvedProblem {
return {
id: row['id'] as string,
description: row['description'] as string,
context: row['context'] as string,
tags: JSON.parse(row['tags'] as string || '[]'),
status: row['status'] as UnsolvedProblem['status'],
priority: row['priority'] as number,
attempts: row['attempts'] as number,
lastAttemptAt: row['last_attempt_at'] ? new Date(row['last_attempt_at'] as string) : null,
createdAt: new Date(row['created_at'] as string),
updatedAt: new Date(row['updated_at'] as string),
solution: row['solution'] as string | null,
relatedNodeIds: JSON.parse(row['related_node_ids'] as string || '[]'),
};
}
close(): void {
this.db.close();
}
}
// ============================================================================
// SHADOW WORK - Background processing
// ============================================================================
import { VestigeDatabase } from './database.js';
/**
* Run Shadow work cycle - look for new insights on unsolved problems
*/
export function runShadowCycle(shadow: ShadowSelf, vestige: VestigeDatabase): {
problemsAnalyzed: number;
insightsGenerated: number;
insights: Array<{ problem: string; insight: string }>;
} {
const result = {
problemsAnalyzed: 0,
insightsGenerated: 0,
insights: [] as Array<{ problem: string; insight: string }>,
};
// Get stale problems that need attention
const problems = shadow.getStaleProblems(1); // Haven't been worked on in 1 hour
for (const problem of problems) {
result.problemsAnalyzed++;
// Extract keywords from problem description
const keywords = problem.description
.toLowerCase()
.split(/\W+/)
.filter(w => w.length > 4);
// Search knowledge base for related content
for (const keyword of keywords.slice(0, 5)) {
try {
const searchResult = vestige.searchNodes(keyword, { limit: 3 });
for (const node of searchResult.items) {
// Check if this node was added after the problem
if (node.createdAt > problem.createdAt) {
// New knowledge! This might help
shadow.addInsight(problem.id, `New knowledge found: "${node.content.slice(0, 100)}..."`, {
source: 'new_knowledge',
confidence: 0.6,
relatedNodeIds: [node.id],
});
result.insightsGenerated++;
result.insights.push({
problem: problem.description.slice(0, 50),
insight: `Found related: ${node.content.slice(0, 50)}...`,
});
}
}
} catch {
// Ignore search errors
}
}
}
return result;
}

314
packages/core/src/core/types.ts
View file

@ -1,314 +0,0 @@
import { z } from 'zod';
// ============================================================================
// SOURCE TYPES
// ============================================================================
export const SourceTypeSchema = z.enum([
'note',
'conversation',
'email',
'book',
'article',
'highlight',
'meeting',
'manual',
'webpage',
]);
export type SourceType = z.infer<typeof SourceTypeSchema>;
export const SourcePlatformSchema = z.enum([
'obsidian',
'notion',
'roam',
'logseq',
'claude',
'chatgpt',
'gmail',
'outlook',
'kindle',
'readwise',
'pocket',
'instapaper',
'manual',
'browser',
]);
export type SourcePlatform = z.infer<typeof SourcePlatformSchema>;
// ============================================================================
// KNOWLEDGE NODE
// ============================================================================
export const KnowledgeNodeSchema = z.object({
id: z.string(),
content: z.string(),
summary: z.string().optional(),
// Temporal metadata
createdAt: z.date(),
updatedAt: z.date(),
lastAccessedAt: z.date(),
accessCount: z.number().default(0),
// Decay modeling (SM-2 inspired spaced repetition)
retentionStrength: z.number().min(0).max(1).default(1),
stabilityFactor: z.number().min(1).optional().default(1), // Grows with reviews, flattens decay curve
sentimentIntensity: z.number().min(0).max(1).optional().default(0), // Emotional weight - higher = decays slower
nextReviewDate: z.date().optional(),
reviewCount: z.number().default(0),
// Dual-Strength Memory Model (Bjork & Bjork, 1992)
storageStrength: z.number().min(1).default(1), // How well encoded (never decreases)
retrievalStrength: z.number().min(0).max(1).default(1), // How accessible now (decays)
// Provenance
sourceType: SourceTypeSchema,
sourcePlatform: SourcePlatformSchema,
sourceId: z.string().optional(), // Original source reference
sourceUrl: z.string().optional(),
sourceChain: z.array(z.string()).default([]), // Full provenance path
// Git-Blame for Thoughts - what code was being worked on when this memory was created?
gitContext: z.object({
branch: z.string().optional(),
commit: z.string().optional(), // Short SHA
commitMessage: z.string().optional(), // First line of commit message
repoPath: z.string().optional(), // Repository root path
dirty: z.boolean().optional(), // Had uncommitted changes?
changedFiles: z.array(z.string()).optional(), // Files with uncommitted changes
}).optional(),
// Confidence & quality
confidence: z.number().min(0).max(1).default(0.8),
isContradicted: z.boolean().default(false),
contradictionIds: z.array(z.string()).default([]),
// Extracted entities
people: z.array(z.string()).default([]),
concepts: z.array(z.string()).default([]),
events: z.array(z.string()).default([]),
tags: z.array(z.string()).default([]),
});
export type KnowledgeNode = z.infer<typeof KnowledgeNodeSchema>;
// Input type where optional/default fields are truly optional (for insertNode)
export type KnowledgeNodeInput = z.input<typeof KnowledgeNodeSchema>;
// ============================================================================
// PERSON NODE (People Memory / Mini-CRM)
// ============================================================================
export const InteractionTypeSchema = z.enum([
'meeting',
'email',
'call',
'message',
'social',
'collaboration',
'mention', // Referenced in notes but not direct interaction
]);
export type InteractionType = z.infer<typeof InteractionTypeSchema>;
export const InteractionSchema = z.object({
id: z.string(),
personId: z.string(),
type: InteractionTypeSchema,
date: z.date(),
summary: z.string(),
topics: z.array(z.string()).default([]),
sentiment: z.number().min(-1).max(1).optional(), // -1 negative, 0 neutral, 1 positive
actionItems: z.array(z.string()).default([]),
sourceNodeId: z.string().optional(), // Link to knowledge node if derived
});
export type Interaction = z.infer<typeof InteractionSchema>;
export const PersonNodeSchema = z.object({
id: z.string(),
name: z.string(),
aliases: z.array(z.string()).default([]),
// Relationship context
howWeMet: z.string().optional(),
relationshipType: z.string().optional(), // colleague, friend, mentor, family, etc.
organization: z.string().optional(),
role: z.string().optional(),
location: z.string().optional(),
// Contact info
email: z.string().optional(),
phone: z.string().optional(),
socialLinks: z.record(z.string()).default({}),
// Communication patterns
lastContactAt: z.date().optional(),
contactFrequency: z.number().default(0), // Interactions per month (calculated)
preferredChannel: z.string().optional(),
// Shared context
sharedTopics: z.array(z.string()).default([]),
sharedProjects: z.array(z.string()).default([]),
// Meta
notes: z.string().optional(),
relationshipHealth: z.number().min(0).max(1).default(0.5), // Calculated from recency + frequency
createdAt: z.date(),
updatedAt: z.date(),
});
export type PersonNode = z.infer<typeof PersonNodeSchema>;
// ============================================================================
// GRAPH EDGES (Relationships)
// ============================================================================
export const EdgeTypeSchema = z.enum([
'relates_to',
'derived_from',
'contradicts',
'supports',
'references',
'part_of',
'follows', // Temporal sequence
'person_mentioned',
'concept_instance',
'similar_to',
]);
export type EdgeType = z.infer<typeof EdgeTypeSchema>;
export const GraphEdgeSchema = z.object({
id: z.string(),
fromId: z.string(),
toId: z.string(),
edgeType: EdgeTypeSchema,
weight: z.number().min(0).max(1).default(0.5),
metadata: z.record(z.unknown()).default({}),
createdAt: z.date(),
});
export type GraphEdge = z.infer<typeof GraphEdgeSchema>;
// ============================================================================
// SOURCE TRACKING
// ============================================================================
export const SourceSchema = z.object({
id: z.string(),
type: SourceTypeSchema,
platform: SourcePlatformSchema,
originalId: z.string().optional(),
url: z.string().optional(),
filePath: z.string().optional(),
title: z.string().optional(),
author: z.string().optional(),
publicationDate: z.date().optional(),
// Sync tracking
ingestedAt: z.date(),
lastSyncedAt: z.date(),
contentHash: z.string().optional(), // For change detection
// Stats
nodeCount: z.number().default(0),
});
export type Source = z.infer<typeof SourceSchema>;
// ============================================================================
// TOOL INPUT/OUTPUT SCHEMAS
// ============================================================================
export const IngestInputSchema = z.object({
content: z.string(),
source: SourceTypeSchema.optional().default('manual'),
platform: SourcePlatformSchema.optional().default('manual'),
sourceId: z.string().optional(),
sourceUrl: z.string().optional(),
timestamp: z.string().datetime().optional(),
people: z.array(z.string()).optional(),
tags: z.array(z.string()).optional(),
title: z.string().optional(),
});
export type IngestInput = z.infer<typeof IngestInputSchema>;
export const RecallOptionsSchema = z.object({
query: z.string(),
timeRange: z.object({
start: z.string().datetime().optional(),
end: z.string().datetime().optional(),
}).optional(),
sources: z.array(SourceTypeSchema).optional(),
platforms: z.array(SourcePlatformSchema).optional(),
people: z.array(z.string()).optional(),
minConfidence: z.number().min(0).max(1).optional(),
limit: z.number().min(1).max(100).optional().default(10),
includeContext: z.boolean().optional().default(true),
});
export type RecallOptions = z.infer<typeof RecallOptionsSchema>;
export const RecallResultSchema = z.object({
node: KnowledgeNodeSchema,
score: z.number(),
matchType: z.enum(['semantic', 'keyword', 'graph']),
context: z.string().optional(),
relatedNodes: z.array(z.string()).optional(),
});
export type RecallResult = z.infer<typeof RecallResultSchema>;
export const SynthesisOptionsSchema = z.object({
topic: z.string(),
depth: z.enum(['shallow', 'deep']).optional().default('shallow'),
format: z.enum(['summary', 'outline', 'narrative']).optional().default('summary'),
maxSources: z.number().optional().default(20),
});
export type SynthesisOptions = z.infer<typeof SynthesisOptionsSchema>;
// ============================================================================
// DECAY MODELING
// ============================================================================
export interface DecayConfig {
// Ebbinghaus forgetting curve parameters
initialRetention: number; // Starting retention (default 1.0)
decayRate: number; // Base decay rate (default ~0.9 for typical forgetting)
minRetention: number; // Floor retention (default 0.1)
reviewBoost: number; // How much review increases retention (default 0.3)
accessBoost: number; // How much access slows decay (default 0.1)
}
export const DEFAULT_DECAY_CONFIG: DecayConfig = {
initialRetention: 1.0,
decayRate: 0.9,
minRetention: 0.1,
reviewBoost: 0.3,
accessBoost: 0.1,
};
// ============================================================================
// DAILY BRIEF
// ============================================================================
export const DailyBriefSchema = z.object({
date: z.date(),
stats: z.object({
totalNodes: z.number(),
addedToday: z.number(),
addedThisWeek: z.number(),
connectionsDiscovered: z.number(),
}),
reviewDue: z.array(z.object({
nodeId: z.string(),
summary: z.string(),
lastAccessed: z.date(),
retentionStrength: z.number(),
})),
peopleToReconnect: z.array(z.object({
personId: z.string(),
name: z.string(),
daysSinceContact: z.number(),
sharedTopics: z.array(z.string()),
})),
interestingConnections: z.array(z.object({
nodeA: z.string(),
nodeB: z.string(),
connectionReason: z.string(),
})),
recentThemes: z.array(z.string()),
});
export type DailyBrief = z.infer<typeof DailyBriefSchema>;

1154
packages/core/src/core/vector-store.ts
View file

File diff suppressed because it is too large Load diff

1353
packages/core/src/index.ts
View file

File diff suppressed because it is too large Load diff

181
packages/core/src/jobs/ConsolidationJob.ts
View file

@ -1,181 +0,0 @@
/**
* ConsolidationJob - Knowledge Consolidation Processing
*
* Consolidates related knowledge nodes by:
* - Merging highly similar nodes
* - Strengthening frequently co-accessed node clusters
* - Pruning orphaned edges
* - Optimizing the database
*
* Designed to run as a scheduled background job (e.g., weekly).
*
* @module jobs/ConsolidationJob
*/
import type { VestigeDatabase } from '../core/database.js';
import type { Job, JobHandler } from './JobQueue.js';
// ============================================================================
// TYPES
// ============================================================================
export interface ConsolidationJobData {
/** Minimum similarity threshold for merging nodes (0-1). Default: 0.95 */
mergeThreshold?: number;
/** Whether to prune orphaned edges. Default: true */
pruneOrphanedEdges?: boolean;
/** Whether to optimize database after consolidation. Default: true */
optimizeDb?: boolean;
/** Whether to run in dry-run mode (analysis only). Default: false */
dryRun?: boolean;
}
export interface ConsolidationJobResult {
/** Number of node pairs analyzed for similarity */
pairsAnalyzed: number;
/** Number of nodes merged (dry run: would be merged) */
nodesMerged: number;
/** Number of orphaned edges pruned */
edgesPruned: number;
/** Number of edge weights updated (strengthened) */
edgesStrengthened: number;
/** Whether database optimization was performed */
databaseOptimized: boolean;
/** Time taken in milliseconds */
duration: number;
/** Timestamp when the job ran */
timestamp: Date;
}
// ============================================================================
// CONSOLIDATION LOGIC
// ============================================================================
/**
* Run knowledge consolidation on the database
*/
async function runConsolidation(
db: VestigeDatabase,
options: {
mergeThreshold?: number;
pruneOrphanedEdges?: boolean;
optimizeDb?: boolean;
dryRun?: boolean;
} = {}
): Promise<ConsolidationJobResult> {
const startTime = Date.now();
const {
mergeThreshold = 0.95,
pruneOrphanedEdges = true,
optimizeDb = true,
dryRun = false,
} = options;
const result: ConsolidationJobResult = {
pairsAnalyzed: 0,
nodesMerged: 0,
edgesPruned: 0,
edgesStrengthened: 0,
databaseOptimized: false,
duration: 0,
timestamp: new Date(),
};
// Step 1: Analyze and strengthen co-accessed clusters
// (Nodes accessed together frequently should have stronger edges)
const stats = db.getStats();
result.pairsAnalyzed = Math.min(stats.totalNodes * (stats.totalNodes - 1) / 2, 10000);
// Step 2: Prune orphaned edges (edges pointing to deleted nodes)
// In a real implementation, this would query for edges with invalid node references
if (pruneOrphanedEdges && !dryRun) {
// The database foreign keys should handle this, but we can do a sanity check
// For now, we just report 0 pruned as SQLite handles this via ON DELETE CASCADE
result.edgesPruned = 0;
}
// Step 3: Optimize database
if (optimizeDb && !dryRun) {
try {
db.optimize();
result.databaseOptimized = true;
} catch {
// Log but don't fail the job
result.databaseOptimized = false;
}
}
result.duration = Date.now() - startTime;
return result;
}
// ============================================================================
// JOB HANDLER FACTORY
// ============================================================================
/**
* Create a consolidation job handler
*
* @param db - VestigeDatabase instance
* @returns Job handler function
*
* @example
* ```typescript
* const db = new VestigeDatabase();
* const queue = new JobQueue();
*
* queue.register('consolidation', createConsolidationJobHandler(db), {
* concurrency: 1, // Only one consolidation at a time
* retryDelay: 3600000, // Wait 1 hour before retry
* });
*
* // Schedule to run weekly on Sunday at 4 AM
* queue.schedule('consolidation', '0 4 * * 0', {});
* ```
*/
export function createConsolidationJobHandler(
db: VestigeDatabase
): JobHandler<ConsolidationJobData, ConsolidationJobResult> {
return async (job: Job<ConsolidationJobData>): Promise<ConsolidationJobResult> => {
return runConsolidation(db, {
mergeThreshold: job.data.mergeThreshold,
pruneOrphanedEdges: job.data.pruneOrphanedEdges,
optimizeDb: job.data.optimizeDb,
dryRun: job.data.dryRun,
});
};
}
// ============================================================================
// HELPER FUNCTIONS
// ============================================================================
/**
* Preview what consolidation would do without making changes
*/
export async function previewConsolidation(
db: VestigeDatabase
): Promise<ConsolidationJobResult> {
return runConsolidation(db, { dryRun: true });
}
/**
* Get database health metrics relevant to consolidation
*/
export function getConsolidationMetrics(db: VestigeDatabase): {
totalNodes: number;
totalEdges: number;
databaseSizeMB: number;
needsOptimization: boolean;
} {
const stats = db.getStats();
const size = db.getDatabaseSize();
const health = db.checkHealth();
return {
totalNodes: stats.totalNodes,
totalEdges: stats.totalEdges,
databaseSizeMB: size.mb,
needsOptimization: health.status !== 'healthy' || size.mb > 50,
};
}

98
packages/core/src/jobs/DecayJob.ts
View file

@ -1,98 +0,0 @@
/**
* DecayJob - Memory Decay Processing
*
* Applies the Ebbinghaus forgetting curve to all knowledge nodes,
* updating their retention strength based on time since last access.
*
* Designed to run as a scheduled background job (e.g., daily at 3 AM).
*
* @module jobs/DecayJob
*/
import type { VestigeDatabase } from '../core/database.js';
import type { Job, JobHandler } from './JobQueue.js';
// ============================================================================
// TYPES
// ============================================================================
export interface DecayJobData {
/** Optional: Minimum retention threshold to skip already-decayed nodes */
minRetention?: number;
/** Optional: Maximum number of nodes to process in one batch */
batchSize?: number;
}
export interface DecayJobResult {
/** Number of nodes whose retention was updated */
updatedCount: number;
/** Total time taken in milliseconds */
processingTime: number;
/** Timestamp when the job ran */
timestamp: Date;
}
// ============================================================================
// JOB HANDLER FACTORY
// ============================================================================
/**
* Create a decay job handler
*
* @param db - VestigeDatabase instance
* @returns Job handler function
*
* @example
* ```typescript
* const db = new VestigeDatabase();
* const queue = new JobQueue();
*
* queue.register('decay', createDecayJobHandler(db), {
* concurrency: 1, // Only one decay job at a time
* retryDelay: 60000, // Wait 1 minute before retry
* });
*
* // Schedule to run daily at 3 AM
* queue.schedule('decay', '0 3 * * *', {});
* ```
*/
export function createDecayJobHandler(
db: VestigeDatabase
): JobHandler<DecayJobData, DecayJobResult> {
return async (job: Job<DecayJobData>): Promise<DecayJobResult> => {
const startTime = Date.now();
// Apply decay to all nodes
// The database method handles the Ebbinghaus curve calculation
const updatedCount = db.applyDecay();
const result: DecayJobResult = {
updatedCount,
processingTime: Date.now() - startTime,
timestamp: new Date(),
};
return result;
};
}
// ============================================================================
// HELPER FUNCTIONS
// ============================================================================
/**
* Get nodes that are critically decayed (retention < threshold)
* Useful for generating review notifications
*/
export async function getCriticallyDecayedNodes(
db: VestigeDatabase,
threshold: number = 0.3
): Promise<{ nodeId: string; retention: number; content: string }[]> {
const result = db.getDecayingNodes(threshold, { limit: 50 });
return result.items.map(node => ({
nodeId: node.id,
retention: node.retentionStrength,
content: node.content.slice(0, 100),
}));
}

809
packages/core/src/jobs/JobQueue.ts
View file

@ -1,809 +0,0 @@
/**
* JobQueue - Background Job Processing for Vestige MCP
*
* A production-ready in-memory job queue with:
* - Priority-based job scheduling
* - Retry logic with exponential backoff
* - Concurrency control per job type
* - Event-driven architecture
* - Cron-like scheduling support
*
* @module jobs/JobQueue
*/
import { EventEmitter } from 'events';
import { nanoid } from 'nanoid';
// ============================================================================
// TYPES
// ============================================================================
export type JobStatus = 'pending' | 'running' | 'completed' | 'failed';
export interface Job<T = unknown> {
id: string;
name: string;
data: T;
priority: number;
createdAt: Date;
scheduledAt?: Date;
startedAt?: Date;
completedAt?: Date;
retryCount: number;
maxRetries: number;
status: JobStatus;
error?: string;
}
export interface JobResult<R = unknown> {
jobId: string;
success: boolean;
result?: R;
error?: Error;
duration: number;
}
export type JobHandler<T, R> = (job: Job<T>) => Promise<R>;
export interface JobOptions {
/** Priority (higher = processed first). Default: 0 */
priority?: number;
/** Delay in milliseconds before job becomes eligible. Default: 0 */
delay?: number;
/** Maximum retry attempts on failure. Default: 3 */
maxRetries?: number;
}
export interface JobDefinition<T = unknown, R = unknown> {
name: string;
handler: JobHandler<T, R>;
concurrency: number;
retryDelay: number;
}
export interface ScheduledJob {
name: string;
cronExpression: string;
data: unknown;
lastRun?: Date;
nextRun?: Date;
}
export interface QueueStats {
pending: number;
running: number;
completed: number;
failed: number;
total: number;
}
// ============================================================================
// JOB QUEUE EVENTS
// ============================================================================
export interface JobQueueEvents {
'job:added': (job: Job) => void;
'job:started': (job: Job) => void;
'job:completed': (job: Job, result: JobResult) => void;
'job:failed': (job: Job, error: Error) => void;
'job:retry': (job: Job, attempt: number, error: Error) => void;
'queue:drained': () => void;
'queue:error': (error: Error) => void;
}
// ============================================================================
// CRON PARSER (Simple Implementation)
// ============================================================================
interface CronFields {
minute: number[];
hour: number[];
dayOfMonth: number[];
month: number[];
dayOfWeek: number[];
}
/**
* Parse a simple cron expression
* Format: minute hour day-of-month month day-of-week
* Supports: numbers, *, /step, ranges (-)
*/
function parseCronField(field: string, min: number, max: number): number[] {
const values: number[] = [];
// Handle wildcard
if (field === '*') {
for (let i = min; i <= max; i++) {
values.push(i);
}
return values;
}
// Handle step values (*/n or n/m)
if (field.includes('/')) {
const [range, stepStr] = field.split('/');
const step = parseInt(stepStr || '1', 10);
let start = min;
let end = max;
if (range && range !== '*') {
if (range.includes('-')) {
const [s, e] = range.split('-');
start = parseInt(s || String(min), 10);
end = parseInt(e || String(max), 10);
} else {
start = parseInt(range, 10);
}
}
for (let i = start; i <= end; i += step) {
values.push(i);
}
return values;
}
// Handle ranges (n-m)
if (field.includes('-')) {
const [start, end] = field.split('-');
const s = parseInt(start || String(min), 10);
const e = parseInt(end || String(max), 10);
for (let i = s; i <= e; i++) {
values.push(i);
}
return values;
}
// Handle comma-separated values
if (field.includes(',')) {
return field.split(',').map(v => parseInt(v.trim(), 10));
}
// Single value
values.push(parseInt(field, 10));
return values;
}
function parseCronExpression(expression: string): CronFields {
const parts = expression.trim().split(/\s+/);
if (parts.length !== 5) {
throw new Error(`Invalid cron expression: ${expression}. Expected 5 fields.`);
}
return {
minute: parseCronField(parts[0] || '*', 0, 59),
hour: parseCronField(parts[1] || '*', 0, 23),
dayOfMonth: parseCronField(parts[2] || '*', 1, 31),
month: parseCronField(parts[3] || '*', 1, 12),
dayOfWeek: parseCronField(parts[4] || '*', 0, 6),
};
}
function getNextCronDate(expression: string, after: Date = new Date()): Date {
const fields = parseCronExpression(expression);
const next = new Date(after);
next.setSeconds(0);
next.setMilliseconds(0);
// Start from next minute
next.setMinutes(next.getMinutes() + 1);
// Find next matching time (limit iterations to prevent infinite loops)
for (let iterations = 0; iterations < 525600; iterations++) { // Max 1 year of minutes
const minute = next.getMinutes();
const hour = next.getHours();
const dayOfMonth = next.getDate();
const month = next.getMonth() + 1; // JS months are 0-indexed
const dayOfWeek = next.getDay();
// Check if current time matches cron expression
if (
fields.minute.includes(minute) &&
fields.hour.includes(hour) &&
fields.dayOfMonth.includes(dayOfMonth) &&
fields.month.includes(month) &&
fields.dayOfWeek.includes(dayOfWeek)
) {
return next;
}
// Advance by one minute
next.setMinutes(next.getMinutes() + 1);
}
throw new Error(`Could not find next cron date within 1 year for: ${expression}`);
}
// ============================================================================
// JOB QUEUE IMPLEMENTATION
// ============================================================================
export class JobQueue extends EventEmitter {
private jobs: Map<string, Job> = new Map();
private handlers: Map<string, JobDefinition> = new Map();
private running: Map<string, number> = new Map();
private interval: NodeJS.Timeout | null = null;
private scheduledJobs: Map<string, ScheduledJob> = new Map();
private schedulerInterval: NodeJS.Timeout | null = null;
private isProcessing = false;
private isPaused = false;
// Completed/failed job history (limited size)
private readonly maxHistorySize = 1000;
private completedJobIds: Set<string> = new Set();
private failedJobIds: Set<string> = new Set();
constructor() {
super();
this.setMaxListeners(100);
}
// ============================================================================
// HANDLER REGISTRATION
// ============================================================================
/**
* Register a job handler
*
* @param name - Unique job type name
* @param handler - Async function to process the job
* @param options - Handler options (concurrency, retryDelay)
*
* @example
* ```typescript
* queue.register('send-email', async (job) => {
* await sendEmail(job.data);
* return { sent: true };
* }, { concurrency: 5, retryDelay: 5000 });
* ```
*/
register<T, R>(
name: string,
handler: JobHandler<T, R>,
options?: { concurrency?: number; retryDelay?: number }
): void {
if (this.handlers.has(name)) {
throw new Error(`Handler already registered for job type: ${name}`);
}
// Store as JobDefinition<unknown, unknown> since we type-erase at runtime
// The type safety is maintained at the call site (add/register)
const definition: JobDefinition = {
name,
handler: handler as unknown as JobHandler<unknown, unknown>,
concurrency: options?.concurrency ?? 1,
retryDelay: options?.retryDelay ?? 1000,
};
this.handlers.set(name, definition);
this.running.set(name, 0);
}
/**
* Unregister a job handler
*/
unregister(name: string): boolean {
const deleted = this.handlers.delete(name);
this.running.delete(name);
return deleted;
}
// ============================================================================
// JOB MANAGEMENT
// ============================================================================
/**
* Add a job to the queue
*
* @param name - Job type name (must have registered handler)
* @param data - Job data payload
* @param options - Job options (priority, delay, maxRetries)
* @returns Job ID
*
* @example
* ```typescript
* const jobId = queue.add('send-email', {
* to: 'user@example.com',
* subject: 'Hello'
* }, { priority: 10, maxRetries: 5 });
* ```
*/
add<T>(
name: string,
data: T,
options?: JobOptions
): string {
if (!this.handlers.has(name)) {
throw new Error(`No handler registered for job type: ${name}`);
}
const id = nanoid();
const now = new Date();
let scheduledAt: Date | undefined;
if (options?.delay && options.delay > 0) {
scheduledAt = new Date(now.getTime() + options.delay);
}
const job: Job<T> = {
id,
name,
data,
priority: options?.priority ?? 0,
createdAt: now,
scheduledAt,
retryCount: 0,
maxRetries: options?.maxRetries ?? 3,
status: 'pending',
};
this.jobs.set(id, job as Job);
this.emit('job:added', job);
// Trigger processing if running
if (this.isProcessing && !this.isPaused) {
this.processNextJobs();
}
return id;
}
/**
* Get a job by ID
*/
getJob(id: string): Job | undefined {
return this.jobs.get(id);
}
/**
* Get all jobs matching a filter
*/
getJobs(filter?: { name?: string; status?: JobStatus }): Job[] {
let jobs = Array.from(this.jobs.values());
if (filter?.name) {
jobs = jobs.filter(j => j.name === filter.name);
}
if (filter?.status) {
jobs = jobs.filter(j => j.status === filter.status);
}
return jobs;
}
/**
* Remove a job from the queue
* Can only remove pending jobs
*/
removeJob(id: string): boolean {
const job = this.jobs.get(id);
if (!job) return false;
if (job.status === 'running') {
throw new Error('Cannot remove a running job');
}
return this.jobs.delete(id);
}
/**
* Clear all completed/failed jobs from history
*/
clearHistory(): void {
for (const id of this.completedJobIds) {
this.jobs.delete(id);
}
for (const id of this.failedJobIds) {
this.jobs.delete(id);
}
this.completedJobIds.clear();
this.failedJobIds.clear();
}
// ============================================================================
// QUEUE STATISTICS
// ============================================================================
/**
* Get queue statistics
*/
getStats(): QueueStats {
let pending = 0;
let running = 0;
let completed = 0;
let failed = 0;
for (const job of this.jobs.values()) {
switch (job.status) {
case 'pending':
pending++;
break;
case 'running':
running++;
break;
case 'completed':
completed++;
break;
case 'failed':
failed++;
break;
}
}
return {
pending,
running,
completed,
failed,
total: this.jobs.size,
};
}
/**
* Check if queue is empty (no pending or running jobs)
*/
isEmpty(): boolean {
for (const job of this.jobs.values()) {
if (job.status === 'pending' || job.status === 'running') {
return false;
}
}
return true;
}
// ============================================================================
// PROCESSING
// ============================================================================
/**
* Start processing jobs
*
* @param pollInterval - How often to check for new jobs (ms). Default: 100
*/
start(pollInterval: number = 100): void {
if (this.isProcessing) {
return;
}
this.isProcessing = true;
this.isPaused = false;
this.interval = setInterval(() => {
if (!this.isPaused) {
this.processNextJobs();
}
}, pollInterval);
// Start scheduler for cron jobs
this.startScheduler();
// Process immediately
this.processNextJobs();
}
/**
* Stop processing jobs
*/
stop(): void {
this.isProcessing = false;
if (this.interval) {
clearInterval(this.interval);
this.interval = null;
}
this.stopScheduler();
}
/**
* Pause processing (jobs stay in queue)
*/
pause(): void {
this.isPaused = true;
}
/**
* Resume processing
*/
resume(): void {
this.isPaused = false;
this.processNextJobs();
}
/**
* Wait for all pending jobs to complete
*/
async drain(): Promise<void> {
return new Promise((resolve) => {
const check = () => {
if (this.isEmpty()) {
resolve();
} else {
setTimeout(check, 50);
}
};
check();
});
}
/**
* Process next eligible jobs
*/
private processNextJobs(): void {
const now = new Date();
// Get pending jobs sorted by priority (descending)
const pendingJobs = Array.from(this.jobs.values())
.filter(job => {
if (job.status !== 'pending') return false;
if (job.scheduledAt && job.scheduledAt > now) return false;
return true;
})
.sort((a, b) => b.priority - a.priority);
// Process jobs respecting concurrency limits
for (const job of pendingJobs) {
const definition = this.handlers.get(job.name);
if (!definition) continue;
const currentRunning = this.running.get(job.name) ?? 0;
if (currentRunning >= definition.concurrency) continue;
// Start processing this job
this.processJob(job, definition);
}
}
/**
* Process a single job
*/
private async processJob(job: Job, definition: JobDefinition): Promise<void> {
// Update job status
job.status = 'running';
job.startedAt = new Date();
// Track running count
const currentRunning = this.running.get(job.name) ?? 0;
this.running.set(job.name, currentRunning + 1);
this.emit('job:started', job);
const startTime = Date.now();
try {
const result = await definition.handler(job);
// Job completed successfully
job.status = 'completed';
job.completedAt = new Date();
const jobResult: JobResult = {
jobId: job.id,
success: true,
result,
duration: Date.now() - startTime,
};
this.emit('job:completed', job, jobResult);
// Track in history
this.addToHistory(job.id, 'completed');
} catch (error) {
const err = error instanceof Error ? error : new Error(String(error));
// Check if we should retry
if (job.retryCount < job.maxRetries) {
job.retryCount++;
job.status = 'pending';
// Schedule retry with exponential backoff
const backoffDelay = definition.retryDelay * Math.pow(2, job.retryCount - 1);
job.scheduledAt = new Date(Date.now() + backoffDelay);
this.emit('job:retry', job, job.retryCount, err);
} else {
// Max retries exceeded - mark as failed
job.status = 'failed';
job.completedAt = new Date();
job.error = err.message;
this.emit('job:failed', job, err);
// Track in history
this.addToHistory(job.id, 'failed');
}
} finally {
// Update running count
const runningCount = this.running.get(job.name) ?? 1;
this.running.set(job.name, Math.max(0, runningCount - 1));
// Check if queue is drained
if (this.isEmpty()) {
this.emit('queue:drained');
}
}
}
/**
* Add job to history tracking (with size limit)
*/
private addToHistory(jobId: string, type: 'completed' | 'failed'): void {
const targetSet = type === 'completed' ? this.completedJobIds : this.failedJobIds;
targetSet.add(jobId);
// Trim history if too large
if (targetSet.size > this.maxHistorySize) {
const iterator = targetSet.values();
const firstValue = iterator.next().value;
if (firstValue) {
targetSet.delete(firstValue);
this.jobs.delete(firstValue);
}
}
}
// ============================================================================
// SCHEDULING (CRON-LIKE)
// ============================================================================
/**
* Schedule a recurring job
*
* @param name - Job type name
* @param cronExpression - Cron expression (minute hour day-of-month month day-of-week)
* @param data - Job data payload
*
* @example
* ```typescript
* // Run decay at 3 AM daily
* queue.schedule('decay', '0 3 * * *', {});
*
* // Run REM cycle every 6 hours
* queue.schedule('rem-cycle', '0 *\\/6 * * *', {});
* ```
*/
schedule<T>(name: string, cronExpression: string, data: T): void {
if (!this.handlers.has(name)) {
throw new Error(`No handler registered for job type: ${name}`);
}
// Validate cron expression by parsing it
try {
parseCronExpression(cronExpression);
} catch (error) {
throw new Error(`Invalid cron expression for ${name}: ${cronExpression}`);
}
const scheduledJob: ScheduledJob = {
name,
cronExpression,
data,
nextRun: getNextCronDate(cronExpression),
};
this.scheduledJobs.set(name, scheduledJob);
}
/**
* Remove a scheduled job
*/
unschedule(name: string): boolean {
return this.scheduledJobs.delete(name);
}
/**
* Get all scheduled jobs
*/
getScheduledJobs(): ScheduledJob[] {
return Array.from(this.scheduledJobs.values());
}
/**
* Start the scheduler
*/
private startScheduler(): void {
if (this.schedulerInterval) return;
// Check every minute for scheduled jobs
this.schedulerInterval = setInterval(() => {
this.checkScheduledJobs();
}, 60000);
// Also check immediately
this.checkScheduledJobs();
}
/**
* Stop the scheduler
*/
private stopScheduler(): void {
if (this.schedulerInterval) {
clearInterval(this.schedulerInterval);
this.schedulerInterval = null;
}
}
/**
* Check and trigger scheduled jobs
*/
private checkScheduledJobs(): void {
const now = new Date();
for (const [name, scheduled] of this.scheduledJobs) {
if (scheduled.nextRun && scheduled.nextRun <= now) {
try {
// Add the job
this.add(name, scheduled.data);
// Update last run and calculate next run
scheduled.lastRun = now;
scheduled.nextRun = getNextCronDate(scheduled.cronExpression, now);
} catch (error) {
const err = error instanceof Error ? error : new Error(String(error));
this.emit('queue:error', err);
}
}
}
}
// ============================================================================
// CLEANUP
// ============================================================================
/**
* Graceful shutdown
*/
async shutdown(timeout: number = 30000): Promise<void> {
this.stop();
this.isPaused = true;
// Wait for running jobs to complete (with timeout)
const waitStart = Date.now();
while (Date.now() - waitStart < timeout) {
const stats = this.getStats();
if (stats.running === 0) {
break;
}
await new Promise(resolve => setTimeout(resolve, 100));
}
// Clear all jobs
this.jobs.clear();
this.completedJobIds.clear();
this.failedJobIds.clear();
this.scheduledJobs.clear();
this.removeAllListeners();
}
}
// ============================================================================
// SINGLETON INSTANCE (Optional)
// ============================================================================
let defaultQueue: JobQueue | null = null;
/**
* Get the default job queue instance
*/
export function getDefaultQueue(): JobQueue {
if (!defaultQueue) {
defaultQueue = new JobQueue();
}
return defaultQueue;
}
/**
* Reset the default queue (for testing)
*/
export function resetDefaultQueue(): void {
if (defaultQueue) {
defaultQueue.shutdown().catch(() => {});
defaultQueue = null;
}
}

132
packages/core/src/jobs/REMCycleJob.ts
View file

@ -1,132 +0,0 @@
/**
* REMCycleJob - Connection Discovery Processing
*
* Runs the REM (Rapid Eye Movement) cycle to discover hidden connections
* between knowledge nodes using semantic similarity, shared concepts,
* and keyword overlap analysis.
*
* Designed to run as a scheduled background job (e.g., every 6 hours).
*
* @module jobs/REMCycleJob
*/
import type { VestigeDatabase } from '../core/database.js';
import { runREMCycle } from '../core/rem-cycle.js';
import type { Job, JobHandler } from './JobQueue.js';
// ============================================================================
// TYPES
// ============================================================================
export interface REMCycleJobData {
/** Maximum number of nodes to analyze per cycle. Default: 50 */
maxAnalyze?: number;
/** Minimum connection strength threshold (0-1). Default: 0.3 */
minStrength?: number;
/** If true, only discover but don't create edges. Default: false */
dryRun?: boolean;
}
export interface REMCycleJobResult {
/** Number of nodes analyzed */
nodesAnalyzed: number;
/** Number of potential connections discovered */
connectionsDiscovered: number;
/** Number of graph edges actually created */
connectionsCreated: number;
/** Time taken in milliseconds */
duration: number;
/** Details of discovered connections */
discoveries: Array<{
nodeA: string;
nodeB: string;
reason: string;
}>;
/** Timestamp when the job ran */
timestamp: Date;
}
// ============================================================================
// JOB HANDLER FACTORY
// ============================================================================
/**
* Create a REM cycle job handler
*
* @param db - VestigeDatabase instance
* @returns Job handler function
*
* @example
* ```typescript
* const db = new VestigeDatabase();
* const queue = new JobQueue();
*
* queue.register('rem-cycle', createREMCycleJobHandler(db), {
* concurrency: 1, // Only one REM cycle at a time
* retryDelay: 300000, // Wait 5 minutes before retry
* });
*
* // Schedule to run every 6 hours
* queue.schedule('rem-cycle', '0 *\/6 * * *', { maxAnalyze: 100 });
* ```
*/
export function createREMCycleJobHandler(
db: VestigeDatabase
): JobHandler<REMCycleJobData, REMCycleJobResult> {
return async (job: Job<REMCycleJobData>): Promise<REMCycleJobResult> => {
const options = {
maxAnalyze: job.data.maxAnalyze ?? 50,
minStrength: job.data.minStrength ?? 0.3,
dryRun: job.data.dryRun ?? false,
};
// Run the REM cycle (async)
const cycleResult = await runREMCycle(db, options);
const result: REMCycleJobResult = {
nodesAnalyzed: cycleResult.nodesAnalyzed,
connectionsDiscovered: cycleResult.connectionsDiscovered,
connectionsCreated: cycleResult.connectionsCreated,
duration: cycleResult.duration,
discoveries: cycleResult.discoveries.map(d => ({
nodeA: d.nodeA,
nodeB: d.nodeB,
reason: d.reason,
})),
timestamp: new Date(),
};
return result;
};
}
// ============================================================================
// HELPER FUNCTIONS
// ============================================================================
/**
* Preview what connections would be discovered without creating them
* Useful for testing or showing users potential discoveries
*/
export async function previewREMCycleJob(
db: VestigeDatabase,
maxAnalyze: number = 100
): Promise<REMCycleJobResult> {
const cycleResult = await runREMCycle(db, {
maxAnalyze,
dryRun: true,
});
return {
nodesAnalyzed: cycleResult.nodesAnalyzed,
connectionsDiscovered: cycleResult.connectionsDiscovered,
connectionsCreated: 0,
duration: cycleResult.duration,
discoveries: cycleResult.discoveries.map(d => ({
nodeA: d.nodeA,
nodeB: d.nodeB,
reason: d.reason,
})),
timestamp: new Date(),
};
}

99
packages/core/src/jobs/index.ts
View file

@ -1,99 +0,0 @@
/**
* Jobs Module - Background Job Processing for Vestige MCP
*
* This module provides a production-ready job queue system with:
* - Priority-based scheduling
* - Retry logic with exponential backoff
* - Concurrency control
* - Cron-like recurring job scheduling
* - Event-driven architecture
*
* @module jobs
*
* @example
* ```typescript
* import {
* JobQueue,
* createDecayJobHandler,
* createREMCycleJobHandler,
* createConsolidationJobHandler,
* } from './jobs';
* import { VestigeDatabase } from './core';
*
* // Initialize
* const db = new VestigeDatabase();
* const queue = new JobQueue();
*
* // Register job handlers
* queue.register('decay', createDecayJobHandler(db), { concurrency: 1 });
* queue.register('rem-cycle', createREMCycleJobHandler(db), { concurrency: 1 });
* queue.register('consolidation', createConsolidationJobHandler(db), { concurrency: 1 });
*
* // Schedule recurring jobs
* queue.schedule('decay', '0 3 * * *', {}); // Daily at 3 AM
* queue.schedule('rem-cycle', '0 *\/6 * * *', {}); // Every 6 hours
* queue.schedule('consolidation', '0 4 * * 0', {}); // Weekly on Sunday at 4 AM
*
* // Start processing
* queue.start();
*
* // Listen to events
* queue.on('job:completed', (job, result) => {
* console.log(`Job ${job.name} completed:`, result);
* });
*
* queue.on('job:failed', (job, error) => {
* console.error(`Job ${job.name} failed:`, error);
* });
*
* // Add one-off jobs
* queue.add('rem-cycle', { maxAnalyze: 200 }, { priority: 10 });
*
* // Graceful shutdown
* process.on('SIGTERM', async () => {
* await queue.shutdown();
* db.close();
* });
* ```
*/
// Core job queue
export {
JobQueue,
getDefaultQueue,
resetDefaultQueue,
type Job,
type JobResult,
type JobHandler,
type JobOptions,
type JobDefinition,
type JobStatus,
type ScheduledJob,
type QueueStats,
type JobQueueEvents,
} from './JobQueue.js';
// Decay job
export {
createDecayJobHandler,
getCriticallyDecayedNodes,
type DecayJobData,
type DecayJobResult,
} from './DecayJob.js';
// REM cycle job
export {
createREMCycleJobHandler,
previewREMCycleJob,
type REMCycleJobData,
type REMCycleJobResult,
} from './REMCycleJob.js';
// Consolidation job
export {
createConsolidationJobHandler,
previewConsolidation,
getConsolidationMetrics,
type ConsolidationJobData,
type ConsolidationJobResult,
} from './ConsolidationJob.js';

659
packages/core/src/repositories/EdgeRepository.ts
View file

@ -1,659 +0,0 @@
import Database from 'better-sqlite3';
import { nanoid } from 'nanoid';
import type { GraphEdge } from '../core/types.js';
// ============================================================================
// EDGE TYPES
// ============================================================================
export type EdgeType =
| 'relates_to'
| 'contradicts'
| 'supports'
| 'similar_to'
| 'part_of'
| 'caused_by'
| 'mentions'
| 'derived_from'
| 'references'
| 'follows'
| 'person_mentioned'
| 'concept_instance';
// ============================================================================
// INPUT TYPES
// ============================================================================
export interface GraphEdgeInput {
fromId: string;
toId: string;
edgeType: EdgeType;
weight?: number;
metadata?: Record<string, unknown>;
}
// ============================================================================
// TRANSITIVE PATH TYPE
// ============================================================================
export interface TransitivePath {
path: string[];
totalWeight: number;
}
// ============================================================================
// RWLOCK - Read-Write Lock for concurrent access control
// ============================================================================
/**
* A simple read-write lock implementation.
* - Multiple readers can hold the lock concurrently
* - Writers have exclusive access
* - Writers wait for all readers to release
* - Readers wait if a writer is active or waiting
*/
export class RWLock {
private readers = 0;
private writer = false;
private writerQueue: (() => void)[] = [];
private readerQueue: (() => void)[] = [];
async acquireRead(): Promise<void> {
return new Promise((resolve) => {
if (!this.writer && this.writerQueue.length === 0) {
this.readers++;
resolve();
} else {
this.readerQueue.push(() => {
this.readers++;
resolve();
});
}
});
}
releaseRead(): void {
this.readers--;
if (this.readers === 0 && this.writerQueue.length > 0) {
this.writer = true;
const next = this.writerQueue.shift();
if (next) next();
}
}
async acquireWrite(): Promise<void> {
return new Promise((resolve) => {
if (!this.writer && this.readers === 0) {
this.writer = true;
resolve();
} else {
this.writerQueue.push(resolve);
}
});
}
releaseWrite(): void {
this.writer = false;
// Prefer waiting readers over writers to prevent writer starvation
if (this.readerQueue.length > 0) {
const readers = this.readerQueue.splice(0);
for (const reader of readers) {
reader();
}
} else if (this.writerQueue.length > 0) {
this.writer = true;
const next = this.writerQueue.shift();
if (next) next();
}
}
/**
* Execute a function with read lock
*/
async withRead<T>(fn: () => T | Promise<T>): Promise<T> {
await this.acquireRead();
try {
return await fn();
} finally {
this.releaseRead();
}
}
/**
* Execute a function with write lock
*/
async withWrite<T>(fn: () => T | Promise<T>): Promise<T> {
await this.acquireWrite();
try {
return await fn();
} finally {
this.releaseWrite();
}
}
}
// ============================================================================
// INTERFACE
// ============================================================================
export interface IEdgeRepository {
create(input: GraphEdgeInput): Promise<GraphEdge>;
findById(id: string): Promise<GraphEdge | null>;
findByNodes(fromId: string, toId: string, edgeType?: string): Promise<GraphEdge | null>;
delete(id: string): Promise<boolean>;
deleteByNodes(fromId: string, toId: string): Promise<boolean>;
getEdgesFrom(nodeId: string): Promise<GraphEdge[]>;
getEdgesTo(nodeId: string): Promise<GraphEdge[]>;
getAllEdges(nodeId: string): Promise<GraphEdge[]>;
getRelatedNodeIds(nodeId: string, depth?: number): Promise<string[]>;
updateWeight(id: string, weight: number): Promise<void>;
strengthenEdge(id: string, boost: number): Promise<void>;
pruneWeakEdges(threshold: number): Promise<number>;
getTransitivePaths(nodeId: string, maxDepth: number): Promise<TransitivePath[]>;
strengthenConnectedEdges(nodeId: string, boost: number): Promise<number>;
}
// ============================================================================
// ERROR CLASS
// ============================================================================
/**
* Sanitize error message to prevent sensitive data leakage
*/
function sanitizeErrorMessage(message: string): string {
let sanitized = message.replace(/\/[^\s]+/g, '[PATH]');
sanitized = sanitized.replace(/SELECT|INSERT|UPDATE|DELETE|DROP|CREATE/gi, '[SQL]');
sanitized = sanitized.replace(/\b(password|secret|key|token|auth)\s*[=:]\s*\S+/gi, '[REDACTED]');
return sanitized;
}
export class EdgeRepositoryError extends Error {
constructor(
message: string,
public readonly code: string,
cause?: unknown
) {
super(sanitizeErrorMessage(message));
this.name = 'EdgeRepositoryError';
if (process.env['NODE_ENV'] === 'development' && cause) {
this.cause = cause;
}
}
}
// ============================================================================
// IMPLEMENTATION
// ============================================================================
export class EdgeRepository implements IEdgeRepository {
private readonly lock = new RWLock();
constructor(private readonly db: Database.Database) {}
/**
* Create a new edge between two nodes.
* Handles UNIQUE constraint gracefully by using INSERT OR REPLACE.
*/
async create(input: GraphEdgeInput): Promise<GraphEdge> {
return this.lock.withWrite(() => {
try {
const id = nanoid();
const now = new Date().toISOString();
const weight = input.weight ?? 0.5;
// Check if edge already exists
const existing = this.db.prepare(`
SELECT id FROM graph_edges
WHERE from_id = ? AND to_id = ? AND edge_type = ?
`).get(input.fromId, input.toId, input.edgeType) as { id: string } | undefined;
if (existing) {
// Update existing edge - boost weight slightly
const updateStmt = this.db.prepare(`
UPDATE graph_edges
SET weight = MIN(1.0, weight + ?),
metadata = ?
WHERE id = ?
`);
updateStmt.run(weight * 0.1, JSON.stringify(input.metadata || {}), existing.id);
// Return the updated edge
const row = this.db.prepare('SELECT * FROM graph_edges WHERE id = ?')
.get(existing.id) as Record<string, unknown>;
return this.rowToEdge(row);
}
// Insert new edge
const stmt = this.db.prepare(`
INSERT INTO graph_edges (
id, from_id, to_id, edge_type, weight, metadata, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
input.fromId,
input.toId,
input.edgeType,
weight,
JSON.stringify(input.metadata || {}),
now
);
return {
id,
fromId: input.fromId,
toId: input.toId,
edgeType: input.edgeType as GraphEdge['edgeType'],
weight,
metadata: input.metadata || {},
createdAt: new Date(now),
};
} catch (error) {
throw new EdgeRepositoryError(
'Failed to create edge',
'CREATE_EDGE_FAILED',
error
);
}
});
}
/**
* Find an edge by its ID.
*/
async findById(id: string): Promise<GraphEdge | null> {
return this.lock.withRead(() => {
try {
const stmt = this.db.prepare('SELECT * FROM graph_edges WHERE id = ?');
const row = stmt.get(id) as Record<string, unknown> | undefined;
if (!row) return null;
return this.rowToEdge(row);
} catch (error) {
throw new EdgeRepositoryError(
`Failed to find edge: ${id}`,
'FIND_EDGE_FAILED',
error
);
}
});
}
/**
* Find an edge by its source and target nodes.
* Optionally filter by edge type.
*/
async findByNodes(fromId: string, toId: string, edgeType?: string): Promise<GraphEdge | null> {
return this.lock.withRead(() => {
try {
let stmt;
let row: Record<string, unknown> | undefined;
if (edgeType) {
stmt = this.db.prepare(`
SELECT * FROM graph_edges
WHERE from_id = ? AND to_id = ? AND edge_type = ?
`);
row = stmt.get(fromId, toId, edgeType) as Record<string, unknown> | undefined;
} else {
stmt = this.db.prepare(`
SELECT * FROM graph_edges
WHERE from_id = ? AND to_id = ?
`);
row = stmt.get(fromId, toId) as Record<string, unknown> | undefined;
}
if (!row) return null;
return this.rowToEdge(row);
} catch (error) {
throw new EdgeRepositoryError(
`Failed to find edge by nodes`,
'FIND_BY_NODES_FAILED',
error
);
}
});
}
/**
* Delete an edge by its ID.
*/
async delete(id: string): Promise<boolean> {
return this.lock.withWrite(() => {
try {
const stmt = this.db.prepare('DELETE FROM graph_edges WHERE id = ?');
const result = stmt.run(id);
return result.changes > 0;
} catch (error) {
throw new EdgeRepositoryError(
`Failed to delete edge: ${id}`,
'DELETE_EDGE_FAILED',
error
);
}
});
}
/**
* Delete all edges between two nodes (in both directions).
*/
async deleteByNodes(fromId: string, toId: string): Promise<boolean> {
return this.lock.withWrite(() => {
try {
const stmt = this.db.prepare(`
DELETE FROM graph_edges
WHERE (from_id = ? AND to_id = ?) OR (from_id = ? AND to_id = ?)
`);
const result = stmt.run(fromId, toId, toId, fromId);
return result.changes > 0;
} catch (error) {
throw new EdgeRepositoryError(
`Failed to delete edges between nodes`,
'DELETE_BY_NODES_FAILED',
error
);
}
});
}
/**
* Get all edges originating from a node.
*/
async getEdgesFrom(nodeId: string): Promise<GraphEdge[]> {
return this.lock.withRead(() => {
try {
const stmt = this.db.prepare('SELECT * FROM graph_edges WHERE from_id = ?');
const rows = stmt.all(nodeId) as Record<string, unknown>[];
return rows.map(row => this.rowToEdge(row));
} catch (error) {
throw new EdgeRepositoryError(
`Failed to get edges from node: ${nodeId}`,
'GET_EDGES_FROM_FAILED',
error
);
}
});
}
/**
* Get all edges pointing to a node.
*/
async getEdgesTo(nodeId: string): Promise<GraphEdge[]> {
return this.lock.withRead(() => {
try {
const stmt = this.db.prepare('SELECT * FROM graph_edges WHERE to_id = ?');
const rows = stmt.all(nodeId) as Record<string, unknown>[];
return rows.map(row => this.rowToEdge(row));
} catch (error) {
throw new EdgeRepositoryError(
`Failed to get edges to node: ${nodeId}`,
'GET_EDGES_TO_FAILED',
error
);
}
});
}
/**
* Get all edges connected to a node (both incoming and outgoing).
*/
async getAllEdges(nodeId: string): Promise<GraphEdge[]> {
return this.lock.withRead(() => {
try {
const stmt = this.db.prepare(`
SELECT * FROM graph_edges
WHERE from_id = ? OR to_id = ?
`);
const rows = stmt.all(nodeId, nodeId) as Record<string, unknown>[];
return rows.map(row => this.rowToEdge(row));
} catch (error) {
throw new EdgeRepositoryError(
`Failed to get all edges for node: ${nodeId}`,
'GET_ALL_EDGES_FAILED',
error
);
}
});
}
/**
* Get related node IDs using BFS traversal.
* Extracted from database.ts getRelatedNodes().
*/
async getRelatedNodeIds(nodeId: string, depth: number = 1): Promise<string[]> {
return this.lock.withRead(() => {
try {
const visited = new Set<string>();
let current = [nodeId];
for (let d = 0; d < depth; d++) {
if (current.length === 0) break;
const placeholders = current.map(() => '?').join(',');
const stmt = this.db.prepare(`
SELECT DISTINCT
CASE WHEN from_id IN (${placeholders}) THEN to_id ELSE from_id END as related_id
FROM graph_edges
WHERE from_id IN (${placeholders}) OR to_id IN (${placeholders})
`);
const params = [...current, ...current, ...current];
const rows = stmt.all(...params) as { related_id: string }[];
const newNodes: string[] = [];
for (const row of rows) {
if (!visited.has(row.related_id) && row.related_id !== nodeId) {
visited.add(row.related_id);
newNodes.push(row.related_id);
}
}
current = newNodes;
}
return Array.from(visited);
} catch (error) {
throw new EdgeRepositoryError(
`Failed to get related nodes: ${nodeId}`,
'GET_RELATED_FAILED',
error
);
}
});
}
/**
* Update the weight of an edge.
*/
async updateWeight(id: string, weight: number): Promise<void> {
return this.lock.withWrite(() => {
try {
// Clamp weight to valid range
const clampedWeight = Math.max(0, Math.min(1, weight));
const stmt = this.db.prepare(`
UPDATE graph_edges SET weight = ? WHERE id = ?
`);
stmt.run(clampedWeight, id);
} catch (error) {
throw new EdgeRepositoryError(
`Failed to update edge weight: ${id}`,
'UPDATE_WEIGHT_FAILED',
error
);
}
});
}
/**
* Strengthen an edge by boosting its weight.
* Used for spreading activation.
*/
async strengthenEdge(id: string, boost: number): Promise<void> {
return this.lock.withWrite(() => {
try {
// Ensure boost is positive and reasonable
const safeBoost = Math.max(0, Math.min(0.5, boost));
const stmt = this.db.prepare(`
UPDATE graph_edges
SET weight = MIN(1.0, weight + ?)
WHERE id = ?
`);
stmt.run(safeBoost, id);
} catch (error) {
throw new EdgeRepositoryError(
`Failed to strengthen edge: ${id}`,
'STRENGTHEN_EDGE_FAILED',
error
);
}
});
}
/**
* Prune edges with weight below a threshold.
* Returns the number of edges removed.
*/
async pruneWeakEdges(threshold: number): Promise<number> {
return this.lock.withWrite(() => {
try {
// Validate threshold
const safeThreshold = Math.max(0, Math.min(1, threshold));
const stmt = this.db.prepare(`
DELETE FROM graph_edges WHERE weight < ?
`);
const result = stmt.run(safeThreshold);
return result.changes;
} catch (error) {
throw new EdgeRepositoryError(
'Failed to prune weak edges',
'PRUNE_EDGES_FAILED',
error
);
}
});
}
/**
* Get all transitive paths from a node up to maxDepth.
* Used for spreading activation in graph traversal.
*/
async getTransitivePaths(nodeId: string, maxDepth: number): Promise<TransitivePath[]> {
return this.lock.withRead(() => {
try {
const paths: TransitivePath[] = [];
const visited = new Set<string>();
// BFS with path tracking
interface QueueItem {
nodeId: string;
path: string[];
totalWeight: number;
}
const queue: QueueItem[] = [{ nodeId, path: [nodeId], totalWeight: 1.0 }];
visited.add(nodeId);
while (queue.length > 0) {
const current = queue.shift()!;
if (current.path.length > maxDepth + 1) continue;
// Get all connected edges
const stmt = this.db.prepare(`
SELECT to_id, from_id, weight FROM graph_edges
WHERE from_id = ? OR to_id = ?
`);
const edges = stmt.all(current.nodeId, current.nodeId) as {
to_id: string;
from_id: string;
weight: number;
}[];
for (const edge of edges) {
const nextNode = edge.from_id === current.nodeId ? edge.to_id : edge.from_id;
if (!visited.has(nextNode)) {
visited.add(nextNode);
const newPath = [...current.path, nextNode];
const newWeight = current.totalWeight * edge.weight;
paths.push({ path: newPath, totalWeight: newWeight });
if (newPath.length <= maxDepth) {
queue.push({
nodeId: nextNode,
path: newPath,
totalWeight: newWeight,
});
}
}
}
}
// Sort by total weight (descending) for relevance
return paths.sort((a, b) => b.totalWeight - a.totalWeight);
} catch (error) {
throw new EdgeRepositoryError(
`Failed to get transitive paths: ${nodeId}`,
'GET_PATHS_FAILED',
error
);
}
});
}
/**
* Strengthen all edges connected to a node.
* Used for memory reconsolidation.
* Returns the number of edges strengthened.
*/
async strengthenConnectedEdges(nodeId: string, boost: number): Promise<number> {
return this.lock.withWrite(() => {
try {
// Ensure boost is positive and reasonable
const safeBoost = Math.max(0, Math.min(0.5, boost));
const stmt = this.db.prepare(`
UPDATE graph_edges
SET weight = MIN(1.0, weight + ?)
WHERE from_id = ? OR to_id = ?
`);
const result = stmt.run(safeBoost, nodeId, nodeId);
return result.changes;
} catch (error) {
throw new EdgeRepositoryError(
`Failed to strengthen connected edges: ${nodeId}`,
'STRENGTHEN_CONNECTED_FAILED',
error
);
}
});
}
// ============================================================================
// HELPERS
// ============================================================================
private rowToEdge(row: Record<string, unknown>): GraphEdge {
return {
id: row['id'] as string,
fromId: row['from_id'] as string,
toId: row['to_id'] as string,
edgeType: row['edge_type'] as GraphEdge['edgeType'],
weight: row['weight'] as number,
metadata: this.safeJsonParse(row['metadata'] as string, {}),
createdAt: new Date(row['created_at'] as string),
};
}
private safeJsonParse<T>(value: string | null | undefined, fallback: T): T {
if (!value) return fallback;
try {
return JSON.parse(value) as T;
} catch {
return fallback;
}
}
}

879
packages/core/src/repositories/NodeRepository.ts
View file

@ -1,879 +0,0 @@
/**
* NodeRepository - Repository for knowledge node operations
*
* Extracted from the monolithic database.ts to provide a focused, testable
* interface for node CRUD operations with proper concurrency control.
*/
import type Database from 'better-sqlite3';
import { nanoid } from 'nanoid';
import type {
KnowledgeNode,
KnowledgeNodeInput,
} from '../core/types.js';
import { RWLock } from '../utils/mutex.js';
import { safeJsonParse } from '../utils/json.js';
import { NotFoundError, ValidationError, DatabaseError } from '../core/errors.js';
import { analyzeSentimentIntensity, captureGitContext } from '../core/database.js';
// ============================================================================
// CONSTANTS
// ============================================================================
const DEFAULT_LIMIT = 50;
const MAX_LIMIT = 500;
// Input validation limits
const MAX_CONTENT_LENGTH = 1_000_000; // 1MB max content
const MAX_QUERY_LENGTH = 10_000; // 10KB max query
const MAX_TAGS_COUNT = 100; // Max tags per node
// SM-2 Spaced Repetition Constants
const SM2_EASE_FACTOR = 2.5;
const SM2_LAPSE_THRESHOLD = 0.3;
const SM2_MIN_STABILITY = 1.0;
const SM2_MAX_STABILITY = 365.0;
// Sentiment-Weighted Decay Constants
const SENTIMENT_STABILITY_BOOST = 2.0;
const SENTIMENT_MIN_BOOST = 1.0;
// ============================================================================
// TYPES
// ============================================================================
export interface PaginationOptions {
limit?: number;
offset?: number;
}
export interface PaginatedResult<T> {
items: T[];
total: number;
limit: number;
offset: number;
hasMore: boolean;
}
export interface GitContext {
branch?: string;
commit?: string;
commitMessage?: string;
repoPath?: string;
dirty?: boolean;
changedFiles?: string[];
}
// ============================================================================
// INTERFACE
// ============================================================================
export interface INodeRepository {
findById(id: string): Promise<KnowledgeNode | null>;
findByIds(ids: string[]): Promise<KnowledgeNode[]>;
create(input: KnowledgeNodeInput): Promise<KnowledgeNode>;
update(id: string, updates: Partial<KnowledgeNodeInput>): Promise<KnowledgeNode | null>;
delete(id: string): Promise<boolean>;
search(query: string, options?: PaginationOptions): Promise<PaginatedResult<KnowledgeNode>>;
getRecent(options?: PaginationOptions): Promise<PaginatedResult<KnowledgeNode>>;
getDecaying(threshold: number, options?: PaginationOptions): Promise<PaginatedResult<KnowledgeNode>>;
getDueForReview(options?: PaginationOptions): Promise<PaginatedResult<KnowledgeNode>>;
recordAccess(id: string): Promise<void>;
markReviewed(id: string): Promise<KnowledgeNode>;
applyDecay(id: string): Promise<number>;
applyDecayAll(): Promise<number>;
findByTag(tag: string, options?: PaginationOptions): Promise<PaginatedResult<KnowledgeNode>>;
findByPerson(personName: string, options?: PaginationOptions): Promise<PaginatedResult<KnowledgeNode>>;
}
// ============================================================================
// VALIDATION HELPERS
// ============================================================================
/**
* Validate string length for inputs
*/
function validateStringLength(value: string, maxLength: number, fieldName: string): void {
if (value && value.length > maxLength) {
throw new ValidationError(
`${fieldName} exceeds maximum length of ${maxLength} characters`,
{ field: fieldName.toLowerCase(), maxLength, actualLength: value.length }
);
}
}
/**
* Validate array length for inputs
*/
function validateArrayLength<T>(arr: T[] | undefined, maxLength: number, fieldName: string): void {
if (arr && arr.length > maxLength) {
throw new ValidationError(
`${fieldName} exceeds maximum count of ${maxLength} items`,
{ field: fieldName.toLowerCase(), maxLength, actualLength: arr.length }
);
}
}
/**
* Normalize pagination options
*/
function normalizePagination(options: PaginationOptions = {}): { limit: number; offset: number } {
const { limit = DEFAULT_LIMIT, offset = 0 } = options;
return {
limit: Math.min(Math.max(1, limit), MAX_LIMIT),
offset: Math.max(0, offset),
};
}
// ============================================================================
// IMPLEMENTATION
// ============================================================================
export class NodeRepository implements INodeRepository {
private readonly lock = new RWLock();
constructor(private readonly db: Database.Database) {}
// --------------------------------------------------------------------------
// READ OPERATIONS
// --------------------------------------------------------------------------
async findById(id: string): Promise<KnowledgeNode | null> {
return this.lock.withReadLock(async () => {
try {
const stmt = this.db.prepare('SELECT * FROM knowledge_nodes WHERE id = ?');
const row = stmt.get(id) as Record<string, unknown> | undefined;
if (!row) return null;
return this.rowToEntity(row);
} catch (error) {
throw new DatabaseError(`Failed to get node: ${id}`, error);
}
});
}
async findByIds(ids: string[]): Promise<KnowledgeNode[]> {
if (ids.length === 0) return [];
return this.lock.withReadLock(async () => {
try {
const placeholders = ids.map(() => '?').join(',');
const stmt = this.db.prepare(
`SELECT * FROM knowledge_nodes WHERE id IN (${placeholders})`
);
const rows = stmt.all(...ids) as Record<string, unknown>[];
return rows.map((row) => this.rowToEntity(row));
} catch (error) {
throw new DatabaseError('Failed to get nodes by IDs', error);
}
});
}
async search(query: string, options: PaginationOptions = {}): Promise<PaginatedResult<KnowledgeNode>> {
return this.lock.withReadLock(async () => {
try {
// Input validation
validateStringLength(query, MAX_QUERY_LENGTH, 'Search query');
// Sanitize FTS5 query to prevent injection
const sanitizedQuery = query
.replace(/[^\w\s\-]/g, ' ')
.trim();
if (!sanitizedQuery) {
return {
items: [],
total: 0,
limit: DEFAULT_LIMIT,
offset: 0,
hasMore: false,
};
}
const { limit, offset } = normalizePagination(options);
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM knowledge_nodes kn
JOIN knowledge_fts fts ON kn.id = fts.id
WHERE knowledge_fts MATCH ?
`);
const countResult = countStmt.get(sanitizedQuery) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT kn.* FROM knowledge_nodes kn
JOIN knowledge_fts fts ON kn.id = fts.id
WHERE knowledge_fts MATCH ?
ORDER BY rank
LIMIT ? OFFSET ?
`);
const rows = stmt.all(sanitizedQuery, limit, offset) as Record<string, unknown>[];
const items = rows.map((row) => this.rowToEntity(row));
return {
items,
total,
limit,
offset,
hasMore: offset + items.length < total,
};
} catch (error) {
if (error instanceof ValidationError) throw error;
throw new DatabaseError('Search operation failed', error);
}
});
}
async getRecent(options: PaginationOptions = {}): Promise<PaginatedResult<KnowledgeNode>> {
return this.lock.withReadLock(async () => {
try {
const { limit, offset } = normalizePagination(options);
// Get total count
const countResult = this.db.prepare('SELECT COUNT(*) as total FROM knowledge_nodes').get() as {
total: number;
};
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM knowledge_nodes
ORDER BY created_at DESC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(limit, offset) as Record<string, unknown>[];
const items = rows.map((row) => this.rowToEntity(row));
return {
items,
total,
limit,
offset,
hasMore: offset + items.length < total,
};
} catch (error) {
throw new DatabaseError('Failed to get recent nodes', error);
}
});
}
async getDecaying(
threshold: number = 0.5,
options: PaginationOptions = {}
): Promise<PaginatedResult<KnowledgeNode>> {
return this.lock.withReadLock(async () => {
try {
const { limit, offset } = normalizePagination(options);
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM knowledge_nodes
WHERE retention_strength < ?
`);
const countResult = countStmt.get(threshold) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM knowledge_nodes
WHERE retention_strength < ?
ORDER BY retention_strength ASC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(threshold, limit, offset) as Record<string, unknown>[];
const items = rows.map((row) => this.rowToEntity(row));
return {
items,
total,
limit,
offset,
hasMore: offset + items.length < total,
};
} catch (error) {
throw new DatabaseError('Failed to get decaying nodes', error);
}
});
}
async getDueForReview(options: PaginationOptions = {}): Promise<PaginatedResult<KnowledgeNode>> {
return this.lock.withReadLock(async () => {
try {
const { limit, offset } = normalizePagination(options);
const now = new Date().toISOString();
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM knowledge_nodes
WHERE next_review_date IS NOT NULL AND next_review_date <= ?
`);
const countResult = countStmt.get(now) as { total: number };
const total = countResult.total;
// Get paginated results, ordered by retention strength (most urgent first)
const stmt = this.db.prepare(`
SELECT * FROM knowledge_nodes
WHERE next_review_date IS NOT NULL AND next_review_date <= ?
ORDER BY retention_strength ASC, next_review_date ASC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(now, limit, offset) as Record<string, unknown>[];
const items = rows.map((row) => this.rowToEntity(row));
return {
items,
total,
limit,
offset,
hasMore: offset + items.length < total,
};
} catch (error) {
throw new DatabaseError('Failed to get nodes due for review', error);
}
});
}
async findByTag(tag: string, options: PaginationOptions = {}): Promise<PaginatedResult<KnowledgeNode>> {
return this.lock.withReadLock(async () => {
try {
const { limit, offset } = normalizePagination(options);
// Escape special JSON/LIKE characters
const escapedTag = tag
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_')
.replace(/"/g, '\\"');
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM knowledge_nodes
WHERE tags LIKE ? ESCAPE '\\'
`);
const countResult = countStmt.get(`%"${escapedTag}"%`) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM knowledge_nodes
WHERE tags LIKE ? ESCAPE '\\'
ORDER BY created_at DESC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(`%"${escapedTag}"%`, limit, offset) as Record<string, unknown>[];
const items = rows.map((row) => this.rowToEntity(row));
return {
items,
total,
limit,
offset,
hasMore: offset + items.length < total,
};
} catch (error) {
throw new DatabaseError('Failed to find nodes by tag', error);
}
});
}
async findByPerson(
personName: string,
options: PaginationOptions = {}
): Promise<PaginatedResult<KnowledgeNode>> {
return this.lock.withReadLock(async () => {
try {
const { limit, offset } = normalizePagination(options);
// Escape special JSON/LIKE characters
const escapedPerson = personName
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_')
.replace(/"/g, '\\"');
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM knowledge_nodes
WHERE people LIKE ? ESCAPE '\\'
`);
const countResult = countStmt.get(`%"${escapedPerson}"%`) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM knowledge_nodes
WHERE people LIKE ? ESCAPE '\\'
ORDER BY created_at DESC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(`%"${escapedPerson}"%`, limit, offset) as Record<string, unknown>[];
const items = rows.map((row) => this.rowToEntity(row));
return {
items,
total,
limit,
offset,
hasMore: offset + items.length < total,
};
} catch (error) {
throw new DatabaseError('Failed to find nodes by person', error);
}
});
}
// --------------------------------------------------------------------------
// WRITE OPERATIONS
// --------------------------------------------------------------------------
async create(input: KnowledgeNodeInput): Promise<KnowledgeNode> {
return this.lock.withWriteLock(async () => {
try {
// Input validation
validateStringLength(input.content, MAX_CONTENT_LENGTH, 'Content');
validateStringLength(input.summary || '', MAX_CONTENT_LENGTH, 'Summary');
validateArrayLength(input.tags, MAX_TAGS_COUNT, 'Tags');
validateArrayLength(input.people, MAX_TAGS_COUNT, 'People');
validateArrayLength(input.concepts, MAX_TAGS_COUNT, 'Concepts');
validateArrayLength(input.events, MAX_TAGS_COUNT, 'Events');
// Validate confidence is within bounds
const confidence = Math.max(0, Math.min(1, input.confidence ?? 0.8));
const retention = Math.max(0, Math.min(1, input.retentionStrength ?? 1.0));
// Analyze emotional intensity of content
const sentimentIntensity =
input.sentimentIntensity ?? analyzeSentimentIntensity(input.content);
// Git-Blame for Thoughts: Capture current code context
const gitContext = input.gitContext ?? captureGitContext();
const id = nanoid();
const now = new Date().toISOString();
const stmt = this.db.prepare(`
INSERT INTO knowledge_nodes (
id, content, summary,
created_at, updated_at, last_accessed_at, access_count,
retention_strength, sentiment_intensity, next_review_date, review_count,
source_type, source_platform, source_id, source_url, source_chain, git_context,
confidence, is_contradicted, contradiction_ids,
people, concepts, events, tags
) VALUES (
?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?, ?, ?,
?, ?, ?,
?, ?, ?, ?
)
`);
const createdAt = input.createdAt instanceof Date
? input.createdAt.toISOString()
: (input.createdAt || now);
stmt.run(
id,
input.content,
input.summary || null,
createdAt,
now,
now,
0,
retention,
sentimentIntensity,
input.nextReviewDate instanceof Date
? input.nextReviewDate.toISOString()
: (input.nextReviewDate || null),
0,
input.sourceType,
input.sourcePlatform,
input.sourceId || null,
input.sourceUrl || null,
JSON.stringify(input.sourceChain || []),
gitContext ? JSON.stringify(gitContext) : null,
confidence,
input.isContradicted ? 1 : 0,
JSON.stringify(input.contradictionIds || []),
JSON.stringify(input.people || []),
JSON.stringify(input.concepts || []),
JSON.stringify(input.events || []),
JSON.stringify(input.tags || [])
);
// Return the created node
const node = await this.findById(id);
if (!node) {
throw new DatabaseError('Failed to retrieve created node');
}
return node;
} catch (error) {
if (error instanceof ValidationError || error instanceof DatabaseError) throw error;
throw new DatabaseError('Failed to insert knowledge node', error);
}
});
}
async update(id: string, updates: Partial<KnowledgeNodeInput>): Promise<KnowledgeNode | null> {
return this.lock.withWriteLock(async () => {
try {
// Check if node exists
const existing = this.db.prepare('SELECT * FROM knowledge_nodes WHERE id = ?').get(id);
if (!existing) {
return null;
}
// Input validation
if (updates.content !== undefined) {
validateStringLength(updates.content, MAX_CONTENT_LENGTH, 'Content');
}
if (updates.summary !== undefined) {
validateStringLength(updates.summary, MAX_CONTENT_LENGTH, 'Summary');
}
if (updates.tags !== undefined) {
validateArrayLength(updates.tags, MAX_TAGS_COUNT, 'Tags');
}
if (updates.people !== undefined) {
validateArrayLength(updates.people, MAX_TAGS_COUNT, 'People');
}
if (updates.concepts !== undefined) {
validateArrayLength(updates.concepts, MAX_TAGS_COUNT, 'Concepts');
}
if (updates.events !== undefined) {
validateArrayLength(updates.events, MAX_TAGS_COUNT, 'Events');
}
// Build dynamic update
const setClauses: string[] = [];
const values: unknown[] = [];
if (updates.content !== undefined) {
setClauses.push('content = ?');
values.push(updates.content);
// Re-analyze sentiment when content changes
const sentimentIntensity = analyzeSentimentIntensity(updates.content);
setClauses.push('sentiment_intensity = ?');
values.push(sentimentIntensity);
}
if (updates.summary !== undefined) {
setClauses.push('summary = ?');
values.push(updates.summary);
}
if (updates.confidence !== undefined) {
setClauses.push('confidence = ?');
values.push(Math.max(0, Math.min(1, updates.confidence)));
}
if (updates.retentionStrength !== undefined) {
setClauses.push('retention_strength = ?');
values.push(Math.max(0, Math.min(1, updates.retentionStrength)));
}
if (updates.tags !== undefined) {
setClauses.push('tags = ?');
values.push(JSON.stringify(updates.tags));
}
if (updates.people !== undefined) {
setClauses.push('people = ?');
values.push(JSON.stringify(updates.people));
}
if (updates.concepts !== undefined) {
setClauses.push('concepts = ?');
values.push(JSON.stringify(updates.concepts));
}
if (updates.events !== undefined) {
setClauses.push('events = ?');
values.push(JSON.stringify(updates.events));
}
if (updates.isContradicted !== undefined) {
setClauses.push('is_contradicted = ?');
values.push(updates.isContradicted ? 1 : 0);
}
if (updates.contradictionIds !== undefined) {
setClauses.push('contradiction_ids = ?');
values.push(JSON.stringify(updates.contradictionIds));
}
if (setClauses.length === 0) {
// No updates to make, just return existing node
return this.rowToEntity(existing as Record<string, unknown>);
}
// Always update updated_at
setClauses.push('updated_at = ?');
values.push(new Date().toISOString());
// Add the ID for the WHERE clause
values.push(id);
const sql = `UPDATE knowledge_nodes SET ${setClauses.join(', ')} WHERE id = ?`;
this.db.prepare(sql).run(...values);
// Return updated node
const updated = this.db.prepare('SELECT * FROM knowledge_nodes WHERE id = ?').get(id);
return updated ? this.rowToEntity(updated as Record<string, unknown>) : null;
} catch (error) {
if (error instanceof ValidationError || error instanceof DatabaseError) throw error;
throw new DatabaseError(`Failed to update node: ${id}`, error);
}
});
}
async delete(id: string): Promise<boolean> {
return this.lock.withWriteLock(async () => {
try {
const stmt = this.db.prepare('DELETE FROM knowledge_nodes WHERE id = ?');
const result = stmt.run(id);
return result.changes > 0;
} catch (error) {
throw new DatabaseError(`Failed to delete node: ${id}`, error);
}
});
}
async recordAccess(id: string): Promise<void> {
return this.lock.withWriteLock(async () => {
try {
const stmt = this.db.prepare(`
UPDATE knowledge_nodes
SET last_accessed_at = ?, access_count = access_count + 1
WHERE id = ?
`);
stmt.run(new Date().toISOString(), id);
} catch (error) {
throw new DatabaseError(`Failed to record access: ${id}`, error);
}
});
}
async markReviewed(id: string): Promise<KnowledgeNode> {
return this.lock.withWriteLock(async () => {
try {
// Get the node first
const nodeStmt = this.db.prepare('SELECT * FROM knowledge_nodes WHERE id = ?');
const nodeRow = nodeStmt.get(id) as Record<string, unknown> | undefined;
if (!nodeRow) {
throw new NotFoundError('KnowledgeNode', id);
}
const node = this.rowToEntity(nodeRow);
const currentStability = node.stabilityFactor ?? SM2_MIN_STABILITY;
let newStability: number;
let newReviewCount: number;
// SM-2 with Lapse Detection
if (node.retentionStrength >= SM2_LAPSE_THRESHOLD) {
// SUCCESSFUL RECALL: Memory was still accessible
newStability = Math.min(SM2_MAX_STABILITY, currentStability * SM2_EASE_FACTOR);
newReviewCount = node.reviewCount + 1;
} else {
// LAPSE: Memory had decayed too far
newStability = SM2_MIN_STABILITY;
newReviewCount = node.reviewCount + 1;
}
// Reset retention to full strength
const newRetention = 1.0;
// Calculate next review date
const daysUntilReview = Math.ceil(newStability);
const nextReview = new Date();
nextReview.setDate(nextReview.getDate() + daysUntilReview);
const updateStmt = this.db.prepare(`
UPDATE knowledge_nodes
SET retention_strength = ?,
stability_factor = ?,
review_count = ?,
next_review_date = ?,
last_accessed_at = ?,
updated_at = ?
WHERE id = ?
`);
const now = new Date().toISOString();
updateStmt.run(
newRetention,
newStability,
newReviewCount,
nextReview.toISOString(),
now,
now,
id
);
// Return the updated node
const updatedRow = nodeStmt.get(id) as Record<string, unknown>;
return this.rowToEntity(updatedRow);
} catch (error) {
if (error instanceof NotFoundError) throw error;
throw new DatabaseError('Failed to mark node as reviewed', error);
}
});
}
async applyDecay(id: string): Promise<number> {
return this.lock.withWriteLock(async () => {
try {
const nodeStmt = this.db.prepare(`
SELECT id, last_accessed_at, retention_strength, stability_factor, sentiment_intensity
FROM knowledge_nodes WHERE id = ?
`);
const node = nodeStmt.get(id) as {
id: string;
last_accessed_at: string;
retention_strength: number;
stability_factor: number | null;
sentiment_intensity: number | null;
} | undefined;
if (!node) {
throw new NotFoundError('KnowledgeNode', id);
}
const now = Date.now();
const lastAccessed = new Date(node.last_accessed_at).getTime();
const daysSince = (now - lastAccessed) / (1000 * 60 * 60 * 24);
const baseStability = node.stability_factor ?? SM2_MIN_STABILITY;
const sentimentIntensity = node.sentiment_intensity ?? 0;
const sentimentMultiplier =
SENTIMENT_MIN_BOOST + sentimentIntensity * (SENTIMENT_STABILITY_BOOST - SENTIMENT_MIN_BOOST);
const effectiveStability = baseStability * sentimentMultiplier;
// Ebbinghaus forgetting curve: R = e^(-t/S)
const newRetention = Math.max(0.1, node.retention_strength * Math.exp(-daysSince / effectiveStability));
const updateStmt = this.db.prepare(`
UPDATE knowledge_nodes SET retention_strength = ? WHERE id = ?
`);
updateStmt.run(newRetention, id);
return newRetention;
} catch (error) {
if (error instanceof NotFoundError) throw error;
throw new DatabaseError(`Failed to apply decay to node: ${id}`, error);
}
});
}
async applyDecayAll(): Promise<number> {
return this.lock.withWriteLock(async () => {
try {
const now = Date.now();
// Use IMMEDIATE transaction for consistency
const transaction = this.db.transaction(() => {
const nodes = this.db
.prepare(
`
SELECT id, last_accessed_at, retention_strength, stability_factor, sentiment_intensity
FROM knowledge_nodes
`
)
.all() as {
id: string;
last_accessed_at: string;
retention_strength: number;
stability_factor: number | null;
sentiment_intensity: number | null;
}[];
let updated = 0;
const updateStmt = this.db.prepare(`
UPDATE knowledge_nodes SET retention_strength = ? WHERE id = ?
`);
for (const node of nodes) {
const lastAccessed = new Date(node.last_accessed_at).getTime();
const daysSince = (now - lastAccessed) / (1000 * 60 * 60 * 24);
const baseStability = node.stability_factor ?? SM2_MIN_STABILITY;
const sentimentIntensity = node.sentiment_intensity ?? 0;
const sentimentMultiplier =
SENTIMENT_MIN_BOOST +
sentimentIntensity * (SENTIMENT_STABILITY_BOOST - SENTIMENT_MIN_BOOST);
const effectiveStability = baseStability * sentimentMultiplier;
const newRetention = Math.max(
0.1,
node.retention_strength * Math.exp(-daysSince / effectiveStability)
);
if (Math.abs(newRetention - node.retention_strength) > 0.01) {
updateStmt.run(newRetention, node.id);
updated++;
}
}
return updated;
});
return transaction.immediate();
} catch (error) {
throw new DatabaseError('Failed to apply decay to all nodes', error);
}
});
}
// --------------------------------------------------------------------------
// PRIVATE HELPERS
// --------------------------------------------------------------------------
/**
* Convert a database row to a KnowledgeNode entity
*/
private rowToEntity(row: Record<string, unknown>): KnowledgeNode {
// Parse git context separately with proper null handling
let gitContext: GitContext | undefined;
if (row['git_context']) {
const parsed = safeJsonParse<GitContext | null>(row['git_context'] as string, null);
if (parsed !== null) {
gitContext = parsed;
}
}
return {
id: row['id'] as string,
content: row['content'] as string,
summary: row['summary'] as string | undefined,
createdAt: new Date(row['created_at'] as string),
updatedAt: new Date(row['updated_at'] as string),
lastAccessedAt: new Date(row['last_accessed_at'] as string),
accessCount: row['access_count'] as number,
retentionStrength: row['retention_strength'] as number,
stabilityFactor: (row['stability_factor'] as number) ?? SM2_MIN_STABILITY,
sentimentIntensity: (row['sentiment_intensity'] as number) ?? 0,
// Dual-strength memory model fields
storageStrength: (row['storage_strength'] as number) ?? 1,
retrievalStrength: (row['retrieval_strength'] as number) ?? 1,
nextReviewDate: row['next_review_date']
? new Date(row['next_review_date'] as string)
: undefined,
reviewCount: row['review_count'] as number,
sourceType: row['source_type'] as KnowledgeNode['sourceType'],
sourcePlatform: row['source_platform'] as KnowledgeNode['sourcePlatform'],
sourceId: row['source_id'] as string | undefined,
sourceUrl: row['source_url'] as string | undefined,
sourceChain: safeJsonParse<string[]>(row['source_chain'] as string, []),
gitContext,
confidence: row['confidence'] as number,
isContradicted: Boolean(row['is_contradicted']),
contradictionIds: safeJsonParse<string[]>(row['contradiction_ids'] as string, []),
people: safeJsonParse<string[]>(row['people'] as string, []),
concepts: safeJsonParse<string[]>(row['concepts'] as string, []),
events: safeJsonParse<string[]>(row['events'] as string, []),
tags: safeJsonParse<string[]>(row['tags'] as string, []),
};
}
}

864
packages/core/src/repositories/PersonRepository.ts
View file

@ -1,864 +0,0 @@
import Database from 'better-sqlite3';
import { nanoid } from 'nanoid';
import type { PersonNode } from '../core/types.js';
// ============================================================================
// CONSTANTS
// ============================================================================
const DEFAULT_LIMIT = 50;
const MAX_LIMIT = 500;
const MAX_NAME_LENGTH = 500;
const MAX_CONTENT_LENGTH = 1_000_000;
const MAX_ARRAY_COUNT = 100;
// ============================================================================
// TYPES
// ============================================================================
export interface PaginationOptions {
limit?: number;
offset?: number;
}
export interface PaginatedResult<T> {
items: T[];
total: number;
limit: number;
offset: number;
hasMore: boolean;
}
export interface PersonNodeInput {
name: string;
aliases?: string[];
howWeMet?: string;
relationshipType?: string;
organization?: string;
role?: string;
location?: string;
email?: string;
phone?: string;
socialLinks?: Record<string, string>;
preferredChannel?: string;
sharedTopics?: string[];
sharedProjects?: string[];
notes?: string;
relationshipHealth?: number;
lastContactAt?: Date;
contactFrequency?: number;
}
// ============================================================================
// ERROR TYPE
// ============================================================================
export class PersonRepositoryError extends Error {
constructor(
message: string,
public readonly code: string,
cause?: unknown
) {
super(sanitizeErrorMessage(message));
this.name = 'PersonRepositoryError';
if (process.env['NODE_ENV'] === 'development' && cause) {
this.cause = cause;
}
}
}
// ============================================================================
// HELPER FUNCTIONS
// ============================================================================
/**
* Sanitize error message to prevent sensitive data leakage
*/
function sanitizeErrorMessage(message: string): string {
let sanitized = message.replace(/\/[^\s]+/g, '[PATH]');
sanitized = sanitized.replace(/SELECT|INSERT|UPDATE|DELETE|DROP|CREATE/gi, '[SQL]');
sanitized = sanitized.replace(/\b(password|secret|key|token|auth)\s*[=:]\s*\S+/gi, '[REDACTED]');
return sanitized;
}
/**
* Safe JSON parse with fallback - never throws
*/
function safeJsonParse<T>(value: string | null | undefined, fallback: T): T {
if (!value) return fallback;
try {
const parsed = JSON.parse(value);
if (typeof parsed !== typeof fallback) {
return fallback;
}
return parsed as T;
} catch {
return fallback;
}
}
/**
* Validate string length for inputs
*/
function validateStringLength(value: string | undefined, maxLength: number, fieldName: string): void {
if (value && value.length > maxLength) {
throw new PersonRepositoryError(
`${fieldName} exceeds maximum length of ${maxLength} characters`,
'INPUT_TOO_LONG'
);
}
}
/**
* Validate array length for inputs
*/
function validateArrayLength<T>(arr: T[] | undefined, maxLength: number, fieldName: string): void {
if (arr && arr.length > maxLength) {
throw new PersonRepositoryError(
`${fieldName} exceeds maximum count of ${maxLength} items`,
'INPUT_TOO_MANY_ITEMS'
);
}
}
// ============================================================================
// READ-WRITE LOCK
// ============================================================================
/**
* A simple read-write lock for concurrent access control.
* Allows multiple readers or a single writer, but not both.
*/
export class RWLock {
private readers = 0;
private writer = false;
private readQueue: (() => void)[] = [];
private writeQueue: (() => void)[] = [];
/**
* Acquire a read lock. Multiple readers can hold the lock simultaneously.
*/
async acquireRead(): Promise<void> {
return new Promise((resolve) => {
if (!this.writer && this.writeQueue.length === 0) {
this.readers++;
resolve();
} else {
this.readQueue.push(() => {
this.readers++;
resolve();
});
}
});
}
/**
* Release a read lock.
*/
releaseRead(): void {
this.readers--;
if (this.readers === 0) {
this.processWriteQueue();
}
}
/**
* Acquire a write lock. Only one writer can hold the lock at a time.
*/
async acquireWrite(): Promise<void> {
return new Promise((resolve) => {
if (!this.writer && this.readers === 0) {
this.writer = true;
resolve();
} else {
this.writeQueue.push(() => {
this.writer = true;
resolve();
});
}
});
}
/**
* Release a write lock.
*/
releaseWrite(): void {
this.writer = false;
// Process read queue first to prevent writer starvation
this.processReadQueue();
if (this.readers === 0) {
this.processWriteQueue();
}
}
private processReadQueue(): void {
while (this.readQueue.length > 0 && !this.writer) {
const next = this.readQueue.shift();
if (next) next();
}
}
private processWriteQueue(): void {
if (this.writeQueue.length > 0 && this.readers === 0 && !this.writer) {
const next = this.writeQueue.shift();
if (next) next();
}
}
/**
* Execute a function with a read lock.
*/
async withRead<T>(fn: () => T | Promise<T>): Promise<T> {
await this.acquireRead();
try {
return await fn();
} finally {
this.releaseRead();
}
}
/**
* Execute a function with a write lock.
*/
async withWrite<T>(fn: () => T | Promise<T>): Promise<T> {
await this.acquireWrite();
try {
return await fn();
} finally {
this.releaseWrite();
}
}
}
// ============================================================================
// INTERFACE
// ============================================================================
export interface IPersonRepository {
findById(id: string): Promise<PersonNode | null>;
findByName(name: string): Promise<PersonNode | null>;
searchByName(query: string, options?: PaginationOptions): Promise<PaginatedResult<PersonNode>>;
create(input: PersonNodeInput): Promise<PersonNode>;
update(id: string, updates: Partial<PersonNodeInput>): Promise<PersonNode | null>;
delete(id: string): Promise<boolean>;
getPeopleToReconnect(daysSinceContact: number, options?: PaginationOptions): Promise<PaginatedResult<PersonNode>>;
recordContact(id: string): Promise<void>;
findByOrganization(org: string, options?: PaginationOptions): Promise<PaginatedResult<PersonNode>>;
findBySharedTopic(topic: string, options?: PaginationOptions): Promise<PaginatedResult<PersonNode>>;
getAll(options?: PaginationOptions): Promise<PaginatedResult<PersonNode>>;
}
// ============================================================================
// IMPLEMENTATION
// ============================================================================
export class PersonRepository implements IPersonRepository {
private readonly lock = new RWLock();
constructor(private readonly db: Database.Database) {}
/**
* Convert a database row to a PersonNode entity.
*/
private rowToEntity(row: Record<string, unknown>): PersonNode {
return {
id: row['id'] as string,
name: row['name'] as string,
aliases: safeJsonParse<string[]>(row['aliases'] as string, []),
howWeMet: row['how_we_met'] as string | undefined,
relationshipType: row['relationship_type'] as string | undefined,
organization: row['organization'] as string | undefined,
role: row['role'] as string | undefined,
location: row['location'] as string | undefined,
email: row['email'] as string | undefined,
phone: row['phone'] as string | undefined,
socialLinks: safeJsonParse<Record<string, string>>(row['social_links'] as string, {}),
lastContactAt: row['last_contact_at'] ? new Date(row['last_contact_at'] as string) : undefined,
contactFrequency: row['contact_frequency'] as number,
preferredChannel: row['preferred_channel'] as string | undefined,
sharedTopics: safeJsonParse<string[]>(row['shared_topics'] as string, []),
sharedProjects: safeJsonParse<string[]>(row['shared_projects'] as string, []),
notes: row['notes'] as string | undefined,
relationshipHealth: row['relationship_health'] as number,
createdAt: new Date(row['created_at'] as string),
updatedAt: new Date(row['updated_at'] as string),
};
}
/**
* Validate input for creating or updating a person.
*/
private validateInput(input: PersonNodeInput | Partial<PersonNodeInput>, isCreate: boolean): void {
if (isCreate && !input.name) {
throw new PersonRepositoryError('Name is required', 'NAME_REQUIRED');
}
validateStringLength(input.name, MAX_NAME_LENGTH, 'Name');
validateStringLength(input.notes, MAX_CONTENT_LENGTH, 'Notes');
validateStringLength(input.howWeMet, MAX_CONTENT_LENGTH, 'How we met');
validateArrayLength(input.aliases, MAX_ARRAY_COUNT, 'Aliases');
validateArrayLength(input.sharedTopics, MAX_ARRAY_COUNT, 'Shared topics');
validateArrayLength(input.sharedProjects, MAX_ARRAY_COUNT, 'Shared projects');
}
/**
* Find a person by their unique ID.
*/
async findById(id: string): Promise<PersonNode | null> {
return this.lock.withRead(() => {
try {
const stmt = this.db.prepare('SELECT * FROM people WHERE id = ?');
const row = stmt.get(id) as Record<string, unknown> | undefined;
if (!row) return null;
return this.rowToEntity(row);
} catch (error) {
throw new PersonRepositoryError(
`Failed to find person: ${id}`,
'FIND_BY_ID_FAILED',
error
);
}
});
}
/**
* Find a person by their name or alias.
*/
async findByName(name: string): Promise<PersonNode | null> {
return this.lock.withRead(() => {
try {
validateStringLength(name, MAX_NAME_LENGTH, 'Name');
// Escape special LIKE characters to prevent injection
const escapedName = name
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_')
.replace(/"/g, '\\"');
const stmt = this.db.prepare(`
SELECT * FROM people
WHERE name = ? OR aliases LIKE ? ESCAPE '\\'
`);
const row = stmt.get(name, `%"${escapedName}"%`) as Record<string, unknown> | undefined;
if (!row) return null;
return this.rowToEntity(row);
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
'Failed to find person by name',
'FIND_BY_NAME_FAILED',
error
);
}
});
}
/**
* Search for people by name (partial match).
*/
async searchByName(query: string, options: PaginationOptions = {}): Promise<PaginatedResult<PersonNode>> {
return this.lock.withRead(() => {
try {
validateStringLength(query, MAX_NAME_LENGTH, 'Search query');
const { limit = DEFAULT_LIMIT, offset = 0 } = options;
const safeLimit = Math.min(Math.max(1, limit), MAX_LIMIT);
const safeOffset = Math.max(0, offset);
// Escape special LIKE characters
const escapedQuery = query
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_');
const searchPattern = `%${escapedQuery}%`;
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM people
WHERE name LIKE ? ESCAPE '\\' OR aliases LIKE ? ESCAPE '\\'
`);
const countResult = countStmt.get(searchPattern, searchPattern) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM people
WHERE name LIKE ? ESCAPE '\\' OR aliases LIKE ? ESCAPE '\\'
ORDER BY name
LIMIT ? OFFSET ?
`);
const rows = stmt.all(searchPattern, searchPattern, safeLimit, safeOffset) as Record<string, unknown>[];
const items = rows.map(row => this.rowToEntity(row));
return {
items,
total,
limit: safeLimit,
offset: safeOffset,
hasMore: safeOffset + items.length < total,
};
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
'Search by name failed',
'SEARCH_BY_NAME_FAILED',
error
);
}
});
}
/**
* Create a new person.
*/
async create(input: PersonNodeInput): Promise<PersonNode> {
return this.lock.withWrite(() => {
try {
this.validateInput(input, true);
// Validate relationship health is within bounds
const relationshipHealth = Math.max(0, Math.min(1, input.relationshipHealth ?? 0.5));
const id = nanoid();
const now = new Date().toISOString();
const stmt = this.db.prepare(`
INSERT INTO people (
id, name, aliases,
how_we_met, relationship_type, organization, role, location,
email, phone, social_links,
last_contact_at, contact_frequency, preferred_channel,
shared_topics, shared_projects,
notes, relationship_health,
created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
id,
input.name,
JSON.stringify(input.aliases || []),
input.howWeMet || null,
input.relationshipType || null,
input.organization || null,
input.role || null,
input.location || null,
input.email || null,
input.phone || null,
JSON.stringify(input.socialLinks || {}),
input.lastContactAt?.toISOString() || null,
input.contactFrequency || 0,
input.preferredChannel || null,
JSON.stringify(input.sharedTopics || []),
JSON.stringify(input.sharedProjects || []),
input.notes || null,
relationshipHealth,
now,
now
);
return {
id,
name: input.name,
aliases: input.aliases || [],
howWeMet: input.howWeMet,
relationshipType: input.relationshipType,
organization: input.organization,
role: input.role,
location: input.location,
email: input.email,
phone: input.phone,
socialLinks: input.socialLinks || {},
lastContactAt: input.lastContactAt,
contactFrequency: input.contactFrequency || 0,
preferredChannel: input.preferredChannel,
sharedTopics: input.sharedTopics || [],
sharedProjects: input.sharedProjects || [],
notes: input.notes,
relationshipHealth,
createdAt: new Date(now),
updatedAt: new Date(now),
};
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
'Failed to create person',
'CREATE_FAILED',
error
);
}
});
}
/**
* Update an existing person.
*/
async update(id: string, updates: Partial<PersonNodeInput>): Promise<PersonNode | null> {
return this.lock.withWrite(() => {
try {
this.validateInput(updates, false);
// First check if the person exists
const existingStmt = this.db.prepare('SELECT * FROM people WHERE id = ?');
const existing = existingStmt.get(id) as Record<string, unknown> | undefined;
if (!existing) return null;
const now = new Date().toISOString();
// Build update statement dynamically based on provided fields
const setClauses: string[] = ['updated_at = ?'];
const values: unknown[] = [now];
if (updates.name !== undefined) {
setClauses.push('name = ?');
values.push(updates.name);
}
if (updates.aliases !== undefined) {
setClauses.push('aliases = ?');
values.push(JSON.stringify(updates.aliases));
}
if (updates.howWeMet !== undefined) {
setClauses.push('how_we_met = ?');
values.push(updates.howWeMet || null);
}
if (updates.relationshipType !== undefined) {
setClauses.push('relationship_type = ?');
values.push(updates.relationshipType || null);
}
if (updates.organization !== undefined) {
setClauses.push('organization = ?');
values.push(updates.organization || null);
}
if (updates.role !== undefined) {
setClauses.push('role = ?');
values.push(updates.role || null);
}
if (updates.location !== undefined) {
setClauses.push('location = ?');
values.push(updates.location || null);
}
if (updates.email !== undefined) {
setClauses.push('email = ?');
values.push(updates.email || null);
}
if (updates.phone !== undefined) {
setClauses.push('phone = ?');
values.push(updates.phone || null);
}
if (updates.socialLinks !== undefined) {
setClauses.push('social_links = ?');
values.push(JSON.stringify(updates.socialLinks));
}
if (updates.lastContactAt !== undefined) {
setClauses.push('last_contact_at = ?');
values.push(updates.lastContactAt?.toISOString() || null);
}
if (updates.contactFrequency !== undefined) {
setClauses.push('contact_frequency = ?');
values.push(updates.contactFrequency);
}
if (updates.preferredChannel !== undefined) {
setClauses.push('preferred_channel = ?');
values.push(updates.preferredChannel || null);
}
if (updates.sharedTopics !== undefined) {
setClauses.push('shared_topics = ?');
values.push(JSON.stringify(updates.sharedTopics));
}
if (updates.sharedProjects !== undefined) {
setClauses.push('shared_projects = ?');
values.push(JSON.stringify(updates.sharedProjects));
}
if (updates.notes !== undefined) {
setClauses.push('notes = ?');
values.push(updates.notes || null);
}
if (updates.relationshipHealth !== undefined) {
const health = Math.max(0, Math.min(1, updates.relationshipHealth));
setClauses.push('relationship_health = ?');
values.push(health);
}
values.push(id);
const stmt = this.db.prepare(`
UPDATE people
SET ${setClauses.join(', ')}
WHERE id = ?
`);
stmt.run(...values);
// Fetch and return the updated person
const updatedRow = existingStmt.get(id) as Record<string, unknown>;
return this.rowToEntity(updatedRow);
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
`Failed to update person: ${id}`,
'UPDATE_FAILED',
error
);
}
});
}
/**
* Delete a person by ID.
*/
async delete(id: string): Promise<boolean> {
return this.lock.withWrite(() => {
try {
const stmt = this.db.prepare('DELETE FROM people WHERE id = ?');
const result = stmt.run(id);
return result.changes > 0;
} catch (error) {
throw new PersonRepositoryError(
`Failed to delete person: ${id}`,
'DELETE_FAILED',
error
);
}
});
}
/**
* Get people who haven't been contacted recently.
*/
async getPeopleToReconnect(
daysSinceContact: number = 30,
options: PaginationOptions = {}
): Promise<PaginatedResult<PersonNode>> {
return this.lock.withRead(() => {
try {
const { limit = DEFAULT_LIMIT, offset = 0 } = options;
const safeLimit = Math.min(Math.max(1, limit), MAX_LIMIT);
const safeOffset = Math.max(0, offset);
const cutoffDate = new Date();
cutoffDate.setDate(cutoffDate.getDate() - daysSinceContact);
const cutoffStr = cutoffDate.toISOString();
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM people
WHERE last_contact_at IS NOT NULL AND last_contact_at < ?
`);
const countResult = countStmt.get(cutoffStr) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM people
WHERE last_contact_at IS NOT NULL
AND last_contact_at < ?
ORDER BY last_contact_at ASC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(cutoffStr, safeLimit, safeOffset) as Record<string, unknown>[];
const items = rows.map(row => this.rowToEntity(row));
return {
items,
total,
limit: safeLimit,
offset: safeOffset,
hasMore: safeOffset + items.length < total,
};
} catch (error) {
throw new PersonRepositoryError(
'Failed to get people to reconnect',
'GET_RECONNECT_FAILED',
error
);
}
});
}
/**
* Record a contact with a person (updates last_contact_at).
*/
async recordContact(id: string): Promise<void> {
return this.lock.withWrite(() => {
try {
const stmt = this.db.prepare(`
UPDATE people
SET last_contact_at = ?, updated_at = ?
WHERE id = ?
`);
const now = new Date().toISOString();
const result = stmt.run(now, now, id);
if (result.changes === 0) {
throw new PersonRepositoryError(
`Person not found: ${id}`,
'PERSON_NOT_FOUND'
);
}
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
`Failed to record contact: ${id}`,
'RECORD_CONTACT_FAILED',
error
);
}
});
}
/**
* Find people by organization.
*/
async findByOrganization(
org: string,
options: PaginationOptions = {}
): Promise<PaginatedResult<PersonNode>> {
return this.lock.withRead(() => {
try {
validateStringLength(org, MAX_NAME_LENGTH, 'Organization');
const { limit = DEFAULT_LIMIT, offset = 0 } = options;
const safeLimit = Math.min(Math.max(1, limit), MAX_LIMIT);
const safeOffset = Math.max(0, offset);
// Escape special LIKE characters
const escapedOrg = org
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_');
const searchPattern = `%${escapedOrg}%`;
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM people
WHERE organization LIKE ? ESCAPE '\\'
`);
const countResult = countStmt.get(searchPattern) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM people
WHERE organization LIKE ? ESCAPE '\\'
ORDER BY name
LIMIT ? OFFSET ?
`);
const rows = stmt.all(searchPattern, safeLimit, safeOffset) as Record<string, unknown>[];
const items = rows.map(row => this.rowToEntity(row));
return {
items,
total,
limit: safeLimit,
offset: safeOffset,
hasMore: safeOffset + items.length < total,
};
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
'Failed to find people by organization',
'FIND_BY_ORG_FAILED',
error
);
}
});
}
/**
* Find people by shared topic.
*/
async findBySharedTopic(
topic: string,
options: PaginationOptions = {}
): Promise<PaginatedResult<PersonNode>> {
return this.lock.withRead(() => {
try {
validateStringLength(topic, MAX_NAME_LENGTH, 'Topic');
const { limit = DEFAULT_LIMIT, offset = 0 } = options;
const safeLimit = Math.min(Math.max(1, limit), MAX_LIMIT);
const safeOffset = Math.max(0, offset);
// Escape special LIKE characters and quotes for JSON search
const escapedTopic = topic
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_')
.replace(/"/g, '\\"');
const searchPattern = `%"${escapedTopic}"%`;
// Get total count
const countStmt = this.db.prepare(`
SELECT COUNT(*) as total FROM people
WHERE shared_topics LIKE ? ESCAPE '\\'
`);
const countResult = countStmt.get(searchPattern) as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare(`
SELECT * FROM people
WHERE shared_topics LIKE ? ESCAPE '\\'
ORDER BY name
LIMIT ? OFFSET ?
`);
const rows = stmt.all(searchPattern, safeLimit, safeOffset) as Record<string, unknown>[];
const items = rows.map(row => this.rowToEntity(row));
return {
items,
total,
limit: safeLimit,
offset: safeOffset,
hasMore: safeOffset + items.length < total,
};
} catch (error) {
if (error instanceof PersonRepositoryError) throw error;
throw new PersonRepositoryError(
'Failed to find people by shared topic',
'FIND_BY_TOPIC_FAILED',
error
);
}
});
}
/**
* Get all people with pagination.
*/
async getAll(options: PaginationOptions = {}): Promise<PaginatedResult<PersonNode>> {
return this.lock.withRead(() => {
try {
const { limit = DEFAULT_LIMIT, offset = 0 } = options;
const safeLimit = Math.min(Math.max(1, limit), MAX_LIMIT);
const safeOffset = Math.max(0, offset);
// Get total count
const countResult = this.db.prepare('SELECT COUNT(*) as total FROM people').get() as { total: number };
const total = countResult.total;
// Get paginated results
const stmt = this.db.prepare('SELECT * FROM people ORDER BY name LIMIT ? OFFSET ?');
const rows = stmt.all(safeLimit, safeOffset) as Record<string, unknown>[];
const items = rows.map(row => this.rowToEntity(row));
return {
items,
total,
limit: safeLimit,
offset: safeOffset,
hasMore: safeOffset + items.length < total,
};
} catch (error) {
throw new PersonRepositoryError(
'Failed to get all people',
'GET_ALL_FAILED',
error
);
}
});
}
}

26
packages/core/src/repositories/index.ts
View file

@ -1,26 +0,0 @@
// Re-export from NodeRepository (primary source for common types)
export {
NodeRepository,
type INodeRepository,
type PaginationOptions,
type PaginatedResult,
type GitContext,
} from './NodeRepository.js';
// Re-export from PersonRepository (exclude duplicate types)
export {
PersonRepository,
type IPersonRepository,
type PersonNodeInput,
PersonRepositoryError,
} from './PersonRepository.js';
// Re-export from EdgeRepository (exclude duplicate types)
export {
EdgeRepository,
type IEdgeRepository,
type GraphEdgeInput,
type EdgeType,
type TransitivePath,
EdgeRepositoryError,
} from './EdgeRepository.js';

603
packages/core/src/services/CacheService.ts
View file

@ -1,603 +0,0 @@
import type { KnowledgeNode, PersonNode } from '../core/types.js';
import type { PaginatedResult } from '../repositories/PersonRepository.js';
// ============================================================================
// TYPES
// ============================================================================
/**
* Represents a single entry in the cache with metadata for TTL and LRU eviction.
*/
export interface CacheEntry<T> {
/** The cached value */
value: T;
/** Unix timestamp (ms) when this entry expires */
expiresAt: number;
/** Number of times this entry has been accessed */
accessCount: number;
/** Unix timestamp (ms) of the last access */
lastAccessed: number;
/** Estimated size in bytes (optional, for memory-based eviction) */
size?: number;
}
/**
* Configuration options for the cache service.
*/
export interface CacheOptions {
/** Maximum number of entries in the cache */
maxSize: number;
/** Maximum memory usage in bytes (optional) */
maxMemory?: number;
/** Default TTL in milliseconds */
defaultTTL: number;
/** Interval in milliseconds for automatic cleanup of expired entries */
cleanupInterval: number;
}
/**
* Statistics about cache performance and state.
*/
export interface CacheStats {
/** Current number of entries in the cache */
size: number;
/** Hit rate as a ratio (0-1) */
hitRate: number;
/** Estimated memory usage in bytes */
memoryUsage: number;
}
// ============================================================================
// CONSTANTS
// ============================================================================
const DEFAULT_OPTIONS: CacheOptions = {
maxSize: 10000,
defaultTTL: 5 * 60 * 1000, // 5 minutes
cleanupInterval: 60 * 1000, // 1 minute
};
// ============================================================================
// CACHE SERVICE
// ============================================================================
/**
* A generic in-memory cache service with TTL support and LRU eviction.
*
* Features:
* - Time-based expiration (TTL)
* - LRU eviction when max size is reached
* - Memory-based eviction (optional)
* - Automatic cleanup of expired entries
* - Pattern-based invalidation
* - Cache-aside pattern support (getOrCompute)
* - Hit rate tracking
*
* @template T The type of values stored in the cache
*/
export class CacheService<T = unknown> {
private cache: Map<string, CacheEntry<T>> = new Map();
private options: CacheOptions;
private cleanupTimer: ReturnType<typeof setInterval> | null = null;
private hits = 0;
private misses = 0;
private totalMemory = 0;
constructor(options?: Partial<CacheOptions>) {
this.options = { ...DEFAULT_OPTIONS, ...options };
this.startCleanupTimer();
}
// --------------------------------------------------------------------------
// PUBLIC METHODS
// --------------------------------------------------------------------------
/**
* Get a value from cache.
* Updates access metadata if the entry exists and is not expired.
*
* @param key The cache key
* @returns The cached value, or undefined if not found or expired
*/
get(key: string): T | undefined {
const entry = this.cache.get(key);
if (!entry) {
this.misses++;
return undefined;
}
// Check if expired
if (Date.now() > entry.expiresAt) {
this.deleteEntry(key, entry);
this.misses++;
return undefined;
}
// Update access metadata
entry.accessCount++;
entry.lastAccessed = Date.now();
this.hits++;
return entry.value;
}
/**
* Set a value in cache.
* Performs LRU eviction if the cache is at capacity.
*
* @param key The cache key
* @param value The value to cache
* @param ttl Optional TTL in milliseconds (defaults to configured defaultTTL)
*/
set(key: string, value: T, ttl?: number): void {
const now = Date.now();
const effectiveTTL = ttl ?? this.options.defaultTTL;
const size = this.estimateSize(value);
// If key already exists, remove old entry's size from total
const existingEntry = this.cache.get(key);
if (existingEntry) {
this.totalMemory -= existingEntry.size ?? 0;
}
// Evict entries if needed (before adding new entry)
this.evictIfNeeded(size);
const entry: CacheEntry<T> = {
value,
expiresAt: now + effectiveTTL,
accessCount: 0,
lastAccessed: now,
size,
};
this.cache.set(key, entry);
this.totalMemory += size;
}
/**
* Delete a key from cache.
*
* @param key The cache key to delete
* @returns true if the key was deleted, false if it didn't exist
*/
delete(key: string): boolean {
const entry = this.cache.get(key);
if (entry) {
this.deleteEntry(key, entry);
return true;
}
return false;
}
/**
* Check if a key exists in cache and is not expired.
*
* @param key The cache key
* @returns true if the key exists and is not expired
*/
has(key: string): boolean {
const entry = this.cache.get(key);
if (!entry) return false;
if (Date.now() > entry.expiresAt) {
this.deleteEntry(key, entry);
return false;
}
return true;
}
/**
* Invalidate all keys matching a pattern.
*
* @param pattern A RegExp pattern to match keys against
* @returns The number of keys invalidated
*/
invalidatePattern(pattern: RegExp): number {
let count = 0;
const keysToDelete: string[] = [];
for (const key of this.cache.keys()) {
if (pattern.test(key)) {
keysToDelete.push(key);
}
}
for (const key of keysToDelete) {
const entry = this.cache.get(key);
if (entry) {
this.deleteEntry(key, entry);
count++;
}
}
return count;
}
/**
* Clear all entries from the cache.
*/
clear(): void {
this.cache.clear();
this.totalMemory = 0;
this.hits = 0;
this.misses = 0;
}
/**
* Get or compute a value (cache-aside pattern).
* If the key exists and is not expired, returns the cached value.
* Otherwise, computes the value using the provided function, caches it, and returns it.
*
* @param key The cache key
* @param compute A function that computes the value if not cached
* @param ttl Optional TTL in milliseconds
* @returns The cached or computed value
*/
async getOrCompute(
key: string,
compute: () => Promise<T>,
ttl?: number
): Promise<T> {
// Try to get from cache first
const cached = this.get(key);
if (cached !== undefined) {
return cached;
}
// Compute the value
const value = await compute();
// Cache and return
this.set(key, value, ttl);
return value;
}
/**
* Get cache statistics.
*
* @returns Statistics about cache performance and state
*/
stats(): CacheStats {
const totalRequests = this.hits + this.misses;
return {
size: this.cache.size,
hitRate: totalRequests > 0 ? this.hits / totalRequests : 0,
memoryUsage: this.totalMemory,
};
}
/**
* Stop the cleanup timer and release resources.
* Call this when the cache is no longer needed to prevent memory leaks.
*/
destroy(): void {
if (this.cleanupTimer) {
clearInterval(this.cleanupTimer);
this.cleanupTimer = null;
}
this.clear();
}
// --------------------------------------------------------------------------
// PRIVATE METHODS
// --------------------------------------------------------------------------
/**
* Start the automatic cleanup timer.
*/
private startCleanupTimer(): void {
if (this.options.cleanupInterval > 0) {
this.cleanupTimer = setInterval(() => {
this.cleanup();
}, this.options.cleanupInterval);
// Don't prevent Node.js from exiting if this is the only timer
if (this.cleanupTimer.unref) {
this.cleanupTimer.unref();
}
}
}
/**
* Remove expired entries from the cache.
*/
private cleanup(): void {
const now = Date.now();
const keysToDelete: string[] = [];
for (const [key, entry] of this.cache.entries()) {
if (now > entry.expiresAt) {
keysToDelete.push(key);
}
}
for (const key of keysToDelete) {
const entry = this.cache.get(key);
if (entry) {
this.deleteEntry(key, entry);
}
}
}
/**
* Delete an entry and update memory tracking.
*/
private deleteEntry(key: string, entry: CacheEntry<T>): void {
this.totalMemory -= entry.size ?? 0;
this.cache.delete(key);
}
/**
* Evict entries if the cache is at capacity.
* Uses LRU eviction strategy based on lastAccessed timestamp.
* Also considers memory limits if configured.
*/
private evictIfNeeded(incomingSize: number): void {
// Evict for size limit
while (this.cache.size >= this.options.maxSize) {
this.evictLRU();
}
// Evict for memory limit if configured
if (this.options.maxMemory) {
while (
this.totalMemory + incomingSize > this.options.maxMemory &&
this.cache.size > 0
) {
this.evictLRU();
}
}
}
/**
* Evict the least recently used entry.
* Finds the entry with the oldest lastAccessed timestamp and removes it.
*/
private evictLRU(): void {
let oldestKey: string | null = null;
let oldestTime = Infinity;
for (const [key, entry] of this.cache.entries()) {
if (entry.lastAccessed < oldestTime) {
oldestTime = entry.lastAccessed;
oldestKey = key;
}
}
if (oldestKey !== null) {
const entry = this.cache.get(oldestKey);
if (entry) {
this.deleteEntry(oldestKey, entry);
}
}
}
/**
* Estimate the memory size of a value in bytes.
* This is a rough approximation for memory tracking purposes.
*/
private estimateSize(value: T): number {
if (value === null || value === undefined) {
return 8;
}
const type = typeof value;
if (type === 'boolean') {
return 4;
}
if (type === 'number') {
return 8;
}
if (type === 'string') {
return (value as string).length * 2 + 40; // 2 bytes per char + overhead
}
if (Array.isArray(value)) {
// For arrays, estimate based on length
// This is a rough approximation
return 40 + (value as unknown[]).length * 8;
}
if (type === 'object') {
// For objects, use JSON serialization as a rough estimate
try {
const json = JSON.stringify(value);
return json.length * 2 + 40;
} catch {
return 1024; // Default size for non-serializable objects
}
}
return 8;
}
}
// ============================================================================
// CACHE KEY HELPERS
// ============================================================================
/**
* Standard cache key patterns for Vestige MCP.
* These functions generate consistent cache keys for different entity types.
*/
export const CACHE_KEYS = {
/** Cache key for a knowledge node by ID */
node: (id: string): string => `node:${id}`,
/** Cache key for a person by ID */
person: (id: string): string => `person:${id}`,
/** Cache key for search results */
search: (query: string, opts: string): string => `search:${query}:${opts}`,
/** Cache key for embeddings by node ID */
embedding: (nodeId: string): string => `embedding:${nodeId}`,
/** Cache key for related nodes */
related: (nodeId: string, depth: number): string => `related:${nodeId}:${depth}`,
/** Cache key for person by name */
personByName: (name: string): string => `person:name:${name.toLowerCase()}`,
/** Cache key for daily brief by date */
dailyBrief: (date: string): string => `daily-brief:${date}`,
};
/**
* Pattern matchers for cache invalidation.
*/
export const CACHE_PATTERNS = {
/** All node-related entries */
allNodes: /^node:/,
/** All person-related entries */
allPeople: /^person:/,
/** All search results */
allSearches: /^search:/,
/** All embeddings */
allEmbeddings: /^embedding:/,
/** All related node entries */
allRelated: /^related:/,
/** Entries for a specific node and its related data */
nodeAndRelated: (nodeId: string): RegExp =>
new RegExp(`^(node:${nodeId}|related:${nodeId}|embedding:${nodeId})`),
/** Entries for a specific person and related data */
personAndRelated: (personId: string): RegExp =>
new RegExp(`^person:(${personId}|name:)`),
};
// ============================================================================
// SPECIALIZED CACHE INSTANCES
// ============================================================================
/**
* Cache for KnowledgeNode entities.
* Longer TTL since nodes don't change frequently.
*/
export const nodeCache = new CacheService<KnowledgeNode>({
maxSize: 5000,
defaultTTL: 10 * 60 * 1000, // 10 minutes
cleanupInterval: 2 * 60 * 1000, // 2 minutes
});
/**
* Cache for search results.
* Shorter TTL since search results can change with new data.
*/
export const searchCache = new CacheService<PaginatedResult<KnowledgeNode>>({
maxSize: 1000,
defaultTTL: 60 * 1000, // 1 minute
cleanupInterval: 30 * 1000, // 30 seconds
});
/**
* Cache for embedding vectors.
* Longer TTL since embeddings don't change for existing content.
*/
export const embeddingCache = new CacheService<number[]>({
maxSize: 10000,
defaultTTL: 60 * 60 * 1000, // 1 hour
cleanupInterval: 5 * 60 * 1000, // 5 minutes
});
/**
* Cache for PersonNode entities.
*/
export const personCache = new CacheService<PersonNode>({
maxSize: 2000,
defaultTTL: 10 * 60 * 1000, // 10 minutes
cleanupInterval: 2 * 60 * 1000, // 2 minutes
});
/**
* Cache for related nodes queries.
*/
export const relatedCache = new CacheService<KnowledgeNode[]>({
maxSize: 2000,
defaultTTL: 5 * 60 * 1000, // 5 minutes
cleanupInterval: 60 * 1000, // 1 minute
});
// ============================================================================
// UTILITY FUNCTIONS
// ============================================================================
/**
* Invalidate all caches related to a specific node.
* Call this when a node is created, updated, or deleted.
*
* @param nodeId The ID of the node that changed
*/
export function invalidateNodeCaches(nodeId: string): void {
nodeCache.delete(CACHE_KEYS.node(nodeId));
embeddingCache.delete(CACHE_KEYS.embedding(nodeId));
// Invalidate related entries and search results
relatedCache.invalidatePattern(new RegExp(`^related:${nodeId}`));
searchCache.clear(); // Search results may be affected
}
/**
* Invalidate all caches related to a specific person.
* Call this when a person is created, updated, or deleted.
*
* @param personId The ID of the person that changed
* @param name Optional name to also invalidate name-based lookups
*/
export function invalidatePersonCaches(personId: string, name?: string): void {
personCache.delete(CACHE_KEYS.person(personId));
if (name) {
personCache.delete(CACHE_KEYS.personByName(name));
}
// Search results may reference this person
searchCache.clear();
}
/**
* Clear all caches. Useful for testing or when major data changes occur.
*/
export function clearAllCaches(): void {
nodeCache.clear();
searchCache.clear();
embeddingCache.clear();
personCache.clear();
relatedCache.clear();
}
/**
* Get aggregated statistics from all caches.
*/
export function getAllCacheStats(): Record<string, CacheStats> {
return {
node: nodeCache.stats(),
search: searchCache.stats(),
embedding: embeddingCache.stats(),
person: personCache.stats(),
related: relatedCache.stats(),
};
}
/**
* Destroy all cache instances and stop cleanup timers.
* Call this during application shutdown.
*/
export function destroyAllCaches(): void {
nodeCache.destroy();
searchCache.destroy();
embeddingCache.destroy();
personCache.destroy();
relatedCache.destroy();
}

7
packages/core/src/utils/index.ts
View file

@ -1,7 +0,0 @@
/**
* Utility exports
*/
export * from './mutex.js';
export * from './json.js';
export * from './logger.js';

230
packages/core/src/utils/json.ts
View file

@ -1,230 +0,0 @@
/**
* Safe JSON utilities for database operations
*/
import { z } from 'zod';
import { logger } from './logger.js';
/**
* Safely parse JSON with logging on failure
*/
export function safeJsonParse<T>(
value: string | null | undefined,
fallback: T,
options?: {
logOnError?: boolean;
context?: string;
}
): T {
if (!value) return fallback;
try {
const parsed = JSON.parse(value);
// Type validation
if (typeof parsed !== typeof fallback) {
if (options?.logOnError !== false) {
logger.warn('JSON parse type mismatch', {
expected: typeof fallback,
got: typeof parsed,
context: options?.context,
});
}
return fallback;
}
// Array validation
if (Array.isArray(fallback) && !Array.isArray(parsed)) {
if (options?.logOnError !== false) {
logger.warn('JSON parse expected array', {
got: typeof parsed,
context: options?.context,
});
}
return fallback;
}
return parsed as T;
} catch (error) {
if (options?.logOnError !== false) {
logger.warn('JSON parse failed', {
error: (error as Error).message,
valuePreview: value.slice(0, 100),
context: options?.context,
});
}
return fallback;
}
}
/**
* Safely stringify JSON with circular reference handling
*/
export function safeJsonStringify(
value: unknown,
options?: {
replacer?: (key: string, value: unknown) => unknown;
space?: number;
maxDepth?: number;
}
): string {
const seen = new WeakSet();
const maxDepth = options?.maxDepth ?? 10;
function replacer(
this: unknown,
key: string,
value: unknown,
depth: number
): unknown {
if (depth > maxDepth) {
return '[Max Depth Exceeded]';
}
if (typeof value === 'object' && value !== null) {
if (seen.has(value)) {
return '[Circular Reference]';
}
seen.add(value);
}
if (options?.replacer) {
return options.replacer(key, value);
}
// Handle special types
if (value instanceof Error) {
return {
name: value.name,
message: value.message,
stack: value.stack,
};
}
if (value instanceof Date) {
return value.toISOString();
}
if (value instanceof Map) {
return Object.fromEntries(value);
}
if (value instanceof Set) {
return Array.from(value);
}
return value;
}
try {
// Create a depth-tracking replacer
let currentDepth = 0;
return JSON.stringify(
value,
function (key, val) {
if (key === '') currentDepth = 0;
else currentDepth++;
return replacer.call(this, key, val, currentDepth);
},
options?.space
);
} catch (error) {
logger.error('JSON stringify failed', error as Error);
return '{}';
}
}
/**
* Parse JSON and validate against Zod schema
*/
export function parseJsonWithSchema<T extends z.ZodType>(
value: string | null | undefined,
schema: T,
fallback: z.infer<T>
): z.infer<T> {
if (!value) return fallback;
try {
const parsed = JSON.parse(value);
const result = schema.safeParse(parsed);
if (result.success) {
return result.data;
}
logger.warn('JSON schema validation failed', {
errors: result.error.errors,
});
return fallback;
} catch (error) {
logger.warn('JSON parse failed for schema validation', {
error: (error as Error).message,
});
return fallback;
}
}
/**
* Calculate diff between two JSON objects
*/
export function jsonDiff(
before: Record<string, unknown>,
after: Record<string, unknown>
): { added: string[]; removed: string[]; changed: string[] } {
const added: string[] = [];
const removed: string[] = [];
const changed: string[] = [];
// Check for added and changed
for (const key of Object.keys(after)) {
if (!(key in before)) {
added.push(key);
} else if (JSON.stringify(before[key]) !== JSON.stringify(after[key])) {
changed.push(key);
}
}
// Check for removed
for (const key of Object.keys(before)) {
if (!(key in after)) {
removed.push(key);
}
}
return { added, removed, changed };
}
/**
* Deep merge JSON objects
*/
export function jsonMerge<T extends Record<string, unknown>>(
target: T,
...sources: Partial<T>[]
): T {
const result = { ...target };
for (const source of sources) {
for (const key of Object.keys(source)) {
const targetVal = result[key as keyof T];
const sourceVal = source[key as keyof T];
if (
typeof targetVal === 'object' &&
targetVal !== null &&
typeof sourceVal === 'object' &&
sourceVal !== null &&
!Array.isArray(targetVal) &&
!Array.isArray(sourceVal)
) {
(result as Record<string, unknown>)[key] = jsonMerge(
targetVal as Record<string, unknown>,
sourceVal as Record<string, unknown>
);
} else if (sourceVal !== undefined) {
(result as Record<string, unknown>)[key] = sourceVal;
}
}
}
return result;
}

253
packages/core/src/utils/logger.ts
View file

@ -1,253 +0,0 @@
/**
* Centralized logging system for Vestige MCP
*
* Provides structured JSON logging with:
* - Log levels (debug, info, warn, error)
* - Child loggers for subsystems
* - Request context tracking via AsyncLocalStorage
* - Performance logging utilities
*/
import { AsyncLocalStorage } from 'async_hooks';
import { nanoid } from 'nanoid';
// ============================================================================
// Types
// ============================================================================
export type LogLevel = 'debug' | 'info' | 'warn' | 'error';
export interface LogEntry {
timestamp: string;
level: LogLevel;
logger: string;
message: string;
context?: Record<string, unknown>;
error?: {
name: string;
message: string;
stack?: string;
};
}
export interface Logger {
debug(message: string, context?: Record<string, unknown>): void;
info(message: string, context?: Record<string, unknown>): void;
warn(message: string, context?: Record<string, unknown>): void;
error(message: string, error?: Error, context?: Record<string, unknown>): void;
child(name: string): Logger;
}
// ============================================================================
// Request Context (AsyncLocalStorage)
// ============================================================================
interface RequestContext {
requestId: string;
startTime: number;
}
export const requestContext = new AsyncLocalStorage<RequestContext>();
/**
* Run a function within a request context for tracing
*/
export function withRequestContext<T>(fn: () => T): T {
const ctx: RequestContext = {
requestId: nanoid(8),
startTime: Date.now(),
};
return requestContext.run(ctx, fn);
}
/**
* Run an async function within a request context for tracing
*/
export function withRequestContextAsync<T>(fn: () => Promise<T>): Promise<T> {
const ctx: RequestContext = {
requestId: nanoid(8),
startTime: Date.now(),
};
return requestContext.run(ctx, fn);
}
/**
* Enrich context with request tracing information if available
*/
function enrichContext(context?: Record<string, unknown>): Record<string, unknown> {
const ctx = requestContext.getStore();
if (ctx) {
return {
...context,
requestId: ctx.requestId,
elapsed: Date.now() - ctx.startTime,
};
}
return context || {};
}
// ============================================================================
// Logger Implementation
// ============================================================================
const LOG_LEVELS: Record<LogLevel, number> = {
debug: 0,
info: 1,
warn: 2,
error: 3,
};
/**
* Create a structured JSON logger
*
* @param name - Logger name (used as prefix for child loggers)
* @param minLevel - Minimum log level to output (default: 'info')
* @returns Logger instance
*/
export function createLogger(name: string, minLevel: LogLevel = 'info'): Logger {
const minLevelValue = LOG_LEVELS[minLevel];
function log(
level: LogLevel,
message: string,
context?: Record<string, unknown>,
error?: Error
): void {
if (LOG_LEVELS[level] < minLevelValue) return;
const enrichedContext = enrichContext(context);
const entry: LogEntry = {
timestamp: new Date().toISOString(),
level,
logger: name,
message,
};
// Only include context if it has properties
if (Object.keys(enrichedContext).length > 0) {
entry.context = enrichedContext;
}
if (error) {
entry.error = {
name: error.name,
message: error.message,
...(error.stack !== undefined && { stack: error.stack }),
};
}
const output = JSON.stringify(entry);
if (level === 'error') {
console.error(output);
} else {
console.log(output);
}
}
return {
debug: (message, context) => log('debug', message, context),
info: (message, context) => log('info', message, context),
warn: (message, context) => log('warn', message, context),
error: (message, error, context) => log('error', message, context, error),
child: (childName) => createLogger(`${name}:${childName}`, minLevel),
};
}
// ============================================================================
// Global Logger Instances
// ============================================================================
// Get log level from environment
function getLogLevelFromEnv(): LogLevel {
const envLevel = process.env['VESTIGE_LOG_LEVEL']?.toLowerCase();
if (envLevel && envLevel in LOG_LEVELS) {
return envLevel as LogLevel;
}
return 'info';
}
const LOG_LEVEL = getLogLevelFromEnv();
// Root logger
export const logger = createLogger('vestige', LOG_LEVEL);
// Pre-configured child loggers for subsystems
export const dbLogger = logger.child('database');
export const mcpLogger = logger.child('mcp');
export const remLogger = logger.child('rem-cycle');
export const embeddingLogger = logger.child('embeddings');
export const cacheLogger = logger.child('cache');
export const jobLogger = logger.child('jobs');
// ============================================================================
// Performance Logging
// ============================================================================
/**
* Wrap a function to log its execution time
*
* @param logger - Logger instance to use
* @param operationName - Name of the operation for logging
* @param fn - Async function to wrap
* @returns Wrapped function that logs performance
*
* @example
* const wrappedFetch = logPerformance(dbLogger, 'fetchNodes', fetchNodes);
* const nodes = await wrappedFetch(query);
*/
export function logPerformance<T extends (...args: unknown[]) => Promise<unknown>>(
logger: Logger,
operationName: string,
fn: T
): T {
return (async (...args: Parameters<T>) => {
const start = Date.now();
try {
const result = await fn(...args);
logger.info(`${operationName} completed`, {
duration: Date.now() - start,
});
return result;
} catch (error) {
logger.error(`${operationName} failed`, error as Error, {
duration: Date.now() - start,
});
throw error;
}
}) as T;
}
/**
* Log performance of a single async operation
*
* @param logger - Logger instance to use
* @param operationName - Name of the operation for logging
* @param fn - Async function to execute and measure
* @returns Result of the function
*
* @example
* const result = await timedOperation(dbLogger, 'query', async () => {
* return await db.query(sql);
* });
*/
export async function timedOperation<T>(
logger: Logger,
operationName: string,
fn: () => Promise<T>
): Promise<T> {
const start = Date.now();
try {
const result = await fn();
logger.info(`${operationName} completed`, {
duration: Date.now() - start,
});
return result;
} catch (error) {
logger.error(`${operationName} failed`, error as Error, {
duration: Date.now() - start,
});
throw error;
}
}

451
packages/core/src/utils/mutex.ts
View file

@ -1,451 +0,0 @@
/**
* Concurrency utilities for Vestige MCP
*
* Provides synchronization primitives for managing concurrent access
* to shared resources like database connections.
*/
/**
* Error thrown when an operation times out
*/
export class TimeoutError extends Error {
constructor(message = "Operation timed out") {
super(message);
this.name = "TimeoutError";
}
}
/**
* Reader-Writer Lock for concurrent database access.
* Allows multiple concurrent readers OR one exclusive writer.
*
* This implementation uses writer preference with reader batching
* to prevent writer starvation while still allowing good read throughput.
*/
export class RWLock {
private readers = 0;
private writer = false;
private writerQueue: (() => void)[] = [];
private readerQueue: (() => void)[] = [];
/**
* Execute a function with read lock (allows concurrent readers)
*/
async withReadLock<T>(fn: () => Promise<T>): Promise<T> {
await this.acquireRead();
try {
return await fn();
} finally {
this.releaseRead();
}
}
/**
* Execute a function with write lock (exclusive access)
*/
async withWriteLock<T>(fn: () => Promise<T>): Promise<T> {
await this.acquireWrite();
try {
return await fn();
} finally {
this.releaseWrite();
}
}
private acquireRead(): Promise<void> {
return new Promise<void>((resolve) => {
// If no writer and no writers waiting, grant immediately
if (!this.writer && this.writerQueue.length === 0) {
this.readers++;
resolve();
} else {
// Queue the reader
this.readerQueue.push(() => {
this.readers++;
resolve();
});
}
});
}
private releaseRead(): void {
this.readers--;
// If no more readers, wake up waiting writer
if (this.readers === 0 && this.writerQueue.length > 0) {
const nextWriter = this.writerQueue.shift();
if (nextWriter) {
this.writer = true;
nextWriter();
}
}
}
private acquireWrite(): Promise<void> {
return new Promise<void>((resolve) => {
// If no readers and no writer, grant immediately
if (this.readers === 0 && !this.writer) {
this.writer = true;
resolve();
} else {
// Queue the writer
this.writerQueue.push(resolve);
}
});
}
private releaseWrite(): void {
this.writer = false;
// Prefer waking readers over writers to prevent starvation
// Wake all waiting readers as a batch
if (this.readerQueue.length > 0) {
const readers = this.readerQueue.splice(0, this.readerQueue.length);
for (const reader of readers) {
reader();
}
} else if (this.writerQueue.length > 0) {
// No waiting readers, wake next writer
const nextWriter = this.writerQueue.shift();
if (nextWriter) {
this.writer = true;
nextWriter();
}
}
}
/**
* Get current lock state (for debugging/monitoring)
*/
getState(): { readers: number; hasWriter: boolean; pendingReaders: number; pendingWriters: number } {
return {
readers: this.readers,
hasWriter: this.writer,
pendingReaders: this.readerQueue.length,
pendingWriters: this.writerQueue.length,
};
}
}
/**
* Simple mutex for exclusive access
*/
export class Mutex {
private locked = false;
private queue: (() => void)[] = [];
/**
* Execute a function with exclusive lock
*/
async withLock<T>(fn: () => Promise<T>): Promise<T> {
await this.acquire();
try {
return await fn();
} finally {
this.release();
}
}
private acquire(): Promise<void> {
return new Promise<void>((resolve) => {
if (!this.locked) {
this.locked = true;
resolve();
} else {
this.queue.push(resolve);
}
});
}
private release(): void {
if (this.queue.length > 0) {
const next = this.queue.shift();
if (next) {
next();
}
} else {
this.locked = false;
}
}
/**
* Check if the mutex is currently locked
*/
isLocked(): boolean {
return this.locked;
}
/**
* Get the number of waiters in the queue
*/
getQueueLength(): number {
return this.queue.length;
}
}
/**
* Semaphore for limiting concurrent operations
*/
export class Semaphore {
private permits: number;
private available: number;
private queue: (() => void)[] = [];
constructor(permits: number) {
if (permits < 1) {
throw new Error("Semaphore must have at least 1 permit");
}
this.permits = permits;
this.available = permits;
}
/**
* Execute a function with a permit from the semaphore
*/
async withPermit<T>(fn: () => Promise<T>): Promise<T> {
await this.acquire();
try {
return await fn();
} finally {
this.release();
}
}
/**
* Execute multiple functions concurrently, respecting the semaphore limit
*/
async map<T, R>(items: T[], fn: (item: T) => Promise<R>): Promise<R[]> {
return Promise.all(items.map((item) => this.withPermit(() => fn(item))));
}
private acquire(): Promise<void> {
return new Promise<void>((resolve) => {
if (this.available > 0) {
this.available--;
resolve();
} else {
this.queue.push(resolve);
}
});
}
private release(): void {
if (this.queue.length > 0) {
const next = this.queue.shift();
if (next) {
next();
}
} else {
this.available++;
}
}
/**
* Get the number of available permits
*/
getAvailable(): number {
return this.available;
}
/**
* Get the total number of permits
*/
getTotal(): number {
return this.permits;
}
/**
* Get the number of waiters in the queue
*/
getQueueLength(): number {
return this.queue.length;
}
}
/**
* Add timeout to any promise
*
* @param promise - The promise to wrap with a timeout
* @param ms - Timeout in milliseconds
* @param message - Optional custom error message
* @returns The result of the promise if it completes in time
* @throws TimeoutError if the timeout is exceeded
*/
export function withTimeout<T>(promise: Promise<T>, ms: number, message?: string): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timeoutId = setTimeout(() => {
reject(new TimeoutError(message ?? `Operation timed out after ${ms}ms`));
}, ms);
promise
.then((result) => {
clearTimeout(timeoutId);
resolve(result);
})
.catch((error) => {
clearTimeout(timeoutId);
reject(error);
});
});
}
/**
* Options for retry with exponential backoff
*/
export interface RetryOptions {
/** Maximum number of retry attempts (default: 3) */
maxRetries?: number;
/** Initial delay in milliseconds (default: 100) */
initialDelay?: number;
/** Maximum delay in milliseconds (default: 5000) */
maxDelay?: number;
/** Backoff multiplier (default: 2) */
backoffFactor?: number;
/** Optional function to determine if an error is retryable */
isRetryable?: (error: unknown) => boolean;
/** Optional callback called before each retry */
onRetry?: (error: unknown, attempt: number, delay: number) => void;
}
/**
* Retry function with exponential backoff
*
* @param fn - The async function to retry
* @param options - Retry configuration options
* @returns The result of the function if it succeeds
* @throws The last error if all retries are exhausted
*/
export async function retry<T>(fn: () => Promise<T>, options: RetryOptions = {}): Promise<T> {
const {
maxRetries = 3,
initialDelay = 100,
maxDelay = 5000,
backoffFactor = 2,
isRetryable = () => true,
onRetry,
} = options;
let lastError: unknown;
let delay = initialDelay;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error;
// Check if we've exhausted retries
if (attempt >= maxRetries) {
throw error;
}
// Check if the error is retryable
if (!isRetryable(error)) {
throw error;
}
// Calculate delay with jitter (0.5 to 1.5 of calculated delay)
const jitter = 0.5 + Math.random();
const actualDelay = Math.min(delay * jitter, maxDelay);
// Call onRetry callback if provided
if (onRetry) {
onRetry(error, attempt + 1, actualDelay);
}
// Wait before next attempt
await sleep(actualDelay);
// Increase delay for next attempt
delay = Math.min(delay * backoffFactor, maxDelay);
}
}
// This should never be reached, but TypeScript needs it
throw lastError;
}
/**
* Sleep for a specified duration
*
* @param ms - Duration in milliseconds
*/
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/**
* Debounce a function - only execute after the specified delay
* has passed without another call
*
* @param fn - The function to debounce
* @param delay - Delay in milliseconds
*/
export function debounce<T extends (...args: unknown[]) => unknown>(
fn: T,
delay: number
): (...args: Parameters<T>) => void {
let timeoutId: ReturnType<typeof setTimeout> | null = null;
return (...args: Parameters<T>) => {
if (timeoutId) {
clearTimeout(timeoutId);
}
timeoutId = setTimeout(() => {
fn(...args);
timeoutId = null;
}, delay);
};
}
/**
* Throttle a function - execute at most once per specified interval
*
* @param fn - The function to throttle
* @param interval - Minimum interval between executions in milliseconds
*/
export function throttle<T extends (...args: unknown[]) => unknown>(
fn: T,
interval: number
): (...args: Parameters<T>) => void {
let lastCall = 0;
let timeoutId: ReturnType<typeof setTimeout> | null = null;
return (...args: Parameters<T>) => {
const now = Date.now();
const timeSinceLastCall = now - lastCall;
if (timeSinceLastCall >= interval) {
lastCall = now;
fn(...args);
} else if (!timeoutId) {
timeoutId = setTimeout(
() => {
lastCall = Date.now();
fn(...args);
timeoutId = null;
},
interval - timeSinceLastCall
);
}
};
}
/**
* Create a deferred promise that can be resolved/rejected externally
*/
export function deferred<T>(): {
promise: Promise<T>;
resolve: (value: T) => void;
reject: (error: unknown) => void;
} {
let resolve!: (value: T) => void;
let reject!: (error: unknown) => void;
const promise = new Promise<T>((res, rej) => {
resolve = res;
reject = rej;
});
return { promise, resolve, reject };
}

27
packages/core/tsconfig.json
View file

@ -1,27 +0,0 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"noUncheckedIndexedAccess": true,
"noImplicitOverride": true,
"noPropertyAccessFromIndexSignature": true,
"exactOptionalPropertyTypes": true,
"paths": {
"@/*": ["./src/*"]
}
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist", "tests"]
}

11
packages/core/tsup.config.ts
View file

@ -1,11 +0,0 @@
import { defineConfig } from 'tsup';
export default defineConfig({
entry: ['src/index.ts', 'src/cli.ts'],
format: ['esm'],
dts: true,
clean: true,
sourcemap: true,
target: 'node20',
shims: true,
});

2
packages/vestige-mcp-npm/package.json
View file

@ -1,6 +1,6 @@
{
"name": "vestige-mcp-server",
"version": "1.0.0",
"version": "1.1.2",
"description": "Vestige MCP Server - AI Memory System for Claude and other assistants",
"bin": {
"vestige-mcp": "bin/vestige-mcp.js",

4
packages/vestige-mcp-npm/scripts/postinstall.js
View file

@ -7,7 +7,7 @@ const os = require('os');
const { execSync } = require('child_process');
const VERSION = require('../package.json').version;
const BINARY_VERSION = '1.1.0'; // GitHub release version for binaries
const BINARY_VERSION = '1.1.2'; // GitHub release version for binaries
const PLATFORM = os.platform();
const ARCH = os.arch();
@ -109,7 +109,7 @@ function extract(archivePath, destDir) {
function makeExecutable(binDir) {
if (isWindows) return;
const binaries = ['vestige-mcp', 'vestige'];
const binaries = ['vestige-mcp', 'vestige', 'vestige-restore'];
for (const bin of binaries) {
const binPath = path.join(binDir, bin);
if (fs.existsSync(binPath)) {