Real concurrency e2e (8 threads adding the same file) surfaced a bug the
mocked unit tests missed: concurrent add_document calls failed with
sqlite3.OperationalError "database is locked". Root cause — under WAL the
dedup SELECT (find_document_by_hash) left a read snapshot on the
connection, and the subsequent INSERT on that stale snapshot raised
SQLITE_BUSY_SNAPSHOT, which busy_timeout does not retry.
Fixes:
- open connections in autocommit (isolation_level=None) so a SELECT never
leaves a lingering read snapshot and each write is its own transaction
- PRAGMA busy_timeout=10000 so concurrent writers wait for the WAL
single-writer lock instead of failing immediately
- an instance-level write lock serializing the fast write methods within
the process (the expensive LLM indexing stays parallel)
Now 8 concurrent adds of one file -> a single doc_id, zero errors.
Adds a real-thread regression test.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- SQLite dedup race: add UNIQUE(collection_name, file_hash) and switch
save_document to a plain INSERT. add_document now catches the
IntegrityError from a concurrent add of the same content, cleans up its
managed files, and returns the winning doc_id — instead of two doc_ids
for one file (each having paid for its own LLM indexing).
- PDF image paths: store the absolute path to each extracted image
instead of a path relative to the indexing process's cwd. The
 references broke as soon as a query ran from a different
directory.
- LegacyCloudAPI: URL-encode doc_id / retrieval_id path segments (added
_enc()), matching CloudBackend. An id containing '/', '?', '#' or a
space previously hit the wrong endpoint or produced a malformed URL.
Adds regression tests: UNIQUE enforcement, the add-race resolving to the
winner, absolute image paths, and encoded legacy URLs.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- AgentRunner.run: offload to a worker-thread event loop when called
from inside a running loop (Jupyter, FastAPI handlers) — mirrors
pipeline._run_async; Runner.run_sync raised RuntimeError there.
- SQLiteStorage: create connections with check_same_thread=False so
close() can actually close connections created by worker threads.
Each thread still gets its own connection via threading.local; with
the default True those closes raised ProgrammingError (silently
swallowed) and leaked every worker connection.
- CloudBackend.query: non-streaming chat completions now use a 300s
timeout and a single attempt. The default 30s ReadTimeout fired
before generation finished and the retry loop re-billed the full
server-side retrieval + generation up to three times. _request gains
retries/timeout overrides; the exhausted-retry path also no longer
sleeps before raising.
- MarkdownParser: content before the first heading (abstract/preamble)
becomes a node instead of being silently dropped and unretrievable;
a file with no headings at all yields a single document node instead
of zero nodes (which pushed an empty page list into the pipeline).
- LegacyCloudAPI.is_retrieval_ready: API failures (revoked key, network
down) now propagate as PageIndexAPIError instead of reading as
"not ready", which turned polling loops into infinite loops.
Adds regression tests for each fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- get_agent_tools branches on doc_ids:
- scoped (doc_ids=[...]): drops list_documents and hard-enforces a
whitelist on the remaining tools; system prompt switches to
SCOPED_SYSTEM_PROMPT (no list_documents instruction); doc list +
summaries are prepended to the user message via wrap_with_doc_context.
- open (doc_ids=None): unchanged 4-tool agent loop.
- list_documents now exposes doc_description (sqlite + cloud).
- Collection.query emits UserWarning when doc_ids is None and the
collection holds >1 documents; PAGEINDEX_EXPERIMENTAL_MULTIDOC=1
silences it. Single-doc collections skip the warning; empty
collections raise ValueError.
- Agents SDK tracing upload disabled by default (avoids SSL timeouts);
PAGEINDEX_AGENTS_TRACING=1 re-enables it.
- README: new SDK Usage section covering local/cloud quick start,
streaming, multi-doc as experimental, and runnable examples.
