Real concurrency e2e (8 threads adding the same file) surfaced a bug the
mocked unit tests missed: concurrent add_document calls failed with
sqlite3.OperationalError "database is locked". Root cause — under WAL the
dedup SELECT (find_document_by_hash) left a read snapshot on the
connection, and the subsequent INSERT on that stale snapshot raised
SQLITE_BUSY_SNAPSHOT, which busy_timeout does not retry.
Fixes:
- open connections in autocommit (isolation_level=None) so a SELECT never
leaves a lingering read snapshot and each write is its own transaction
- PRAGMA busy_timeout=10000 so concurrent writers wait for the WAL
single-writer lock instead of failing immediately
- an instance-level write lock serializing the fast write methods within
the process (the expensive LLM indexing stays parallel)
Now 8 concurrent adds of one file -> a single doc_id, zero errors.
Adds a real-thread regression test.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The dedup refactor dropped the explicit load_dotenv() that the old
top-level utils.py ran on import. Since then, .env was only loaded as a
side effect of importing litellm — which would silently break both
local mode (needs OPENAI_API_KEY in the environment) and cloud usage
(callers read PAGEINDEX_API_KEY via os.environ) if litellm changed that
behavior or its import were made lazy. Restore an explicit load_dotenv()
at the top of pageindex/__init__.py so PageIndex owns .env loading.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
main advanced (litellm 1.84.0 #342, #188 TOC fixes, #281, README) while
dev turned pageindex/page_index.py and utils.py into deprecation shims
over pageindex/index/*. Both sides touched those two files, hence the
conflict.
Resolution:
- Keep dev's shims for the two top-level modules (the real implementation
lives in pageindex/index/*). requirements.txt auto-merged to
litellm==1.84.0.
- #188 ("prevent KeyError crash and context exhaustion in TOC
processing") landed on main's top-level page_index.py, which is now a
shim on dev — so its fixes were NOT in dev's index/page_index.py.
Ported them into pageindex/index/page_index.py (preserving dev's
IndexConfig/bool integration): .get() on the TOC check functions +
detect_page_index, incremental-chat_history retry loops in
extract_toc_content and toc_transformer, truncation moved before the
loop, .get('table_of_contents', []) and the single_toc_item_index_fixer
None guard.
- Repoint #188's merged test (tests/test_issue_163.py) at
pageindex.index.page_index so its patches hit the module where the code
now lives (they were silently hitting the shim → real LLM calls).
Full suite: 158 passed, 2 skipped.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Replace the bare `except Exception: pass` around resp.close() in the
query_stream finally block with an explanatory comment and a debug log
(flagged by github-code-quality on PR #272). Behavior unchanged — the
close is best-effort to unblock the background thread.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
For faithful 0.2.x cloud SDK drop-in compatibility, is_retrieval_ready
again swallows PageIndexAPIError and returns False (instead of raising),
so existing `while not is_retrieval_ready(...)` polling loops behave
exactly as before. Documented that this can loop forever on a permanent
error — that is the legacy contract; callers guard their own loops.
(The new SDK's own indexing path doesn't use this method — it polls
document status with a bounded 120-attempt cap — so the infinite-loop
risk is confined to legacy-SDK usage that already had it.)
Test updated to assert the swallow behavior.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Engineering-quality cleanups from the SDK review (no behavior change):
- Return-type discoverability: add pageindex/types.py with TypedDicts
(DocumentInfo, DocumentDetail, PageContent) and annotate Collection /
Backend methods with them; add docstrings to every public Collection
method (including the get_page_content `pages` spec). Exported from the
package. Zero runtime cost — these are plain dicts.
- Backend protocol as a real contract:
* query_stream is an async generator, so the protocol now declares it
as `def ... -> AsyncIterator[QueryEvent]` (not `async def`, which
typed it as a coroutine and never matched the implementations).
* custom-parser support is expressed as a runtime_checkable
SupportsParserRegistration capability protocol; the client uses
isinstance(...) instead of hasattr(...) duck-typing.
- Parser layering: move count_tokens into a leaf module pageindex/tokens.py
so parser/* imports it from there instead of reaching back into
pageindex.index (a reverse dependency). index.utils re-exports it for
backward compatibility.
Adds tests/test_architecture.py enforcing: parser never imports index,
count_tokens is a single shared leaf, the capability protocol works,
both backends satisfy Backend, and the TypedDicts are exported.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- SQLite dedup race: add UNIQUE(collection_name, file_hash) and switch
save_document to a plain INSERT. add_document now catches the
IntegrityError from a concurrent add of the same content, cleans up its
managed files, and returns the winning doc_id — instead of two doc_ids
for one file (each having paid for its own LLM indexing).
- PDF image paths: store the absolute path to each extracted image
instead of a path relative to the indexing process's cwd. The
 references broke as soon as a query ran from a different
directory.
- LegacyCloudAPI: URL-encode doc_id / retrieval_id path segments (added
_enc()), matching CloudBackend. An id containing '/', '?', '#' or a
space previously hit the wrong endpoint or produced a malformed URL.
Adds regression tests: UNIQUE enforcement, the add-race resolving to the
winner, absolute image paths, and encoded legacy URLs.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- legacy call_llm: open AsyncOpenAI via `async with` so the client (and
its HTTP connection pool) is closed instead of leaked.
- LocalBackend.add_document: fail fast with CollectionNotFoundError when
the collection doesn't exist, before the expensive parse + LLM index
(previously the missing FK only tripped at save time, after paying for
the LLM work). Also raise builtin FileNotFoundError for a missing path
instead of FileTypeError (which now means only "unsupported extension").
- Collection.query(doc_ids=None): the empty-collection guard now always
runs — previously it was skipped once PAGEINDEX_EXPERIMENTAL_MULTIDOC
was set. A single list_documents call serves both the guard and the
multi-doc warning (no separate call just to decide whether to warn).
- CloudBackend.query_stream: on early consumer break / GeneratorExit,
signal the background SSE thread to stop and force-close the response,
so it no longer drains the whole stream in the background.
Adds regression tests for each (client closed, fail-fast on unknown
collection, FileNotFoundError, empty-check under the multidoc env flag,
single list call, early-break thread stop).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The new SDK copied the legacy indexing pipeline into pageindex/index/
instead of moving it, leaving two divergent copies of page_index.py /
page_index_md.py / utils.py. They had already drifted (the legacy copy
still compared IndexConfig booleans against 'yes' — a separate fix),
and every pipeline change had to be applied twice.
Make pageindex/index/ the single source of truth (same pattern as the
LegacyCloudAPI shim for the 0.2.x cloud SDK):
- pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes
(get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers,
...) so it's the sole utils module. Reconciled the diverged funcs:
kept the modern versions, backported the #331 get_leaf_nodes .get()
fix, and restored remove_fields' max_len parameter (superset).
- index/page_index*.py now import `from .utils import *`;
index/legacy_utils.py (a re-export of the old top-level utils) deleted.
- Top-level page_index.py / page_index_md.py / utils.py become thin
re-export shims that emit PendingDeprecationWarning. The md_to_tree
shim coerces legacy 'yes'/'no' string flags to bool (the canonical
version is boolean-typed).
- ConfigLoader no longer reads the deleted config.yaml; it builds
defaults from IndexConfig (was an unconditional FileNotFoundError).
- __init__.py and retrieve.py import from pageindex.index.* directly so
`import pageindex` does not trip the shims.
Adds tests/test_legacy_shims.py pinning the contract: clean top-level
import doesn't warn, legacy submodule imports warn, symbols still
resolve, shim and canonical share one implementation, the #331 fix and
ConfigLoader-without-yaml both hold, and the md_to_tree coercion works.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- AgentRunner.run: offload to a worker-thread event loop when called
from inside a running loop (Jupyter, FastAPI handlers) — mirrors
pipeline._run_async; Runner.run_sync raised RuntimeError there.
- SQLiteStorage: create connections with check_same_thread=False so
close() can actually close connections created by worker threads.
Each thread still gets its own connection via threading.local; with
the default True those closes raised ProgrammingError (silently
swallowed) and leaked every worker connection.
- CloudBackend.query: non-streaming chat completions now use a 300s
timeout and a single attempt. The default 30s ReadTimeout fired
before generation finished and the retry loop re-billed the full
server-side retrieval + generation up to three times. _request gains
retries/timeout overrides; the exhausted-retry path also no longer
sleeps before raising.
- MarkdownParser: content before the first heading (abstract/preamble)
becomes a node instead of being silently dropped and unretrievable;
a file with no headings at all yields a single document node instead
of zero nodes (which pushed an empty page list into the pipeline).
- LegacyCloudAPI.is_retrieval_ready: API failures (revoked key, network
down) now propagate as PageIndexAPIError instead of reading as
"not ready", which turned polling loops into infinite loops.
Adds regression tests for each fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- local delete_collection: validate the collection name before rmtree.
An unvalidated name like "../.." escaped files_dir and deleted
arbitrary directories (path traversal).
- legacy page_index(): restore the node_id/summary/text/description
enhancements. IndexConfig now carries booleans (pydantic coerces the
legacy 'yes'/'no' strings at the boundary), but page_index_main still
compared `opt.if_add_node_id == 'yes'` — always False — so every
enhancement was silently skipped for legacy-API callers. Conditions
now branch on the booleans, matching pageindex/index/page_index.py.
- LegacyCloudAPI._request: bound every request with a timeout
(30s, 120s read timeout for streamed responses) so a dead connection
can't hang legacy submit/poll/chat callers forever. The legacy
contract tests pinned the missing timeout; updated to pin its
presence instead.
Adds regression tests: path-traversal rejection, 'yes'/'no' -> bool
coercion, and timeout assertions.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Fixes the cloud/local contract mismatches from the PR #272 review
(verified against the official API docs — the cloud API has no
folder/collection endpoints publicly, GET /docs supports limit<=100
with offset):
- query_stream: emit a terminal answer_done event with the full answer
(same contract as the local backend); raise CloudAPIError instead of
disguising HTTP errors as answer events; move the initial connect
inside try so a connection failure can no longer strand the consumer
awaiting a sentinel that never arrives
- _request: rewind file objects before retrying so a transient 5xx/429
during upload no longer re-sends an empty multipart body; carry the
HTTP status on CloudAPIError (status_code) and keep the last status
in the max-retries error
- list_documents: paginate with limit/offset instead of a hard-coded
limit=100, so >100-doc collections are no longer silently truncated
(whole-collection queries rely on this list)
- folders: treat only 403/404 as "folders unavailable" (warned via
warnings.warn instead of an invisible logger.warning, matched on
status_code instead of a "403" substring); transient errors now
propagate instead of being permanently cached as folder_id=None
- error taxonomy parity: cloud doc endpoints map HTTP 404 to
DocumentNotFoundError; local get_document raises DocumentNotFoundError
instead of returning {}; local delete_document raises on missing
doc_id instead of silently deleting nothing
- cloud get_document warns that include_text is not supported instead
of silently ignoring it
Adds regression tests for each fix (11 new tests).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
* fix: prevent KeyError crash and context exhaustion in TOC processing
- Use .get() with safe defaults for all LLM response dict accesses
- Optimize extract_toc_content retry loop to grow chat_history
incrementally instead of rebuilding with full accumulated response
- Optimize toc_transformer retry loop to use chat_history instead of
re-embedding the entire raw TOC and incomplete JSON in each prompt
- Return best-effort results on max retries instead of raising
- Add 14 mock-based tests covering all fix scenarios
Closes#163
* fix: address review feedback on retry behavior and None guard
- Restore explicit Exception on max retries instead of silent warning
- Move truncation logic before the retry loop so it only runs once
on the initial incomplete response, not on every iteration
- Add explicit None guard for physical_index before passing to
convert_physical_index_to_int to prevent potential TypeError
- Update test to expect Exception on max retries
---------
Co-authored-by: Your Name <you@example.com>
litellm 1.83.7 hard-pins python-dotenv==1.0.1, which conflicts with
python-dotenv==1.2.2 in requirements.txt and makes a fresh install fail
(ResolutionImpossible under both pip 25.2 and uv). Downgrading dotenv to
1.0.1 is not an option: python-dotenv < 1.2.2 is affected by
GHSA-mf9w-mj56-hr94 (moderate).
litellm 1.84.0 relaxed its pin to python-dotenv<2.0,>=1.0.0, allowing the
patched python-dotenv==1.2.2 to remain. Full requirements.txt resolves
cleanly under both pip and uv.
Closes#286
list_to_tree() deletes the 'nodes' key from leaf nodes entirely via
clean_node(). Direct access via structure['nodes'] raises KeyError on
these nodes. Using structure.get('nodes') returns None (falsy) safely,
consistent with how 'nodes' is accessed elsewhere in the codebase.
Fixes#330
llm_completion / llm_acompletion (pageindex/utils.py and
pageindex/index/utils.py) set `litellm.drop_params = True` on the litellm
module. litellm is a process-wide singleton, so this leaked into every other
library sharing it (e.g. a host app like OpenKB that exposes its own litellm
config) and could not be turned off.
Replace the hardcoded `temperature=0` + global `drop_params` with a single
PageIndex-owned per-call kwargs mechanism (config._LLM_PARAMS):
- defaults preserve behavior: {"temperature": 0, "drop_params": True};
- passed per call via **get_llm_params(), never writing litellm's globals, so
nothing leaks into other litellm users in the same process;
- externally configurable: pageindex.set_llm_params(drop_params=False,
temperature=1, num_retries=5, ...) or the PAGEINDEX_DROP_PARAMS env shortcut;
- model/messages are reserved (PageIndex supplies them) and rejected.
- Add Website and Book a Demo buttons; reorder and recolor to brand palette
- Replace removed/invalid shields logos (LinkedIn, envelope) and the Website
icon with inline white SVGs so all badges show an icon
- Reword header tagline (No Vector DB, No Chunking; Context-Aware Retrieval; Human-like)
- Rename top-nav Homepage -> Website
scripts/e2e_legacy_sdk.py becomes examples/demo_legacy_sdk.py to sit
alongside the other runnable demos (local/cloud/query-modes), and the
README's Runnable examples list now points at it. Docstring command
updated to the new path; the legacy script docstring also calls out
that it exercises the 0.2.x compatibility methods.
The scripts/ directory had no other entries and is removed.
- Collection.query and Backend.query/query_stream accept doc_ids as
str, list[str] or None. Single str is normalized to [str] inside each
backend; bare [] is rejected with ValueError at both layers.
- wrap_with_doc_context wraps the scoped doc list in <docs>...</docs>
and SCOPED_SYSTEM_PROMPT instructs the agent to treat that block as
data, not instructions (defense against prompt injection via
auto-generated doc_description).
- _require_cloud_api now distinguishes api_key="" from api_key=None;
the former gives a targeted error pointing at the empty-string vs
fall-back-to-local situation when legacy SDK methods are called.
- Legacy PageIndexClient.list_documents docstring spells out the
return-shape difference vs collection.list_documents() to flag a
silent migration footgun (paginated dict with id/name keys vs plain
list[dict] with doc_id/doc_name keys).
- Remove dead CloudBackend.get_agent_tools stub (not on the Backend
protocol; only ever returned an empty AgentTools()) and the
SYSTEM_PROMPT alias (OPEN_/SCOPED_SYSTEM_PROMPT are the explicit
names now).
- README quick start and streaming example now pass doc_ids; new
multi-document section shows both str and list forms.
- examples/demo_query_modes.py exercises all five query-mode cases
(single-doc, multi-doc with/without env var, scoped single, scoped
multi) for manual verification.
- get_agent_tools branches on doc_ids:
- scoped (doc_ids=[...]): drops list_documents and hard-enforces a
whitelist on the remaining tools; system prompt switches to
SCOPED_SYSTEM_PROMPT (no list_documents instruction); doc list +
summaries are prepended to the user message via wrap_with_doc_context.
- open (doc_ids=None): unchanged 4-tool agent loop.
- list_documents now exposes doc_description (sqlite + cloud).
- Collection.query emits UserWarning when doc_ids is None and the
collection holds >1 documents; PAGEINDEX_EXPERIMENTAL_MULTIDOC=1
silences it. Single-doc collections skip the warning; empty
collections raise ValueError.
- Agents SDK tracing upload disabled by default (avoids SSL timeouts);
PAGEINDEX_AGENTS_TRACING=1 re-enables it.
- README: new SDK Usage section covering local/cloud quick start,
streaming, multi-doc as experimental, and runnable examples.
pageindex/config.py imports `from pydantic import BaseModel` in production
code, but pyproject.toml only pulled pydantic in transitively via litellm.
A future litellm release could drop or re-pin pydantic and break installs.
Pin to `>=2.5.0,<3.0.0` to match the v2-style BaseModel usage already in the
codebase, and to stay compatible with litellm's own pydantic constraint.
* feat:compatible with Pageindex SDK
* corner cases fixed
* fix: mock behavior of old SDK
* fix: close streaming response and warn on empty api_key
- LegacyCloudAPI: close response in `finally` for both _stream_chat_response
variants so abandoned iterators no longer leak the TCP connection.
- PageIndexClient: emit a warning instead of silently falling back to local
when api_key is the empty string, surfacing typical env-var-unset misconfig.
- FakeResponse: add close()/closed to match the real requests.Response API.
- Add unit coverage for stream close (both paths) and the empty-api_key warning.
- Add scripts/e2e_legacy_sdk.py to smoke-test the legacy SDK contract end-to-end
against api.pageindex.ai.
* chore: mark legacy SDK methods with @deprecated and docstring pointers
- Decorate the 12 PageIndexClient cloud-SDK compat methods with
@typing_extensions.deprecated(..., category=PendingDeprecationWarning):
- IDE/type-checkers render them with a strikethrough hint
- runtime warnings stay silent by default (no spam for existing callers),
surfaceable via `python -W default::PendingDeprecationWarning`
- Add a one-line docstring on each pointing to the Collection-based equivalent.
- Promote typing-extensions to a direct dependency (was transitive via litellm).
---------
Co-authored-by: XinyanZhou <xinyanzhou@XinyanZhoudeMacBook-Pro.local>
Co-authored-by: saccharin98 <xinyanzhou938@gmail.com>
Co-authored-by: mountain <kose2livs@gmail.com>
Weekly scan to open PRs upgrading third-party actions used in CI.
pip dependencies remain pinned and are covered by Dependabot security
updates separately.
The cloud backend previously polled tree_resp["retrieval_ready"]
as the ready signal. Empirically this flag is not a reliable
indicator — docs can reach status=="completed" without
retrieval_ready flipping, causing col.add() to wait until the 10
min timeout before giving up on otherwise-successful uploads.
The cloud API's canonical ready signal is status=="completed";
switch the poll to check that instead.
