Verified against current dev; the compat/behavior decisions (#7 api_key
semantics, #10 CLI flags, #11 doc-description default) are deferred.
Crashes:
- page_index(): snapshot args before importing IndexConfig — locals() was
capturing the imported class and IndexConfig(extra='forbid') made every call
raise ValidationError.
- process_none_page_numbers: pop('page', None) instead of del (a TOC item
without 'page' raised KeyError mid-pipeline).
- pipeline._run_async: guard only the loop detection, not the run, so a real
RuntimeError from the coroutine isn't masked as "asyncio.run() cannot be
called from a running event loop".
Silent-wrong / robustness:
- LocalBackend.get_document_structure and the agent get_document /
get_document_structure tools now surface a missing doc (raise / error-JSON)
instead of returning empty, matching get_page_content and the cloud backend.
- cloud delete_collection drops the cached folder_id.
- cloud query raises on an empty collection instead of POSTing doc_id:[].
- LocalClient skips the API-key check for keyless providers (ollama, lm_studio,
…) so keyless LiteLLM models aren't rejected at construction.
Compat / cleanup:
- md_to_tree coerces legacy 'yes'/'no' string flags (a bare 'no' was truthy).
- FileTypeError also subclasses ValueError (0.2.x raised ValueError).
- _validate_llm_provider no longer mutates global litellm.model_cost_map_url.
- __all__ re-includes legacy exports (page_index, md_to_tree, get_*).
- Rewrite examples/agentic_vectorless_rag_demo.py to the Collection API and use
the in-repo attention.pdf (the old workspace=/client.index/client.documents
API no longer exists).
Adds tests/test_review_fixes.py (10 regressions). Full suite: 189 passed.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Address PR #272 review:
- get_md_page_content / retrieve._get_md_page_content returned every node
whose line_num fell in [min(pages), max(pages)], so a non-contiguous spec
like "5,100" over-fetched everything in between. Match the exact requested
line numbers instead, mirroring the PDF path. (Same bug as #280.)
- Restore the CHATGPT_API_KEY -> OPENAI_API_KEY backward-compat alias dropped
when pageindex/utils.py became a re-export shim; users with only
CHATGPT_API_KEY set would otherwise fail auth after upgrading. It now runs
in __init__.py right after load_dotenv.
Adds regression tests for both.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The previous approach (bounded_gather building a fresh semaphore per call)
did NOT compose: the indexing call graph nests gathers
(tree_parser -> process_large_node_recursively -> recurse, plus each node's
check_title gather), and each level got its own independent cap. Peak
in-flight LLM calls grew ~N^depth, so a deep/wide document still exhausted
file descriptors (Errno 24) — the exact failure the cap was meant to
prevent — while the flat summary phase was over-serialized. Naively sharing
one semaphore across gather levels would instead deadlock (a parent holds a
slot while awaiting children that need slots).
Move the throttle to the single chokepoint every LLM call funnels through,
llm_acompletion: one shared semaphore per event loop, acquired only around
the litellm.acompletion network call. This gives a true global cap that
composes across any nesting and can't deadlock (a parent awaiting children
holds no slot). bounded_gather is gone; the call sites revert to plain
asyncio.gather.
Also:
- Reject bool in max_concurrency validation (bool is an int subclass, so
set_max_concurrency(True) / IndexConfig(max_concurrency=True) previously
became Semaphore(1) and silently serialized). Shared _validate_max_concurrency
+ a pydantic field_validator.
- Guard check_title_appearance_in_start_concurrent against an out-of-range
or 0 physical_index (LLM can emit one): it was dereferenced during task
construction, outside the gather's return_exceptions protection, aborting
the whole build; 0 silently wrapped to the last page. Now marked 'no'.
- Propagate contextvars into agent.py's worker-thread run (mirrors
pipeline._run_async) so ContextVar settings stay consistent.
- Tests rewritten to cover the nested case the old flat tests missed, the
leaf-level throttle in llm_acompletion, bool rejection, and the
out-of-range physical_index guard.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Cap concurrent in-flight LLM calls during indexing via a shared
semaphore (bounded_gather), so a many-node document no longer schedules
one socket per node and exhausts the process fd limit (Errno 24).
Make the per-index max_concurrency override correct under concurrency:
- Scope IndexConfig(max_concurrency=...) to the build_index call via a
ContextVar (max_concurrency_scope) instead of mutating a process
global. A one-off value no longer sticks as the new default, and
concurrent indexing of other documents isn't affected.
- Propagate the context through _run_async's worker-thread fallback so
the override survives the sync-over-async thread hop.
- set_max_concurrency() stays as the explicit process-wide setter.
Also stop `from .utils import *` leaking a `config` name (SimpleNamespace
alias) that shadowed the real pageindex.config submodule for the
page_index modules; the alias is now `_config`.
Adds regression tests for cap enforcement, scope stickiness/isolation,
worker-thread propagation, and the config-namespace fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
main advanced (litellm 1.84.0 #342, #188 TOC fixes, #281, README) while
dev turned pageindex/page_index.py and utils.py into deprecation shims
over pageindex/index/*. Both sides touched those two files, hence the
conflict.
Resolution:
- Keep dev's shims for the two top-level modules (the real implementation
lives in pageindex/index/*). requirements.txt auto-merged to
litellm==1.84.0.
- #188 ("prevent KeyError crash and context exhaustion in TOC
processing") landed on main's top-level page_index.py, which is now a
shim on dev — so its fixes were NOT in dev's index/page_index.py.
Ported them into pageindex/index/page_index.py (preserving dev's
IndexConfig/bool integration): .get() on the TOC check functions +
detect_page_index, incremental-chat_history retry loops in
extract_toc_content and toc_transformer, truncation moved before the
loop, .get('table_of_contents', []) and the single_toc_item_index_fixer
None guard.
- Repoint #188's merged test (tests/test_issue_163.py) at
pageindex.index.page_index so its patches hit the module where the code
now lives (they were silently hitting the shim → real LLM calls).
Full suite: 158 passed, 2 skipped.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Engineering-quality cleanups from the SDK review (no behavior change):
- Return-type discoverability: add pageindex/types.py with TypedDicts
(DocumentInfo, DocumentDetail, PageContent) and annotate Collection /
Backend methods with them; add docstrings to every public Collection
method (including the get_page_content `pages` spec). Exported from the
package. Zero runtime cost — these are plain dicts.
- Backend protocol as a real contract:
* query_stream is an async generator, so the protocol now declares it
as `def ... -> AsyncIterator[QueryEvent]` (not `async def`, which
typed it as a coroutine and never matched the implementations).
* custom-parser support is expressed as a runtime_checkable
SupportsParserRegistration capability protocol; the client uses
isinstance(...) instead of hasattr(...) duck-typing.
- Parser layering: move count_tokens into a leaf module pageindex/tokens.py
so parser/* imports it from there instead of reaching back into
pageindex.index (a reverse dependency). index.utils re-exports it for
backward compatibility.
Adds tests/test_architecture.py enforcing: parser never imports index,
count_tokens is a single shared leaf, the capability protocol works,
both backends satisfy Backend, and the TypedDicts are exported.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- legacy call_llm: open AsyncOpenAI via `async with` so the client (and
its HTTP connection pool) is closed instead of leaked.
- LocalBackend.add_document: fail fast with CollectionNotFoundError when
the collection doesn't exist, before the expensive parse + LLM index
(previously the missing FK only tripped at save time, after paying for
the LLM work). Also raise builtin FileNotFoundError for a missing path
instead of FileTypeError (which now means only "unsupported extension").
- Collection.query(doc_ids=None): the empty-collection guard now always
runs — previously it was skipped once PAGEINDEX_EXPERIMENTAL_MULTIDOC
was set. A single list_documents call serves both the guard and the
multi-doc warning (no separate call just to decide whether to warn).
- CloudBackend.query_stream: on early consumer break / GeneratorExit,
signal the background SSE thread to stop and force-close the response,
so it no longer drains the whole stream in the background.
Adds regression tests for each (client closed, fail-fast on unknown
collection, FileNotFoundError, empty-check under the multidoc env flag,
single list call, early-break thread stop).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The new SDK copied the legacy indexing pipeline into pageindex/index/
instead of moving it, leaving two divergent copies of page_index.py /
page_index_md.py / utils.py. They had already drifted (the legacy copy
still compared IndexConfig booleans against 'yes' — a separate fix),
and every pipeline change had to be applied twice.
Make pageindex/index/ the single source of truth (same pattern as the
LegacyCloudAPI shim for the 0.2.x cloud SDK):
- pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes
(get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers,
...) so it's the sole utils module. Reconciled the diverged funcs:
kept the modern versions, backported the #331 get_leaf_nodes .get()
fix, and restored remove_fields' max_len parameter (superset).
- index/page_index*.py now import `from .utils import *`;
index/legacy_utils.py (a re-export of the old top-level utils) deleted.
- Top-level page_index.py / page_index_md.py / utils.py become thin
re-export shims that emit PendingDeprecationWarning. The md_to_tree
shim coerces legacy 'yes'/'no' string flags to bool (the canonical
version is boolean-typed).
- ConfigLoader no longer reads the deleted config.yaml; it builds
defaults from IndexConfig (was an unconditional FileNotFoundError).
- __init__.py and retrieve.py import from pageindex.index.* directly so
`import pageindex` does not trip the shims.
Adds tests/test_legacy_shims.py pinning the contract: clean top-level
import doesn't warn, legacy submodule imports warn, symbols still
resolve, shim and canonical share one implementation, the #331 fix and
ConfigLoader-without-yaml both hold, and the md_to_tree coercion works.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
llm_completion / llm_acompletion (pageindex/utils.py and
pageindex/index/utils.py) set `litellm.drop_params = True` on the litellm
module. litellm is a process-wide singleton, so this leaked into every other
library sharing it (e.g. a host app like OpenKB that exposes its own litellm
config) and could not be turned off.
Replace the hardcoded `temperature=0` + global `drop_params` with a single
PageIndex-owned per-call kwargs mechanism (config._LLM_PARAMS):
- defaults preserve behavior: {"temperature": 0, "drop_params": True};
- passed per call via **get_llm_params(), never writing litellm's globals, so
nothing leaks into other litellm users in the same process;
- externally configurable: pageindex.set_llm_params(drop_params=False,
temperature=1, num_retries=5, ...) or the PAGEINDEX_DROP_PARAMS env shortcut;
- model/messages are reserved (PageIndex supplies them) and rejected.
