Verified against current dev; the compat/behavior decisions (#7 api_key
semantics, #10 CLI flags, #11 doc-description default) are deferred.
Crashes:
- page_index(): snapshot args before importing IndexConfig — locals() was
capturing the imported class and IndexConfig(extra='forbid') made every call
raise ValidationError.
- process_none_page_numbers: pop('page', None) instead of del (a TOC item
without 'page' raised KeyError mid-pipeline).
- pipeline._run_async: guard only the loop detection, not the run, so a real
RuntimeError from the coroutine isn't masked as "asyncio.run() cannot be
called from a running event loop".
Silent-wrong / robustness:
- LocalBackend.get_document_structure and the agent get_document /
get_document_structure tools now surface a missing doc (raise / error-JSON)
instead of returning empty, matching get_page_content and the cloud backend.
- cloud delete_collection drops the cached folder_id.
- cloud query raises on an empty collection instead of POSTing doc_id:[].
- LocalClient skips the API-key check for keyless providers (ollama, lm_studio,
…) so keyless LiteLLM models aren't rejected at construction.
Compat / cleanup:
- md_to_tree coerces legacy 'yes'/'no' string flags (a bare 'no' was truthy).
- FileTypeError also subclasses ValueError (0.2.x raised ValueError).
- _validate_llm_provider no longer mutates global litellm.model_cost_map_url.
- __all__ re-includes legacy exports (page_index, md_to_tree, get_*).
- Rewrite examples/agentic_vectorless_rag_demo.py to the Collection API and use
the in-repo attention.pdf (the old workspace=/client.index/client.documents
API no longer exists).
Adds tests/test_review_fixes.py (10 regressions). Full suite: 189 passed.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The previous approach (bounded_gather building a fresh semaphore per call)
did NOT compose: the indexing call graph nests gathers
(tree_parser -> process_large_node_recursively -> recurse, plus each node's
check_title gather), and each level got its own independent cap. Peak
in-flight LLM calls grew ~N^depth, so a deep/wide document still exhausted
file descriptors (Errno 24) — the exact failure the cap was meant to
prevent — while the flat summary phase was over-serialized. Naively sharing
one semaphore across gather levels would instead deadlock (a parent holds a
slot while awaiting children that need slots).
Move the throttle to the single chokepoint every LLM call funnels through,
llm_acompletion: one shared semaphore per event loop, acquired only around
the litellm.acompletion network call. This gives a true global cap that
composes across any nesting and can't deadlock (a parent awaiting children
holds no slot). bounded_gather is gone; the call sites revert to plain
asyncio.gather.
Also:
- Reject bool in max_concurrency validation (bool is an int subclass, so
set_max_concurrency(True) / IndexConfig(max_concurrency=True) previously
became Semaphore(1) and silently serialized). Shared _validate_max_concurrency
+ a pydantic field_validator.
- Guard check_title_appearance_in_start_concurrent against an out-of-range
or 0 physical_index (LLM can emit one): it was dereferenced during task
construction, outside the gather's return_exceptions protection, aborting
the whole build; 0 silently wrapped to the last page. Now marked 'no'.
- Propagate contextvars into agent.py's worker-thread run (mirrors
pipeline._run_async) so ContextVar settings stay consistent.
- Tests rewritten to cover the nested case the old flat tests missed, the
leaf-level throttle in llm_acompletion, bool rejection, and the
out-of-range physical_index guard.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Cap concurrent in-flight LLM calls during indexing via a shared
semaphore (bounded_gather), so a many-node document no longer schedules
one socket per node and exhausts the process fd limit (Errno 24).
Make the per-index max_concurrency override correct under concurrency:
- Scope IndexConfig(max_concurrency=...) to the build_index call via a
ContextVar (max_concurrency_scope) instead of mutating a process
global. A one-off value no longer sticks as the new default, and
concurrent indexing of other documents isn't affected.
- Propagate the context through _run_async's worker-thread fallback so
the override survives the sync-over-async thread hop.
- set_max_concurrency() stays as the explicit process-wide setter.
Also stop `from .utils import *` leaking a `config` name (SimpleNamespace
alias) that shadowed the real pageindex.config submodule for the
page_index modules; the alias is now `_config`.
Adds regression tests for cap enforcement, scope stickiness/isolation,
worker-thread propagation, and the config-namespace fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
main advanced (litellm 1.84.0 #342, #188 TOC fixes, #281, README) while
dev turned pageindex/page_index.py and utils.py into deprecation shims
over pageindex/index/*. Both sides touched those two files, hence the
conflict.
Resolution:
- Keep dev's shims for the two top-level modules (the real implementation
lives in pageindex/index/*). requirements.txt auto-merged to
litellm==1.84.0.
- #188 ("prevent KeyError crash and context exhaustion in TOC
processing") landed on main's top-level page_index.py, which is now a
shim on dev — so its fixes were NOT in dev's index/page_index.py.
Ported them into pageindex/index/page_index.py (preserving dev's
IndexConfig/bool integration): .get() on the TOC check functions +
detect_page_index, incremental-chat_history retry loops in
extract_toc_content and toc_transformer, truncation moved before the
loop, .get('table_of_contents', []) and the single_toc_item_index_fixer
None guard.
- Repoint #188's merged test (tests/test_issue_163.py) at
pageindex.index.page_index so its patches hit the module where the code
now lives (they were silently hitting the shim → real LLM calls).
Full suite: 158 passed, 2 skipped.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The new SDK copied the legacy indexing pipeline into pageindex/index/
instead of moving it, leaving two divergent copies of page_index.py /
page_index_md.py / utils.py. They had already drifted (the legacy copy
still compared IndexConfig booleans against 'yes' — a separate fix),
and every pipeline change had to be applied twice.
Make pageindex/index/ the single source of truth (same pattern as the
LegacyCloudAPI shim for the 0.2.x cloud SDK):
- pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes
(get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers,
...) so it's the sole utils module. Reconciled the diverged funcs:
kept the modern versions, backported the #331 get_leaf_nodes .get()
fix, and restored remove_fields' max_len parameter (superset).
- index/page_index*.py now import `from .utils import *`;
index/legacy_utils.py (a re-export of the old top-level utils) deleted.
- Top-level page_index.py / page_index_md.py / utils.py become thin
re-export shims that emit PendingDeprecationWarning. The md_to_tree
shim coerces legacy 'yes'/'no' string flags to bool (the canonical
version is boolean-typed).
- ConfigLoader no longer reads the deleted config.yaml; it builds
defaults from IndexConfig (was an unconditional FileNotFoundError).
- __init__.py and retrieve.py import from pageindex.index.* directly so
`import pageindex` does not trip the shims.
Adds tests/test_legacy_shims.py pinning the contract: clean top-level
import doesn't warn, legacy submodule imports warn, symbols still
resolve, shim and canonical share one implementation, the #331 fix and
ConfigLoader-without-yaml both hold, and the md_to_tree coercion works.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
