llm_completion/llm_acompletion raised RuntimeError once retries were
exhausted, which propagated through the unprotected sync TOC-detection
chain (find_toc_pages -> toc_transformer -> toc_extractor -> ...) and
aborted the whole document index — a regression vs. the pre-SDK behavior
that returned "" and let callers degrade (extract_json('') -> {} ->
.get(default), falling back to no-TOC indexing).
Restore the empty-result contract ("" / ("", "error") with
return_finish_reason), now logged at WARNING so the failure is visible
rather than silent. The async gather sites keep return_exceptions=True to
absorb any non-LLM error. A single persistently-failing call no longer
blocks indexing the rest of the document.
The sync-concurrency fix marked the scoped-override semaphore for release
before it was actually acquired, so a coroutine cancelled while polling for
a scoped permit (or a sync acquire interrupted mid-wait) ran the finally and
released a permit it never held — inflating the scoped cap for later calls
(the mirror of the ceiling-leak fix). Only bind the release guard after the
acquire succeeds, in both the async and sync semaphores. Regression test
included: cancelling a waiter leaves the scoped permit count at 1, not 2.
A stalled or half-open connection (e.g. a flaky local proxy that keeps
a socket ESTABLISHED but never sends data) could hang indexing forever
with no error — litellm/httpx had no timeout applied. Add a default
per-request timeout (120s, tunable via PAGEINDEX_LLM_TIMEOUT or
set_llm_params(timeout=...)) so a hung call fails fast with a clear
litellm Timeout, which the existing retry loops surface. Rides
get_llm_params(), so it flows through both llm_completion and
llm_acompletion automatically and is overridable per-index via
llm_params_scope / IndexConfig(llm_params=...).
asyncio.to_thread(ceiling_sem.acquire) blocks a worker thread that
can't be interrupted. If the awaiting coroutine is cancelled (Ctrl-C,
an outer timeout) while that thread is still parked inside acquire(),
the thread can go on to actually acquire the permit after the
coroutine has already unwound — leaking it forever, since the
matching finally: release() never runs for that attempt. Poll with
the non-blocking acquire(False) form instead, which returns instantly
and closes the leak window entirely.
Addresses items 4-8 from the max-effort review of PR #272 (VectifyAI/PageIndex#272).
- agent.py: wrap_with_doc_context() strips '<'/'>' from doc_name/doc_description
(untrusted: unsanitized filename / LLM-generated from document content) before
inserting them into the <docs>...</docs> block, so embedded content can never
form a literal </docs> that closes the delimiter early and escapes the
untrusted-data boundary SCOPED_SYSTEM_PROMPT relies on. Deterministic
per-field transform, doesn't touch the (cacheable) static system prompt.
- ConfigLoader.load() (legacy 0.2.x compat) now routes merged overrides through
IndexConfig before returning, so a legacy 'no' string gets pydantic's bool
coercion instead of surviving as a truthy non-empty string — page_index_main's
bare `if opt.if_add_node_summary:` checks (changed from `== 'yes'` elsewhere
in this PR) were silently inverting caller intent and firing unwanted billed
LLM calls.
- verify_toc, process_large_node_recursively, tree_parser,
generate_summaries_for_structure, generate_summaries_for_structure_md: added
return_exceptions=True to their asyncio.gather calls (llm_completion/
llm_acompletion raise RuntimeError on retry exhaustion, added earlier in this
PR), each with a degrade path matching the pattern already used by sibling
hardened gathers in the same files. One transient LLM failure no longer
aborts the whole document's indexing.
- _llm_semaphore is now a true process-wide ceiling (threading.Semaphore,
shared across every thread/event loop) instead of one asyncio.Semaphore per
event loop -- concurrently indexing N documents on N threads no longer
multiplies the effective cap by N. A max_concurrency_scope() override is
layered as a second, nested, per-loop restriction that can only tighten the
effective cap within the ceiling, never widen past it.
- set_llm_params() mutated a bare process-wide dict with no per-call isolation,
unlike max_concurrency which already had ContextVar scoping. Added
llm_params_scope() (mirrors max_concurrency_scope) + IndexConfig.llm_params,
wired into build_index() the same way max_concurrency already was, so
concurrent indexing jobs with different llm kwargs don't leak into each
other.
Adds regression tests for all five. Full suite: 221 passed, 2 skipped (one
pre-existing, unrelated flaky cloud-streaming test intermittently fails on
rerun; confirmed independent of this change).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The previous approach (bounded_gather building a fresh semaphore per call)
did NOT compose: the indexing call graph nests gathers
(tree_parser -> process_large_node_recursively -> recurse, plus each node's
check_title gather), and each level got its own independent cap. Peak
in-flight LLM calls grew ~N^depth, so a deep/wide document still exhausted
file descriptors (Errno 24) — the exact failure the cap was meant to
prevent — while the flat summary phase was over-serialized. Naively sharing
one semaphore across gather levels would instead deadlock (a parent holds a
slot while awaiting children that need slots).
Move the throttle to the single chokepoint every LLM call funnels through,
llm_acompletion: one shared semaphore per event loop, acquired only around
the litellm.acompletion network call. This gives a true global cap that
composes across any nesting and can't deadlock (a parent awaiting children
holds no slot). bounded_gather is gone; the call sites revert to plain
asyncio.gather.
Also:
- Reject bool in max_concurrency validation (bool is an int subclass, so
set_max_concurrency(True) / IndexConfig(max_concurrency=True) previously
became Semaphore(1) and silently serialized). Shared _validate_max_concurrency
+ a pydantic field_validator.
- Guard check_title_appearance_in_start_concurrent against an out-of-range
or 0 physical_index (LLM can emit one): it was dereferenced during task
construction, outside the gather's return_exceptions protection, aborting
the whole build; 0 silently wrapped to the last page. Now marked 'no'.
- Propagate contextvars into agent.py's worker-thread run (mirrors
pipeline._run_async) so ContextVar settings stay consistent.
- Tests rewritten to cover the nested case the old flat tests missed, the
leaf-level throttle in llm_acompletion, bool rejection, and the
out-of-range physical_index guard.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Cap concurrent in-flight LLM calls during indexing via a shared
semaphore (bounded_gather), so a many-node document no longer schedules
one socket per node and exhausts the process fd limit (Errno 24).
Make the per-index max_concurrency override correct under concurrency:
- Scope IndexConfig(max_concurrency=...) to the build_index call via a
ContextVar (max_concurrency_scope) instead of mutating a process
global. A one-off value no longer sticks as the new default, and
concurrent indexing of other documents isn't affected.
- Propagate the context through _run_async's worker-thread fallback so
the override survives the sync-over-async thread hop.
- set_max_concurrency() stays as the explicit process-wide setter.
Also stop `from .utils import *` leaking a `config` name (SimpleNamespace
alias) that shadowed the real pageindex.config submodule for the
page_index modules; the alias is now `_config`.
Adds regression tests for cap enforcement, scope stickiness/isolation,
worker-thread propagation, and the config-namespace fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
