- index/utils.py: fix an asyncio deadlock in _sync_llm_semaphore. Sync LLM
calls (check_toc / process_no_toc / toc_transformer) run on the event-loop
thread nested inside the async meta_processor, and its blocking ceiling
acquire could wait forever for a permit held by async _llm_semaphore holders
that can only release it once the (now-frozen) loop runs. Take the slot
non-blocking when on a running loop; keep the blocking acquire off-loop.
- index/utils.py: bound parse_pages ranges before materializing range() into
the list, so a huge span like '1-2000000000' is rejected up front instead of
exhausting memory before the 1000-page cap is ever checked (DoS).
- storage/sqlite.py: bump a generation counter on close() so a thread that
cached a connection in thread-local storage reconnects on its next call
instead of reusing a closed handle (ProgrammingError).
- backend/cloud.py: after connect, bail out of the SSE background thread if the
consumer already abandoned the stream, instead of draining it in the background.
Adds regression tests for each fix.
Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272.
- pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without-
alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png
then raises "unsupported colorspace", silently dropped by the surrounding
except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves
RGBA untouched).
- pipeline.py: detect_strategy([]) (an empty/whitespace-only source file)
returned "content_based", routing into the PDF-oriented TOC-detection
pipeline -- wasting a real LLM call before raising IndexingError. Empty
node lists now route to level_based, whose build_tree_from_levels([])
returns an empty structure instantly with zero LLM calls.
- page_index.py (shim): pageindex/__init__.py binds the canonical `page_index`
function as the package attribute, but this file is ALSO a real submodule
of the same name -- importing it anywhere (import machinery, unconditional)
overwrites that attribute with the module object, breaking
`from pageindex import page_index; page_index(x)` for the rest of the
process. Made the shim module itself callable (delegates to the real
function via a ModuleType subclass), so whichever object ends up in that
slot is callable regardless of import order.
- storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape
on a duplicate name (new CollectionAlreadyExistsError); the collections
table's CHECK constraint only validated the name's first character (GLOB
'*' is a wildcard, not a regex quantifier over the preceding class) --
fixed to validate the whole string, and SQLiteStorage now also validates in
Python (it's a public StorageEngine usable directly, bypassing
LocalBackend's own check).
- tests/test_review_fixes_2.py: two tests used a ContentNode with no `level`
set, so build_index took the content_based path and made real (retried,
slow, and -- with a valid key -- billable) LLM calls instead of testing the
text-stripping logic they claimed to. Mocked out _content_based_pipeline.
- retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of
the canonical parse_pages/get_pdf_page_content that had already drifted
(missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical
now, so the legacy pageindex.get_page_content path can't silently regress
again.
- parser/markdown.py: a leading UTF-8 BOM broke first-header detection
(not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only
backtick fences were recognized as code blocks, so a '#'-prefixed line
inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading --
recognize both fence styles.
- run_pageindex.py: --if-thinning wasn't migrated to the bare-flag +
legacy-yes/no convention the other four --if-add-* flags got; bare usage
raised an argparse error and it never went through the shared coercion.
- types.py: DocumentDetail's `structure` field was inside the class's
total=False body, so TypedDict rules made it optional even though every
backend always populates it. Split into a required base class.
Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped
(one pre-existing, unrelated flaky cloud-streaming test).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Address PR #272 review:
- get_md_page_content / retrieve._get_md_page_content returned every node
whose line_num fell in [min(pages), max(pages)], so a non-contiguous spec
like "5,100" over-fetched everything in between. Match the exact requested
line numbers instead, mirroring the PDF path. (Same bug as #280.)
- Restore the CHATGPT_API_KEY -> OPENAI_API_KEY backward-compat alias dropped
when pageindex/utils.py became a re-export shim; users with only
CHATGPT_API_KEY set would otherwise fail auth after upgrading. It now runs
in __init__.py right after load_dotenv.
Adds regression tests for both.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
