- local get_agent_tools: `set(doc_ids) if doc_ids is not None else None` so
doc_ids=[] is a scope of nothing (reject all), not open mode. The public
query path already guarded []; this hardens direct callers.
- legacy get_tree: send summary=true/false (lowercase) instead of Python's
capitalized True/False, matching the modern CloudBackend and the API.
- markdown parser: track the opening fence character so a ```-fence isn't
closed by a ~~~ line (CommonMark), keeping '#'-lines inside it out of headings.
- dedupe the cloud base URL: single API_BASE in cloud_api, referenced by
CloudBackend and PageIndexClient (was three independent copies).
Regression tests for each.
Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272.
- pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without-
alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png
then raises "unsupported colorspace", silently dropped by the surrounding
except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves
RGBA untouched).
- pipeline.py: detect_strategy([]) (an empty/whitespace-only source file)
returned "content_based", routing into the PDF-oriented TOC-detection
pipeline -- wasting a real LLM call before raising IndexingError. Empty
node lists now route to level_based, whose build_tree_from_levels([])
returns an empty structure instantly with zero LLM calls.
- page_index.py (shim): pageindex/__init__.py binds the canonical `page_index`
function as the package attribute, but this file is ALSO a real submodule
of the same name -- importing it anywhere (import machinery, unconditional)
overwrites that attribute with the module object, breaking
`from pageindex import page_index; page_index(x)` for the rest of the
process. Made the shim module itself callable (delegates to the real
function via a ModuleType subclass), so whichever object ends up in that
slot is callable regardless of import order.
- storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape
on a duplicate name (new CollectionAlreadyExistsError); the collections
table's CHECK constraint only validated the name's first character (GLOB
'*' is a wildcard, not a regex quantifier over the preceding class) --
fixed to validate the whole string, and SQLiteStorage now also validates in
Python (it's a public StorageEngine usable directly, bypassing
LocalBackend's own check).
- tests/test_review_fixes_2.py: two tests used a ContentNode with no `level`
set, so build_index took the content_based path and made real (retried,
slow, and -- with a valid key -- billable) LLM calls instead of testing the
text-stripping logic they claimed to. Mocked out _content_based_pipeline.
- retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of
the canonical parse_pages/get_pdf_page_content that had already drifted
(missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical
now, so the legacy pageindex.get_page_content path can't silently regress
again.
- parser/markdown.py: a leading UTF-8 BOM broke first-header detection
(not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only
backtick fences were recognized as code blocks, so a '#'-prefixed line
inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading --
recognize both fence styles.
- run_pageindex.py: --if-thinning wasn't migrated to the bare-flag +
legacy-yes/no convention the other four --if-add-* flags got; bare usage
raised an argparse error and it never went through the shared coercion.
- types.py: DocumentDetail's `structure` field was inside the class's
total=False body, so TypedDict rules made it optional even though every
backend always populates it. Split into a required base class.
Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped
(one pre-existing, unrelated flaky cloud-streaming test).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- AgentRunner.run: offload to a worker-thread event loop when called
from inside a running loop (Jupyter, FastAPI handlers) — mirrors
pipeline._run_async; Runner.run_sync raised RuntimeError there.
- SQLiteStorage: create connections with check_same_thread=False so
close() can actually close connections created by worker threads.
Each thread still gets its own connection via threading.local; with
the default True those closes raised ProgrammingError (silently
swallowed) and leaked every worker connection.
- CloudBackend.query: non-streaming chat completions now use a 300s
timeout and a single attempt. The default 30s ReadTimeout fired
before generation finished and the retry loop re-billed the full
server-side retrieval + generation up to three times. _request gains
retries/timeout overrides; the exhausted-retry path also no longer
sleeps before raising.
- MarkdownParser: content before the first heading (abstract/preamble)
becomes a node instead of being silently dropped and unretrievable;
a file with no headings at all yields a single document node instead
of zero nodes (which pushed an empty page list into the pipeline).
- LegacyCloudAPI.is_retrieval_ready: API failures (revoked key, network
down) now propagate as PageIndexAPIError instead of reading as
"not ready", which turned polling loops into infinite loops.
Adds regression tests for each fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
