- local get_agent_tools: `set(doc_ids) if doc_ids is not None else None` so
doc_ids=[] is a scope of nothing (reject all), not open mode. The public
query path already guarded []; this hardens direct callers.
- legacy get_tree: send summary=true/false (lowercase) instead of Python's
capitalized True/False, matching the modern CloudBackend and the API.
- markdown parser: track the opening fence character so a ```-fence isn't
closed by a ~~~ line (CommonMark), keeping '#'-lines inside it out of headings.
- dedupe the cloud base URL: single API_BASE in cloud_api, referenced by
CloudBackend and PageIndexClient (was three independent copies).
Regression tests for each.
Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272.
- pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without-
alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png
then raises "unsupported colorspace", silently dropped by the surrounding
except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves
RGBA untouched).
- pipeline.py: detect_strategy([]) (an empty/whitespace-only source file)
returned "content_based", routing into the PDF-oriented TOC-detection
pipeline -- wasting a real LLM call before raising IndexingError. Empty
node lists now route to level_based, whose build_tree_from_levels([])
returns an empty structure instantly with zero LLM calls.
- page_index.py (shim): pageindex/__init__.py binds the canonical `page_index`
function as the package attribute, but this file is ALSO a real submodule
of the same name -- importing it anywhere (import machinery, unconditional)
overwrites that attribute with the module object, breaking
`from pageindex import page_index; page_index(x)` for the rest of the
process. Made the shim module itself callable (delegates to the real
function via a ModuleType subclass), so whichever object ends up in that
slot is callable regardless of import order.
- storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape
on a duplicate name (new CollectionAlreadyExistsError); the collections
table's CHECK constraint only validated the name's first character (GLOB
'*' is a wildcard, not a regex quantifier over the preceding class) --
fixed to validate the whole string, and SQLiteStorage now also validates in
Python (it's a public StorageEngine usable directly, bypassing
LocalBackend's own check).
- tests/test_review_fixes_2.py: two tests used a ContentNode with no `level`
set, so build_index took the content_based path and made real (retried,
slow, and -- with a valid key -- billable) LLM calls instead of testing the
text-stripping logic they claimed to. Mocked out _content_based_pipeline.
- retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of
the canonical parse_pages/get_pdf_page_content that had already drifted
(missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical
now, so the legacy pageindex.get_page_content path can't silently regress
again.
- parser/markdown.py: a leading UTF-8 BOM broke first-header detection
(not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only
backtick fences were recognized as code blocks, so a '#'-prefixed line
inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading --
recognize both fence styles.
- run_pageindex.py: --if-thinning wasn't migrated to the bare-flag +
legacy-yes/no convention the other four --if-add-* flags got; bare usage
raised an argparse error and it never went through the shared coercion.
- types.py: DocumentDetail's `structure` field was inside the class's
total=False body, so TypedDict rules made it optional even though every
backend always populates it. Split into a required base class.
Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped
(one pre-existing, unrelated flaky cloud-streaming test).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Replace the `...` bodies in DocumentParser / StorageEngine protocol methods
with one-line docstrings: silences the CodeQL "statement has no effect" false
positives on #272 (`...` is idiomatic for typing.Protocol, but docstrings
document the contract and don't trip the analyzer) with no behavior change.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Engineering-quality cleanups from the SDK review (no behavior change):
- Return-type discoverability: add pageindex/types.py with TypedDicts
(DocumentInfo, DocumentDetail, PageContent) and annotate Collection /
Backend methods with them; add docstrings to every public Collection
method (including the get_page_content `pages` spec). Exported from the
package. Zero runtime cost — these are plain dicts.
- Backend protocol as a real contract:
* query_stream is an async generator, so the protocol now declares it
as `def ... -> AsyncIterator[QueryEvent]` (not `async def`, which
typed it as a coroutine and never matched the implementations).
* custom-parser support is expressed as a runtime_checkable
SupportsParserRegistration capability protocol; the client uses
isinstance(...) instead of hasattr(...) duck-typing.
- Parser layering: move count_tokens into a leaf module pageindex/tokens.py
so parser/* imports it from there instead of reaching back into
pageindex.index (a reverse dependency). index.utils re-exports it for
backward compatibility.
Adds tests/test_architecture.py enforcing: parser never imports index,
count_tokens is a single shared leaf, the capability protocol works,
both backends satisfy Backend, and the TypedDicts are exported.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- SQLite dedup race: add UNIQUE(collection_name, file_hash) and switch
save_document to a plain INSERT. add_document now catches the
IntegrityError from a concurrent add of the same content, cleans up its
managed files, and returns the winning doc_id — instead of two doc_ids
for one file (each having paid for its own LLM indexing).
- PDF image paths: store the absolute path to each extracted image
instead of a path relative to the indexing process's cwd. The
 references broke as soon as a query ran from a different
directory.
- LegacyCloudAPI: URL-encode doc_id / retrieval_id path segments (added
_enc()), matching CloudBackend. An id containing '/', '?', '#' or a
space previously hit the wrong endpoint or produced a malformed URL.
Adds regression tests: UNIQUE enforcement, the add-race resolving to the
winner, absolute image paths, and encoded legacy URLs.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- AgentRunner.run: offload to a worker-thread event loop when called
from inside a running loop (Jupyter, FastAPI handlers) — mirrors
pipeline._run_async; Runner.run_sync raised RuntimeError there.
- SQLiteStorage: create connections with check_same_thread=False so
close() can actually close connections created by worker threads.
Each thread still gets its own connection via threading.local; with
the default True those closes raised ProgrammingError (silently
swallowed) and leaked every worker connection.
- CloudBackend.query: non-streaming chat completions now use a 300s
timeout and a single attempt. The default 30s ReadTimeout fired
before generation finished and the retry loop re-billed the full
server-side retrieval + generation up to three times. _request gains
retries/timeout overrides; the exhausted-retry path also no longer
sleeps before raising.
- MarkdownParser: content before the first heading (abstract/preamble)
becomes a node instead of being silently dropped and unretrievable;
a file with no headings at all yields a single document node instead
of zero nodes (which pushed an empty page list into the pipeline).
- LegacyCloudAPI.is_retrieval_ready: API failures (revoked key, network
down) now propagate as PageIndexAPIError instead of reading as
"not ready", which turned polling loops into infinite loops.
Adds regression tests for each fix.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
