Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272.
- pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without-
alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png
then raises "unsupported colorspace", silently dropped by the surrounding
except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves
RGBA untouched).
- pipeline.py: detect_strategy([]) (an empty/whitespace-only source file)
returned "content_based", routing into the PDF-oriented TOC-detection
pipeline -- wasting a real LLM call before raising IndexingError. Empty
node lists now route to level_based, whose build_tree_from_levels([])
returns an empty structure instantly with zero LLM calls.
- page_index.py (shim): pageindex/__init__.py binds the canonical `page_index`
function as the package attribute, but this file is ALSO a real submodule
of the same name -- importing it anywhere (import machinery, unconditional)
overwrites that attribute with the module object, breaking
`from pageindex import page_index; page_index(x)` for the rest of the
process. Made the shim module itself callable (delegates to the real
function via a ModuleType subclass), so whichever object ends up in that
slot is callable regardless of import order.
- storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape
on a duplicate name (new CollectionAlreadyExistsError); the collections
table's CHECK constraint only validated the name's first character (GLOB
'*' is a wildcard, not a regex quantifier over the preceding class) --
fixed to validate the whole string, and SQLiteStorage now also validates in
Python (it's a public StorageEngine usable directly, bypassing
LocalBackend's own check).
- tests/test_review_fixes_2.py: two tests used a ContentNode with no `level`
set, so build_index took the content_based path and made real (retried,
slow, and -- with a valid key -- billable) LLM calls instead of testing the
text-stripping logic they claimed to. Mocked out _content_based_pipeline.
- retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of
the canonical parse_pages/get_pdf_page_content that had already drifted
(missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical
now, so the legacy pageindex.get_page_content path can't silently regress
again.
- parser/markdown.py: a leading UTF-8 BOM broke first-header detection
(not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only
backtick fences were recognized as code blocks, so a '#'-prefixed line
inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading --
recognize both fence styles.
- run_pageindex.py: --if-thinning wasn't migrated to the bare-flag +
legacy-yes/no convention the other four --if-add-* flags got; bare usage
raised an argparse error and it never went through the shared coercion.
- types.py: DocumentDetail's `structure` field was inside the class's
total=False body, so TypedDict rules made it optional even though every
backend always populates it. Split into a required base class.
Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped
(one pre-existing, unrelated flaky cloud-streaming test).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
The new SDK copied the legacy indexing pipeline into pageindex/index/
instead of moving it, leaving two divergent copies of page_index.py /
page_index_md.py / utils.py. They had already drifted (the legacy copy
still compared IndexConfig booleans against 'yes' — a separate fix),
and every pipeline change had to be applied twice.
Make pageindex/index/ the single source of truth (same pattern as the
LegacyCloudAPI shim for the 0.2.x cloud SDK):
- pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes
(get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers,
...) so it's the sole utils module. Reconciled the diverged funcs:
kept the modern versions, backported the #331 get_leaf_nodes .get()
fix, and restored remove_fields' max_len parameter (superset).
- index/page_index*.py now import `from .utils import *`;
index/legacy_utils.py (a re-export of the old top-level utils) deleted.
- Top-level page_index.py / page_index_md.py / utils.py become thin
re-export shims that emit PendingDeprecationWarning. The md_to_tree
shim coerces legacy 'yes'/'no' string flags to bool (the canonical
version is boolean-typed).
- ConfigLoader no longer reads the deleted config.yaml; it builds
defaults from IndexConfig (was an unconditional FileNotFoundError).
- __init__.py and retrieve.py import from pageindex.index.* directly so
`import pageindex` does not trip the shims.
Adds tests/test_legacy_shims.py pinning the contract: clean top-level
import doesn't warn, legacy submodule imports warn, symbols still
resolve, shim and canonical share one implementation, the #331 fix and
ConfigLoader-without-yaml both hold, and the md_to_tree coercion works.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- local delete_collection: validate the collection name before rmtree.
An unvalidated name like "../.." escaped files_dir and deleted
arbitrary directories (path traversal).
- legacy page_index(): restore the node_id/summary/text/description
enhancements. IndexConfig now carries booleans (pydantic coerces the
legacy 'yes'/'no' strings at the boundary), but page_index_main still
compared `opt.if_add_node_id == 'yes'` — always False — so every
enhancement was silently skipped for legacy-API callers. Conditions
now branch on the booleans, matching pageindex/index/page_index.py.
- LegacyCloudAPI._request: bound every request with a timeout
(30s, 120s read timeout for streamed responses) so a dead connection
can't hang legacy submit/poll/chat callers forever. The legacy
contract tests pinned the missing timeout; updated to pin its
presence instead.
Adds regression tests: path-traversal rejection, 'yes'/'no' -> bool
coercion, and timeout assertions.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
* Consolidate tests/ into examples/documents/
* Add line_count and reorder structure keys
* Lazy-load documents with _meta.json index
* Update demo script and add pre-shipped workspace
* Extract shared helpers for JSON reading and meta entry building
* Add PageIndexClient with retrieve, streaming support and litellm integration
* Add OpenAI agents demo example
* Update README with example agent demo section
* Support separate retrieve_model configuration for index and retrieve
* Integrate litellm for multi-provider LLM support
* recover the default config yaml
* Use litellm.acompletion for native async support
* fix tob
* Rename llm_complete/allm_complete to llm_completion/llm_acompletion, remove unused llm_complete_stream
* Pin litellm to version 1.82.0
* resolve comments
* args from cli is used to overrides config.yaml
* Fix get_page_tokens hardcoded model default
Pass opt.model to get_page_tokens so tokenization respects the
configured model instead of always using gpt-4o-2024-11-20.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Remove explicit openai dependency from requirements.txt
openai is no longer directly imported; it comes in as a transitive
dependency of litellm. Pinning it explicitly risks version conflicts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Restore openai==1.101.0 pin in requirements.txt
litellm==1.82.0 and openai-agents have conflicting openai version
requirements, but openai==1.101.0 works at runtime for both.
The pin is necessary to prevent litellm from pulling in openai>=2.x
which would break openai-agents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Remove explicit openai dependency from requirements.txt
openai is not directly used; it comes in as a transitive dependency
of litellm. No openai-agents in this branch so no pin needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix an litellm error log
* resolve comments
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The loop variable `list_index = page_index - start_index` was
overwriting the outer `list_index = incorrect_item['list_index']`,
causing results to be written back to wrong index positions.
Rename the loop variable to `page_list_idx` to avoid shadowing.
Closes#66
The while loop exit condition used len(chat_history), but chat_history
was rebuilt every iteration with exactly 2 elements, making the check
len(chat_history) > 5 never true.
Replace with explicit attempt counter and max_attempts limit.
