PageIndex/pageindex/errors.py

58 lines
1.6 KiB
Python
Raw Normal View History

class PageIndexError(Exception):
"""Base exception for all PageIndex SDK errors."""
pass
class CollectionNotFoundError(PageIndexError):
"""Collection does not exist."""
pass
fix: CMYK image drop, empty-doc crash, page_index shadowing, sqlite hardening, flaky tests Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272. - pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without- alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png then raises "unsupported colorspace", silently dropped by the surrounding except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves RGBA untouched). - pipeline.py: detect_strategy([]) (an empty/whitespace-only source file) returned "content_based", routing into the PDF-oriented TOC-detection pipeline -- wasting a real LLM call before raising IndexingError. Empty node lists now route to level_based, whose build_tree_from_levels([]) returns an empty structure instantly with zero LLM calls. - page_index.py (shim): pageindex/__init__.py binds the canonical `page_index` function as the package attribute, but this file is ALSO a real submodule of the same name -- importing it anywhere (import machinery, unconditional) overwrites that attribute with the module object, breaking `from pageindex import page_index; page_index(x)` for the rest of the process. Made the shim module itself callable (delegates to the real function via a ModuleType subclass), so whichever object ends up in that slot is callable regardless of import order. - storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape on a duplicate name (new CollectionAlreadyExistsError); the collections table's CHECK constraint only validated the name's first character (GLOB '*' is a wildcard, not a regex quantifier over the preceding class) -- fixed to validate the whole string, and SQLiteStorage now also validates in Python (it's a public StorageEngine usable directly, bypassing LocalBackend's own check). - tests/test_review_fixes_2.py: two tests used a ContentNode with no `level` set, so build_index took the content_based path and made real (retried, slow, and -- with a valid key -- billable) LLM calls instead of testing the text-stripping logic they claimed to. Mocked out _content_based_pipeline. - retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of the canonical parse_pages/get_pdf_page_content that had already drifted (missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical now, so the legacy pageindex.get_page_content path can't silently regress again. - parser/markdown.py: a leading UTF-8 BOM broke first-header detection (not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only backtick fences were recognized as code blocks, so a '#'-prefixed line inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading -- recognize both fence styles. - run_pageindex.py: --if-thinning wasn't migrated to the bare-flag + legacy-yes/no convention the other four --if-add-* flags got; bare usage raised an argparse error and it never went through the shared coercion. - types.py: DocumentDetail's `structure` field was inside the class's total=False body, so TypedDict rules made it optional even though every backend always populates it. Split into a required base class. Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped (one pre-existing, unrelated flaky cloud-streaming test). Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-09 11:58:59 +08:00
class CollectionAlreadyExistsError(PageIndexError):
"""Collection already exists (create_collection, not get_or_create)."""
pass
class DocumentNotFoundError(PageIndexError):
"""Document ID not found."""
pass
class IndexingError(PageIndexError):
"""Indexing pipeline failure."""
pass
2026-05-11 21:06:23 +08:00
class PageIndexAPIError(PageIndexError):
"""PageIndex cloud API returned an error.
Kept for compatibility with the pageindex 0.2.x cloud SDK.
"""
pass
class CloudAPIError(PageIndexAPIError):
fix(cloud): align cloud/local backend contracts and harden the HTTP layer Fixes the cloud/local contract mismatches from the PR #272 review (verified against the official API docs — the cloud API has no folder/collection endpoints publicly, GET /docs supports limit<=100 with offset): - query_stream: emit a terminal answer_done event with the full answer (same contract as the local backend); raise CloudAPIError instead of disguising HTTP errors as answer events; move the initial connect inside try so a connection failure can no longer strand the consumer awaiting a sentinel that never arrives - _request: rewind file objects before retrying so a transient 5xx/429 during upload no longer re-sends an empty multipart body; carry the HTTP status on CloudAPIError (status_code) and keep the last status in the max-retries error - list_documents: paginate with limit/offset instead of a hard-coded limit=100, so >100-doc collections are no longer silently truncated (whole-collection queries rely on this list) - folders: treat only 403/404 as "folders unavailable" (warned via warnings.warn instead of an invisible logger.warning, matched on status_code instead of a "403" substring); transient errors now propagate instead of being permanently cached as folder_id=None - error taxonomy parity: cloud doc endpoints map HTTP 404 to DocumentNotFoundError; local get_document raises DocumentNotFoundError instead of returning {}; local delete_document raises on missing doc_id instead of silently deleting nothing - cloud get_document warns that include_text is not supported instead of silently ignoring it Adds regression tests for each fix (11 new tests). Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-07 09:40:09 +08:00
"""Cloud API returned error.
``status_code`` carries the HTTP status when the error came from an HTTP
response (None for transport-level failures), so callers can branch on it
instead of parsing the message.
"""
def __init__(self, message: str, status_code: int | None = None):
super().__init__(message)
self.status_code = status_code
fix: address PR #272 review findings (directly-fixable items) Verified against current dev; the compat/behavior decisions (#7 api_key semantics, #10 CLI flags, #11 doc-description default) are deferred. Crashes: - page_index(): snapshot args before importing IndexConfig — locals() was capturing the imported class and IndexConfig(extra='forbid') made every call raise ValidationError. - process_none_page_numbers: pop('page', None) instead of del (a TOC item without 'page' raised KeyError mid-pipeline). - pipeline._run_async: guard only the loop detection, not the run, so a real RuntimeError from the coroutine isn't masked as "asyncio.run() cannot be called from a running event loop". Silent-wrong / robustness: - LocalBackend.get_document_structure and the agent get_document / get_document_structure tools now surface a missing doc (raise / error-JSON) instead of returning empty, matching get_page_content and the cloud backend. - cloud delete_collection drops the cached folder_id. - cloud query raises on an empty collection instead of POSTing doc_id:[]. - LocalClient skips the API-key check for keyless providers (ollama, lm_studio, …) so keyless LiteLLM models aren't rejected at construction. Compat / cleanup: - md_to_tree coerces legacy 'yes'/'no' string flags (a bare 'no' was truthy). - FileTypeError also subclasses ValueError (0.2.x raised ValueError). - _validate_llm_provider no longer mutates global litellm.model_cost_map_url. - __all__ re-includes legacy exports (page_index, md_to_tree, get_*). - Rewrite examples/agentic_vectorless_rag_demo.py to the Collection API and use the in-repo attention.pdf (the old workspace=/client.index/client.documents API no longer exists). Adds tests/test_review_fixes.py (10 regressions). Full suite: 189 passed. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-08 18:56:51 +08:00
class FileTypeError(PageIndexError, ValueError):
"""Unsupported file type.
Also subclasses ValueError so pre-SDK ``except ValueError`` around indexing
(0.2.x raised ValueError for an unsupported file format) still catches it.
fix: address xhigh code-review findings on 2d46d68..8f536cb Verified 12 findings from an xhigh-effort review of the prior review-fix batch; all confirmed real. Most trace back to one root cause: the build_index() text-stripping fix (8f536cb) correctly stopped Markdown from leaking full text by default, but broke every path that assumed text could be re-read later. Correctness: - LocalBackend._fill_node_text (get_document(include_text=True)) only handled PDF's start_index/end_index convention; Markdown nodes use line_num and got silently empty text. Now handles both. - get_page_content's Markdown fallback (triggered when a StorageEngine legitimately returns None from get_pages()) read from the now-text-stripped structure. It now re-derives from the source file, mirroring the PDF fallback, so it no longer depends on structure text at all. - add_document's PDF-only text-stripping branch (with the stale "markdown needs text in structure for fallback retrieval" comment) is now dead/wrong since build_index() already applies if_add_node_text uniformly — removed. - _validate_llm_provider's keyless-provider allowlist was missing several local LiteLLM providers (xinference, llamafile, triton, oobabooga, openai_like, docker_model_runner, custom, custom_openai, petals) that need no API key just like ollama/lm_studio; expanded. - The three agent-tool closures (get_document, get_document_structure, get_page_content) had three different not-found patterns; two bypassed the backend's DocumentNotFoundError entirely. Extracted LocalBackend. _require_document as the single existence check every method/tool now uses. - examples/agentic_vectorless_rag_demo.py's hand-rolled Agent() didn't apply the litellm/ prefix normalization the SDK does internally, so its own documented "any LiteLLM provider" claim broke for non-openai models. - cloud delete_collection's cache eviction removed the "folders unavailable" None sentinel too, forcing a wasted re-fetch; now only pops on a real id. Cleanup / altitude: - build_index() skips the remove_structure_text walk entirely when text was never added (content_based + if_add_node_summary=False + if_add_node_text= False) instead of a guaranteed no-op tree walk. - page_index()'s locals()-capture-as-kwargs (fragile by construction) replaced with an explicit dict of the named parameters. - run_pageindex.py's _cli_bool and the page_index_md.py legacy shim's _coerce_bool were duplicate, diverging implementations; both now bind directly to the canonical pageindex.index.page_index_md._coerce_bool. - retrieve.py's _get_md_page_content delegated its own traversal instead of calling the canonical get_md_page_content; now a one-line delegation. - FileTypeError's docstring now calls out the except-ordering gotcha from also subclassing ValueError. 17 new regression tests (tests/test_review_fixes_2.py) plus 2 updated in tests/test_legacy_shims.py for the simplified md_to_tree shim. Full suite: 210 passed, 2 skipped. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-08 21:56:41 +08:00
Note: because of this, an ``except ValueError`` clause ahead of an
``except FileTypeError`` clause in the same try block will catch it first
if you need FileTypeError-specific handling, put that except before (or
instead of) a bare ValueError one.
fix: address PR #272 review findings (directly-fixable items) Verified against current dev; the compat/behavior decisions (#7 api_key semantics, #10 CLI flags, #11 doc-description default) are deferred. Crashes: - page_index(): snapshot args before importing IndexConfig — locals() was capturing the imported class and IndexConfig(extra='forbid') made every call raise ValidationError. - process_none_page_numbers: pop('page', None) instead of del (a TOC item without 'page' raised KeyError mid-pipeline). - pipeline._run_async: guard only the loop detection, not the run, so a real RuntimeError from the coroutine isn't masked as "asyncio.run() cannot be called from a running event loop". Silent-wrong / robustness: - LocalBackend.get_document_structure and the agent get_document / get_document_structure tools now surface a missing doc (raise / error-JSON) instead of returning empty, matching get_page_content and the cloud backend. - cloud delete_collection drops the cached folder_id. - cloud query raises on an empty collection instead of POSTing doc_id:[]. - LocalClient skips the API-key check for keyless providers (ollama, lm_studio, …) so keyless LiteLLM models aren't rejected at construction. Compat / cleanup: - md_to_tree coerces legacy 'yes'/'no' string flags (a bare 'no' was truthy). - FileTypeError also subclasses ValueError (0.2.x raised ValueError). - _validate_llm_provider no longer mutates global litellm.model_cost_map_url. - __all__ re-includes legacy exports (page_index, md_to_tree, get_*). - Rewrite examples/agentic_vectorless_rag_demo.py to the Collection API and use the in-repo attention.pdf (the old workspace=/client.index/client.documents API no longer exists). Adds tests/test_review_fixes.py (10 regressions). Full suite: 189 passed. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-08 18:56:51 +08:00
"""
pass