mirror of
https://github.com/VectifyAI/PageIndex.git
synced 2026-07-15 21:11:05 +02:00
4e6a13576d
Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272. - pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without- alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png then raises "unsupported colorspace", silently dropped by the surrounding except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves RGBA untouched). - pipeline.py: detect_strategy([]) (an empty/whitespace-only source file) returned "content_based", routing into the PDF-oriented TOC-detection pipeline -- wasting a real LLM call before raising IndexingError. Empty node lists now route to level_based, whose build_tree_from_levels([]) returns an empty structure instantly with zero LLM calls. - page_index.py (shim): pageindex/__init__.py binds the canonical `page_index` function as the package attribute, but this file is ALSO a real submodule of the same name -- importing it anywhere (import machinery, unconditional) overwrites that attribute with the module object, breaking `from pageindex import page_index; page_index(x)` for the rest of the process. Made the shim module itself callable (delegates to the real function via a ModuleType subclass), so whichever object ends up in that slot is callable regardless of import order. - storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape on a duplicate name (new CollectionAlreadyExistsError); the collections table's CHECK constraint only validated the name's first character (GLOB '*' is a wildcard, not a regex quantifier over the preceding class) -- fixed to validate the whole string, and SQLiteStorage now also validates in Python (it's a public StorageEngine usable directly, bypassing LocalBackend's own check). - tests/test_review_fixes_2.py: two tests used a ContentNode with no `level` set, so build_index took the content_based path and made real (retried, slow, and -- with a valid key -- billable) LLM calls instead of testing the text-stripping logic they claimed to. Mocked out _content_based_pipeline. - retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of the canonical parse_pages/get_pdf_page_content that had already drifted (missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical now, so the legacy pageindex.get_page_content path can't silently regress again. - parser/markdown.py: a leading UTF-8 BOM broke first-header detection (not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only backtick fences were recognized as code blocks, so a '#'-prefixed line inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading -- recognize both fence styles. - run_pageindex.py: --if-thinning wasn't migrated to the bare-flag + legacy-yes/no convention the other four --if-add-* flags got; bare usage raised an argparse error and it never went through the shared coercion. - types.py: DocumentDetail's `structure` field was inside the class's total=False body, so TypedDict rules made it optional even though every backend always populates it. Split into a required base class. Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped (one pre-existing, unrelated flaky cloud-streaming test). Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
40 lines
1.8 KiB
Python
40 lines
1.8 KiB
Python
# pageindex/page_index.py
|
|
# Deprecation shim. The PDF indexing pipeline now lives in
|
|
# pageindex/index/page_index.py (the single source of truth). This module
|
|
# re-exports it so legacy imports (`from pageindex.page_index import ...`,
|
|
# `from pageindex import page_index`) keep working.
|
|
import sys
|
|
import types
|
|
import warnings
|
|
|
|
warnings.warn(
|
|
"pageindex.page_index has moved to pageindex.index.page_index; importing it "
|
|
"from the top level is deprecated and will be removed in a future release.",
|
|
PendingDeprecationWarning,
|
|
stacklevel=2,
|
|
)
|
|
|
|
from .index.page_index import * # noqa: F401,F403,E402
|
|
|
|
# pageindex/__init__.py binds the FUNCTION `page_index` as the package
|
|
# attribute `pageindex.page_index` (`from .index.page_index import *`). But
|
|
# this file is ALSO a real submodule of the same name — the moment anything,
|
|
# anywhere in the process, does `import pageindex.page_index` (exactly what
|
|
# `from pageindex.page_index import X` triggers), Python's import machinery
|
|
# overwrites that package attribute with THIS module object, clobbering the
|
|
# function binding. Afterwards `from pageindex import page_index; page_index(x)`
|
|
# would raise "TypeError: 'module' object is not callable" — silently, and
|
|
# depending entirely on whether this submodule happened to be imported yet.
|
|
#
|
|
# Fix: make this module itself callable, delegating to the real function, so
|
|
# whichever object ends up sitting in the `pageindex.page_index` slot — the
|
|
# function or this module — is callable either way. Both `from pageindex.page_index
|
|
# import page_index_main` (module attribute access) and
|
|
# `from pageindex import page_index; page_index(x)` (call) keep working
|
|
# regardless of import order.
|
|
class _CallableModule(types.ModuleType):
|
|
def __call__(self, *args, **kwargs):
|
|
return page_index(*args, **kwargs)
|
|
|
|
|
|
sys.modules[__name__].__class__ = _CallableModule
|