apunkt/PageIndex
1
0
Fork 0
mirror of https://github.com/VectifyAI/PageIndex.git synced 2026-07-15 21:11:05 +02:00
Code Issues Projects Releases Packages Wiki Activity
349 commits 13 branches 2 tags 68 MiB
Commit graph

4 commits

Author SHA1 Message Date
mountain
4e6a13576d fix: CMYK image drop, empty-doc crash, page_index shadowing, sqlite hardening, flaky tests 
Addresses items 9-13 and a/b/c/f from the max-effort review of PR #272.

- pdf.py: image colorspace check was `pix.n > 4`, which treats CMYK-without-
  alpha (n==4, same as RGBA) as not needing RGB conversion; pix.save() as .png
  then raises "unsupported colorspace", silently dropped by the surrounding
  except. Fixed to `pix.n - pix.alpha >= 4` (correctly converts CMYK, leaves
  RGBA untouched).

- pipeline.py: detect_strategy([]) (an empty/whitespace-only source file)
  returned "content_based", routing into the PDF-oriented TOC-detection
  pipeline -- wasting a real LLM call before raising IndexingError. Empty
  node lists now route to level_based, whose build_tree_from_levels([])
  returns an empty structure instantly with zero LLM calls.

- page_index.py (shim): pageindex/__init__.py binds the canonical `page_index`
  function as the package attribute, but this file is ALSO a real submodule
  of the same name -- importing it anywhere (import machinery, unconditional)
  overwrites that attribute with the module object, breaking
  `from pageindex import page_index; page_index(x)` for the rest of the
  process. Made the shim module itself callable (delegates to the real
  function via a ModuleType subclass), so whichever object ends up in that
  slot is callable regardless of import order.

- storage/sqlite.py: create_collection let a raw sqlite3.IntegrityError escape
  on a duplicate name (new CollectionAlreadyExistsError); the collections
  table's CHECK constraint only validated the name's first character (GLOB
  '*' is a wildcard, not a regex quantifier over the preceding class) --
  fixed to validate the whole string, and SQLiteStorage now also validates in
  Python (it's a public StorageEngine usable directly, bypassing
  LocalBackend's own check).

- tests/test_review_fixes_2.py: two tests used a ContentNode with no `level`
  set, so build_index took the content_based path and made real (retried,
  slow, and -- with a valid key -- billable) LLM calls instead of testing the
  text-stripping logic they claimed to. Mocked out _content_based_pipeline.

- retrieve.py: _parse_pages/_get_pdf_page_content were independent copies of
  the canonical parse_pages/get_pdf_page_content that had already drifted
  (missing the p>=1 filter and 1000-page DoS cap) -- delegate to canonical
  now, so the legacy pageindex.get_page_content path can't silently regress
  again.

- parser/markdown.py: a leading UTF-8 BOM broke first-header detection
  (not whitespace, .strip() doesn't remove it) -- decode utf-8-sig. Only
  backtick fences were recognized as code blocks, so a '#'-prefixed line
  inside a ~~~-fenced block (valid CommonMark) was misparsed as a heading --
  recognize both fence styles.

- run_pageindex.py: --if-thinning wasn't migrated to the bare-flag +
  legacy-yes/no convention the other four --if-add-* flags got; bare usage
  raised an argparse error and it never went through the shared coercion.

- types.py: DocumentDetail's `structure` field was inside the class's
  total=False body, so TypedDict rules made it optional even though every
  backend always populates it. Split into a required base class.

Adds regression tests for all of the above. Full suite: 244 passed, 2 skipped
(one pre-existing, unrelated flaky cloud-streaming test).

Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
 2026-07-09 11:58:59 +08:00
mountain
b9d021916f fix: prompt-injection delimiter escape, legacy config coercion, gather resilience, true cross-thread concurrency bound 
Addresses items 4-8 from the max-effort review of PR #272 (VectifyAI/PageIndex#272).

- agent.py: wrap_with_doc_context() strips '<'/'>' from doc_name/doc_description
  (untrusted: unsanitized filename / LLM-generated from document content) before
  inserting them into the <docs>...</docs> block, so embedded content can never
  form a literal </docs> that closes the delimiter early and escapes the
  untrusted-data boundary SCOPED_SYSTEM_PROMPT relies on. Deterministic
  per-field transform, doesn't touch the (cacheable) static system prompt.

- ConfigLoader.load() (legacy 0.2.x compat) now routes merged overrides through
  IndexConfig before returning, so a legacy 'no' string gets pydantic's bool
  coercion instead of surviving as a truthy non-empty string — page_index_main's
  bare `if opt.if_add_node_summary:` checks (changed from `== 'yes'` elsewhere
  in this PR) were silently inverting caller intent and firing unwanted billed
  LLM calls.

- verify_toc, process_large_node_recursively, tree_parser,
  generate_summaries_for_structure, generate_summaries_for_structure_md: added
  return_exceptions=True to their asyncio.gather calls (llm_completion/
  llm_acompletion raise RuntimeError on retry exhaustion, added earlier in this
  PR), each with a degrade path matching the pattern already used by sibling
  hardened gathers in the same files. One transient LLM failure no longer
  aborts the whole document's indexing.

- _llm_semaphore is now a true process-wide ceiling (threading.Semaphore,
  shared across every thread/event loop) instead of one asyncio.Semaphore per
  event loop -- concurrently indexing N documents on N threads no longer
  multiplies the effective cap by N. A max_concurrency_scope() override is
  layered as a second, nested, per-loop restriction that can only tighten the
  effective cap within the ceiling, never widen past it.

- set_llm_params() mutated a bare process-wide dict with no per-call isolation,
  unlike max_concurrency which already had ContextVar scoping. Added
  llm_params_scope() (mirrors max_concurrency_scope) + IndexConfig.llm_params,
  wired into build_index() the same way max_concurrency already was, so
  concurrent indexing jobs with different llm kwargs don't leak into each
  other.

Adds regression tests for all five. Full suite: 221 passed, 2 skipped (one
pre-existing, unrelated flaky cloud-streaming test intermittently fails on
rerun; confirmed independent of this change).

Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
 2026-07-09 11:15:58 +08:00
mountain
04cb9cb02d fix: address xhigh code-review findings on 2d46d68..8f536cb 
Verified 12 findings from an xhigh-effort review of the prior review-fix
batch; all confirmed real. Most trace back to one root cause: the
build_index() text-stripping fix (8f536cb) correctly stopped Markdown from
leaking full text by default, but broke every path that assumed text could
be re-read later.

Correctness:
- LocalBackend._fill_node_text (get_document(include_text=True)) only handled
  PDF's start_index/end_index convention; Markdown nodes use line_num and got
  silently empty text. Now handles both.
- get_page_content's Markdown fallback (triggered when a StorageEngine
  legitimately returns None from get_pages()) read from the now-text-stripped
  structure. It now re-derives from the source file, mirroring the PDF
  fallback, so it no longer depends on structure text at all.
- add_document's PDF-only text-stripping branch (with the stale "markdown
  needs text in structure for fallback retrieval" comment) is now dead/wrong
  since build_index() already applies if_add_node_text uniformly — removed.
- _validate_llm_provider's keyless-provider allowlist was missing several
  local LiteLLM providers (xinference, llamafile, triton, oobabooga,
  openai_like, docker_model_runner, custom, custom_openai, petals) that need
  no API key just like ollama/lm_studio; expanded.
- The three agent-tool closures (get_document, get_document_structure,
  get_page_content) had three different not-found patterns; two bypassed the
  backend's DocumentNotFoundError entirely. Extracted LocalBackend.
  _require_document as the single existence check every method/tool now uses.
- examples/agentic_vectorless_rag_demo.py's hand-rolled Agent() didn't apply
  the litellm/ prefix normalization the SDK does internally, so its own
  documented "any LiteLLM provider" claim broke for non-openai models.
- cloud delete_collection's cache eviction removed the "folders unavailable"
  None sentinel too, forcing a wasted re-fetch; now only pops on a real id.

Cleanup / altitude:
- build_index() skips the remove_structure_text walk entirely when text was
  never added (content_based + if_add_node_summary=False + if_add_node_text=
  False) instead of a guaranteed no-op tree walk.
- page_index()'s locals()-capture-as-kwargs (fragile by construction) replaced
  with an explicit dict of the named parameters.
- run_pageindex.py's _cli_bool and the page_index_md.py legacy shim's
  _coerce_bool were duplicate, diverging implementations; both now bind
  directly to the canonical pageindex.index.page_index_md._coerce_bool.
- retrieve.py's _get_md_page_content delegated its own traversal instead of
  calling the canonical get_md_page_content; now a one-line delegation.
- FileTypeError's docstring now calls out the except-ordering gotcha from
  also subclassing ValueError.

17 new regression tests (tests/test_review_fixes_2.py) plus 2 updated in
tests/test_legacy_shims.py for the simplified md_to_tree shim. Full suite:
210 passed, 2 skipped.

Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
 2026-07-08 21:56:41 +08:00
mountain
6c948a332b refactor(index): dedupe the copied indexing pipeline behind deprecation shims 
The new SDK copied the legacy indexing pipeline into pageindex/index/
instead of moving it, leaving two divergent copies of page_index.py /
page_index_md.py / utils.py. They had already drifted (the legacy copy
still compared IndexConfig booleans against 'yes' — a separate fix),
and every pipeline change had to be applied twice.

Make pageindex/index/ the single source of truth (same pattern as the
LegacyCloudAPI shim for the 0.2.x cloud SDK):

- pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes
  (get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers,
  ...) so it's the sole utils module. Reconciled the diverged funcs:
  kept the modern versions, backported the #331 get_leaf_nodes .get()
  fix, and restored remove_fields' max_len parameter (superset).
- index/page_index*.py now import `from .utils import *`;
  index/legacy_utils.py (a re-export of the old top-level utils) deleted.
- Top-level page_index.py / page_index_md.py / utils.py become thin
  re-export shims that emit PendingDeprecationWarning. The md_to_tree
  shim coerces legacy 'yes'/'no' string flags to bool (the canonical
  version is boolean-typed).
- ConfigLoader no longer reads the deleted config.yaml; it builds
  defaults from IndexConfig (was an unconditional FileNotFoundError).
- __init__.py and retrieve.py import from pageindex.index.* directly so
  `import pageindex` does not trip the shims.

Adds tests/test_legacy_shims.py pinning the contract: clean top-level
import doesn't warn, legacy submodule imports warn, symbols still
resolve, shim and canonical share one implementation, the #331 fix and
ConfigLoader-without-yaml both hold, and the md_to_tree coercion works.

Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
 2026-07-07 10:42:23 +08:00