PageIndex/pageindex/__init__.py

72 lines
2.4 KiB
Python
Raw Normal View History

# pageindex/__init__.py
# Load .env explicitly, before anything else, so environment-based credentials
# (OPENAI_API_KEY for local mode, PAGEINDEX_API_KEY that callers read via
# os.environ for cloud mode) are populated by PageIndex itself — not left to
# litellm's incidental dotenv loading, which would vanish if litellm changes or
# its import is ever made lazy.
from dotenv import load_dotenv as _load_dotenv
_load_dotenv()
# Backward compatibility: honor CHATGPT_API_KEY as an alias for OPENAI_API_KEY
# (kept from the pre-SDK pageindex.utils). Runs after load_dotenv so a value in
# .env is picked up too; only fills OPENAI_API_KEY when it isn't already set.
import os as _os
if not _os.getenv("OPENAI_API_KEY") and _os.getenv("CHATGPT_API_KEY"):
_os.environ["OPENAI_API_KEY"] = _os.getenv("CHATGPT_API_KEY")
refactor(index): dedupe the copied indexing pipeline behind deprecation shims The new SDK copied the legacy indexing pipeline into pageindex/index/ instead of moving it, leaving two divergent copies of page_index.py / page_index_md.py / utils.py. They had already drifted (the legacy copy still compared IndexConfig booleans against 'yes' — a separate fix), and every pipeline change had to be applied twice. Make pageindex/index/ the single source of truth (same pattern as the LegacyCloudAPI shim for the 0.2.x cloud SDK): - pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes (get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers, ...) so it's the sole utils module. Reconciled the diverged funcs: kept the modern versions, backported the #331 get_leaf_nodes .get() fix, and restored remove_fields' max_len parameter (superset). - index/page_index*.py now import `from .utils import *`; index/legacy_utils.py (a re-export of the old top-level utils) deleted. - Top-level page_index.py / page_index_md.py / utils.py become thin re-export shims that emit PendingDeprecationWarning. The md_to_tree shim coerces legacy 'yes'/'no' string flags to bool (the canonical version is boolean-typed). - ConfigLoader no longer reads the deleted config.yaml; it builds defaults from IndexConfig (was an unconditional FileNotFoundError). - __init__.py and retrieve.py import from pageindex.index.* directly so `import pageindex` does not trip the shims. Adds tests/test_legacy_shims.py pinning the contract: clean top-level import doesn't warn, legacy submodule imports warn, symbols still resolve, shim and canonical share one implementation, the #331 fix and ConfigLoader-without-yaml both hold, and the md_to_tree coercion works. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-07 10:41:29 +08:00
# Upstream exports (backward compatibility). Import from the canonical
# pageindex.index.* modules directly so `import pageindex` does NOT trip the
# top-level deprecation shims (pageindex.page_index / .page_index_md / .utils).
from .index.page_index import * # noqa: E402
refactor(index): dedupe the copied indexing pipeline behind deprecation shims The new SDK copied the legacy indexing pipeline into pageindex/index/ instead of moving it, leaving two divergent copies of page_index.py / page_index_md.py / utils.py. They had already drifted (the legacy copy still compared IndexConfig booleans against 'yes' — a separate fix), and every pipeline change had to be applied twice. Make pageindex/index/ the single source of truth (same pattern as the LegacyCloudAPI shim for the 0.2.x cloud SDK): - pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes (get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers, ...) so it's the sole utils module. Reconciled the diverged funcs: kept the modern versions, backported the #331 get_leaf_nodes .get() fix, and restored remove_fields' max_len parameter (superset). - index/page_index*.py now import `from .utils import *`; index/legacy_utils.py (a re-export of the old top-level utils) deleted. - Top-level page_index.py / page_index_md.py / utils.py become thin re-export shims that emit PendingDeprecationWarning. The md_to_tree shim coerces legacy 'yes'/'no' string flags to bool (the canonical version is boolean-typed). - ConfigLoader no longer reads the deleted config.yaml; it builds defaults from IndexConfig (was an unconditional FileNotFoundError). - __init__.py and retrieve.py import from pageindex.index.* directly so `import pageindex` does not trip the shims. Adds tests/test_legacy_shims.py pinning the contract: clean top-level import doesn't warn, legacy submodule imports warn, symbols still resolve, shim and canonical share one implementation, the #331 fix and ConfigLoader-without-yaml both hold, and the md_to_tree coercion works. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-07 10:41:29 +08:00
from .index.page_index_md import md_to_tree
from .retrieve import get_document, get_document_structure, get_page_content
# SDK exports
from .client import PageIndexClient, LocalClient, CloudClient
from .config import IndexConfig, set_llm_params
from .collection import Collection
from .types import DocumentInfo, DocumentDetail, PageContent
from .parser.protocol import ContentNode, ParsedDocument, DocumentParser
from .storage.protocol import StorageEngine
from .events import QueryEvent
from .errors import (
PageIndexError,
2026-05-11 21:06:23 +08:00
PageIndexAPIError,
CollectionNotFoundError,
DocumentNotFoundError,
IndexingError,
CloudAPIError,
FileTypeError,
)
__all__ = [
"PageIndexClient",
"LocalClient",
"CloudClient",
"IndexConfig",
"set_llm_params",
"Collection",
"DocumentInfo",
"DocumentDetail",
"PageContent",
"ContentNode",
"ParsedDocument",
"DocumentParser",
"StorageEngine",
"QueryEvent",
"PageIndexError",
2026-05-11 21:06:23 +08:00
"PageIndexAPIError",
"CollectionNotFoundError",
"DocumentNotFoundError",
"IndexingError",
"CloudAPIError",
"FileTypeError",
fix: address PR #272 review findings (directly-fixable items) Verified against current dev; the compat/behavior decisions (#7 api_key semantics, #10 CLI flags, #11 doc-description default) are deferred. Crashes: - page_index(): snapshot args before importing IndexConfig — locals() was capturing the imported class and IndexConfig(extra='forbid') made every call raise ValidationError. - process_none_page_numbers: pop('page', None) instead of del (a TOC item without 'page' raised KeyError mid-pipeline). - pipeline._run_async: guard only the loop detection, not the run, so a real RuntimeError from the coroutine isn't masked as "asyncio.run() cannot be called from a running event loop". Silent-wrong / robustness: - LocalBackend.get_document_structure and the agent get_document / get_document_structure tools now surface a missing doc (raise / error-JSON) instead of returning empty, matching get_page_content and the cloud backend. - cloud delete_collection drops the cached folder_id. - cloud query raises on an empty collection instead of POSTing doc_id:[]. - LocalClient skips the API-key check for keyless providers (ollama, lm_studio, …) so keyless LiteLLM models aren't rejected at construction. Compat / cleanup: - md_to_tree coerces legacy 'yes'/'no' string flags (a bare 'no' was truthy). - FileTypeError also subclasses ValueError (0.2.x raised ValueError). - _validate_llm_provider no longer mutates global litellm.model_cost_map_url. - __all__ re-includes legacy exports (page_index, md_to_tree, get_*). - Rewrite examples/agentic_vectorless_rag_demo.py to the Collection API and use the in-repo attention.pdf (the old workspace=/client.index/client.documents API no longer exists). Adds tests/test_review_fixes.py (10 regressions). Full suite: 189 passed. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-08 18:56:51 +08:00
# Legacy top-level exports (pre-SDK API), kept so `from pageindex import *`
# still binds them.
"page_index",
"md_to_tree",
"get_document",
"get_document_structure",
"get_page_content",
]