2026-04-06 22:51:04 +08:00
|
|
|
# pageindex/__init__.py
|
refactor(index): dedupe the copied indexing pipeline behind deprecation shims
The new SDK copied the legacy indexing pipeline into pageindex/index/
instead of moving it, leaving two divergent copies of page_index.py /
page_index_md.py / utils.py. They had already drifted (the legacy copy
still compared IndexConfig booleans against 'yes' — a separate fix),
and every pipeline change had to be applied twice.
Make pageindex/index/ the single source of truth (same pattern as the
LegacyCloudAPI shim for the 0.2.x cloud SDK):
- pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes
(get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers,
...) so it's the sole utils module. Reconciled the diverged funcs:
kept the modern versions, backported the #331 get_leaf_nodes .get()
fix, and restored remove_fields' max_len parameter (superset).
- index/page_index*.py now import `from .utils import *`;
index/legacy_utils.py (a re-export of the old top-level utils) deleted.
- Top-level page_index.py / page_index_md.py / utils.py become thin
re-export shims that emit PendingDeprecationWarning. The md_to_tree
shim coerces legacy 'yes'/'no' string flags to bool (the canonical
version is boolean-typed).
- ConfigLoader no longer reads the deleted config.yaml; it builds
defaults from IndexConfig (was an unconditional FileNotFoundError).
- __init__.py and retrieve.py import from pageindex.index.* directly so
`import pageindex` does not trip the shims.
Adds tests/test_legacy_shims.py pinning the contract: clean top-level
import doesn't warn, legacy submodule imports warn, symbols still
resolve, shim and canonical share one implementation, the #331 fix and
ConfigLoader-without-yaml both hold, and the md_to_tree coercion works.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-07 10:41:29 +08:00
|
|
|
# Upstream exports (backward compatibility). Import from the canonical
|
|
|
|
|
# pageindex.index.* modules directly so `import pageindex` does NOT trip the
|
|
|
|
|
# top-level deprecation shims (pageindex.page_index / .page_index_md / .utils).
|
|
|
|
|
from .index.page_index import *
|
|
|
|
|
from .index.page_index_md import md_to_tree
|
2026-03-26 23:19:50 +08:00
|
|
|
from .retrieve import get_document, get_document_structure, get_page_content
|
2026-04-06 22:51:04 +08:00
|
|
|
|
|
|
|
|
# SDK exports
|
|
|
|
|
from .client import PageIndexClient, LocalClient, CloudClient
|
2026-06-25 16:47:27 +08:00
|
|
|
from .config import IndexConfig, set_llm_params
|
2026-04-06 22:51:04 +08:00
|
|
|
from .collection import Collection
|
refactor(sdk): typed returns, protocol contract, parser layering
Engineering-quality cleanups from the SDK review (no behavior change):
- Return-type discoverability: add pageindex/types.py with TypedDicts
(DocumentInfo, DocumentDetail, PageContent) and annotate Collection /
Backend methods with them; add docstrings to every public Collection
method (including the get_page_content `pages` spec). Exported from the
package. Zero runtime cost — these are plain dicts.
- Backend protocol as a real contract:
* query_stream is an async generator, so the protocol now declares it
as `def ... -> AsyncIterator[QueryEvent]` (not `async def`, which
typed it as a coroutine and never matched the implementations).
* custom-parser support is expressed as a runtime_checkable
SupportsParserRegistration capability protocol; the client uses
isinstance(...) instead of hasattr(...) duck-typing.
- Parser layering: move count_tokens into a leaf module pageindex/tokens.py
so parser/* imports it from there instead of reaching back into
pageindex.index (a reverse dependency). index.utils re-exports it for
backward compatibility.
Adds tests/test_architecture.py enforcing: parser never imports index,
count_tokens is a single shared leaf, the capability protocol works,
both backends satisfy Backend, and the TypedDicts are exported.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-07 12:15:34 +08:00
|
|
|
from .types import DocumentInfo, DocumentDetail, PageContent
|
2026-04-06 22:51:04 +08:00
|
|
|
from .parser.protocol import ContentNode, ParsedDocument, DocumentParser
|
|
|
|
|
from .storage.protocol import StorageEngine
|
|
|
|
|
from .events import QueryEvent
|
|
|
|
|
from .errors import (
|
|
|
|
|
PageIndexError,
|
2026-05-11 21:06:23 +08:00
|
|
|
PageIndexAPIError,
|
2026-04-06 22:51:04 +08:00
|
|
|
CollectionNotFoundError,
|
|
|
|
|
DocumentNotFoundError,
|
|
|
|
|
IndexingError,
|
|
|
|
|
CloudAPIError,
|
|
|
|
|
FileTypeError,
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
__all__ = [
|
|
|
|
|
"PageIndexClient",
|
|
|
|
|
"LocalClient",
|
|
|
|
|
"CloudClient",
|
|
|
|
|
"IndexConfig",
|
2026-06-25 16:47:27 +08:00
|
|
|
"set_llm_params",
|
2026-04-06 22:51:04 +08:00
|
|
|
"Collection",
|
refactor(sdk): typed returns, protocol contract, parser layering
Engineering-quality cleanups from the SDK review (no behavior change):
- Return-type discoverability: add pageindex/types.py with TypedDicts
(DocumentInfo, DocumentDetail, PageContent) and annotate Collection /
Backend methods with them; add docstrings to every public Collection
method (including the get_page_content `pages` spec). Exported from the
package. Zero runtime cost — these are plain dicts.
- Backend protocol as a real contract:
* query_stream is an async generator, so the protocol now declares it
as `def ... -> AsyncIterator[QueryEvent]` (not `async def`, which
typed it as a coroutine and never matched the implementations).
* custom-parser support is expressed as a runtime_checkable
SupportsParserRegistration capability protocol; the client uses
isinstance(...) instead of hasattr(...) duck-typing.
- Parser layering: move count_tokens into a leaf module pageindex/tokens.py
so parser/* imports it from there instead of reaching back into
pageindex.index (a reverse dependency). index.utils re-exports it for
backward compatibility.
Adds tests/test_architecture.py enforcing: parser never imports index,
count_tokens is a single shared leaf, the capability protocol works,
both backends satisfy Backend, and the TypedDicts are exported.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-07 12:15:34 +08:00
|
|
|
"DocumentInfo",
|
|
|
|
|
"DocumentDetail",
|
|
|
|
|
"PageContent",
|
2026-04-06 22:51:04 +08:00
|
|
|
"ContentNode",
|
|
|
|
|
"ParsedDocument",
|
|
|
|
|
"DocumentParser",
|
|
|
|
|
"StorageEngine",
|
|
|
|
|
"QueryEvent",
|
|
|
|
|
"PageIndexError",
|
2026-05-11 21:06:23 +08:00
|
|
|
"PageIndexAPIError",
|
2026-04-06 22:51:04 +08:00
|
|
|
"CollectionNotFoundError",
|
|
|
|
|
"DocumentNotFoundError",
|
|
|
|
|
"IndexingError",
|
|
|
|
|
"CloudAPIError",
|
|
|
|
|
"FileTypeError",
|
|
|
|
|
]
|