Verified 12 findings from an xhigh-effort review of the prior review-fix
batch; all confirmed real. Most trace back to one root cause: the
build_index() text-stripping fix (8f536cb) correctly stopped Markdown from
leaking full text by default, but broke every path that assumed text could
be re-read later.
Correctness:
- LocalBackend._fill_node_text (get_document(include_text=True)) only handled
PDF's start_index/end_index convention; Markdown nodes use line_num and got
silently empty text. Now handles both.
- get_page_content's Markdown fallback (triggered when a StorageEngine
legitimately returns None from get_pages()) read from the now-text-stripped
structure. It now re-derives from the source file, mirroring the PDF
fallback, so it no longer depends on structure text at all.
- add_document's PDF-only text-stripping branch (with the stale "markdown
needs text in structure for fallback retrieval" comment) is now dead/wrong
since build_index() already applies if_add_node_text uniformly — removed.
- _validate_llm_provider's keyless-provider allowlist was missing several
local LiteLLM providers (xinference, llamafile, triton, oobabooga,
openai_like, docker_model_runner, custom, custom_openai, petals) that need
no API key just like ollama/lm_studio; expanded.
- The three agent-tool closures (get_document, get_document_structure,
get_page_content) had three different not-found patterns; two bypassed the
backend's DocumentNotFoundError entirely. Extracted LocalBackend.
_require_document as the single existence check every method/tool now uses.
- examples/agentic_vectorless_rag_demo.py's hand-rolled Agent() didn't apply
the litellm/ prefix normalization the SDK does internally, so its own
documented "any LiteLLM provider" claim broke for non-openai models.
- cloud delete_collection's cache eviction removed the "folders unavailable"
None sentinel too, forcing a wasted re-fetch; now only pops on a real id.
Cleanup / altitude:
- build_index() skips the remove_structure_text walk entirely when text was
never added (content_based + if_add_node_summary=False + if_add_node_text=
False) instead of a guaranteed no-op tree walk.
- page_index()'s locals()-capture-as-kwargs (fragile by construction) replaced
with an explicit dict of the named parameters.
- run_pageindex.py's _cli_bool and the page_index_md.py legacy shim's
_coerce_bool were duplicate, diverging implementations; both now bind
directly to the canonical pageindex.index.page_index_md._coerce_bool.
- retrieve.py's _get_md_page_content delegated its own traversal instead of
calling the canonical get_md_page_content; now a one-line delegation.
- FileTypeError's docstring now calls out the except-ordering gotcha from
also subclassing ValueError.
17 new regression tests (tests/test_review_fixes_2.py) plus 2 updated in
tests/test_legacy_shims.py for the simplified md_to_tree shim. Full suite:
210 passed, 2 skipped.
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
Resolves PR #272 review #10/P5. The --if-add-node-id / -node-summary /
-doc-description / -node-text args were store_true, which rejected the
documented yes/no values and left default-on options impossible to disable
from the CLI. They now use nargs='?' + const=True + a yes/no-coercing type:
--if-add-node-id -> on
--if-add-node-id no -> off (legacy form still works)
(omitted) -> use the IndexConfig default
README updated to the flag usage (noting the legacy `no` off-switch), and
--if-add-node-text is now documented too.
Decisions from the review:
- #7 (api_key semantics): verified FALSE POSITIVE — 0.2.x is a cloud SDK whose
api_key is a PageIndex cloud key (cloud_api.LegacyCloudAPI + docs.pageindex.ai/sdk),
matching the new SDK. No change.
- #11 (if_add_doc_description default True): kept intentionally (open mode).
Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
* Integrate litellm for multi-provider LLM support
* recover the default config yaml
* Use litellm.acompletion for native async support
* fix tob
* Rename llm_complete/allm_complete to llm_completion/llm_acompletion, remove unused llm_complete_stream
* Pin litellm to version 1.82.0
* resolve comments
* args from cli is used to overrides config.yaml
* Fix get_page_tokens hardcoded model default
Pass opt.model to get_page_tokens so tokenization respects the
configured model instead of always using gpt-4o-2024-11-20.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Remove explicit openai dependency from requirements.txt
openai is no longer directly imported; it comes in as a transitive
dependency of litellm. Pinning it explicitly risks version conflicts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Restore openai==1.101.0 pin in requirements.txt
litellm==1.82.0 and openai-agents have conflicting openai version
requirements, but openai==1.101.0 works at runtime for both.
The pin is necessary to prevent litellm from pulling in openai>=2.x
which would break openai-agents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Remove explicit openai dependency from requirements.txt
openai is not directly used; it comes in as a transitive dependency
of litellm. No openai-agents in this branch so no pin needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix an litellm error log
* resolve comments
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
