The prompt-injection hardening ported from main did not fit dev's direction
(graceful degradation + content-faithful, reasoning-based index). Adjust it:
- process_toc_no_page_numbers: a count-mismatched or reordered/renamed LLM
response no longer raises ValueError (which aborted the entire index at the
uncaught top-level path and defeated the return_exceptions degradation the
rest of the pipeline uses). Skip the untrusted chunk and continue; the
accuracy check falls back to another mode when too little gets filled.
- _secure_doc_text: drop the keyword blocklist that replaced phrases like
"act as"/"disregard" with [REDACTED]. Those occur in legitimate titles and
prose, so redaction corrupted document content. Keep the <user_document>
framing + _SYSTEM_HARDENING system instruction as the injection defense.
- _validate_chunk_physical_indices: tolerate non-list input (extract_json
returns {} on parse failure and may return a JSON object) instead of
crashing while iterating a dict.
- Remove _validate_physical_indices / _parse_physical_index: range validation
is already done by validate_and_truncate_physical_indices in meta_processor
and marker->int by convert_physical_index_to_int; the helpers duplicated both.
Tests updated to assert the skip-not-raise behavior and lock in no-redaction
and non-list tolerance. 314 passed, 2 skipped.
main's changes landed on the top-level pageindex/page_index.py and
page_index_md.py, which are deprecation shims on dev (the pipeline now
lives in pageindex/index/). Resolving the conflict by keeping the shims
would silently drop main's work, so port it into index/ instead:
- page_index.py: prompt-injection hardening (_SYSTEM_HARDENING,
_secure_doc_text and the redact/wrap helpers) applied to every prompt
builder, plus physical_index validation (_validate_physical_indices,
_validate_chunk_physical_indices) wired into toc_index_extractor,
process_no_toc and process_toc_no_page_numbers, and TOC-identity
validation (reject reordered/modified/count-mismatched LLM output).
Combined with dev's existing robustness edits in the same functions.
- page_index_md.py: recognize whole-line **bold** as a level-1 heading,
skip empty bold headings, use the stored node level.
- tests: retarget the ported tests at pageindex.index.* instead of the
shim (underscore helpers aren't re-exported by the shim's `import *`,
and patching the shim wouldn't intercept the real pipeline calls).
311 passed, 2 skipped.
