merge upstream/main into dev; port #188 into the index pipeline

main advanced (litellm 1.84.0 #342, #188 TOC fixes, #281, README) while dev turned pageindex/page_index.py and utils.py into deprecation shims over pageindex/index/*. Both sides touched those two files, hence the conflict. Resolution: - Keep dev's shims for the two top-level modules (the real implementation lives in pageindex/index/*). requirements.txt auto-merged to litellm==1.84.0. - #188 ("prevent KeyError crash and context exhaustion in TOC processing") landed on main's top-level page_index.py, which is now a shim on dev — so its fixes were NOT in dev's index/page_index.py. Ported them into pageindex/index/page_index.py (preserving dev's IndexConfig/bool integration): .get() on the TOC check functions + detect_page_index, incremental-chat_history retry loops in extract_toc_content and toc_transformer, truncation moved before the loop, .get('table_of_contents', []) and the single_toc_item_index_fixer None guard. - Repoint #188's merged test (tests/test_issue_163.py) at pageindex.index.page_index so its patches hit the module where the code now lives (they were silently hitting the shim → real LLM calls). Full suite: 158 passed, 2 skipped. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
2026-07-15 21:11:05 +02:00 · 2026-07-07 16:34:02 +08:00 · 2026-07-07 16:34:02 +08:00 · a139bd53d9
commit a139bd53d9
parent 4a6a948ac2 f413c66fee
10 changed files with 287 additions and 82 deletions
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@ -0,0 +1,9 @@
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+    labels:
+      - dependencies
+      - github-actions
--- a/.github/workflows/autoclose-labeled-issues.yml
+++ b/.github/workflows/autoclose-labeled-issues.yml
@ -26,7 +26,7 @@ jobs:
    timeout-minutes: 10
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@v7

      - name: Close inactive duplicate issues
        env:
--- a/.github/workflows/backfill-dedupe.yml
+++ b/.github/workflows/backfill-dedupe.yml
@ -21,7 +21,7 @@ jobs:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7

      - name: Fetch issues and run dedupe
        env:
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@ -0,0 +1,34 @@
+name: CodeQL
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  schedule:
+    - cron: '37 9 * * 1'
+
+permissions:
+  security-events: write
+  contents: read
+  actions: read
+
+jobs:
+  analyze:
+    name: Analyze (actions)
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v7
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v4
+        with:
+          languages: actions
+          build-mode: none
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v4
+        with:
+          category: /language:actions
--- a/.github/workflows/dependency-review.yml
+++ b/.github/workflows/dependency-review.yml
@ -0,0 +1,22 @@
+name: Dependency Review
+
+on:
+  pull_request:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  dependency-review:
+    name: Dependency Review
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v7
+
+      - name: Dependency Review
+        uses: actions/dependency-review-action@v4
+        with:
+          fail-on-severity: moderate
--- a/.github/workflows/issue-dedupe.yml
+++ b/.github/workflows/issue-dedupe.yml
@ -31,7 +31,7 @@ jobs:
       !endsWith(github.actor, '[bot]') &&
       github.actor != 'github-actions')
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7

      - name: Determine issue number
        id: issue
--- a/README.md
+++ b/README.md
@ -13,10 +13,10 @@

 # PageIndex: Vectorless, Reasoning-based RAG

-<p align="center"><b>Reasoning-based RAG&nbsp; ◦ &nbsp;No Vector DB&nbsp; ◦ &nbsp;No Chunking&nbsp; ◦ &nbsp;Human-like Retrieval</b></p>
+<p align="center"><b>Reasoning-based RAG&nbsp; ◦ &nbsp;No Vector DB, No Chunking&nbsp; ◦ &nbsp;Context-Aware Retrieval&nbsp; ◦ &nbsp;Reads Like a Human</b></p>

 <h4 align="center">
-  <a href="https://vectify.ai">🌐 Homepage</a>&nbsp; • &nbsp;
+  <a href="https://vectify.ai">🌐 Website</a>&nbsp; • &nbsp;
  <a href="https://chat.pageindex.ai">🖥️ Chat Platform</a>&nbsp; • &nbsp;
  <a href="https://pageindex.ai/developer">🔌 MCP & API</a>&nbsp; • &nbsp;
  <a href="https://docs.pageindex.ai">📖 Docs</a>&nbsp; • &nbsp;
@ -30,9 +30,10 @@
 <details open>
 <summary><h2>📢 Updates</h2></summary>

- 🔥 [**Agentic Vectorless RAG**](https://github.com/VectifyAI/PageIndex/blob/main/examples/agentic_vectorless_rag_demo.py) — A simple *agentic, vectorless RAG* [example](https://github.com/VectifyAI/PageIndex/blob/main/examples/agentic_vectorless_rag_demo.py) with self-hosted PageIndex, using OpenAI Agents SDK.
+- 🔥 [**Agentic Vectorless RAG**](https://github.com/VectifyAI/PageIndex/blob/main/examples/agentic_vectorless_rag_demo.py) — A simple agentic, vectorless RAG [example](#-agentic-vectorless-rag-an-example) with *self-hosted PageIndex*, using OpenAI Agents SDK.
+- [**Scale PageIndex to Millions of Documents**](https://pageindex.ai/blog/pageindex-filesystem) — *PageIndex File System* is a file-level tree indexing layer that lets PageIndex reason over an entire corpus, not just a single document, enabling massive-scale document search.
 - [PageIndex Chat](https://chat.pageindex.ai) — Human-like document analysis agent [platform](https://chat.pageindex.ai) for professional long documents. Also available via [MCP](https://pageindex.ai/developer) or [API](https://pageindex.ai/developer).
- [PageIndex Framework](https://pageindex.ai/blog/pageindex-intro) — Deep dive into PageIndex: an *agentic, in-context tree index* that enables LLMs to perform *reasoning-based, human-like retrieval* over long documents.
+- [PageIndex Framework](https://pageindex.ai/blog/pageindex-intro) — Deep dive into PageIndex: an *agentic, in-context tree index* that enables LLMs to perform *reasoning-based, context-aware retrieval* over long documents.

 <!-- **🧪 Cookbooks:**
 - [Vectorless RAG](https://docs.pageindex.ai/cookbook/vectorless-rag-pageindex): A minimal, hands-on example of reasoning-based RAG using PageIndex. No vectors, no chunking, and human-like retrieval.
@ -44,13 +45,13 @@

 # 📑 Introduction to PageIndex

-Are you frustrated with vector database retrieval accuracy for long professional documents? Traditional vector-based RAG relies on semantic *similarity* rather than true *relevance*. But **similarity ≠ relevance** — what we truly need in retrieval is **relevance**, and that requires **reasoning**. When working with professional documents that demand domain expertise and multi-step reasoning, similarity search often falls short.
+Are you frustrated with vector database retrieval accuracy for long professional documents? Traditional vector-based RAG relies on semantic *similarity* rather than true *relevance*. But **similarity ≠ relevance** — what we truly need in retrieval is **relevance**, and that requires **reasoning**. When working with professional documents that demand *contextual understanding*, domain expertise, and multi-step reasoning, similarity search often falls short — missing what's relevant but not similar, and returning what's similar yet not relevant.

-Inspired by AlphaGo, we propose **[PageIndex](https://vectify.ai/pageindex)** — a **vectorless**, **reasoning-based RAG** system that builds a **hierarchical tree index** from long documents and uses LLMs to **reason** *over that index* for **agentic, context-aware retrieval**.
-It simulates how *human experts* navigate and extract knowledge from complex documents through *tree search*, enabling LLMs to *think* and *reason* their way to the most relevant document sections. PageIndex performs retrieval in two steps:
+Inspired by AlphaGo, we propose **[PageIndex](https://vectify.ai/pageindex)** — a **vectorless**, **reasoning-based RAG** system that builds a **hierarchical tree index** from long documents, and uses LLMs to **reason** *over that index* for **agentic, context-aware retrieval**. The retrieval is *traceable* and *explainable*, with no vector DBs or chunking.
+PageIndex simulates how *human experts* navigate and extract knowledge from complex documents through *tree search*, enabling LLMs to *think* and *reason* their way to the most relevant document sections. It performs retrieval in two steps:

 1. Generate a “Table-of-Contents” **tree structure index** of documents
-2. Perform reasoning-based retrieval through **tree search**
+2. Perform (agentic) reasoning-based retrieval through **tree search**

 <div align="center">
  <a href="https://pageindex.ai/blog/pageindex-intro" target="_blank" title="The PageIndex Framework">
@ -60,28 +61,31 @@ It simulates how *human experts* navigate and extract knowledge from complex doc

 ### 🎯 Core Features

+> PageIndex is a vectorless, reasoning-based RAG engine that mirrors how humans read, delivering traceable, explainable, and context-aware retrieval, without vector databases or chunking.
+
 Compared to traditional vector-based RAG, **PageIndex** features:
 - **No Vector DB**: Uses document structure and LLM reasoning for retrieval, instead of vector similarity search.
 - **No Chunking**: Documents are organized into natural sections, not artificial chunks.
- **Human-like Retrieval**: Simulates how human experts navigate and extract knowledge from complex documents.
- **Better Explainability and Traceability**: Retrieval is based on reasoning — traceable and interpretable, with page and section references. No more opaque, approximate vector search (“vibe retrieval”).
+- **Better Traceability & Explainability**: Retrieval is reasoning-driven and grounded in explicit page and section references, making every result traceable and interpretable — no more “vibe retrieval” with opaque, approximate vector search.
+- **Context-Aware Retrieval**: Retrieval depends on your full context (e.g., conversation history and domain knowledge), and easily incorporates new context.
+- **Human-like Retrieval**: Mirrors how human experts navigate and extract knowledge from complex documents.

-PageIndex powers a reasoning-based RAG system that achieved **state-of-the-art** [98.7% accuracy](https://github.com/VectifyAI/Mafin2.5-FinanceBench) on FinanceBench, demonstrating superior performance over vector-based RAG solutions in professional document analysis. See our [blog post](https://vectify.ai/blog/Mafin2.5) for details.
+PageIndex achieved **state-of-the-art** [98.7% accuracy](https://github.com/VectifyAI/Mafin2.5-FinanceBench) on FinanceBench (financial document QA benchmark), vastly outperforming vector RAG solutions on professional document analysis ([blog post](https://vectify.ai/blog/Mafin2.5)).

 ### 📍 Explore PageIndex

-To learn more, please see a detailed introduction to the [PageIndex framework](https://pageindex.ai/blog/pageindex-intro). Check out this GitHub repo for open-source code, and the [cookbooks](https://docs.pageindex.ai/cookbook), [tutorials](https://docs.pageindex.ai/tutorials), and [blog](https://pageindex.ai/blog) for additional usage guides and examples.
+To learn more, please see a detailed introduction to the [PageIndex framework](https://pageindex.ai/blog/pageindex-intro). Check out [our GitHub](https://docs.pageindex.ai/open-source) for open-source code, and the [cookbooks](https://docs.pageindex.ai/cookbook), [tutorials](https://docs.pageindex.ai/tutorials), and [blog](https://pageindex.ai/blog) for more usage guides and examples.

-The PageIndex service is available as a ChatGPT-style [chat platform](https://chat.pageindex.ai), or can be integrated via [MCP](https://pageindex.ai/developer) or [API](https://pageindex.ai/developer).
+The PageIndex service is available as a ChatGPT-style [chat platform](https://chat.pageindex.ai), or can be integrated via [MCP](https://pageindex.ai/developer) or [API](https://pageindex.ai/developer), with [enterprise](https://pageindex.ai/enterprise) deployment available.

 ### 🛠️ Deployment Options
- Self-host — run locally with this open-source repo.
- Cloud Service — try instantly with our [Chat Platform](https://chat.pageindex.ai/), or integrate via [MCP](https://pageindex.ai/developer) or [API](https://pageindex.ai/developer).
- _Enterprise_ — private or on-prem deployment. [Contact us](https://ii2abc2jejf.typeform.com/to/tK3AXl8T) or [book a demo](https://calendly.com/pageindex/meet) for more details.
+- **Self-host** — run locally with this open-source repo (using standard PDF parsing).
+- **Cloud Service** — production-grade pipeline with enhanced OCR, tree building, and retrieval for best results. Try instantly on our [Chat Platform](https://chat.pageindex.ai/), or integrate via [MCP](https://pageindex.ai/developer) or [API](https://pageindex.ai/developer).
+- **Enterprise** — dedicated or private deployment (VPC, on-prem). [Contact us](https://ii2abc2jejf.typeform.com/to/gVv7qkaN) or [book a demo](https://calendly.com/pageindex/meet) to learn more.

 ### 🧪 Quick Hands-on

- 🔥 [**Agentic Vectorless RAG**](examples/agentic_vectorless_rag_demo.py) (**latest**) — a simple but complete **agentic vectorless RAG** [example](https://github.com/VectifyAI/PageIndex/blob/main/examples/agentic_vectorless_rag_demo.py) with *self-hosted* PageIndex, using OpenAI Agents SDK.
+- 🔥 [**Agentic Vectorless RAG**](examples/agentic_vectorless_rag_demo.py) *(latest)* — a simple but complete **agentic vectorless RAG** [example](#-agentic-vectorless-rag-an-example) with *self-hosted* PageIndex, using OpenAI Agents SDK.
 - Try the [Vectorless RAG](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb) notebook — a *minimal*, hands-on example of reasoning-based RAG using PageIndex.
 - Check out [Vision-based Vectorless RAG](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb) — no OCR; a minimal, vision-based & reasoning-native RAG pipeline that works directly over page images.
  
@ -103,7 +107,7 @@ The PageIndex service is available as a ChatGPT-style [chat platform](https://ch

 # 🌲 PageIndex Tree Structure

-PageIndex can transform lengthy PDF documents into a semantic **tree structure**, similar to a _"table of contents"_ but optimized for use with Large Language Models (LLMs). It's ideal for: financial reports, regulatory filings, academic textbooks, legal or technical manuals, and any document that exceeds LLM context limits.
+PageIndex can transform lengthy PDF documents into a semantic **tree structure**, similar to a _“table of contents”_ but optimized for use with LLMs and AI agents. It's ideal for: financial reports, legal documents, regulatory filings, technical manuals, medical literature, academic textbooks, and any long, complex professional documents.

 Below is an example PageIndex tree structure. Also see more example [documents](https://github.com/VectifyAI/PageIndex/tree/main/examples/documents) and generated [tree structures](https://github.com/VectifyAI/PageIndex/tree/main/examples/documents/results).

@ -135,7 +139,7 @@ Below is an example PageIndex tree structure. Also see more example [documents](
 ...
 ```

-You can generate the PageIndex tree structure with this open-source repo, or use our [API](https://pageindex.ai/developer).
+You can generate PageIndex tree structures with this open-source repo. Or use our [API](https://pageindex.ai/developer) for higher-quality results powered by our enhanced OCR and tree building pipeline.

 ---

@ -200,6 +204,8 @@ Omitting `doc_ids` queries the **entire collection** and lets the agent pick whi

 # ⚙️ Package Usage

+> **Note:** This package uses standard PDF parsing. For use cases with complex PDFs, our [cloud service](https://pageindex.ai/developer) (via MCP and API) offers enhanced OCR, tree building, and retrieval.
+
 You can follow these steps to generate a PageIndex tree from a PDF document.

 ### 1. Install dependencies
@ -210,7 +216,7 @@ pip3 install --upgrade -r requirements.txt

 ### 2. Set your LLM API key

-Create a `.env` file in the root directory with your LLM API key, with multi-LLM support via [LiteLLM](https://docs.litellm.ai/docs/providers):
+Create a `.env` file in the root directory with your LLM API key. Multi-LLM is supported via [LiteLLM](https://docs.litellm.ai/docs/providers):

 ```bash
 OPENAI_API_KEY=your_openai_key_here
@ -247,12 +253,12 @@ We also provide markdown support for PageIndex. You can use the `--md_path` flag
 python3 run_pageindex.py --md_path /path/to/your/document.md
 ```

-> Note: in this mode, we use "#" to determine node headings and their levels. For example, "##" is level 2, "###" is level 3, etc. Make sure your markdown file is formatted correctly. If your Markdown file was converted from a PDF or HTML, we don't recommend using this mode, since most existing conversion tools cannot preserve the original hierarchy. Instead, use our [PageIndex OCR](https://pageindex.ai/blog/ocr), which is designed to preserve the original hierarchy, to convert the PDF to a markdown file and then use this mode.
+> Note: in this mode, we use "#" to determine node headings and their levels. For example, "##" is level 2, "###" is level 3, etc. Make sure your markdown file is formatted correctly. If your Markdown file was converted from a PDF or HTML, we don't recommend using this mode, since most existing conversion tools cannot preserve the original hierarchy. Instead, use our [PageIndex OCR](https://pageindex.ai/blog/ocr), which is designed to preserve it, to convert the PDF to a markdown file and then use this mode.
 </details>

-## Agentic Vectorless RAG: An Example
+## 🚀 Agentic Vectorless RAG: An Example

-For a simple, end-to-end _**agentic vectorless RAG**_ example using PageIndex with OpenAI Agents SDK, see [`examples/agentic_vectorless_rag_demo.py`](examples/agentic_vectorless_rag_demo.py).
+For a simple, end-to-end **agentic vectorless RAG** example using **self-hosted PageIndex** (with OpenAI Agents SDK), see [`examples/agentic_vectorless_rag_demo.py`](examples/agentic_vectorless_rag_demo.py).

 ```bash
 # Install optional dependency
@ -281,7 +287,7 @@ To address this, we introduced PageIndex OCR — the first long-context OCR mode

 # 📈 Case Study: PageIndex Leads Finance QA Benchmark

-[Mafin 2.5](https://vectify.ai/mafin) is a reasoning-based RAG system for financial document analysis, powered by **PageIndex**. It achieved a state-of-the-art [**98.7% accuracy**](https://vectify.ai/blog/Mafin2.5) on the [FinanceBench](https://arxiv.org/abs/2311.11944) benchmark, significantly outperforming traditional vector-based RAG systems.
+[Mafin 2.5](https://vectify.ai/mafin) is a reasoning-based RAG system for financial document analysis, powered by **PageIndex**. It achieved a state-of-the-art [**98.7% accuracy**](https://vectify.ai/blog/Mafin2.5) on [FinanceBench](https://arxiv.org/abs/2311.11944) (financial document QA benchmark), significantly outperforming traditional vector-based RAG systems.

 PageIndex's hierarchical indexing and reasoning-driven retrieval enable precise navigation and extraction of relevant context from complex financial reports, such as SEC filings and earnings disclosures.

@ -335,14 +341,20 @@ PageIndex Blog, Sep 2025.
 </details>


+### 🌐 Open-Source Ecosystem
+
+[PageIndex](https://github.com/VectifyAI/PageIndex) anchors a growing open-source [ecosystem](https://docs.pageindex.ai/open-source) of **long-context AI infra** — [OpenKB](https://github.com/VectifyAI/OpenKB) is an LLM knowledge base that compiles documents into an interlinked wiki. [ChatIndex](https://github.com/VectifyAI/ChatIndex) provides tree indexing and retrieval for long conversational histories and memory. [ConDB](https://github.com/VectifyAI/ConDB) is a KV-cache native context database for tree-based retrieval at scale. [PageIndex MCP](https://github.com/VectifyAI/pageindex-mcp) is PageIndex's MCP server.
+
 ### Connect with Us

 <div align="center">

-[![Twitter](https://img.shields.io/badge/Twitter-000000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/PageIndexAI)&ensp;
-[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/company/vectify-ai/)&ensp;
-[![Discord](https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.com/invite/VuXuf29EUj)&ensp;
-[![Contact Us](https://img.shields.io/badge/Contact_Us-3B82F6?style=for-the-badge&logo=envelope&logoColor=white)](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)
+[![Website](https://img.shields.io/badge/Website-2D72CF?style=for-the-badge&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI%2BPHBhdGggZmlsbD0iI2ZmZiIgZD0iTTEyIDEgMSAxMWgyLjV2MTJoNnYtN2g1djdoNlYxMUgyM3oiLz48L3N2Zz4%3D)](https://pageindex.ai)&nbsp;
+[![Twitter](https://img.shields.io/badge/Twitter-000000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/PageIndexAI)&nbsp;
+[![LinkedIn](https://img.shields.io/badge/LinkedIn-0A66C2?style=for-the-badge&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI%2BPHBhdGggZmlsbD0iI2ZmZiIgZD0iTTIwLjQ1IDIwLjQ1aC0zLjU1di01LjU3YzAtMS4zMy0uMDMtMy4wNC0xLjg1LTMuMDQtMS44NSAwLTIuMTQgMS40NS0yLjE0IDIuOTR2NS42N0g5LjM1VjloMy40MXYxLjU2aC4wNWMuNDgtLjkgMS42NC0xLjg1IDMuMzctMS44NSAzLjYgMCA0LjI3IDIuMzcgNC4yNyA1LjQ2djYuMjh6TTUuMzQgNy40M2EyLjA2IDIuMDYgMCAxIDEgMC00LjEzIDIuMDYgMi4wNiAwIDAgMSAwIDQuMTN6TTcuMTIgMjAuNDVIMy41NlY5aDMuNTZ2MTEuNDV6TTIyLjIyIDBIMS43N0MuNzkgMCAwIC43NyAwIDEuNzN2MjAuNTRDMCAyMy4yMy43OSAyNCAxLjc3IDI0aDIwLjQ1QzIzLjIgMjQgMjQgMjMuMjMgMjQgMjIuMjdWMS43M0MyNCAuNzcgMjMuMiAwIDIyLjIyIDB6Ii8%2BPC9zdmc%2B)](https://www.linkedin.com/company/vectify-ai/)&nbsp;
+[![Discord](https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.com/invite/VuXuf29EUj)&nbsp;
+[![Book a Demo](https://img.shields.io/badge/Book_a_Demo-6E7E96?style=for-the-badge&logo=googlecalendar&logoColor=white)](https://calendly.com/pageindex/meet)&nbsp;
+[![Contact Us](https://img.shields.io/badge/Contact_Us-3B82F6?style=for-the-badge&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjIgNCAyMCAxNiI%2BPHBhdGggZmlsbD0iI2ZmZiIgZD0iTTIwIDRINGMtMS4xIDAtMiAuOS0yIDJ2MTJjMCAxLjEuOSAyIDIgMmgxNmMxLjEgMCAyLS45IDItMlY2YzAtMS4xLS45LTItMi0yem0wIDQtOCA1LTgtNVY2bDggNSA4LTV6Ii8%2BPC9zdmc%2B)](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)

 </div>

--- a/pageindex/index/page_index.py
+++ b/pageindex/index/page_index.py
@ -118,8 +118,8 @@ def toc_detector_single_page(content, model=None):

    response = llm_completion(model=model, prompt=prompt)
    # print('response', response)
-    json_content = extract_json(response)    
-    return json_content['toc_detected']
+    json_content = extract_json(response)
+    return json_content.get('toc_detected', 'no')


 def check_if_toc_extraction_is_complete(content, toc, model=None):
@ -137,7 +137,7 @@ def check_if_toc_extraction_is_complete(content, toc, model=None):
    prompt = prompt + '\n Document:\n' + content + '\n Table of contents:\n' + toc
    response = llm_completion(model=model, prompt=prompt)
    json_content = extract_json(response)
-    return json_content['completed']
+    return json_content.get('completed', 'no')


 def check_if_toc_transformation_is_complete(content, toc, model=None):
@ -155,7 +155,7 @@ def check_if_toc_transformation_is_complete(content, toc, model=None):
    prompt = prompt + '\n Raw Table of contents:\n' + content + '\n Cleaned Table of contents:\n' + toc
    response = llm_completion(model=model, prompt=prompt)
    json_content = extract_json(response)
-    return json_content['completed']
+    return json_content.get('completed', 'no')

 def extract_toc_content(content, model=None):
    prompt = f"""
@ -175,27 +175,19 @@ def extract_toc_content(content, model=None):
        {"role": "user", "content": prompt}, 
        {"role": "assistant", "content": response},    
    ]
-    prompt = f"""please continue the generation of table of contents , directly output the remaining part of the structure"""
-    new_response, finish_reason = llm_completion(model=model, prompt=prompt, chat_history=chat_history, return_finish_reason=True)
-    response = response + new_response
-    if_complete = check_if_toc_transformation_is_complete(content, response, model)
+    continue_prompt = "please continue the generation of table of contents, directly output the remaining part of the structure"
    
-    attempt = 0
    max_attempts = 5
-
-    while not (if_complete == "yes" and finish_reason == "finished"):
-        attempt += 1
-        if attempt > max_attempts:
-            raise Exception('Failed to complete table of contents after maximum retries')
-
-        chat_history = [
-            {"role": "user", "content": prompt},
-            {"role": "assistant", "content": response},
-        ]
-        prompt = f"""please continue the generation of table of contents , directly output the remaining part of the structure"""
-        new_response, finish_reason = llm_completion(model=model, prompt=prompt, chat_history=chat_history, return_finish_reason=True)
+    for attempt in range(max_attempts):
+        new_response, finish_reason = llm_completion(model=model, prompt=continue_prompt, chat_history=chat_history, return_finish_reason=True)
        response = response + new_response
+        chat_history.append({"role": "user", "content": continue_prompt})
+        chat_history.append({"role": "assistant", "content": new_response})
        if_complete = check_if_toc_transformation_is_complete(content, response, model)
+        if if_complete == "yes" and finish_reason == "finished":
+            break
+    else:
+        raise Exception('Failed to complete table of contents extraction after maximum retries')
    
    return response

@ -217,7 +209,7 @@ def detect_page_index(toc_content, model=None):

    response = llm_completion(model=model, prompt=prompt)
    json_content = extract_json(response)
-    return json_content['page_index_given_in_toc']
+    return json_content.get('page_index_given_in_toc', 'no')

 def toc_extractor(page_list, toc_page_list, model):
    def transform_dots_to_colon(text):
@ -296,43 +288,41 @@ def toc_transformer(toc_content, model=None):
    if_complete = check_if_toc_transformation_is_complete(toc_content, last_complete, model)
    if if_complete == "yes" and finish_reason == "finished":
        last_complete = extract_json(last_complete)
-        cleaned_response=convert_page_to_int(last_complete['table_of_contents'])
+        cleaned_response = convert_page_to_int(last_complete.get('table_of_contents', []))
        return cleaned_response
    
    last_complete = get_json_content(last_complete)
-    attempt = 0
+    chat_history = [
+        {"role": "user", "content": prompt},
+        {"role": "assistant", "content": last_complete},
+    ]
+    continue_prompt = "Please continue the table of contents JSON structure from where you left off. Directly output only the remaining part."
+
+    position = last_complete.rfind('}')
+    if position != -1:
+        last_complete = last_complete[:position+2]
+
    max_attempts = 5
-    while not (if_complete == "yes" and finish_reason == "finished"):
-        attempt += 1
-        if attempt > max_attempts:
-            raise Exception('Failed to complete toc transformation after maximum retries')
-        position = last_complete.rfind('}')
-        if position != -1:
-            last_complete = last_complete[:position+2]
-        prompt = f"""
-        Your task is to continue the table of contents json structure, directly output the remaining part of the json structure.
-        The response should be in the following JSON format: 
+    for attempt in range(max_attempts):

-        The raw table of contents json structure is:
-        {toc_content}
-
-        The incomplete transformed table of contents json structure is:
-        {last_complete}
-
-        Please continue the json structure, directly output the remaining part of the json structure."""
-
-        new_complete, finish_reason = llm_completion(model=model, prompt=prompt, return_finish_reason=True)
+        new_complete, finish_reason = llm_completion(model=model, prompt=continue_prompt, chat_history=chat_history, return_finish_reason=True)

        if new_complete.startswith('```json'):
-            new_complete =  get_json_content(new_complete)
-            last_complete = last_complete+new_complete
+            new_complete = get_json_content(new_complete)
+        last_complete = last_complete + new_complete
+
+        chat_history.append({"role": "user", "content": continue_prompt})
+        chat_history.append({"role": "assistant", "content": new_complete})

        if_complete = check_if_toc_transformation_is_complete(toc_content, last_complete, model)
-        
+        if if_complete == "yes" and finish_reason == "finished":
+            break
+    else:
+        raise Exception('Failed to complete TOC transformation after maximum retries')

    last_complete = extract_json(last_complete)

-    cleaned_response=convert_page_to_int(last_complete['table_of_contents'])
+    cleaned_response = convert_page_to_int(last_complete.get('table_of_contents', []))
    return cleaned_response
    

@ -752,8 +742,11 @@ async def single_toc_item_index_fixer(section_title, content, model=None):

    prompt = toc_extractor_prompt + '\nSection Title:\n' + str(section_title) + '\nDocument pages:\n' + content
    response = await llm_acompletion(model=model, prompt=prompt)
-    json_content = extract_json(response)    
-    return convert_physical_index_to_int(json_content['physical_index'])
+    json_content = extract_json(response)
+    physical_index = json_content.get('physical_index')
+    if physical_index is None:
+        return None
+    return convert_physical_index_to_int(physical_index)



--- a/requirements.txt
+++ b/requirements.txt
@ -1,6 +1,6 @@
-litellm==1.83.0
+litellm==1.84.0
 # openai-agents  # optional: required for examples/agentic_vectorless_rag_demo.py
 pymupdf==1.26.4
 PyPDF2==3.0.1
-python-dotenv==1.1.0
+python-dotenv==1.2.2
 pyyaml==6.0.2
--- a/tests/test_issue_163.py
+++ b/tests/test_issue_163.py
@ -0,0 +1,135 @@
+import pytest
+import sys
+import os
+from unittest.mock import patch, MagicMock
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from pageindex.index.page_index import (
+    check_if_toc_extraction_is_complete,
+    check_if_toc_transformation_is_complete,
+    toc_detector_single_page,
+    detect_page_index,
+    extract_toc_content,
+    toc_transformer,
+)
+
+
+class TestRobustKeyAccess:
+    @patch("pageindex.index.page_index.llm_completion", return_value="")
+    def test_toc_detector_empty_response(self, mock_llm):
+        result = toc_detector_single_page("some content", model="test")
+        assert result == "no"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value='{"toc_detected": "yes"}')
+    def test_toc_detector_valid_response(self, mock_llm):
+        result = toc_detector_single_page("some content", model="test")
+        assert result == "yes"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value="not json at all")
+    def test_toc_detector_malformed_response(self, mock_llm):
+        result = toc_detector_single_page("some content", model="test")
+        assert result == "no"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value="")
+    def test_extraction_complete_empty_response(self, mock_llm):
+        result = check_if_toc_extraction_is_complete("doc", "toc", model="test")
+        assert result == "no"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value='{"completed": "yes"}')
+    def test_extraction_complete_valid_response(self, mock_llm):
+        result = check_if_toc_extraction_is_complete("doc", "toc", model="test")
+        assert result == "yes"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value="")
+    def test_transformation_complete_empty_response(self, mock_llm):
+        result = check_if_toc_transformation_is_complete("raw", "cleaned", model="test")
+        assert result == "no"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value='{"thinking": "looks fine", "completed": "yes"}')
+    def test_transformation_complete_valid_response(self, mock_llm):
+        result = check_if_toc_transformation_is_complete("raw", "cleaned", model="test")
+        assert result == "yes"
+
+    @patch("pageindex.index.page_index.llm_completion", return_value="")
+    def test_detect_page_index_empty_response(self, mock_llm):
+        result = detect_page_index("toc text", model="test")
+        assert result == "no"
+
+
+class TestExtractTocContentRetryLoop:
+    @patch("pageindex.index.page_index.check_if_toc_transformation_is_complete")
+    @patch("pageindex.index.page_index.llm_completion")
+    def test_completes_on_first_try(self, mock_llm, mock_check):
+        mock_llm.return_value = ("full toc content", "finished")
+        mock_check.return_value = "yes"
+        result = extract_toc_content("raw content", model="test")
+        assert result == "full toc content"
+        assert mock_llm.call_count == 1
+
+    @patch("pageindex.index.page_index.check_if_toc_transformation_is_complete")
+    @patch("pageindex.index.page_index.llm_completion")
+    def test_continues_on_incomplete(self, mock_llm, mock_check):
+        mock_llm.side_effect = [
+            ("partial toc", "max_output_reached"),
+            (" continued toc", "finished"),
+        ]
+        mock_check.side_effect = ["no", "yes"]
+        result = extract_toc_content("raw content", model="test")
+        assert result == "partial toc continued toc"
+        assert mock_llm.call_count == 2
+
+    @patch("pageindex.index.page_index.check_if_toc_transformation_is_complete")
+    @patch("pageindex.index.page_index.llm_completion")
+    def test_max_retries_raises_exception(self, mock_llm, mock_check):
+        mock_llm.return_value = ("chunk", "max_output_reached")
+        mock_check.return_value = "no"
+        with pytest.raises(Exception, match="Failed to complete table of contents extraction"):
+            extract_toc_content("raw content", model="test")
+        assert mock_llm.call_count == 6
+
+    @patch("pageindex.index.page_index.check_if_toc_transformation_is_complete")
+    @patch("pageindex.index.page_index.llm_completion")
+    def test_chat_history_grows_incrementally(self, mock_llm, mock_check):
+        call_count = [0]
+
+        def side_effect(*args, **kwargs):
+            call_count[0] += 1
+            if call_count[0] == 1:
+                return ("initial", "max_output_reached")
+            if call_count[0] == 2:
+                history = kwargs.get("chat_history", [])
+                assert len(history) == 2
+                return (" part2", "max_output_reached")
+            if call_count[0] == 3:
+                history = kwargs.get("chat_history", [])
+                assert len(history) == 4
+                return (" part3", "finished")
+            return ("", "finished")
+
+        mock_llm.side_effect = side_effect
+        mock_check.side_effect = ["no", "no", "yes"]
+        result = extract_toc_content("raw content", model="test")
+        assert result == "initial part2 part3"
+
+
+class TestTocTransformerRetryLoop:
+    @patch("pageindex.index.page_index.check_if_toc_transformation_is_complete")
+    @patch("pageindex.index.page_index.llm_completion")
+    def test_completes_on_first_try(self, mock_llm, mock_check):
+        mock_llm.return_value = (
+            '{"table_of_contents": [{"structure": "1", "title": "Intro", "page": 1}]}',
+            "finished",
+        )
+        mock_check.return_value = "yes"
+        result = toc_transformer("raw toc", model="test")
+        assert len(result) == 1
+        assert result[0]["title"] == "Intro"
+
+    @patch("pageindex.index.page_index.check_if_toc_transformation_is_complete")
+    @patch("pageindex.index.page_index.llm_completion")
+    def test_handles_missing_table_of_contents_key(self, mock_llm, mock_check):
+        mock_llm.return_value = ('{"other_key": "value"}', "finished")
+        mock_check.return_value = "yes"
+        result = toc_transformer("raw toc", model="test")
+        assert result == []