PageIndex/README.md

271 lines
14 KiB
Markdown
Raw Normal View History

2025-04-30 20:51:59 +07:00
<div align="center">
<a href="https://vectify.ai/pageindex" target="_blank">
<img src="https://github.com/user-attachments/assets/46201e72-675b-43bc-bfbd-081cc6b65a1d" alt="PageIndex Banner" />
</a>
<br/>
<br/>
<p align="center">
<a href="https://trendshift.io/repositories/14736" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14736" alt="VectifyAI%2FPageIndex | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
2026-01-25 17:33:36 +08:00
# PageIndex: Vectorless, Reasoning-based RAG
2025-11-21 01:25:46 +08:00
<p align="center"><b>Reasoning-based RAG&nbsp;&nbsp;No Vector DB&nbsp;&nbsp;No Chunking&nbsp;&nbsp;Human-like Retrieval</b></p>
2025-10-14 23:11:39 +08:00
<h4 align="center">
<a href="https://vectify.ai">🏠 Homepage</a>&nbsp;&nbsp;
2025-12-19 03:50:44 +08:00
<a href="https://chat.pageindex.ai">🖥️ Chat Platform</a>&nbsp;&nbsp;
2025-10-14 23:11:39 +08:00
<a href="https://pageindex.ai/mcp">🔌 MCP</a>&nbsp;&nbsp;
2025-12-22 08:26:04 +08:00
<a href="https://docs.pageindex.ai">📚 Docs</a>&nbsp;&nbsp;
<a href="https://discord.com/invite/VuXuf29EUj">💬 Discord</a>&nbsp;&nbsp;
<a href="https://ii2abc2jejf.typeform.com/to/tK3AXl8T">✉️ Contact</a>&nbsp;
2025-10-14 23:11:39 +08:00
</h4>
2025-08-07 23:05:02 +01:00
</div>
2025-06-17 12:25:25 +01:00
2025-04-01 18:54:08 +08:00
2025-11-13 03:01:09 +08:00
<details open>
2026-01-25 22:11:25 +08:00
<summary><h3>📢 Latest Updates</h3></summary>
2025-09-20 01:10:25 +08:00
2025-12-19 10:05:35 +08:00
**🔥 Releases:**
2025-12-22 08:26:04 +08:00
- [**PageIndex Chat**](https://chat.pageindex.ai): The first human-like document-analysis agent [platform](https://chat.pageindex.ai) built for professional long documents. Can also be integrated via [MCP](https://pageindex.ai/mcp) or [API](https://docs.pageindex.ai/quickstart) (beta).
2025-12-21 01:15:33 +08:00
<!-- - [**PageIndex Chat API**](https://docs.pageindex.ai/quickstart): An API that brings PageIndex's advanced long-document intelligence directly into your applications and workflows. -->
2025-12-19 03:50:44 +08:00
<!-- - [PageIndex MCP](https://pageindex.ai/mcp): Bring PageIndex into Claude, Cursor, or any MCP-enabled agent. Chat with long PDFs in a reasoning-based, human-like way. -->
2025-12-21 01:15:33 +08:00
**📝 Articles:**
2025-12-19 03:50:44 +08:00
- [**PageIndex Framework**](https://pageindex.ai/blog/pageindex-intro): Introduces the PageIndex framework — an *agentic, in-context* *tree index* that enables LLMs to perform *reasoning-based*, *human-like retrieval* over long documents, without vector DB or chunking.
<!-- - [Do We Still Need OCR?](https://pageindex.ai/blog/do-we-need-ocr): Explores how vision-based, reasoning-native RAG challenges the traditional OCR pipeline, and why the future of document AI might be *vectorless* and *vision-based*. -->
2025-11-05 23:07:32 +08:00
2025-11-13 03:01:09 +08:00
**🧪 Cookbooks:**
2025-12-19 11:53:31 +08:00
- [Vectorless RAG](https://docs.pageindex.ai/cookbook/vectorless-rag-pageindex): A minimal, hands-on example of reasoning-based RAG using PageIndex. No vectors, no chunking, and human-like retrieval.
2025-12-21 01:15:33 +08:00
- [Vision-based Vectorless RAG](https://docs.pageindex.ai/cookbook/vision-rag-pageindex): OCR-free, vision-only RAG with PageIndex's reasoning-native retrieval workflow that works directly over PDF page images.
2025-11-13 03:01:09 +08:00
</details>
2025-11-05 23:07:32 +08:00
2025-12-19 10:05:35 +08:00
---
2025-11-05 01:20:46 +08:00
# 📑 Introduction to PageIndex
2025-08-20 01:52:48 +08:00
Are you frustrated with vector database retrieval accuracy for long professional documents? Traditional vector-based RAG relies on semantic *similarity* rather than true *relevance*. But **similarity ≠ relevance** — what we truly need in retrieval is **relevance**, and that requires **reasoning**. When working with professional documents that demand domain expertise and multi-step reasoning, similarity search often falls short.
2025-05-20 12:31:27 +01:00
2025-12-21 01:15:33 +08:00
Inspired by AlphaGo, we propose **[PageIndex](https://vectify.ai/pageindex)** — a **vectorless**, **reasoning-based RAG** system that builds a **hierarchical tree index** from long documents and uses LLMs to **reason** *over that index* for **agentic, context-aware retrieval**.
It simulates how *human experts* navigate and extract knowledge from complex documents through *tree search*, enabling LLMs to *think* and *reason* their way to the most relevant document sections. PageIndex performs retrieval in two steps:
2025-08-24 17:30:21 +08:00
2025-12-19 10:05:35 +08:00
1. Generate a “Table-of-Contents” **tree structure index** of documents
2. Perform reasoning-based retrieval through **tree search**
2025-08-24 17:30:21 +08:00
2025-08-25 17:50:53 +08:00
<div align="center">
2025-12-19 12:09:06 +08:00
<a href="https://pageindex.ai/blog/pageindex-intro" target="_blank" title="The PageIndex Framework">
2025-11-05 01:20:46 +08:00
<img src="https://docs.pageindex.ai/images/cookbook/vectorless-rag.png" width="70%">
2025-12-19 12:09:06 +08:00
</a>
2025-08-25 17:50:53 +08:00
</div>
2025-08-24 17:30:21 +08:00
2026-01-25 22:11:25 +08:00
### 🎯 Core Features
2025-12-21 01:15:33 +08:00
Compared to traditional vector-based RAG, **PageIndex** features:
2025-12-19 05:06:46 +08:00
- **No Vector DB**: Uses document structure and LLM reasoning for retrieval, instead of vector similarity search.
2025-11-07 15:05:57 +08:00
- **No Chunking**: Documents are organized into natural sections, not artificial chunks.
- **Human-like Retrieval**: Simulates how human experts navigate and extract knowledge from complex documents.
2025-12-19 10:05:35 +08:00
- **Better Explainability and Traceability**: Retrieval is based on reasoning — traceable and interpretable, with page and section references. No more opaque, approximate vector search (“vibe retrieval”).
2025-04-01 18:54:08 +08:00
2025-12-19 12:09:06 +08:00
PageIndex powers a reasoning-based RAG system that achieved **state-of-the-art** [98.7% accuracy](https://github.com/VectifyAI/Mafin2.5-FinanceBench) on FinanceBench, demonstrating superior performance over vector-based RAG solutions in professional document analysis (see our [blog post](https://vectify.ai/blog/Mafin2.5) for details).
2025-04-01 18:54:08 +08:00
2025-12-19 03:50:44 +08:00
### 📍 Explore PageIndex
2025-12-21 01:15:33 +08:00
To learn more, please see a detailed introduction of the [PageIndex framework](https://pageindex.ai/blog/pageindex-intro). Check out this GitHub repo for open-source code, and the [cookbooks](https://docs.pageindex.ai/cookbook), [tutorials](https://docs.pageindex.ai/tutorials), and [blog](https://pageindex.ai/blog) for additional usage guides and examples.
2025-12-19 03:50:44 +08:00
2025-12-21 01:15:33 +08:00
The PageIndex service is available as a ChatGPT-style [chat platform](https://chat.pageindex.ai), or can be integrated via [MCP](https://pageindex.ai/mcp) or [API](https://docs.pageindex.ai/quickstart).
### 🛠️ Deployment Options
2025-12-19 10:05:35 +08:00
- Self-host — run locally with this open-source repo.
- Cloud Service — try instantly with our [Chat Platform](https://chat.pageindex.ai/), or integrate with [MCP](https://pageindex.ai/mcp) or [API](https://docs.pageindex.ai/quickstart).
2025-12-21 01:15:33 +08:00
- _Enterprise_ — private or on-prem deployment. [Contact us](https://ii2abc2jejf.typeform.com/to/tK3AXl8T) or [book a demo](https://calendly.com/pageindex/meet) for more details.
2025-11-05 01:02:46 +08:00
### 🧪 Quick Hands-on
2025-12-19 10:05:35 +08:00
- Try the [**Vectorless RAG**](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb) notebook — a *minimal*, hands-on example of reasoning-based RAG using PageIndex.
- Experiment with [*Vision-based Vectorless RAG*](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb) — no OCR; a minimal, reasoning-native RAG pipeline that works directly over page images.
2025-11-05 01:20:46 +08:00
2025-11-06 00:06:26 +08:00
<div align="center">
<a href="https://colab.research.google.com/github/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb" target="_blank" rel="noopener">
<img src="https://img.shields.io/badge/Open_In_Colab-Vectorless_RAG-orange?style=for-the-badge&logo=googlecolab" alt="Open in Colab: Vectorless RAG" />
</a>
2025-11-06 00:06:26 +08:00
&nbsp;&nbsp;
<a href="https://colab.research.google.com/github/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb" target="_blank" rel="noopener">
<img src="https://img.shields.io/badge/Open_In_Colab-Vision_RAG-orange?style=for-the-badge&logo=googlecolab" alt="Open in Colab: Vision RAG" />
</a>
</div>
2025-05-06 05:21:08 +08:00
2025-08-25 17:50:53 +08:00
---
2025-04-12 01:27:24 +08:00
2025-11-05 01:33:30 +08:00
# 🌲 PageIndex Tree Structure
PageIndex can transform lengthy PDF documents into a semantic **tree structure**, similar to a _"table of contents"_ but optimized for use with Large Language Models (LLMs). It's ideal for: financial reports, regulatory filings, academic textbooks, legal or technical manuals, and any document that exceeds LLM context limits.
2025-04-01 18:54:08 +08:00
2025-12-19 10:05:35 +08:00
Below is an example PageIndex tree structure. Also see more example [documents](https://github.com/VectifyAI/PageIndex/tree/main/tests/pdfs) and generated [tree structures](https://github.com/VectifyAI/PageIndex/tree/main/tests/results).
2025-04-01 18:54:08 +08:00
2025-11-07 03:06:26 +08:00
```jsonc
...
{
"title": "Financial Stability",
"node_id": "0006",
"start_index": 21,
"end_index": 22,
"summary": "The Federal Reserve ...",
"nodes": [
{
"title": "Monitoring Financial Vulnerabilities",
"node_id": "0007",
"start_index": 22,
"end_index": 28,
"summary": "The Federal Reserve's monitoring ..."
},
{
"title": "Domestic and International Cooperation and Coordination",
"node_id": "0008",
"start_index": 28,
"end_index": 31,
"summary": "In 2023, the Federal Reserve collaborated ..."
}
]
}
...
```
2025-04-12 01:27:24 +08:00
2025-12-21 01:15:33 +08:00
You can generate the PageIndex tree structure with this open-source repo, or use our [API](https://docs.pageindex.ai/quickstart)
2025-08-25 17:50:53 +08:00
---
2025-04-12 01:27:24 +08:00
2025-12-21 01:15:33 +08:00
# ⚙️ Package Usage
2025-04-01 18:54:08 +08:00
You can follow these steps to generate a PageIndex tree from a PDF document.
2025-04-12 01:27:24 +08:00
### 1. Install dependencies
2025-04-12 01:27:24 +08:00
```bash
pip3 install --upgrade -r requirements.txt
2025-04-01 18:54:08 +08:00
```
2025-04-10 11:38:54 +08:00
### 2. Set your OpenAI API key
2025-04-12 01:27:24 +08:00
Create a `.env` file in the root directory and add your API key:
2025-04-12 01:27:24 +08:00
```bash
CHATGPT_API_KEY=your_openai_key_here
```
2025-04-12 01:27:24 +08:00
### 3. Run PageIndex on your PDF
```bash
python3 run_pageindex.py --pdf_path /path/to/your/document.pdf
2025-04-12 01:27:24 +08:00
```
2025-08-25 17:50:53 +08:00
<details>
<summary><strong>Optional parameters</strong></summary>
<br>
You can customize the processing with additional optional arguments:
2025-04-12 01:27:24 +08:00
```
--model OpenAI model to use (default: gpt-4o-2024-11-20)
--toc-check-pages Pages to check for table of contents (default: 20)
--max-pages-per-node Max pages per node (default: 10)
--max-tokens-per-node Max tokens per node (default: 20000)
--if-add-node-id Add node ID (yes/no, default: yes)
--if-add-node-summary Add node summary (yes/no, default: yes)
--if-add-doc-description Add doc description (yes/no, default: yes)
```
</details>
2025-04-12 01:27:24 +08:00
<details>
<summary><strong>Markdown support</strong></summary>
<br>
2025-12-19 05:06:46 +08:00
We also provide markdown support for PageIndex. You can use the `-md_path` flag to generate a tree structure for a markdown file.
2025-09-02 10:18:28 +01:00
```bash
python3 run_pageindex.py --md_path /path/to/your/document.md
2025-09-02 10:18:28 +01:00
```
2025-12-21 01:15:33 +08:00
> Note: in this function, we use "#" to determine node heading and their levels. For example, "##" is level 2, "###" is level 3, etc. Make sure your markdown file is formatted correctly. If your Markdown file was converted from a PDF or HTML, we don't recommend using this function, since most existing conversion tools cannot preserve the original hierarchy. Instead, use our [PageIndex OCR](https://pageindex.ai/blog/ocr), which is designed to preserve the original hierarchy, to convert the PDF to a markdown file and then use this function.
</details>
2025-12-19 10:05:35 +08:00
<!--
# ☁️ Improved Tree Generation with PageIndex OCR
2025-11-05 04:07:05 +08:00
This repo is designed for generating PageIndex tree structure for simple PDFs, but many real-world use cases involve complex PDFs that are hard to parse by classic Python tools. However, extracting high-quality text from PDF documents remains a non-trivial challenge. Most OCR tools only extract page-level content, losing the broader document context and hierarchy.
To address this, we introduced PageIndex OCR — the first long-context OCR model designed to preserve the global structure of documents. PageIndex OCR significantly outperforms other leading OCR tools, such as those from Mistral and Contextual AI, in recognizing true hierarchy and semantic relationships across document pages.
- Experience next-level OCR quality with PageIndex OCR at our [Dashboard](https://dash.pageindex.ai/).
2025-11-05 04:07:05 +08:00
- Integrate PageIndex OCR seamlessly into your stack via our [API](https://docs.pageindex.ai/quickstart).
2025-08-20 01:23:03 +08:00
<p align="center">
2025-11-05 01:33:30 +08:00
<img src="https://github.com/user-attachments/assets/eb35d8ae-865c-4e60-a33b-ebbd00c41732" width="80%">
</p>
2025-12-19 10:05:35 +08:00
-->
---
2025-12-19 05:06:46 +08:00
# 📈 Case Study: PageIndex Leads Finance QA Benchmark
2025-12-19 11:53:31 +08:00
[Mafin 2.5](https://vectify.ai/mafin) is a reasoning-based RAG system for financial document analysis, powered by **PageIndex**. It achieved a state-of-the-art [**98.7% accuracy**](https://vectify.ai/blog/Mafin2.5) on the [FinanceBench](https://arxiv.org/abs/2311.11944) benchmark, significantly outperforming traditional vector-based RAG systems.
2025-12-19 10:05:35 +08:00
PageIndex's hierarchical indexing and reasoning-driven retrieval enable precise navigation and extraction of relevant context from complex financial reports, such as SEC filings and earnings disclosures.
2025-04-12 01:27:24 +08:00
2025-12-19 10:05:35 +08:00
Explore the full [benchmark results](https://github.com/VectifyAI/Mafin2.5-FinanceBench) and our [blog post](https://vectify.ai/blog/Mafin2.5) for detailed comparisons and performance metrics.
<div align="center">
<a href="https://github.com/VectifyAI/Mafin2.5-FinanceBench">
2025-11-05 01:42:57 +08:00
<img src="https://github.com/user-attachments/assets/571aa074-d803-43c7-80c4-a04254b782a3" width="70%">
</a>
</div>
---
2025-11-21 01:25:46 +08:00
# 🧭 Resources
* 🧪 [Cookbooks](https://docs.pageindex.ai/cookbook/vectorless-rag-pageindex): hands-on, runnable examples and advanced use cases.
2025-12-19 10:05:35 +08:00
* 📖 [Tutorials](https://docs.pageindex.ai/doc-search): practical guides and strategies, including *Document Search* and *Tree Search*.
2025-12-19 11:53:31 +08:00
* 📝 [Blog](https://pageindex.ai/blog): technical articles, research insights, and product updates.
2025-12-21 01:15:33 +08:00
* 🔌 [MCP setup](https://pageindex.ai/mcp#quick-setup) & [API docs](https://docs.pageindex.ai/quickstart): integration details and configuration options.
2025-11-21 01:25:46 +08:00
---
2025-12-19 10:05:35 +08:00
# ⭐ Support Us
2026-02-10 14:31:30 +08:00
Please cite this work as:
```
Mingtian Zhang, Yu Tang and PageIndex Team,
"PageIndex: Next-Generation Vectorless, Reasoning-based RAG",
PageIndex Blog, Sep 2025.
```
Or use the BibTeX citation:
```
@article{zhang2025pageindex,
author = {Mingtian Zhang and Yu Tang and PageIndex Team},
title = {PageIndex: Next-Generation Vectorless, Reasoning-based RAG},
journal = {PageIndex Blog},
year = {2025},
month = {September},
note = {https://pageindex.ai/blog/pageindex-intro},
}
```
2025-12-19 11:53:31 +08:00
Leave us a star 🌟 if you like our project. Thank you!
<p>
2025-11-07 03:06:26 +08:00
<img src="https://github.com/user-attachments/assets/eae4ff38-48ae-4a7c-b19f-eab81201d794" width="80%">
</p>
2025-12-19 12:09:06 +08:00
### Connect with Us
2026-01-24 14:09:21 +08:00
[![Twitter](https://img.shields.io/badge/Twitter-000000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/PageIndexAI)&nbsp;
[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/company/vectify-ai/)&nbsp;
[![Discord](https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.com/invite/VuXuf29EUj)&nbsp;
[![Contact Us](https://img.shields.io/badge/Contact_Us-3B82F6?style=for-the-badge&logo=envelope&logoColor=white)](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)
---
2025-09-03 18:56:29 +08:00
© 2025 [Vectify AI](https://vectify.ai)