mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-05-17 18:35:19 +02:00
9bcd50164d
Adds the full parser_compare experiment for the multimodal_doc suite:
six arms compared on 30 PDFs / 171 questions from MMLongBench-Doc with
anthropic/claude-sonnet-4.5 across the board.
Source code:
- core/parsers/{azure_di,llamacloud,pdf_pages}.py: direct parser SDK
callers (Azure Document Intelligence prebuilt-read/layout, LlamaParse
parse_page_with_llm/parse_page_with_agent) used by the LC arms,
bypassing the SurfSense backend so each (basic/premium) extraction
is a clean A/B independent of backend ETL routing.
- suites/multimodal_doc/parser_compare/{ingest,runner,prompt}.py:
six-arm benchmark (native_pdf, azure_basic_lc, azure_premium_lc,
llamacloud_basic_lc, llamacloud_premium_lc, surfsense_agentic) with
byte-identical prompts per question, deterministic grader, Wilson
CIs, and the per-page preprocessing tariff cost overlay.
Reproducibility:
- pyproject.toml + uv.lock pin pypdf, azure-ai-documentintelligence,
llama-cloud-services as new deps.
- .env.example documents the AZURE_DI_* and LLAMA_CLOUD_API_KEY env
vars now required for parser_compare.
- 12 analysis scripts under scripts/: retry pass with exponential
backoff, post-retry accuracy merge, McNemar / latency / per-PDF
stats, context-overflow hypothesis test, etc. Each produces one
number cited by the blog report.
Citation surface:
- reports/blog/multimodal_doc_parser_compare_n171_report.md: 1219-line
technical writeup (16 sections) covering headline accuracy, per-format
accuracy, McNemar pairwise significance, latency / token / per-PDF
distributions, error analysis, retry experiment, post-retry final
accuracy, cost amortization model with closed-form derivation, threats
to validity, and reproducibility appendix.
- data/multimodal_doc/runs/2026-05-14T00-53-19Z/parser_compare/{raw,
raw_retries,raw_post_retry}.jsonl + run_artifact.json + retry summary
whitelisted via data/.gitignore as the verifiable numbers source.
Gitignore:
- ignore logs_*.txt + retry_run.log; structured artifacts cover the
citation surface, debug logs are noise.
- data/.gitignore default-ignores everything, whitelists the n=171 run
artifacts only (parser manifest left ignored to avoid leaking local
Windows usernames in absolute paths; manifest is fully regenerable
via 'ingest multimodal_doc parser_compare').
- reports/.gitignore now whitelists hand-curated reports/blog/.
Also retires the abandoned CRAG Task 3 implementation (download script,
streaming Task 3 ingest, CragTask3Benchmark + tests) and trims the
runner / ingest module APIs to match.
Co-authored-by: Cursor <cursoragent@cursor.com>
60 lines
2 KiB
Python
60 lines
2 KiB
Python
"""Sanity check extraction sizes against Sonnet 4.5's context window.
|
|
|
|
Sonnet 4.5 supports ~200k tokens. As a *very* rough heuristic, English
|
|
markdown is ~4 chars/token, so anything over ~750k chars likely won't
|
|
fit alongside the system + question + 512 max_output_tokens. Print
|
|
warnings for any extraction that's at risk.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
REPO = Path(__file__).resolve().parents[1]
|
|
MAP = REPO / "data" / "multimodal_doc" / "maps" / "parser_compare_doc_map.jsonl"
|
|
|
|
CHARS_PER_TOKEN = 4
|
|
CTX_TOKENS = 200_000
|
|
PROMPT_OVERHEAD_TOKENS = 1_000 # system + question + format hint
|
|
MAX_OUTPUT_TOKENS = 512
|
|
SAFE_CHARS = (CTX_TOKENS - PROMPT_OVERHEAD_TOKENS - MAX_OUTPUT_TOKENS) * CHARS_PER_TOKEN
|
|
|
|
|
|
def main() -> None:
|
|
rows = [
|
|
json.loads(line)
|
|
for line in MAP.read_text(encoding="utf-8").splitlines()
|
|
if line.strip()
|
|
]
|
|
|
|
total = len(rows)
|
|
arm_max: dict[str, tuple[int, str]] = {}
|
|
overflows: list[tuple[str, str, int]] = []
|
|
for row in rows:
|
|
for arm, ext in (row.get("extractions") or {}).items():
|
|
chars = int(ext.get("chars") or 0)
|
|
if arm not in arm_max or arm_max[arm][0] < chars:
|
|
arm_max[arm] = (chars, row["doc_id"])
|
|
if chars > SAFE_CHARS:
|
|
overflows.append((row["doc_id"], arm, chars))
|
|
|
|
print(f"PDFs in manifest: {total}")
|
|
print(f"safe char budget: {SAFE_CHARS:,} (~{(SAFE_CHARS // CHARS_PER_TOKEN):,} tokens)")
|
|
print()
|
|
print("largest extraction per arm:")
|
|
for arm, (chars, doc_id) in sorted(arm_max.items()):
|
|
print(f" {arm:25s} {chars:>10,} chars ({doc_id})")
|
|
|
|
print()
|
|
if overflows:
|
|
print(f"OVERFLOW RISK ({len(overflows)} extractions > safe budget):")
|
|
for doc_id, arm, chars in overflows:
|
|
est_tokens = chars // CHARS_PER_TOKEN
|
|
print(f" {doc_id} :: {arm} :: {chars:,} chars (~{est_tokens:,} tokens)")
|
|
else:
|
|
print("no overflow risk — all extractions fit Sonnet 4.5's 200k context.")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|