mirror of
https://github.com/VectifyAI/PageIndex.git
synced 2026-07-24 21:41:04 +02:00
41f4a0a6f5
index/utils.py mixes LLM callers with plain string/dict helpers, so any consumer of parse_pages/create_node_mapping (retrieve, page_index_md, the cloud backend's tree normalization) paid litellm's multi-second import and network fetch. The heavy imports now happen at first real use; sys.modules caches them after that.
12 lines
457 B
Python
12 lines
457 B
Python
# pageindex/tokens.py
|
|
# Leaf utility so both the parser and index layers can count tokens without
|
|
# the parser reaching back into pageindex.index (a reverse/horizontal
|
|
# dependency). Depends only on litellm (imported lazily — it's several
|
|
# seconds and a network fetch, and cloud-only paths never need it).
|
|
|
|
|
|
def count_tokens(text, model=None):
|
|
if not text:
|
|
return 0
|
|
import litellm
|
|
return litellm.token_counter(model=model, text=text)
|