apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add num2words dependency to demo notebook and Quick Start section to README

Browse source
This commit is contained in:
Sarvesh 2026-06-09 16:06:49 +05:30
parent 65b6e1cc2c
commit 834e979c20
2 changed files with 15 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

15
README.md
View file

@ -15,7 +15,7 @@ Official implementation of **Video2LoRA**
---
### [Install](#install)  •  [Checkpoints](#checkpoints)  •  [Inference](#inference)  •  [Data Format](#data-format)  •  [Training](#train)
### [Quick Start](#-quick-start)  •  [Install](#install)  •  [Checkpoints](#checkpoints)  •  [Inference](#inference)  •  [Data Format](#data-format)  •  [Training](#train)
<p align="center">
<img src="assets/video2lora-diagram-white.svg" alt="Animated Video2LoRA method diagram" width="900">
@ -25,6 +25,19 @@ Official implementation of **Video2LoRA**
Video2LoRA trains a hypernetwork that converts a video into LoRA weights for a frozen vision-language model. The generated adapter lets the model answer later text prompts without feeding the video tokens again.
## 🚀 Quick Start
Try zero-visual-token inference instantly in your browser via our self-contained interactive tutorial notebook:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video2lora/Video2LoRA/blob/demo/demo.ipynb)
This interactive notebook guides you through:
1. Downloading the 2.2B SmolVLM2 hypernetwork checkpoint.
2. Linking and resolving the project's qualitative videos.
3. Parametric internalization (extracting visual features and generating custom adapter weights).
4. Running comparative Q&A inference (comparing the base vision-language model against the Video2LoRA adapter with zero visual tokens).
5. Visualizing the comparison using a custom dashboard layout.
## Install
Install `uv`.

2
demo.ipynb
View file

@ -98,7 +98,7 @@
"\n",
"# Install necessary packages\n",
"!pip install torch torchvision torchaudio\n",
"!pip install transformers accelerate peft huggingface_hub decord av opencv-python matplotlib einops jaxtyping"
"!pip install transformers accelerate peft huggingface_hub decord av opencv-python matplotlib einops jaxtyping num2words"
]
},
{