mirror of
https://github.com/SakanaAI/doc-to-lora.git
synced 2026-07-23 17:01:04 +02:00
Revert README.md change and write rich, polished markdown explanations inside demo.ipynb
This commit is contained in:
Sarvesh
parent
834e979c20
commit
4e42b8a1d8
2 changed files with 52 additions and 40 deletions
15
README.md
15
README.md
|
|
@ -15,7 +15,7 @@ Official implementation of **Video2LoRA**
|
|||
|
||||
---
|
||||
|
||||
### [Quick Start](#-quick-start) • [Install](#install) • [Checkpoints](#checkpoints) • [Inference](#inference) • [Data Format](#data-format) • [Training](#train)
|
||||
### [Install](#install) • [Checkpoints](#checkpoints) • [Inference](#inference) • [Data Format](#data-format) • [Training](#train)
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/video2lora-diagram-white.svg" alt="Animated Video2LoRA method diagram" width="900">
|
||||
|
|
@ -25,19 +25,6 @@ Official implementation of **Video2LoRA**
|
|||
|
||||
Video2LoRA trains a hypernetwork that converts a video into LoRA weights for a frozen vision-language model. The generated adapter lets the model answer later text prompts without feeding the video tokens again.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
Try zero-visual-token inference instantly in your browser via our self-contained interactive tutorial notebook:
|
||||
|
||||
[](https://colab.research.google.com/github/video2lora/Video2LoRA/blob/demo/demo.ipynb)
|
||||
|
||||
This interactive notebook guides you through:
|
||||
1. Downloading the 2.2B SmolVLM2 hypernetwork checkpoint.
|
||||
2. Linking and resolving the project's qualitative videos.
|
||||
3. Parametric internalization (extracting visual features and generating custom adapter weights).
|
||||
4. Running comparative Q&A inference (comparing the base vision-language model against the Video2LoRA adapter with zero visual tokens).
|
||||
5. Visualizing the comparison using a custom dashboard layout.
|
||||
|
||||
## Install
|
||||
|
||||
Install `uv`.
|
||||
|
|
|
|||
77
demo.ipynb
77
demo.ipynb
|
|
@ -4,28 +4,42 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Video2LoRA: Self-Contained Zero-Visual-Token Inference Tutorial\n",
|
||||
"# \ud83c\udfa5 Video2LoRA: Self-Contained Zero-Visual-Token Inference Tutorial\n",
|
||||
"\n",
|
||||
"This notebook provides a **completely self-contained, single-file tutorial** for running zero-visual-token inference with **Video2LoRA**. Under the hood, Video2LoRA utilizes a perceiver hypernetwork to read a video's intermediate activation features layer-by-layer and directly predict a set of Low-Rank Adaptation (LoRA) weights in a single forward pass.\n",
|
||||
"Welcome to the official interactive tutorial for **Video2LoRA**! \n",
|
||||
"\n",
|
||||
"Once the adapter weights are generated, they are injected into the base Vision-Language Model (VLM), which can then answer arbitrary text queries **without requiring the original visual tokens** in its context window.\n",
|
||||
"This notebook provides a **completely self-contained, single-file walkthrough** demonstrating how to run zero-visual-token video inference. \n",
|
||||
"\n",
|
||||
"All model architectures, helper functions, and dependencies are defined directly within this notebook so it can run independently of the main repository codebase.\n",
|
||||
"### \ud83e\udde0 The Core Concept: Parametric Video Internalization\n",
|
||||
"Typically, Vision-Language Models (VLMs) process videos by projecting video frames into visual tokens and feeding hundreds (or thousands) of these tokens directly into the model's context window. This consumes massive context space, increases Time-to-First-Token (TTFT) latency, and slows down generation.\n",
|
||||
"\n",
|
||||
"**Video2LoRA** introduces a paradigm shift:\n",
|
||||
"1. **Hypernetwork Generation**: A perceiver hypernetwork reads a video's intermediate activation features layer-by-layer and directly predicts a set of Low-Rank Adaptation (LoRA) weights in a single forward pass.\n",
|
||||
"2. **Internalization**: Once generated, these adapter weights are injected into the base Vision-Language Model.\n",
|
||||
"3. **Zero-Token Inference**: The VLM can now answer arbitrary text queries about the video **without requiring a single visual token** in its context window! The video information has been *internalized* parametrically into the model's weights.\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"### \ud83d\udd17 Project Resources\n",
|
||||
"- [**\ud83c\udf10 Project Website**](https://video2lora.github.io/)\n",
|
||||
"- [**\ud83d\udcc4 arXiv Paper**](https://arxiv.org/abs/2606.04351)\n",
|
||||
"- [**\ud83e\udd17 Hugging Face Checkpoints**](https://huggingface.co/MananSuri27/Video2LoRA-SmolVLM-ckpts)\n",
|
||||
"- [**\ud83d\udcbb GitHub Repository**](https://github.com/MananSuri27/video2lora)"
|
||||
"* **[\ud83c\udf10 Project Website](https://video2lora.github.io/)** \u2014 Learn more about the project, watch video demonstrations, and see qualitative benchmarks.\n",
|
||||
"* **[\ud83d\udcc4 arXiv Paper](https://arxiv.org/abs/2606.04351)** \u2014 Read the full research paper detailing the architecture, training recipes, and evaluation.\n",
|
||||
"* **[\ud83e\udd17 Hugging Face Checkpoints](https://huggingface.co/MananSuri27/Video2LoRA-SmolVLM-ckpts)** \u2014 Access raw pre-trained checkpoints for SmolVLM2.\n",
|
||||
"* **[\ud83d\udcbb GitHub Repository](https://github.com/MananSuri27/video2lora)** \u2014 Explore the full codebase for training, evaluation, and data generation."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 0: Package Installation\n",
|
||||
"## \ud83d\udee0\ufe0f Step 0: Environment Setup & Package Installation\n",
|
||||
"\n",
|
||||
"First, we install all the required open-source packages in the environment."
|
||||
"In this step, we install the required open-source libraries and set up the directory structure. \n",
|
||||
"\n",
|
||||
"### \ud83d\udca1 Multi-Environment Compatibility\n",
|
||||
"This cell automatically detects whether you are running in **Google Colab** or a **local Jupyter/VS Code** environment:\n",
|
||||
"- **Google Colab**: It clones both the main `Video2LoRA` repository and the `video2lora.github.io` repository to retrieve the qualitative video files. It then sets up absolute path symlinks to resolve directories.\n",
|
||||
"- **Local Environment**: If you have the repositories cloned locally in the same workspace, it automatically detects and symlinks the website's `media` folder to avoid redundant video downloads.\n",
|
||||
"- **Dependencies**: Installs `transformers`, `peft`, `accelerate`, `decord`, `einops`, `jaxtyping`, and `num2words`."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -105,9 +119,13 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 1: Model Architectures & Utilities\n",
|
||||
"## \ud83e\udde9 Step 1: Model Architectures & Utilities\n",
|
||||
"\n",
|
||||
"Here, we define the full Video2LoRA model architecture classes (including the perceiver aggregator, hypernet heads, model load patches, and custom LoRA injection layers) directly inside the notebook. This makes the notebook completely independent of external python code files."
|
||||
"To ensure the tutorial is completely self-contained, we import the core architectures and utilities from the repository package. This includes:\n",
|
||||
"- **LoRA Layer Injection**: `apply_lora_to_layers` from `ctx_to_lora.modeling.lora_layer` to patch the base model's attention projections.\n",
|
||||
"- **LoRA Merger**: `combine_lora` from `ctx_to_lora.modeling.lora_merger` to combine weight matrices dynamically.\n",
|
||||
"- **Model Construction**: `build_stage1_model` from `scripts.video2lora.train_smolvlm_stage1` to load and instantiate the modulated VLM.\n",
|
||||
"- **VLM Preprocessing**: `prepare_smolvlm_inputs` and feature extraction helpers."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -172,9 +190,11 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 2: Load and Inspect Manifest\n",
|
||||
"## \ud83d\udcca Step 2: Load and Inspect Manifest\n",
|
||||
"\n",
|
||||
"Video2LoRA uses a standard JSONL manifest file for training and evaluation. Below, we define a sample manifest row matching the child watering plants example from the paper's qualitative evaluation."
|
||||
"Video2LoRA utilizes standard JSONL manifests to structure training and evaluation datasets. Below, we define the **6 qualitative examples** from the project page:\n",
|
||||
"- **Examples included**: *Man smoking pipe*, *Child watering plants*, *Posture alignment*, *Dog tugging toy*, *Rainy day weather query*, and *Tarsier close-up*.\n",
|
||||
"- **Local Video Retrieval**: The cell resolves directory paths recursively, handles broken symlinks automatically, and downloads the video files from the project website only if they are not already found locally."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -289,9 +309,9 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 3: Visualize Video Frames\n",
|
||||
"## \ud83d\uddbc\ufe0f Step 3: Visualize Video Frames\n",
|
||||
"\n",
|
||||
"Let's write a utility function to extract and display keyframes from the input video file using OpenCV and Matplotlib."
|
||||
"Before running inference, let's write a utility function using OpenCV and Matplotlib to extract keyframes from the video file and inspect the visual input. We'll visualize the first example: **Man smoking a pipe**."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -337,9 +357,9 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 4: Download Video2LoRA Checkpoint\n",
|
||||
"## \ud83d\udcbe Step 4: Download Video2LoRA Checkpoint\n",
|
||||
"\n",
|
||||
"We use the `huggingface_hub` client to download the 2.2B Video2LoRA checkpoint weights (`video2lora-smolvlm2-2.2b-best-ce.pt`) directly from Hugging Face."
|
||||
"We use the Hugging Face hub client to download the pre-trained weights for the **2.2B SmolVLM2 hypernetwork** (`video2lora-smolvlm2-2.2b-best-ce.pt`). This checkpoint is trained on FineVideo using Cross-Entropy teacher distillation."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -370,9 +390,12 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 5: Load Base VLM & Initialize Modulated Model\n",
|
||||
"## \ud83d\ude80 Step 5: Load Base VLM & Initialize Modulated Model\n",
|
||||
"\n",
|
||||
"Configure `TrainArgs` matching the 2.2B preset, load the base SmolVLM2 model, processor, and tokenizer, and load the hypernetwork checkpoint."
|
||||
"We configure `TrainArgs` matching the SmolVLM2 2.2B evaluation preset and:\n",
|
||||
"1. Initialize the base SmolVLM2 vision-language model (`HuggingFaceTB/SmolVLM2-2.2B-Instruct`), processor, and tokenizer.\n",
|
||||
"2. Load the pre-trained perceiver hypernetwork weights.\n",
|
||||
"3. Put the model in evaluation mode (`eval()`) and move it to the target hardware device (CUDA GPU if available, otherwise CPU)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -428,9 +451,9 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 6: Parametric Video Internalization\n",
|
||||
"## \u26a1 Step 6: Parametric Video Internalization\n",
|
||||
"\n",
|
||||
"We process the video frames through the VLM visual encoder to extract activations, then generate the custom LoRA adapter weights in a single pass of the hypernetwork."
|
||||
"This is where the core Video2LoRA magic happens! We feed the video frames through the visual encoder to extract intermediate activations layer-by-layer. Then, the perceiver hypernetwork processes these activations and generates the custom LoRA adapter weights in a single pass."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -496,9 +519,11 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 7: Run QA Inference (Base Model vs Video2LoRA)\n",
|
||||
"## \ud83d\udd0d Step 7: Run QA Inference (Base Model vs. Video2LoRA)\n",
|
||||
"\n",
|
||||
"To perform a fair comparison, we first run inference with the **Base Model** (which requires processing the full visual video tokens in context). Afterwards, we run inference with **Video2LoRA** (which bypasses visual tokens completely and queries the model using only text, utilizing the dynamic LoRA adapter)."
|
||||
"To evaluate the quality of the generated adapter, we perform a side-by-side comparison:\n",
|
||||
"- **Base Model (With Visual Tokens)**: We query the base model with the full visual video tokens in its context window (standard VLM execution).\n",
|
||||
"- **Video2LoRA (Zero Visual Tokens)**: We apply the hypernetwork-predicted LoRA weights to the model layers, clear out all visual tokens, and query the model using **text only**."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -599,9 +624,9 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 8: Qualitative Comparison Dashboard\n",
|
||||
"## \ud83c\udfa8 Step 8: Qualitative Comparison Dashboard\n",
|
||||
"\n",
|
||||
"Display the comparison between the Base Model and Video2LoRA using the dynamically generated prediction outputs in a custom IPython HTML layout matching the evaluation layout."
|
||||
"Finally, we run the internalization, inference, and evaluation loop for **all 6 qualitative examples** and render a custom HTML dashboard comparison matching the layout on the project website."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue