Video2LoRA: Parametric Video Internalization for Vision-Language Models

Official implementation of Video2LoRA

Manan Suri  ·  Sarvesh Baskar  ·  Dinesh Manocha

University of Maryland, College Park

---

Quick Start  •  Install  •  Checkpoints  •  Inference  •  Data Format  •  Training

Video2LoRA trains a hypernetwork that converts a video into LoRA weights for a frozen vision-language model. The generated adapter lets the model answer later text prompts without feeding the video tokens again.

🚀 Quick Start

Try zero-visual-token inference instantly in your browser via our self-contained interactive tutorial notebook:

This interactive notebook guides you through:

1. Downloading the 2.2B SmolVLM2 hypernetwork checkpoint.
2. Linking and resolving the project's qualitative videos.
3. Parametric internalization (extracting visual features and generating custom adapter weights).
4. Running comparative Q&A inference (comparing the base vision-language model against the Video2LoRA adapter with zero visual tokens).
5. Visualizing the comparison using a custom dashboard layout.

Install

Install uv.

git clone https://github.com/MananSuri27/video2lora.git
cd video2lora
uv sync

For local video path resolution, set:

export VIDEO2LORA_DATA_ROOT=$PWD/data/video2lora

If unset, VIDEO2LORA_DATA_ROOT defaults to data/video2lora.

Checkpoints

Download the released checkpoints from Hugging Face:

uv run huggingface-cli download MananSuri27/Video2LoRA-SmolVLM-ckpts \
  --local-dir checkpoints/Video2LoRA-SmolVLM-ckpts

The repo contains:

checkpoints/Video2LoRA-SmolVLM-ckpts/
  video2lora-smolvlm2-500m-best-ce.pt
  video2lora-smolvlm2-2.2b-best-ce.pt

Inference

Create a JSONL manifest with one row per video. For example:

{"id":"sample-0001","video_path":"/path/to/video.mp4","prompt":"Describe what is happening in this video.","task_type":"caption"}

uv run python -m scripts.video2lora.infer \
  --checkpoint checkpoints/Video2LoRA-SmolVLM-ckpts/video2lora-smolvlm2-500m-best-ce.pt \
  --manifest /path/to/manifest.jsonl \
  --output outputs/tiny_generations.jsonl

For the 2.2B checkpoint, change --checkpoint to:

checkpoints/Video2LoRA-SmolVLM-ckpts/video2lora-smolvlm2-2.2b-best-ce.pt

The output JSONL includes the original row fields plus prediction and checkpoint.

Data Format

Training and inference use JSONL manifests. Each line is one example:

{
  "id": "sample-0001",
  "video_path": "raw/finevideo/sample.mp4",
  "task_type": "caption",
  "prompt": "Describe what is happening in this video.",
  "target_text": "A person is cooking.",
  "dataset": "finevideo",
  "split": "train",
  "metadata": {}
}

Relative video_path values are resolved against VIDEO2LORA_DATA_ROOT. Absolute paths are used as-is.

Generate Teacher Data

The final CE training recipe uses cached teacher-generated targets. Starting from readable manifests, generate teacher targets with:

export VIDEO2LORA_DATA_ROOT=$PWD/data/video2lora

CUDA_VISIBLE_DEVICES=0 uv run python -m scripts.video2lora.generate_finevideo_teacher_targets \
  --input-manifest $VIDEO2LORA_DATA_ROOT/processed/finevideo/train.jsonl \
  --output-manifest $VIDEO2LORA_DATA_ROOT/processed/finevideo/train.teacher_visual_smolvlm.jsonl \
  --smolvlm-name-or-path HuggingFaceTB/SmolVLM2-2.2B-Instruct \
  --per-device-batch-size 8 \
  --max-frames 12 \
  --video-size-longest-edge 384

To shard generation manually, run the same command per GPU with --num-shards N --shard-index R, then merge with --merge-only.

Train

The main CE training entrypoint is:

uv run accelerate launch \
  --config_file accelerate_config.yaml \
  --num_processes 4 \
  -m scripts.video2lora.train_smolvlm_stage1 \
  --smolvlm-name-or-path HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
  --train-manifest data/video2lora/processed/finevideo/train.teacher_visual_smolvlm.jsonl \
  --val-manifest data/video2lora/processed/finevideo/val.teacher_visual_smolvlm.jsonl \
  --val-core-manifest data/video2lora/processed/finevideo/val.teacher_visual_smolvlm.jsonl \
  --output-dir runs/video2lora-smoke \
  --per-device-batch-size 2 \
  --gradient-accumulation-steps 8 \
  --max-steps 1000 \
  --target-modules down_proj \
  --lora-r 16 \
  --latent-size 512 \
  --n-latent-queries 8 \
  --num-blocks 9 \
  --max-frames 12 \
  --video-size-longest-edge 384 \
  --kl-weight 0.0 \
  --wandb-mode disabled

The self-distillation variant is also included:

uv run accelerate launch \
  --config_file accelerate_config.yaml \
  --num_processes 4 \
  -m scripts.video2lora.train_smolvlm_stage1_selfdistill \
  --smolvlm-name-or-path HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
  --train-manifest data/video2lora/processed/finevideo/train.jsonl \
  --val-manifest data/video2lora/processed/finevideo/val.jsonl \
  --val-gen-manifest data/video2lora/processed/finevideo/val_gen_100.jsonl \
  --output-dir runs/video2lora-selfdistill \
  --wandb-mode disabled

Build FineVideo Manifests

If you have raw FineVideo metadata/videos laid out under $VIDEO2LORA_DATA_ROOT/raw/finevideo, build Stage 1 manifests with:

uv run python -m scripts.video2lora.build_finevideo_stage1_manifest