apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
doc-to-lora/demo.ipynb

772 lines
No EOL
36 KiB
Text
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# \ud83c\udfa5 Video2LoRA: Self-Contained Zero-Visual-Token Inference Tutorial\n",
"\n",
"Welcome to the official interactive tutorial for **Video2LoRA**! \n",
"\n",
"This notebook provides a **completely self-contained, single-file walkthrough** demonstrating how to run zero-visual-token video inference. \n",
"\n",
"### \ud83e\udde0 The Core Concept: Parametric Video Internalization\n",
"Typically, Vision-Language Models (VLMs) process videos by projecting video frames into visual tokens and feeding hundreds (or thousands) of these tokens directly into the model's context window. This consumes massive context space, increases Time-to-First-Token (TTFT) latency, and slows down generation.\n",
"\n",
"**Video2LoRA** introduces a paradigm shift:\n",
"1. **Hypernetwork Generation**: A perceiver hypernetwork reads a video's intermediate activation features layer-by-layer and directly predicts a set of Low-Rank Adaptation (LoRA) weights in a single forward pass.\n",
"2. **Internalization**: Once generated, these adapter weights are injected into the base Vision-Language Model.\n",
"3. **Zero-Token Inference**: The VLM can now answer arbitrary text queries about the video **without requiring a single visual token** in its context window! The video information has been *internalized* parametrically into the model's weights.\n",
"\n",
"---\n",
"\n",
"### \ud83d\udd17 Project Resources\n",
"* **[\ud83c\udf10 Project Website](https://video2lora.github.io/)** \u2014 Learn more about the project, watch video demonstrations, and see qualitative benchmarks.\n",
"* **[\ud83d\udcc4 arXiv Paper](https://arxiv.org/abs/2606.04351)** \u2014 Read the full research paper detailing the architecture, training recipes, and evaluation.\n",
"* **[\ud83e\udd17 Hugging Face Checkpoints](https://huggingface.co/MananSuri27/Video2LoRA-SmolVLM-ckpts)** \u2014 Access raw pre-trained checkpoints for SmolVLM2.\n",
"* **[\ud83d\udcbb GitHub Repository](https://github.com/MananSuri27/video2lora)** \u2014 Explore the full codebase for training, evaluation, and data generation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83d\udee0\ufe0f Step 0: Environment Setup & Package Installation\n",
"\n",
"In this step, we install the required open-source libraries and set up the directory structure. \n",
"\n",
"### \ud83d\udca1 Multi-Environment Compatibility\n",
"This cell automatically detects whether you are running in **Google Colab** or a **local Jupyter/VS Code** environment:\n",
"- **Google Colab**: It clones both the main `Video2LoRA` repository and the `video2lora.github.io` repository to retrieve the qualitative video files. It then sets up absolute path symlinks to resolve directories.\n",
"- **Local Environment**: If you have the repositories cloned locally in the same workspace, it automatically detects and symlinks the website's `media` folder to avoid redundant video downloads.\n",
"- **Dependencies**: Installs `transformers`, `peft`, `accelerate`, `decord`, `einops`, `jaxtyping`, and `num2words`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Detect if running in Google Colab\n",
"try:\n",
" import google.colab\n",
" IN_COLAB = True\n",
"except ImportError:\n",
" IN_COLAB = False\n",
"\n",
"import os\n",
"\n",
"if IN_COLAB:\n",
" print(\"Running in Google Colab. Setting up workspace and website repositories...\")\n",
" # 1. Clone the main Video2LoRA repository to absolute path\n",
" if not os.path.exists(\"/content/Video2LoRA\"):\n",
" os.system(\"git clone -b demo https://github.com/video2lora/Video2LoRA.git /content/Video2LoRA\")\n",
" # 2. Clone the website repository to absolute path\n",
" if not os.path.exists(\"/content/video2lora.github.io\"):\n",
" os.system(\"git clone https://github.com/video2lora/video2lora.github.io.git /content/video2lora.github.io\")\n",
" # 3. Change directory to the codebase workspace\n",
" os.chdir(\"/content/Video2LoRA\")\n",
"\n",
"# Automatically set up the media symlink if the website repository is found nearby.\n",
"# This ensures that both local and Colab environments can resolve paths correctly.\n",
"if os.path.lexists(\"media\"):\n",
" # If media exists but is a broken symlink or not a directory, remove it\n",
" if not os.path.isdir(\"media\") or os.path.islink(\"media\"):\n",
" print(\"Removing conflicting/broken media symlink or file...\")\n",
" try:\n",
" if os.path.islink(\"media\"):\n",
" os.unlink(\"media\")\n",
" else:\n",
" import shutil\n",
" shutil.rmtree(\"media\")\n",
" except Exception as e:\n",
" os.system(\"rm -rf media\")\n",
"\n",
"# Dynamically find the website's media directory\n",
"website_media_path = None\n",
"possible_paths = [\n",
" \"/content/video2lora.github.io/media\",\n",
" \"/content/Video2LoRA/video2lora.github.io/media\",\n",
" \"video2lora.github.io/media\",\n",
" \"../video2lora.github.io/media\",\n",
" \"../../video2lora.github.io/media\",\n",
"]\n",
"for p in possible_paths:\n",
" abs_p = os.path.abspath(p)\n",
" if os.path.exists(abs_p) and os.path.isdir(abs_p):\n",
" website_media_path = abs_p\n",
" break\n",
"\n",
"if website_media_path:\n",
" print(f\"Linking media directory from website repository at: {website_media_path}\")\n",
" try:\n",
" os.symlink(website_media_path, \"media\")\n",
" except Exception as e:\n",
" print(f\"Failed to create symlink using os.symlink, trying shell: {e}\")\n",
" os.system(f\"ln -s {website_media_path} media\")\n",
"else:\n",
" print(\"Warning: Website media directory not found. Will download videos on demand.\")\n",
"\n",
"print(f\"Workspace directory: {os.getcwd()}\")\n",
"\n",
"# Install necessary packages\n",
"!pip install torch torchvision torchaudio\n",
"!pip install transformers accelerate peft huggingface_hub decord av opencv-python matplotlib einops jaxtyping num2words \"torchao>=0.16.0\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83e\udde9 Step 1: Model Architectures & Utilities\n",
"\n",
"To ensure the tutorial is completely self-contained, we import the core architectures and utilities from the repository package. This includes:\n",
"- **LoRA Layer Injection**: `apply_lora_to_layers` from `ctx_to_lora.modeling.lora_layer` to patch the base model's attention projections.\n",
"- **LoRA Merger**: `combine_lora` from `ctx_to_lora.modeling.lora_merger` to combine weight matrices dynamically.\n",
"- **Model Construction**: `build_stage1_model` from `scripts.video2lora.train_smolvlm_stage1` to load and instantiate the modulated VLM.\n",
"- **VLM Preprocessing**: `prepare_smolvlm_inputs` and feature extraction helpers."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import os\n",
"from pathlib import Path\n",
"\n",
"# Add repository root and src directory to sys.path to enable local imports\n",
"repo_root = str(Path(os.getcwd()).resolve())\n",
"if repo_root not in sys.path:\n",
" sys.path.insert(0, repo_root)\n",
"src_dir = str(Path(repo_root) / \"src\")\n",
"if os.path.exists(src_dir) and src_dir not in sys.path:\n",
" sys.path.insert(0, src_dir)\n",
"\n",
"import torch\n",
"import cv2\n",
"import matplotlib.pyplot as plt\n",
"from dataclasses import dataclass\n",
"from typing import List, Optional\n",
"\n",
"# Import custom Video2LoRA architectures and utilities from the repository modules\n",
"from ctx_to_lora.modeling.lora_layer import apply_lora_to_layers\n",
"from ctx_to_lora.modeling.lora_merger import combine_lora\n",
"from scripts.video2lora.train_smolvlm_stage1 import build_stage1_model\n",
"from scripts.video2lora.train_smolvlm_online import (\n",
" prepare_smolvlm_inputs,\n",
" extract_l2l_fused_text_features,\n",
")\n",
"\n",
"@dataclass\n",
"class TrainArgs:\n",
" smolvlm_name_or_path: str\n",
" train_manifest: str\n",
" val_manifest: str\n",
" output_dir: str\n",
" lora_r: int = 16\n",
" lora_dropout: float = 0.0\n",
" target_modules: Optional[List[str]] = None\n",
" latent_size: int = 512\n",
" dropout_rate: float = 0.0\n",
" n_latent_queries: int = 8\n",
" num_blocks: int = 9\n",
" num_self_attn_per_block: int = 0\n",
" video_fps: Optional[float] = None\n",
" max_frames: int = 12\n",
" video_size_longest_edge: int = 384\n",
" video_load_backend: str = \"auto\"\n",
" internalization_prompt: str = \"Internalize this video for later captioning.\"\n",
" kl_weight: float = 0.0\n",
" generation_max_new_tokens: int = 128\n",
"\n",
"print(\"Video2LoRA imported architectures and utility functions successfully.\")\n",
"\n",
"# Force-refresh Hugging Face dependency cache for num2words if installed mid-session\n",
"import sys\n",
"import importlib.util\n",
"if importlib.util.find_spec(\"num2words\") is not None:\n",
" try:\n",
" import num2words\n",
" import transformers.utils.import_utils\n",
" transformers.utils.import_utils._num2words_available = True\n",
" for mod_name in list(sys.modules.keys()):\n",
" if \"processing_smolvlm\" in mod_name or \"smolvlm\" in mod_name:\n",
" mod = sys.modules[mod_name]\n",
" if hasattr(mod, \"num2words\") and getattr(mod, \"num2words\") is None:\n",
" setattr(mod, \"num2words\", num2words)\n",
" except Exception:\n",
" pass\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83d\udcca Step 2: Load and Inspect Manifest\n",
"\n",
"Video2LoRA utilizes standard JSONL manifests to structure training and evaluation datasets. Below, we define the **6 qualitative examples** from the project page:\n",
"- **Examples included**: *Man smoking pipe*, *Child watering plants*, *Posture alignment*, *Dog tugging toy*, *Rainy day weather query*, and *Tarsier close-up*.\n",
"- **Local Video Retrieval**: The cell resolves directory paths recursively, handles broken symlinks automatically, and downloads the video files from the project website only if they are not already found locally."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Define the 6 qualitative examples from the Video2LoRA project website\n",
"examples = [\n",
" {\n",
" \"id\": \"man-smoking-pipe\",\n",
" \"video_path\": \"media/benchmarks/carebench/v_00014063_0.mp4\",\n",
" \"dataset\": \"CaReBench: Caption\",\n",
" \"prompt\": \"Describe the video in as much useful visual detail as possible. Include the main activity, visible people or objects, scene context, appearance, and any important visual details that help explain what is happening.\",\n",
" \"target_text\": \"This video depicts a scene of a man lighting a pipe with a lighter. The man in the video is smoking a pipe held in his mouth, supported by his left hand, while his right hand grips the lighter. His right forearm features a large black tattoo. He has short, thick hair that is a deep brown color and is dressed in a loose-fitting black tank top. He is seated next to a window, which has a wooden frame and blue curtains, with a brick wall behind him and a wooden door on the right. The door has a square pattern, adorned with silver hinges and a doorknob. In the video, he first ignites the lighter with his right hand and then brings the flame to the pipe, holding it in that position for several seconds. Throughout this time, his left hand remains steady on the pipe, and his gaze is fixed intently on it, ensuring that the pipe is fully lit before setting the lighter down. He then continues to hold the pipe with his left hand and begins to smoke. The video is shot from the front, clearly illustrating how relaxed he is while smoking at home.\"\n",
" },\n",
" {\n",
" \"id\": \"child-watering-plants\",\n",
" \"video_path\": \"media/benchmarks/carebench/v_00016555_0.mp4\",\n",
" \"dataset\": \"CaReBench: Events\",\n",
" \"prompt\": \"Describe the key visible events in chronological order. Include all important actions and changes you can observe, with enough detail to distinguish each event clearly.\",\n",
" \"target_text\": \"Little boy watering plants outdoors; Using watering can to pour water into flower pot; Shifting camera angle from side view to rear view; Tapping edge of flower pot a few times; Setting down watering can\"\n",
" },\n",
" {\n",
" \"id\": \"posture-alignment\",\n",
" \"video_path\": \"media/benchmarks/plm/f522598789220c70_122_155.mp4\",\n",
" \"dataset\": \"PLM-SGQA\",\n",
" \"prompt\": \"Does this look like the same posture she's holding?\",\n",
" \"target_text\": \"Yes, it appears you're mirroring the same posture. Your alignment, knee bend, and spine position match the demonstration, indicating proper form and engagement of the targeted muscle groups for optimal effectiveness and safety.\"\n",
" },\n",
" {\n",
" \"id\": \"dog-tugging\",\n",
" \"video_path\": \"media/benchmarks/plm/b5bdb7f254cb1727_369_400.mp4\",\n",
" \"dataset\": \"PLM-SGQA\",\n",
" \"prompt\": \"Is he trying to tug?\",\n",
" \"target_text\": \"Yes, your dog is likely inviting a tug-of-war game. Holding the toy in his mouth and possibly looking at you or wagging his tail indicates he's ready to engage in a playful tug.\"\n",
" },\n",
" {\n",
" \"id\": \"rainy-day\",\n",
" \"video_path\": \"media/benchmarks/vidcapbench/132065802449.mp4\",\n",
" \"dataset\": \"VidCapBench\",\n",
" \"prompt\": \"What is the weather like in the scene? Answer only the question, in one sentence.\",\n",
" \"target_text\": \"Rainy day.\"\n",
" },\n",
" {\n",
" \"id\": \"tarsier-creature\",\n",
" \"video_path\": \"media/benchmarks/vidcapbench/Tarsier_20.mp4\",\n",
" \"dataset\": \"VidCapBench\",\n",
" \"prompt\": \"Which parts of the creature are highlighted in the video? Answer only the question, in one sentence.\",\n",
" \"target_text\": \"A close-up of its face, eyes, and hair.\"\n",
" }\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import urllib.request\n",
"\n",
"# Download the qualitative videos from the project website repo\n",
"print(\"Checking/downloading qualitative benchmark video files from project website...\")\n",
"for item in examples:\n",
" video_path = item[\"video_path\"]\n",
" dir_name = os.path.dirname(video_path)\n",
" \n",
" # 1. Clean up any broken symlinks or conflicting files along the directories path\n",
" # to avoid FileNotFoundError during folder creation.\n",
" parts = video_path.split('/')\n",
" current_path = \"\"\n",
" for part in parts[:-1]: # skip the filename\n",
" if current_path:\n",
" current_path = os.path.join(current_path, part)\n",
" else:\n",
" current_path = part\n",
" if os.path.lexists(current_path) and not os.path.isdir(current_path):\n",
" print(f\"Removing conflict at '{current_path}' (broken link or file) to allow directory creation...\")\n",
" try:\n",
" if os.path.islink(current_path):\n",
" os.unlink(current_path)\n",
" else:\n",
" os.remove(current_path)\n",
" except Exception as e:\n",
" print(f\"Failed to remove conflict at {current_path}: {e}\")\n",
" \n",
" # 2. Safely create directory structure\n",
" try:\n",
" os.makedirs(dir_name, exist_ok=True)\n",
" except Exception as e:\n",
" print(f\"Warning: Could not create directory {dir_name}: {e}\")\n",
" \n",
" # 3. Check and retrieve the video file\n",
" if not os.path.exists(video_path):\n",
" url = f\"https://video2lora.github.io/{video_path}\"\n",
" print(f\"Downloading {video_path} from {url}...\")\n",
" try:\n",
" urllib.request.urlretrieve(url, video_path)\n",
" print(\"Download successful.\")\n",
" except Exception as e:\n",
" print(f\"Failed to download: {e}\")\n",
" else:\n",
" print(f\"Found local video file: {video_path}\")\n",
"\n",
"print(f\"\\nLoaded {len(examples)} qualitative examples successfully.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83d\uddbc\ufe0f Step 3: Visualize Video Frames\n",
"\n",
"Before running inference, let's write a utility function using OpenCV and Matplotlib to extract keyframes from the video file and inspect the visual input. We'll visualize the first example: **Man smoking a pipe**."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def show_video_frames(video_path, num_frames=4):\n",
" \"\"\"\n",
" Helper function to load keyframes from a video path and plot them in a grid.\n",
" \"\"\"\n",
" if not os.path.exists(video_path):\n",
" print(f\"Video file not found at: {video_path} (Please provide a valid video to inspect).\")\n",
" return\n",
" \n",
" cap = cv2.VideoCapture(video_path)\n",
" total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n",
" indices = [int(i * total_frames / num_frames) for i in range(num_frames)]\n",
" \n",
" fig, axes = plt.subplots(1, num_frames, figsize=(16, 4))\n",
" for i, idx in enumerate(indices):\n",
" cap.set(cv2.CAP_PROP_POS_FRAMES, idx)\n",
" ret, frame = cap.read()\n",
" if ret:\n",
" frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
" axes[i].imshow(frame)\n",
" axes[i].axis('off')\n",
" axes[i].set_title(f\"Frame {idx}\")\n",
" else:\n",
" axes[i].text(0.5, 0.5, \"Frame Read Error\", ha=\"center\", va=\"center\")\n",
" axes[i].axis('off')\n",
" plt.tight_layout()\n",
" plt.show()\n",
" cap.release()\n",
"\n",
"# Visualize the first qualitative example (man smoking a pipe)\n",
"print(f\"Visualizing keyframes for Example 1: {examples[0]['id']}\")\n",
"show_video_frames(examples[0][\"video_path\"], num_frames=4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83d\udcbe Step 4: Download Video2LoRA Checkpoint\n",
"\n",
"We use the Hugging Face hub client to download the pre-trained weights for the **2.2B SmolVLM2 hypernetwork** (`video2lora-smolvlm2-2.2b-best-ce.pt`). This checkpoint is trained on FineVideo using Cross-Entropy teacher distillation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"from huggingface_hub import hf_hub_download\n",
"\n",
"print(\"Downloading 2.2B Video2LoRA Checkpoint from Hugging Face...\")\n",
"\n",
"checkpoint_dir = Path(\"checkpoints/Video2LoRA-SmolVLM-ckpts\")\n",
"checkpoint_dir.mkdir(parents=True, exist_ok=True)\n",
"\n",
"# Download checkpoint weights programmatically\n",
"ckpt_path = hf_hub_download(\n",
" repo_id=\"MananSuri27/Video2LoRA-SmolVLM-ckpts\",\n",
" filename=\"video2lora-smolvlm2-2.2b-best-ce.pt\",\n",
" local_dir=str(checkpoint_dir)\n",
")\n",
"\n",
"print(f\"Checkpoint successfully downloaded to: {ckpt_path}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83d\ude80 Step 5: Load Base VLM & Initialize Modulated Model\n",
"\n",
"We configure `TrainArgs` matching the SmolVLM2 2.2B evaluation preset and:\n",
"1. Initialize the base SmolVLM2 vision-language model (`HuggingFaceTB/SmolVLM2-2.2B-Instruct`), processor, and tokenizer.\n",
"2. Load the pre-trained perceiver hypernetwork weights.\n",
"3. Put the model in evaluation mode (`eval()`) and move it to the target hardware device (CUDA GPU if available, otherwise CPU)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set configuration arguments for the 2.2B SmolVLM2 model preset\n",
"train_args = TrainArgs(\n",
" smolvlm_name_or_path=\"HuggingFaceTB/SmolVLM2-2.2B-Instruct\",\n",
" train_manifest=\"\",\n",
" val_manifest=\"\",\n",
" output_dir=\"\",\n",
" lora_r=16,\n",
" lora_dropout=0.0,\n",
" target_modules=[\"down_proj\"],\n",
" latent_size=512,\n",
" dropout_rate=0.0,\n",
" n_latent_queries=8,\n",
" num_blocks=9,\n",
" num_self_attn_per_block=0,\n",
" video_fps=None,\n",
" max_frames=12,\n",
" video_size_longest_edge=384,\n",
" video_load_backend=\"auto\",\n",
" internalization_prompt=\"Internalize this video for later captioning.\",\n",
" kl_weight=0.0,\n",
" generation_max_new_tokens=128\n",
")\n",
"\n",
"# Configure device\n",
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"\n",
"# 1. Initialize SmolVLM2 base model, processor, and tokenizer\n",
"print(\"Loading SmolVLM2 model (2.2B-Instruct)...\")\n",
"\n",
"# Force-refresh Hugging Face dependency cache for num2words if installed mid-session\n",
"import sys\n",
"import importlib.util\n",
"if importlib.util.find_spec(\"num2words\") is not None:\n",
" try:\n",
" import num2words\n",
" import transformers.utils.import_utils\n",
" transformers.utils.import_utils._num2words_available = True\n",
" for mod_name in list(sys.modules.keys()):\n",
" if \"processing_smolvlm\" in mod_name or \"smolvlm\" in mod_name:\n",
" mod = sys.modules[mod_name]\n",
" if hasattr(mod, \"num2words\") and getattr(mod, \"num2words\") is None:\n",
" setattr(mod, \"num2words\", num2words)\n",
" except Exception:\n",
" pass\n",
"\n",
"model, raw_model, processor, tokenizer = build_stage1_model(train_args, device=device)\n",
"\n",
"# 2. Load the hypernetwork weights\n",
"print(\"Loading hypernetwork weights...\")\n",
"state_dict = torch.load(ckpt_path, map_location=\"cpu\", weights_only=False)\n",
"model.load_state_dict(state_dict)\n",
"\n",
"# Place models in evaluation mode\n",
"model.to(device)\n",
"model.eval()\n",
"raw_model.eval()\n",
"\n",
"print(\"Model load complete.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \u26a1 Step 6: Parametric Video Internalization\n",
"\n",
"This is where the core Video2LoRA magic happens! We feed the video frames through the visual encoder to extract intermediate activations layer-by-layer. Then, the perceiver hypernetwork processes these activations and generates the custom LoRA adapter weights in a single pass."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def run_internalization(example, model, raw_model, processor, train_args, device):\n",
" \"\"\"\n",
" Extracts visual features layer-by-layer and generates the dynamic LoRA adapter.\n",
" \"\"\"\n",
" internalize_messages = [\n",
" [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"video\", \"path\": example[\"video_path\"]},\n",
" {\"type\": \"text\", \"text\": train_args.internalization_prompt}\n",
" ]\n",
" }\n",
" ]\n",
" ]\n",
"\n",
" vlm_inputs = prepare_smolvlm_inputs(\n",
" processor,\n",
" internalize_messages,\n",
" device,\n",
" video_fps=train_args.video_fps,\n",
" max_frames=train_args.max_frames,\n",
" video_size_longest_edge=train_args.video_size_longest_edge,\n",
" video_load_backend=train_args.video_load_backend\n",
" )\n",
"\n",
" ctx_features, ctx_attn_mask, ctx_position_ids = extract_l2l_fused_text_features(\n",
" raw_model,\n",
" vlm_inputs,\n",
" num_target_layers=model.hypernet.n_layers\n",
" )\n",
"\n",
" generated_loras, _ = model.generate_weights(\n",
" ctx_ids=None,\n",
" ctx_features=ctx_features,\n",
" ctx_attn_mask=ctx_attn_mask,\n",
" ctx_position_ids=ctx_position_ids\n",
" )\n",
"\n",
" generated_loras = combine_lora(\n",
" generated_loras,\n",
" torch.ones(1, dtype=torch.int32, device=device),\n",
" lora_bias=model.hypernet.get_head_bias() if model.hypernet.config.use_bias else None\n",
" )\n",
"\n",
" return generated_loras\n",
"\n",
"# Example run of internalization for child watering plants (index 1)\n",
"print(f\"Running internalization for Example 2: {examples[1]['id']}...\")\n",
"sample_loras = run_internalization(examples[1], model, raw_model, processor, train_args, device)\n",
"print(\"Internalization successful.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83d\udd0d Step 7: Run QA Inference (Base Model vs. Video2LoRA)\n",
"\n",
"To evaluate the quality of the generated adapter, we perform a side-by-side comparison:\n",
"- **Base Model (With Visual Tokens)**: We query the base model with the full visual video tokens in its context window (standard VLM execution).\n",
"- **Video2LoRA (Zero Visual Tokens)**: We apply the hypernetwork-predicted LoRA weights to the model layers, clear out all visual tokens, and query the model using **text only**."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def run_inference(example, generated_loras, model, raw_model, processor, tokenizer, train_args, device):\n",
" \"\"\"\n",
" Runs inference comparing base model (using visual tokens) vs Video2LoRA (zero visual tokens).\n",
" \"\"\"\n",
" # === Part A: Base Model Inference (with visual tokens) ===\n",
" base_messages = [\n",
" [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"video\", \"path\": example[\"video_path\"]},\n",
" {\"type\": \"text\", \"text\": example[\"prompt\"]}\n",
" ]\n",
" }\n",
" ]\n",
" ]\n",
"\n",
" base_inputs = prepare_smolvlm_inputs(\n",
" processor,\n",
" base_messages,\n",
" device,\n",
" video_fps=train_args.video_fps,\n",
" max_frames=train_args.max_frames,\n",
" video_size_longest_edge=train_args.video_size_longest_edge,\n",
" video_load_backend=train_args.video_load_backend\n",
" )\n",
"\n",
" base_generated_ids = raw_model.generate(\n",
" **base_inputs,\n",
" max_new_tokens=train_args.generation_max_new_tokens,\n",
" do_sample=False,\n",
" pad_token_id=tokenizer.pad_token_id,\n",
" eos_token_id=tokenizer.eos_token_id\n",
" )\n",
"\n",
" base_prediction = tokenizer.decode(\n",
" base_generated_ids[0][base_inputs[\"input_ids\"].shape[1]:],\n",
" skip_special_tokens=True\n",
" ).strip()\n",
"\n",
" # === Part B: Video2LoRA Inference (zero visual tokens) ===\n",
" apply_lora_to_layers(\n",
" model.base_model,\n",
" model.hypernet.layer_indices,\n",
" generated_loras,\n",
" torch.ones(1, dtype=torch.int32, device=device),\n",
" position_ids=None\n",
" )\n",
"\n",
" prompt_ids = tokenizer.apply_chat_template(\n",
" [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [{\"type\": \"text\", \"text\": example[\"prompt\"]}]\n",
" }\n",
" ],\n",
" tokenize=True,\n",
" add_generation_prompt=True,\n",
" return_tensors=\"pt\"\n",
" ).to(device)\n",
"\n",
" generated_ids = model.base_model.generate(\n",
" input_ids=prompt_ids,\n",
" attention_mask=torch.ones_like(prompt_ids),\n",
" max_new_tokens=train_args.generation_max_new_tokens,\n",
" do_sample=False,\n",
" pad_token_id=tokenizer.pad_token_id,\n",
" eos_token_id=tokenizer.eos_token_id\n",
" )\n",
"\n",
" video2lora_prediction = tokenizer.decode(\n",
" generated_ids[0][prompt_ids.shape[1]:],\n",
" skip_special_tokens=True\n",
" ).strip()\n",
"\n",
" # Reset model LoRA patches\n",
" model.reset()\n",
"\n",
" return base_prediction, video2lora_prediction\n",
"\n",
"# Run comparison inference on the sample example\n",
"print(f\"Running inference for Example 2: {examples[1]['id']}...\")\n",
"base_prediction, video2lora_prediction = run_inference(\n",
" examples[1], sample_loras, model, raw_model, processor, tokenizer, train_args, device\n",
")\n",
"print(f\"\\nPredictions Generated:\\nBase Model: {base_prediction}\\nVideo2LoRA: {video2lora_prediction}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## \ud83c\udfa8 Step 8: Qualitative Comparison Dashboard\n",
"\n",
"Finally, we run the internalization, inference, and evaluation loop for **all 6 qualitative examples** and render a custom HTML dashboard comparison matching the layout on the project website."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import HTML, display\n",
"\n",
"def display_comparison(\n",
" video_path,\n",
" question_prompt,\n",
" ground_truth,\n",
" base_model_output,\n",
" video2lora_output,\n",
" dataset_name\n",
"):\n",
" \"\"\"\n",
" Renders a beautifully styled comparison board with local HTML5 video player.\n",
" \"\"\"\n",
" html_content = f\"\"\"\n",
" <div style=\"font-family: 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; max-width: 900px; margin: 20px auto; color: #1e293b; background-color: #f8fafc; padding: 24px; border-radius: 16px; box-shadow: 0 4px 20px rgba(0,0,0,0.05);\">\n",
" \n",
" <!-- Header Source Badge -->\n",
" <div style=\"display: flex; justify-content: space-between; align-items: center; margin-bottom: 16px;\">\n",
" <h3 style=\"margin: 0; font-size: 18px; font-weight: 700; color: #0f172a;\">Qualitative Comparison</h3>\n",
" <span style=\"background-color: #e2e8f0; color: #475569; padding: 4px 10px; border-radius: 9999px; font-size: 12px; font-weight: 600;\">{dataset_name}</span>\n",
" </div>\n",
"\n",
" <!-- Local HTML5 Video Player -->\n",
" <div style=\"text-align: center; margin-bottom: 20px; background: #000; border-radius: 12px; overflow: hidden; box-shadow: 0 4px 12px rgba(0,0,0,0.15);\">\n",
" <video src=\"{video_path}\" controls loop muted autoplay style=\"width: 100%; max-height: 400px; object-fit: contain; display: block; margin: 0 auto;\"></video>\n",
" </div>\n",
" \n",
" <!-- Question Prompt Card -->\n",
" <div style=\"background-color: #f0f4ff; border: 1px solid #dbeafe; border-radius: 8px; padding: 16px; margin-bottom: 16px;\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #1e40af; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">QUESTION PROMPT</h5>\n",
" <p style=\"margin: 0; font-size: 15px; line-height: 1.5; color: #1e293b;\">{question_prompt}</p>\n",
" </div>\n",
" \n",
" <!-- Ground Truth Card -->\n",
" <div style=\"background-color: #fefbeb; border: 1px solid #fef3c7; border-radius: 8px; padding: 16px; margin-bottom: 16px;\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #854d0e; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">GROUND TRUTH</h5>\n",
" <p style=\"margin: 0; font-size: 15px; line-height: 1.5; color: #1e293b; font-weight: 500;\">{ground_truth}</p>\n",
" </div>\n",
" \n",
" <!-- Model Comparisons Grid -->\n",
" <div style=\"display: grid; grid-template-columns: 1fr 1fr; gap: 16px;\">\n",
" \n",
" <!-- Base Model Card -->\n",
" <div style=\"background-color: #fff; border: 1.5px solid #ea580c; border-radius: 8px; padding: 16px; box-shadow: 0 2px 4px rgba(0,0,0,0.02);\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #ea580c; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">BASE MODEL (with visual tokens)</h5>\n",
" <p style=\"margin: 0; font-size: 14px; line-height: 1.5; color: #334155;\">{base_model_output}</p>\n",
" </div>\n",
" \n",
" <!-- Video2LoRA Card -->\n",
" <div style=\"background-color: #f0fdf4; border: 1.5px solid #16a34a; border-radius: 8px; padding: 16px; box-shadow: 0 2px 4px rgba(0,0,0,0.02);\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #16a34a; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">VIDEO2LORA (zero visual tokens)</h5>\n",
" <p style=\"margin: 0; font-size: 14px; line-height: 1.5; color: #1e293b; font-weight: 500;\">{video2lora_output}</p>\n",
" </div>\n",
" \n",
" </div>\n",
" \n",
" </div>\n",
" \"\"\"\n",
" display(HTML(html_content))\n",
"\n",
"# Loop through all 6 qualitative examples, run internalization + inference, and display the dashboards\n",
"print(\"Processing and running qualitative comparison inference for all 6 examples...\")\n",
"for idx, example in enumerate(examples):\n",
" print(f\"\\n--- Running Example {idx+1}/{len(examples)}: {example['id']} ---\")\n",
" # 1. Internalize\n",
" loras = run_internalization(example, model, raw_model, processor, train_args, device)\n",
" # 2. Run inference\n",
" base_pred, v2l_pred = run_inference(example, loras, model, raw_model, processor, tokenizer, train_args, device)\n",
" # 3. Render dashboard\n",
" display_comparison(\n",
" video_path=example[\"video_path\"],\n",
" question_prompt=example[\"prompt\"],\n",
" ground_truth=example[\"target_text\"],\n",
" base_model_output=base_pred,\n",
" video2lora_output=v2l_pred,\n",
" dataset_name=example[\"dataset\"]\n",
" )"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in a new issue View git blame Copy permalink