{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# \ud83c\udfa5 Video2LoRA: Self-Contained Zero-Visual-Token Inference Tutorial\n", "\n", "Welcome to the official interactive tutorial for **Video2LoRA**! \n", "\n", "This notebook provides a **completely self-contained, single-file walkthrough** demonstrating how to run zero-visual-token video inference. \n", "\n", "### \ud83e\udde0 The Core Concept: Parametric Video Internalization\n", "Typically, Vision-Language Models (VLMs) process videos by projecting video frames into visual tokens and feeding hundreds (or thousands) of these tokens directly into the model's context window. This consumes massive context space, increases Time-to-First-Token (TTFT) latency, and slows down generation.\n", "\n", "**Video2LoRA** introduces a paradigm shift:\n", "1. **Hypernetwork Generation**: A perceiver hypernetwork reads a video's intermediate activation features layer-by-layer and directly predicts a set of Low-Rank Adaptation (LoRA) weights in a single forward pass.\n", "2. **Internalization**: Once generated, these adapter weights are injected into the base Vision-Language Model.\n", "3. **Zero-Token Inference**: The VLM can now answer arbitrary text queries about the video **without requiring a single visual token** in its context window! The video information has been *internalized* parametrically into the model's weights.\n", "\n", "---\n", "\n", "### \ud83d\udd17 Project Resources\n", "* **[\ud83c\udf10 Project Website](https://video2lora.github.io/)** \u2014 Learn more about the project, watch video demonstrations, and see qualitative benchmarks.\n", "* **[\ud83d\udcc4 arXiv Paper](https://arxiv.org/abs/2606.04351)** \u2014 Read the full research paper detailing the architecture, training recipes, and evaluation.\n", "* **[\ud83e\udd17 Hugging Face Checkpoints](https://huggingface.co/MananSuri27/Video2LoRA-SmolVLM-ckpts)** \u2014 Access raw pre-trained checkpoints for SmolVLM2.\n", "* **[\ud83d\udcbb GitHub Repository](https://github.com/MananSuri27/video2lora)** \u2014 Explore the full codebase for training, evaluation, and data generation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\udee0\ufe0f Step 0: Environment Setup & Package Installation\n", "\n", "In this step, we install the required open-source libraries and set up the directory structure. \n", "\n", "### \ud83d\udca1 Multi-Environment Compatibility\n", "This cell automatically detects whether you are running in **Google Colab** or a **local Jupyter/VS Code** environment:\n", "- **Google Colab**: It clones both the main `Video2LoRA` repository and the `video2lora.github.io` repository to retrieve the qualitative video files. It then sets up absolute path symlinks to resolve directories.\n", "- **Local Environment**: If you have the repositories cloned locally in the same workspace, it automatically detects and symlinks the website's `media` folder to avoid redundant video downloads.\n", "- **Dependencies**: Installs `transformers`, `peft`, `accelerate`, `decord`, `einops`, `jaxtyping`, and `num2words`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Detect if running in Google Colab\n", "try:\n", " import google.colab\n", " IN_COLAB = True\n", "except ImportError:\n", " IN_COLAB = False\n", "\n", "import os\n", "\n", "if IN_COLAB:\n", " print(\"Running in Google Colab. Setting up workspace and website repositories...\")\n", " # 1. Clone the main Video2LoRA repository to absolute path\n", " if not os.path.exists(\"/content/Video2LoRA\"):\n", " os.system(\"git clone -b demo https://github.com/video2lora/Video2LoRA.git /content/Video2LoRA\")\n", " # 2. Clone the website repository to absolute path\n", " if not os.path.exists(\"/content/video2lora.github.io\"):\n", " os.system(\"git clone https://github.com/video2lora/video2lora.github.io.git /content/video2lora.github.io\")\n", " # 3. Change directory to the codebase workspace\n", " os.chdir(\"/content/Video2LoRA\")\n", "\n", "# Automatically set up the media symlink if the website repository is found nearby.\n", "# This ensures that both local and Colab environments can resolve paths correctly.\n", "if os.path.lexists(\"media\"):\n", " # If media exists but is a broken symlink or not a directory, remove it\n", " if not os.path.isdir(\"media\") or os.path.islink(\"media\"):\n", " print(\"Removing conflicting/broken media symlink or file...\")\n", " try:\n", " if os.path.islink(\"media\"):\n", " os.unlink(\"media\")\n", " else:\n", " import shutil\n", " shutil.rmtree(\"media\")\n", " except Exception as e:\n", " os.system(\"rm -rf media\")\n", "\n", "# Dynamically find the website's media directory\n", "website_media_path = None\n", "possible_paths = [\n", " \"/content/video2lora.github.io/media\",\n", " \"/content/Video2LoRA/video2lora.github.io/media\",\n", " \"video2lora.github.io/media\",\n", " \"../video2lora.github.io/media\",\n", " \"../../video2lora.github.io/media\",\n", "]\n", "for p in possible_paths:\n", " abs_p = os.path.abspath(p)\n", " if os.path.exists(abs_p) and os.path.isdir(abs_p):\n", " website_media_path = abs_p\n", " break\n", "\n", "if website_media_path:\n", " print(f\"Linking media directory from website repository at: {website_media_path}\")\n", " try:\n", " os.symlink(website_media_path, \"media\")\n", " except Exception as e:\n", " print(f\"Failed to create symlink using os.symlink, trying shell: {e}\")\n", " os.system(f\"ln -s {website_media_path} media\")\n", "else:\n", " print(\"Warning: Website media directory not found. Will download videos on demand.\")\n", "\n", "print(f\"Workspace directory: {os.getcwd()}\")\n", "\n", "# Install necessary packages\n", "!pip install torch torchvision torchaudio\n", "!pip install transformers accelerate peft huggingface_hub decord av opencv-python matplotlib einops jaxtyping num2words \"torchao>=0.16.0\"\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83e\udde9 Step 1: Model Architectures & Utilities\n", "\n", "To ensure the tutorial is completely self-contained, we import the core architectures and utilities from the repository package. This includes:\n", "- **LoRA Layer Injection**: `apply_lora_to_layers` from `ctx_to_lora.modeling.lora_layer` to patch the base model's attention projections.\n", "- **LoRA Merger**: `combine_lora` from `ctx_to_lora.modeling.lora_merger` to combine weight matrices dynamically.\n", "- **Model Construction**: `build_stage1_model` from `scripts.video2lora.train_smolvlm_stage1` to load and instantiate the modulated VLM.\n", "- **VLM Preprocessing**: `prepare_smolvlm_inputs` and feature extraction helpers." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sys\n", "import os\n", "from pathlib import Path\n", "\n", "# Add repository root and src directory to sys.path to enable local imports\n", "repo_root = str(Path(os.getcwd()).resolve())\n", "if repo_root not in sys.path:\n", " sys.path.insert(0, repo_root)\n", "src_dir = str(Path(repo_root) / \"src\")\n", "if os.path.exists(src_dir) and src_dir not in sys.path:\n", " sys.path.insert(0, src_dir)\n", "\n", "import torch\n", "import cv2\n", "import matplotlib.pyplot as plt\n", "from dataclasses import dataclass\n", "from typing import List, Optional\n", "\n", "# Import custom Video2LoRA architectures and utilities from the repository modules\n", "from ctx_to_lora.modeling.lora_layer import apply_lora_to_layers\n", "from ctx_to_lora.modeling.lora_merger import combine_lora\n", "from scripts.video2lora.train_smolvlm_stage1 import build_stage1_model\n", "from scripts.video2lora.train_smolvlm_online import (\n", " prepare_smolvlm_inputs,\n", " extract_l2l_fused_text_features,\n", ")\n", "\n", "@dataclass\n", "class TrainArgs:\n", " smolvlm_name_or_path: str\n", " train_manifest: str\n", " val_manifest: str\n", " output_dir: str\n", " lora_r: int = 16\n", " lora_dropout: float = 0.0\n", " target_modules: Optional[List[str]] = None\n", " latent_size: int = 512\n", " dropout_rate: float = 0.0\n", " n_latent_queries: int = 8\n", " num_blocks: int = 9\n", " num_self_attn_per_block: int = 0\n", " video_fps: Optional[float] = None\n", " max_frames: int = 12\n", " video_size_longest_edge: int = 384\n", " video_load_backend: str = \"auto\"\n", " internalization_prompt: str = \"Internalize this video for later captioning.\"\n", " kl_weight: float = 0.0\n", " generation_max_new_tokens: int = 128\n", "\n", "print(\"Video2LoRA imported architectures and utility functions successfully.\")\n", "\n", "# Force-refresh Hugging Face dependency cache for num2words if installed mid-session\n", "import sys\n", "import importlib.util\n", "if importlib.util.find_spec(\"num2words\") is not None:\n", " try:\n", " from num2words import num2words as num2words_func\n", " import transformers.utils.import_utils\n", " transformers.utils.import_utils._num2words_available = True\n", " for mod_name in list(sys.modules.keys()):\n", " if \"processing_smolvlm\" in mod_name or \"smolvlm\" in mod_name:\n", " mod = sys.modules[mod_name]\n", " if hasattr(mod, \"num2words\"):\n", " setattr(mod, \"num2words\", num2words_func)\n", " except Exception:\n", " pass\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\udcca Step 2: Load and Inspect Manifest\n", "\n", "Video2LoRA utilizes standard JSONL manifests to structure training and evaluation datasets. Below, we define the **6 qualitative examples** from the project page:\n", "- **Examples included**: *Man smoking pipe*, *Child watering plants*, *Posture alignment*, *Dog tugging toy*, *Rainy day weather query*, and *Tarsier close-up*.\n", "- **Local Video Retrieval**: The cell resolves directory paths recursively, handles broken symlinks automatically, and downloads the video files from the project website only if they are not already found locally." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Define the 6 qualitative examples from the Video2LoRA project website\n", "examples = [\n", " {\n", " \"id\": \"man-smoking-pipe\",\n", " \"video_path\": \"media/benchmarks/carebench/v_00014063_0.mp4\",\n", " \"dataset\": \"CaReBench: Caption\",\n", " \"prompt\": \"Describe the video in as much useful visual detail as possible. Include the main activity, visible people or objects, scene context, appearance, and any important visual details that help explain what is happening.\",\n", " \"target_text\": \"This video depicts a scene of a man lighting a pipe with a lighter. The man in the video is smoking a pipe held in his mouth, supported by his left hand, while his right hand grips the lighter. His right forearm features a large black tattoo. He has short, thick hair that is a deep brown color and is dressed in a loose-fitting black tank top. He is seated next to a window, which has a wooden frame and blue curtains, with a brick wall behind him and a wooden door on the right. The door has a square pattern, adorned with silver hinges and a doorknob. In the video, he first ignites the lighter with his right hand and then brings the flame to the pipe, holding it in that position for several seconds. Throughout this time, his left hand remains steady on the pipe, and his gaze is fixed intently on it, ensuring that the pipe is fully lit before setting the lighter down. He then continues to hold the pipe with his left hand and begins to smoke. The video is shot from the front, clearly illustrating how relaxed he is while smoking at home.\"\n", " },\n", " {\n", " \"id\": \"child-watering-plants\",\n", " \"video_path\": \"media/benchmarks/carebench/v_00016555_0.mp4\",\n", " \"dataset\": \"CaReBench: Events\",\n", " \"prompt\": \"Describe the key visible events in chronological order. Include all important actions and changes you can observe, with enough detail to distinguish each event clearly.\",\n", " \"target_text\": \"Little boy watering plants outdoors; Using watering can to pour water into flower pot; Shifting camera angle from side view to rear view; Tapping edge of flower pot a few times; Setting down watering can\"\n", " },\n", " {\n", " \"id\": \"posture-alignment\",\n", " \"video_path\": \"media/benchmarks/plm/f522598789220c70_122_155.mp4\",\n", " \"dataset\": \"PLM-SGQA\",\n", " \"prompt\": \"Does this look like the same posture she's holding?\",\n", " \"target_text\": \"Yes, it appears you're mirroring the same posture. Your alignment, knee bend, and spine position match the demonstration, indicating proper form and engagement of the targeted muscle groups for optimal effectiveness and safety.\"\n", " },\n", " {\n", " \"id\": \"dog-tugging\",\n", " \"video_path\": \"media/benchmarks/plm/b5bdb7f254cb1727_369_400.mp4\",\n", " \"dataset\": \"PLM-SGQA\",\n", " \"prompt\": \"Is he trying to tug?\",\n", " \"target_text\": \"Yes, your dog is likely inviting a tug-of-war game. Holding the toy in his mouth and possibly looking at you or wagging his tail indicates he's ready to engage in a playful tug.\"\n", " },\n", " {\n", " \"id\": \"rainy-day\",\n", " \"video_path\": \"media/benchmarks/vidcapbench/132065802449.mp4\",\n", " \"dataset\": \"VidCapBench\",\n", " \"prompt\": \"What is the weather like in the scene? Answer only the question, in one sentence.\",\n", " \"target_text\": \"Rainy day.\"\n", " },\n", " {\n", " \"id\": \"tarsier-creature\",\n", " \"video_path\": \"media/benchmarks/vidcapbench/Tarsier_20.mp4\",\n", " \"dataset\": \"VidCapBench\",\n", " \"prompt\": \"Which parts of the creature are highlighted in the video? Answer only the question, in one sentence.\",\n", " \"target_text\": \"A close-up of its face, eyes, and hair.\"\n", " }\n", "]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import urllib.request\n", "\n", "# Download the qualitative videos from the project website repo\n", "print(\"Checking/downloading qualitative benchmark video files from project website...\")\n", "for item in examples:\n", " video_path = item[\"video_path\"]\n", " dir_name = os.path.dirname(video_path)\n", " \n", " # 1. Clean up any broken symlinks or conflicting files along the directories path\n", " # to avoid FileNotFoundError during folder creation.\n", " parts = video_path.split('/')\n", " current_path = \"\"\n", " for part in parts[:-1]: # skip the filename\n", " if current_path:\n", " current_path = os.path.join(current_path, part)\n", " else:\n", " current_path = part\n", " if os.path.lexists(current_path) and not os.path.isdir(current_path):\n", " print(f\"Removing conflict at '{current_path}' (broken link or file) to allow directory creation...\")\n", " try:\n", " if os.path.islink(current_path):\n", " os.unlink(current_path)\n", " else:\n", " os.remove(current_path)\n", " except Exception as e:\n", " print(f\"Failed to remove conflict at {current_path}: {e}\")\n", " \n", " # 2. Safely create directory structure\n", " try:\n", " os.makedirs(dir_name, exist_ok=True)\n", " except Exception as e:\n", " print(f\"Warning: Could not create directory {dir_name}: {e}\")\n", " \n", " # 3. Check and retrieve the video file\n", " if not os.path.exists(video_path):\n", " url = f\"https://video2lora.github.io/{video_path}\"\n", " print(f\"Downloading {video_path} from {url}...\")\n", " try:\n", " urllib.request.urlretrieve(url, video_path)\n", " print(\"Download successful.\")\n", " except Exception as e:\n", " print(f\"Failed to download: {e}\")\n", " else:\n", " print(f\"Found local video file: {video_path}\")\n", "\n", "print(f\"\\nLoaded {len(examples)} qualitative examples successfully.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\uddbc\ufe0f Step 3: Visualize Video Frames\n", "\n", "Before running inference, let's write a utility function using OpenCV and Matplotlib to extract keyframes from the video file and inspect the visual input. We'll visualize the first example: **Man smoking a pipe**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def show_video_frames(video_path, num_frames=4):\n", " \"\"\"\n", " Helper function to load keyframes from a video path and plot them in a grid.\n", " \"\"\"\n", " if not os.path.exists(video_path):\n", " print(f\"Video file not found at: {video_path} (Please provide a valid video to inspect).\")\n", " return\n", " \n", " cap = cv2.VideoCapture(video_path)\n", " total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n", " indices = [int(i * total_frames / num_frames) for i in range(num_frames)]\n", " \n", " fig, axes = plt.subplots(1, num_frames, figsize=(16, 4))\n", " for i, idx in enumerate(indices):\n", " cap.set(cv2.CAP_PROP_POS_FRAMES, idx)\n", " ret, frame = cap.read()\n", " if ret:\n", " frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n", " axes[i].imshow(frame)\n", " axes[i].axis('off')\n", " axes[i].set_title(f\"Frame {idx}\")\n", " else:\n", " axes[i].text(0.5, 0.5, \"Frame Read Error\", ha=\"center\", va=\"center\")\n", " axes[i].axis('off')\n", " plt.tight_layout()\n", " plt.show()\n", " cap.release()\n", "\n", "# Visualize the first qualitative example (man smoking a pipe)\n", "print(f\"Visualizing keyframes for Example 1: {examples[0]['id']}\")\n", "show_video_frames(examples[0][\"video_path\"], num_frames=4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\udcbe Step 4: Download Video2LoRA Checkpoint\n", "\n", "We use the Hugging Face hub client to download the pre-trained weights for the **2.2B SmolVLM2 hypernetwork** (`video2lora-smolvlm2-2.2b-best-ce.pt`). This checkpoint is trained on FineVideo using Cross-Entropy teacher distillation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "from huggingface_hub import hf_hub_download\n", "\n", "print(\"Downloading 2.2B Video2LoRA Checkpoint from Hugging Face...\")\n", "\n", "checkpoint_dir = Path(\"checkpoints/Video2LoRA-SmolVLM-ckpts\")\n", "checkpoint_dir.mkdir(parents=True, exist_ok=True)\n", "\n", "# Download checkpoint weights programmatically\n", "ckpt_path = hf_hub_download(\n", " repo_id=\"MananSuri27/Video2LoRA-SmolVLM-ckpts\",\n", " filename=\"video2lora-smolvlm2-2.2b-best-ce.pt\",\n", " local_dir=str(checkpoint_dir)\n", ")\n", "\n", "print(f\"Checkpoint successfully downloaded to: {ckpt_path}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\ude80 Step 5: Load Base VLM & Initialize Modulated Model\n", "\n", "We configure `TrainArgs` matching the SmolVLM2 2.2B evaluation preset and:\n", "1. Initialize the base SmolVLM2 vision-language model (`HuggingFaceTB/SmolVLM2-2.2B-Instruct`), processor, and tokenizer.\n", "2. Load the pre-trained perceiver hypernetwork weights.\n", "3. Put the model in evaluation mode (`eval()`) and move it to the target hardware device (CUDA GPU if available, otherwise CPU)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Set configuration arguments for the 2.2B SmolVLM2 model preset\n", "train_args = TrainArgs(\n", " smolvlm_name_or_path=\"HuggingFaceTB/SmolVLM2-2.2B-Instruct\",\n", " train_manifest=\"\",\n", " val_manifest=\"\",\n", " output_dir=\"\",\n", " lora_r=16,\n", " lora_dropout=0.0,\n", " target_modules=[\"down_proj\"],\n", " latent_size=512,\n", " dropout_rate=0.0,\n", " n_latent_queries=8,\n", " num_blocks=9,\n", " num_self_attn_per_block=0,\n", " video_fps=None,\n", " max_frames=12,\n", " video_size_longest_edge=384,\n", " video_load_backend=\"auto\",\n", " internalization_prompt=\"Internalize this video for later captioning.\",\n", " kl_weight=0.0,\n", " generation_max_new_tokens=128\n", ")\n", "\n", "# Configure device\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "# 1. Initialize SmolVLM2 base model, processor, and tokenizer\n", "print(\"Loading SmolVLM2 model (2.2B-Instruct)...\")\n", "\n", "\n", "# Force-refresh Hugging Face dependency cache for num2words if installed mid-session\n", "import sys\n", "import importlib.util\n", "if importlib.util.find_spec(\"num2words\") is not None:\n", " try:\n", " from num2words import num2words as num2words_func\n", " import transformers.utils.import_utils\n", " transformers.utils.import_utils._num2words_available = True\n", " for mod_name in list(sys.modules.keys()):\n", " if \"processing_smolvlm\" in mod_name or \"smolvlm\" in mod_name:\n", " mod = sys.modules[mod_name]\n", " if hasattr(mod, \"num2words\"):\n", " setattr(mod, \"num2words\", num2words_func)\n", " except Exception:\n", " pass\n", "\n", "model, raw_model, processor, tokenizer = build_stage1_model(train_args, device=device)\n", "\n", "# 2. Load the hypernetwork weights\n", "print(\"Loading hypernetwork weights...\")\n", "state_dict = torch.load(ckpt_path, map_location=\"cpu\", weights_only=False)\n", "model.load_state_dict(state_dict)\n", "\n", "# Place models in evaluation mode\n", "model.to(device)\n", "model.eval()\n", "raw_model.eval()\n", "\n", "print(\"Model load complete.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \u26a1 Step 6: Parametric Video Internalization\n", "\n", "This is where the core Video2LoRA magic happens! We feed the video frames through the visual encoder to extract intermediate activations layer-by-layer. Then, the perceiver hypernetwork processes these activations and generates the custom LoRA adapter weights in a single pass." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def run_internalization(example, model, raw_model, processor, train_args, device):\n", " \"\"\"\n", " Extracts visual features layer-by-layer and generates the dynamic LoRA adapter.\n", " \"\"\"\n", " # Ensure any patched LoRA hooks are cleared before feature extraction\n", " model.reset()\n", "\n", " internalize_messages = [\n", " [\n", " {\n", " \"role\": \"user\",\n", " \"content\": [\n", " {\"type\": \"video\", \"path\": example[\"video_path\"]},\n", " {\"type\": \"text\", \"text\": train_args.internalization_prompt}\n", " ]\n", " }\n", " ]\n", " ]\n", "\n", " vlm_inputs = prepare_smolvlm_inputs(\n", " processor,\n", " internalize_messages,\n", " device,\n", " video_fps=train_args.video_fps,\n", " max_frames=train_args.max_frames,\n", " video_size_longest_edge=train_args.video_size_longest_edge,\n", " video_load_backend=train_args.video_load_backend\n", " )\n", "\n", " ctx_features, ctx_attn_mask, ctx_position_ids = extract_l2l_fused_text_features(\n", " raw_model,\n", " vlm_inputs,\n", " num_target_layers=model.hypernet.n_layers\n", " )\n", "\n", " generated_loras, _ = model.generate_weights(\n", " ctx_ids=None,\n", " ctx_features=ctx_features,\n", " ctx_attn_mask=ctx_attn_mask,\n", " ctx_position_ids=ctx_position_ids\n", " )\n", "\n", " generated_loras = combine_lora(\n", " generated_loras,\n", " torch.ones(1, dtype=torch.int32, device=device),\n", " lora_bias=model.hypernet.get_head_bias() if model.hypernet.config.use_bias else None\n", " )\n", "\n", " return generated_loras\n", "\n", "# Example run of internalization for child watering plants (index 1)\n", "print(f\"Running internalization for Example 2: {examples[1]['id']}...\")\n", "sample_loras = run_internalization(examples[1], model, raw_model, processor, train_args, device)\n", "print(\"Internalization successful.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\udd0d Step 7: Base Model Inference (With Visual Tokens)\n", "\n", "We first run inference using the **Base VLM Model** (with original video frames and visual tokens passed into the context window). We will query the base model for all 6 qualitative examples and save their predictions in a list `base_predictions`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch\n", "import gc\n", "\n", "# Ensure the model is reset to the clean base model state (LoRA hooks disabled)\n", "model.reset()\n", "torch.cuda.empty_cache()\n", "gc.collect()\n", "\n", "base_predictions = []\n", "print(\"Running Base Model Inference (with visual tokens) for all 6 examples...\")\n", "\n", "with torch.inference_mode():\n", " for idx, example in enumerate(examples):\n", " print(f\"Processing Example {idx+1}/{len(examples)}: {example['id']}...\")\n", " \n", " base_messages = [\n", " [\n", " {\n", " \"role\": \"user\",\n", " \"content\": [\n", " {\"type\": \"video\", \"path\": example[\"video_path\"]},\n", " {\"type\": \"text\", \"text\": example[\"prompt\"]}\n", " ]\n", " }\n", " ]\n", " ]\n", "\n", " base_inputs = prepare_smolvlm_inputs(\n", " processor,\n", " base_messages,\n", " device,\n", " video_fps=train_args.video_fps,\n", " max_frames=train_args.max_frames,\n", " video_size_longest_edge=train_args.video_size_longest_edge,\n", " video_load_backend=train_args.video_load_backend\n", " )\n", "\n", " base_generated_ids = raw_model.generate(\n", " **base_inputs,\n", " max_new_tokens=train_args.generation_max_new_tokens,\n", " do_sample=False,\n", " pad_token_id=tokenizer.pad_token_id,\n", " eos_token_id=tokenizer.eos_token_id\n", " )\n", "\n", " base_pred = tokenizer.decode(\n", " base_generated_ids[0][base_inputs[\"input_ids\"].shape[1]:],\n", " skip_special_tokens=True\n", " ).strip()\n", " \n", " base_predictions.append(base_pred)\n", " \n", " # Clean up temporary GPU tensors for this step\n", " del base_inputs, base_generated_ids\n", " torch.cuda.empty_cache()\n", " gc.collect()\n", "\n", "print(\"\\nBase Model inferences completed successfully.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \u26a1 Step 8: Video2LoRA Internalization & Inference (Zero Visual Tokens)\n", "\n", "Next, we generate and run the dynamic adapter weights using **Video2LoRA** for all 6 qualitative examples. For each example:\n", "1. We run the visual frames through the base vision encoder to extract layer-by-layer activations.\n", "2. We pass the activations into the Perceiver Hypernetwork to generate the custom LoRA weights.\n", "3. We patch the base model's attention projections (`model.patch_lora_forward()`) and inject the generated adapter weights (`apply_lora_to_layers`).\n", "4. We query the VLM using **text only** (zero visual tokens in the context window) and save the outputs in `video2lora_predictions`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch\n", "import gc\n", "\n", "video2lora_predictions = []\n", "print(\"Running Video2LoRA Internalization & Inference (zero visual tokens) for all 6 examples...\")\n", "\n", "with torch.inference_mode():\n", " for idx, example in enumerate(examples):\n", " print(f\"Processing Example {idx+1}/{len(examples)}: {example['id']}...\")\n", " \n", " # 1. Reset model to ensure clean base features are extracted\n", " model.reset()\n", " torch.cuda.empty_cache()\n", " gc.collect()\n", " \n", " # 2. Extract context activations\n", " internalize_messages = [\n", " [\n", " {\n", " \"role\": \"user\",\n", " \"content\": [\n", " {\"type\": \"video\", \"path\": example[\"video_path\"]},\n", " {\"type\": \"text\", \"text\": train_args.internalization_prompt}\n", " ]\n", " }\n", " ]\n", " ]\n", " \n", " vlm_inputs = prepare_smolvlm_inputs(\n", " processor,\n", " internalize_messages,\n", " device,\n", " video_fps=train_args.video_fps,\n", " max_frames=train_args.max_frames,\n", " video_size_longest_edge=train_args.video_size_longest_edge,\n", " video_load_backend=train_args.video_load_backend\n", " )\n", "\n", " ctx_features, ctx_attn_mask, ctx_position_ids = extract_l2l_fused_text_features(\n", " raw_model,\n", " vlm_inputs,\n", " num_target_layers=model.hypernet.n_layers\n", " )\n", "\n", " # 3. Generate dynamic LoRA weights\n", " generated_loras, _ = model.generate_weights(\n", " ctx_ids=None,\n", " ctx_features=ctx_features,\n", " ctx_attn_mask=ctx_attn_mask,\n", " ctx_position_ids=ctx_position_ids\n", " )\n", "\n", " generated_loras = combine_lora(\n", " generated_loras,\n", " torch.ones(1, dtype=torch.int32, device=device),\n", " lora_bias=model.hypernet.get_head_bias() if model.hypernet.config.use_bias else None\n", " )\n", " \n", " # 4. Patch base layers and inject generated weights\n", " model.patch_lora_forward()\n", " apply_lora_to_layers(\n", " model.base_model,\n", " model.hypernet.layer_indices,\n", " generated_loras,\n", " torch.ones(1, dtype=torch.int32, device=device),\n", " position_ids=None\n", " )\n", " \n", " # 5. Run text-only Q&A inference\n", " prompt_ids = tokenizer.apply_chat_template(\n", " [\n", " {\n", " \"role\": \"user\",\n", " \"content\": [{\"type\": \"text\", \"text\": example[\"prompt\"]}]\n", " }\n", " ],\n", " tokenize=True,\n", " add_generation_prompt=True,\n", " return_tensors=\"pt\"\n", " ).to(device)\n", "\n", " if hasattr(prompt_ids, \"keys\"):\n", " input_ids = prompt_ids[\"input_ids\"]\n", " attention_mask = prompt_ids.get(\"attention_mask\", torch.ones_like(input_ids))\n", " else:\n", " input_ids = prompt_ids\n", " attention_mask = torch.ones_like(input_ids)\n", "\n", " generated_ids = model.base_model.generate(\n", " input_ids=input_ids,\n", " attention_mask=attention_mask,\n", " max_new_tokens=train_args.generation_max_new_tokens,\n", " do_sample=False,\n", " pad_token_id=tokenizer.pad_token_id,\n", " eos_token_id=tokenizer.eos_token_id\n", " )\n", "\n", " video2lora_pred = tokenizer.decode(\n", " generated_ids[0][input_ids.shape[1]:],\n", " skip_special_tokens=True\n", " ).strip()\n", " \n", " video2lora_predictions.append(video2lora_pred)\n", " \n", " # Clean up temporary GPU tensors for this step\n", " del vlm_inputs, ctx_features, ctx_attn_mask, ctx_position_ids, generated_loras, prompt_ids, input_ids, attention_mask, generated_ids\n", " model.reset()\n", " torch.cuda.empty_cache()\n", " gc.collect()\n", "\n", "# Reset model hooks back to clean base state\n", "model.reset()\n", "torch.cuda.empty_cache()\n", "gc.collect()\n", "\n", "print(\"\\nVideo2LoRA internalization and inferences completed successfully.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83c\udfa8 Step 9: Qualitative Comparison Dashboard\n", "\n", "Below, we display the generated predictions. First, we print the text output summary directly to compare predictions side-by-side. Then, we render the beautiful visual qualitative comparison board for each example using our HTML template." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.display import HTML, display\n", "\n", "def display_comparison(\n", " video_path,\n", " question_prompt,\n", " ground_truth,\n", " base_model_output,\n", " video2lora_output,\n", " dataset_name\n", "):\n", " \"\"\"\n", " Renders a beautifully styled comparison board with local HTML5 video player.\n", " \"\"\"\n", " html_content = f\"\"\"\n", "
\n", " \n", " \n", "
\n", "

Qualitative Comparison

\n", " {dataset_name}\n", "
\n", "\n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", "
QUESTION PROMPT
\n", "

{question_prompt}

\n", "
\n", " \n", " \n", "
\n", "
GROUND TRUTH
\n", "

{ground_truth}

\n", "
\n", " \n", " \n", "
\n", " \n", " \n", "
\n", "
BASE MODEL (with visual tokens)
\n", "

{base_model_output}

\n", "
\n", " \n", " \n", "
\n", "
VIDEO2LORA (zero visual tokens)
\n", "

{video2lora_output}

\n", "
\n", " \n", "
\n", " \n", "
\n", " \"\"\"\n", " display(HTML(html_content))\n", "\n", "# Print summary text responses first\n", "print(\"=== Summary of Qualitative Responses ===\")\n", "for idx, example in enumerate(examples):\n", " print(f\"\\n--- Example {idx+1}: {example['id']} ({example['dataset']}) ---\")\n", " print(f\"Question: {example['prompt']}\")\n", " print(f\"Ground Truth: {example['target_text']}\")\n", " print(f\"Base Model : {base_predictions[idx]}\")\n", " print(f\"Video2LoRA : {video2lora_predictions[idx]}\")\n", "\n", "print(\"\\nRendering HTML Dashboards...\")\n", "# Display comparison boards\n", "for idx, example in enumerate(examples):\n", " display_comparison(\n", " video_path=example[\"video_path\"],\n", " question_prompt=example[\"prompt\"],\n", " ground_truth=example[\"target_text\"],\n", " base_model_output=base_predictions[idx],\n", " video2lora_output=video2lora_predictions[idx],\n", " dataset_name=example[\"dataset\"]\n", " )" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 2 }