apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
doc-to-lora/demo.ipynb

618 lines
No EOL
27 KiB
Text
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Video2LoRA: Self-Contained Zero-Visual-Token Inference Tutorial\n",
"\n",
"This notebook provides a **completely self-contained, single-file tutorial** for running zero-visual-token inference with **Video2LoRA**. Under the hood, Video2LoRA utilizes a perceiver hypernetwork to read a video's intermediate activation features layer-by-layer and directly predict a set of Low-Rank Adaptation (LoRA) weights in a single forward pass.\n",
"\n",
"Once the adapter weights are generated, they are injected into the base Vision-Language Model (VLM), which can then answer arbitrary text queries **without requiring the original visual tokens** in its context window.\n",
"\n",
"All model architectures, helper functions, and dependencies are defined directly within this notebook so it can run independently of the main repository codebase.\n",
"\n",
"### \ud83d\udd17 Project Resources\n",
"- [**\ud83c\udf10 Project Website**](https://video2lora.github.io/)\n",
"- [**\ud83d\udcc4 arXiv Paper**](https://arxiv.org/abs/2606.04351)\n",
"- [**\ud83e\udd17 Hugging Face Checkpoints**](https://huggingface.co/MananSuri27/Video2LoRA-SmolVLM-ckpts)\n",
"- [**\ud83d\udcbb GitHub Repository**](https://github.com/MananSuri27/video2lora)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 0: Package Installation\n",
"\n",
"First, we install all the required open-source packages in the environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Detect if running in Google Colab\n",
"try:\n",
" import google.colab\n",
" IN_COLAB = True\n",
"except ImportError:\n",
" IN_COLAB = False\n",
"\n",
"if IN_COLAB:\n",
" print(\"Running in Google Colab. Setting up workspace repository...\")\n",
" import os\n",
" if not os.path.exists(\"/content/Video2LoRA\"):\n",
" os.system(\"git clone -b demo https://github.com/video2lora/Video2LoRA.git\")\n",
" os.chdir(\"/content/Video2LoRA\")\n",
" print(f\"Workspace directory changed to: {os.getcwd()}\")\n",
"\n",
"# Install necessary packages\n",
"!pip install torch torchvision torchaudio\n",
"!pip install transformers accelerate peft huggingface_hub decord av opencv-python matplotlib einops jaxtyping"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Model Architectures & Utilities\n",
"\n",
"Here, we define the full Video2LoRA model architecture classes (including the perceiver aggregator, hypernet heads, model load patches, and custom LoRA injection layers) directly inside the notebook. This makes the notebook completely independent of external python code files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import os\n",
"from pathlib import Path\n",
"\n",
"# Add repository root and src directory to sys.path to enable local imports\n",
"repo_root = str(Path(os.getcwd()).resolve())\n",
"if repo_root not in sys.path:\n",
" sys.path.insert(0, repo_root)\n",
"src_dir = str(Path(repo_root) / \"src\")\n",
"if os.path.exists(src_dir) and src_dir not in sys.path:\n",
" sys.path.insert(0, src_dir)\n",
"\n",
"import torch\n",
"import cv2\n",
"import matplotlib.pyplot as plt\n",
"from dataclasses import dataclass\n",
"from typing import List, Optional\n",
"\n",
"# Import custom Video2LoRA architectures and utilities from the repository modules\n",
"from ctx_to_lora.modeling.lora_layer import apply_lora_to_layers\n",
"from ctx_to_lora.modeling.lora_merger import combine_lora\n",
"from scripts.video2lora.train_smolvlm_stage1 import build_stage1_model\n",
"from scripts.video2lora.train_smolvlm_online import (\n",
" prepare_smolvlm_inputs,\n",
" extract_l2l_fused_text_features,\n",
")\n",
"\n",
"@dataclass\n",
"class TrainArgs:\n",
" smolvlm_name_or_path: str\n",
" train_manifest: str\n",
" val_manifest: str\n",
" output_dir: str\n",
" lora_r: int = 16\n",
" lora_dropout: float = 0.0\n",
" target_modules: Optional[List[str]] = None\n",
" latent_size: int = 512\n",
" dropout_rate: float = 0.0\n",
" n_latent_queries: int = 8\n",
" num_blocks: int = 9\n",
" num_self_attn_per_block: int = 0\n",
" video_fps: Optional[float] = None\n",
" max_frames: int = 12\n",
" video_size_longest_edge: int = 384\n",
" video_load_backend: str = \"auto\"\n",
" internalization_prompt: str = \"Internalize this video for later captioning.\"\n",
" kl_weight: float = 0.0\n",
" generation_max_new_tokens: int = 128\n",
"\n",
"print(\"Video2LoRA imported architectures and utility functions successfully.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Load and Inspect Manifest\n",
"\n",
"Video2LoRA uses a standard JSONL manifest file for training and evaluation. Below, we define a sample manifest row matching the child watering plants example from the paper's qualitative evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Define the 6 qualitative examples from the Video2LoRA project website\n",
"examples = [\n",
" {\n",
" \"id\": \"man-smoking-pipe\",\n",
" \"video_path\": \"media/benchmarks/carebench/v_00014063_0.mp4\",\n",
" \"dataset\": \"CaReBench: Caption\",\n",
" \"prompt\": \"Describe the video in as much useful visual detail as possible. Include the main activity, visible people or objects, scene context, appearance, and any important visual details that help explain what is happening.\",\n",
" \"target_text\": \"This video depicts a scene of a man lighting a pipe with a lighter. The man in the video is smoking a pipe held in his mouth, supported by his left hand, while his right hand grips the lighter. His right forearm features a large black tattoo. He has short, thick hair that is a deep brown color and is dressed in a loose-fitting black tank top. He is seated next to a window, which has a wooden frame and blue curtains, with a brick wall behind him and a wooden door on the right. The door has a square pattern, adorned with silver hinges and a doorknob. In the video, he first ignites the lighter with his right hand and then brings the flame to the pipe, holding it in that position for several seconds. Throughout this time, his left hand remains steady on the pipe, and his gaze is fixed intently on it, ensuring that the pipe is fully lit before setting the lighter down. He then continues to hold the pipe with his left hand and begins to smoke. The video is shot from the front, clearly illustrating how relaxed he is while smoking at home.\"\n",
" },\n",
" {\n",
" \"id\": \"child-watering-plants\",\n",
" \"video_path\": \"media/benchmarks/carebench/v_00016555_0.mp4\",\n",
" \"dataset\": \"CaReBench: Events\",\n",
" \"prompt\": \"Describe the key visible events in chronological order. Include all important actions and changes you can observe, with enough detail to distinguish each event clearly.\",\n",
" \"target_text\": \"Little boy watering plants outdoors; Using watering can to pour water into flower pot; Shifting camera angle from side view to rear view; Tapping edge of flower pot a few times; Setting down watering can\"\n",
" },\n",
" {\n",
" \"id\": \"posture-alignment\",\n",
" \"video_path\": \"media/benchmarks/plm/f522598789220c70_122_155.mp4\",\n",
" \"dataset\": \"PLM-SGQA\",\n",
" \"prompt\": \"Does this look like the same posture she's holding?\",\n",
" \"target_text\": \"Yes, it appears you're mirroring the same posture. Your alignment, knee bend, and spine position match the demonstration, indicating proper form and engagement of the targeted muscle groups for optimal effectiveness and safety.\"\n",
" },\n",
" {\n",
" \"id\": \"dog-tugging\",\n",
" \"video_path\": \"media/benchmarks/plm/b5bdb7f254cb1727_369_400.mp4\",\n",
" \"dataset\": \"PLM-SGQA\",\n",
" \"prompt\": \"Is he trying to tug?\",\n",
" \"target_text\": \"Yes, your dog is likely inviting a tug-of-war game. Holding the toy in his mouth and possibly looking at you or wagging his tail indicates he's ready to engage in a playful tug.\"\n",
" },\n",
" {\n",
" \"id\": \"rainy-day\",\n",
" \"video_path\": \"media/benchmarks/vidcapbench/132065802449.mp4\",\n",
" \"dataset\": \"VidCapBench\",\n",
" \"prompt\": \"What is the weather like in the scene? Answer only the question, in one sentence.\",\n",
" \"target_text\": \"Rainy day.\"\n",
" },\n",
" {\n",
" \"id\": \"tarsier-creature\",\n",
" \"video_path\": \"media/benchmarks/vidcapbench/Tarsier_20.mp4\",\n",
" \"dataset\": \"VidCapBench\",\n",
" \"prompt\": \"Which parts of the creature are highlighted in the video? Answer only the question, in one sentence.\",\n",
" \"target_text\": \"A close-up of its face, eyes, and hair.\"\n",
" }\n",
"]\n",
"\n",
"import os\n",
"import urllib.request\n",
"\n",
"# Download the qualitative videos from the project website repo\n",
"print(\"Checking/downloading qualitative benchmark video files from project website...\")\n",
"for item in examples:\n",
" video_path = item[\"video_path\"]\n",
" os.makedirs(os.path.dirname(video_path), exist_ok=True)\n",
" if not os.path.exists(video_path):\n",
" url = f\"https://video2lora.github.io/{video_path}\"\n",
" print(f\"Downloading {video_path} from {url}...\")\n",
" try:\n",
" urllib.request.urlretrieve(url, video_path)\n",
" print(\"Download successful.\")\n",
" except Exception as e:\n",
" print(f\"Failed to download: {e}\")\n",
" else:\n",
" print(f\"Found local video file: {video_path}\")\n",
"\n",
"print(f\"\\nLoaded {len(examples)} qualitative examples successfully.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Visualize Video Frames\n",
"\n",
"Let's write a utility function to extract and display keyframes from the input video file using OpenCV and Matplotlib."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def show_video_frames(video_path, num_frames=4):\n",
" \"\"\"\n",
" Helper function to load keyframes from a video path and plot them in a grid.\n",
" \"\"\"\n",
" if not os.path.exists(video_path):\n",
" print(f\"Video file not found at: {video_path} (Please provide a valid video to inspect).\")\n",
" return\n",
" \n",
" cap = cv2.VideoCapture(video_path)\n",
" total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n",
" indices = [int(i * total_frames / num_frames) for i in range(num_frames)]\n",
" \n",
" fig, axes = plt.subplots(1, num_frames, figsize=(16, 4))\n",
" for i, idx in enumerate(indices):\n",
" cap.set(cv2.CAP_PROP_POS_FRAMES, idx)\n",
" ret, frame = cap.read()\n",
" if ret:\n",
" frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
" axes[i].imshow(frame)\n",
" axes[i].axis('off')\n",
" axes[i].set_title(f\"Frame {idx}\")\n",
" else:\n",
" axes[i].text(0.5, 0.5, \"Frame Read Error\", ha=\"center\", va=\"center\")\n",
" axes[i].axis('off')\n",
" plt.tight_layout()\n",
" plt.show()\n",
" cap.release()\n",
"\n",
"# Visualize the first qualitative example (man smoking a pipe)\n",
"print(f\"Visualizing keyframes for Example 1: {examples[0]['id']}\")\n",
"show_video_frames(examples[0][\"video_path\"], num_frames=4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Download Video2LoRA Checkpoint\n",
"\n",
"We use the `huggingface_hub` client to download the 2.2B Video2LoRA checkpoint weights (`video2lora-smolvlm2-2.2b-best-ce.pt`) directly from Hugging Face."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"Downloading 2.2B Video2LoRA Checkpoint from Hugging Face...\")\n",
"\n",
"checkpoint_dir = Path(\"checkpoints/Video2LoRA-SmolVLM-ckpts\")\n",
"checkpoint_dir.mkdir(parents=True, exist_ok=True)\n",
"\n",
"# Download checkpoint weights programmatically\n",
"ckpt_path = hf_hub_download(\n",
" repo_id=\"MananSuri27/Video2LoRA-SmolVLM-ckpts\",\n",
" filename=\"video2lora-smolvlm2-2.2b-best-ce.pt\",\n",
" local_dir=str(checkpoint_dir)\n",
")\n",
"\n",
"print(f\"Checkpoint successfully downloaded to: {ckpt_path}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Load Base VLM & Initialize Modulated Model\n",
"\n",
"Configure `TrainArgs` matching the 2.2B preset, load the base SmolVLM2 model, processor, and tokenizer, and load the hypernetwork checkpoint."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set configuration arguments for the 2.2B SmolVLM2 model preset\n",
"train_args = TrainArgs(\n",
" smolvlm_name_or_path=\"HuggingFaceTB/SmolVLM2-2.2B-Instruct\",\n",
" train_manifest=\"\",\n",
" val_manifest=\"\",\n",
" output_dir=\"\",\n",
" lora_r=16,\n",
" lora_dropout=0.0,\n",
" target_modules=[\"down_proj\"],\n",
" latent_size=512,\n",
" dropout_rate=0.0,\n",
" n_latent_queries=8,\n",
" num_blocks=9,\n",
" num_self_attn_per_block=0,\n",
" video_fps=None,\n",
" max_frames=12,\n",
" video_size_longest_edge=384,\n",
" video_load_backend=\"auto\",\n",
" internalization_prompt=\"Internalize this video for later captioning.\",\n",
" kl_weight=0.0,\n",
" generation_max_new_tokens=128\n",
")\n",
"\n",
"# Configure device\n",
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"\n",
"# 1. Initialize SmolVLM2 base model, processor, and tokenizer\n",
"print(\"Loading SmolVLM2 model (2.2B-Instruct)...\")\n",
"model, raw_model, processor, tokenizer = build_stage1_model(train_args, device=device)\n",
"\n",
"# 2. Load the hypernetwork weights\n",
"print(\"Loading hypernetwork weights...\")\n",
"state_dict = torch.load(ckpt_path, map_location=\"cpu\", weights_only=False)\n",
"model.load_state_dict(state_dict)\n",
"\n",
"# Place models in evaluation mode\n",
"model.to(device)\n",
"model.eval()\n",
"raw_model.eval()\n",
"\n",
"print(\"Model load complete.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Parametric Video Internalization\n",
"\n",
"We process the video frames through the VLM visual encoder to extract activations, then generate the custom LoRA adapter weights in a single pass of the hypernetwork."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def run_internalization(example, model, raw_model, processor, train_args, device):\n",
" \"\"\"\n",
" Extracts visual features layer-by-layer and generates the dynamic LoRA adapter.\n",
" \"\"\"\n",
" internalize_messages = [\n",
" [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"video\", \"path\": example[\"video_path\"]},\n",
" {\"type\": \"text\", \"text\": train_args.internalization_prompt}\n",
" ]\n }\n",
" ]\n",
" ]\n",
"\n",
" vlm_inputs = prepare_smolvlm_inputs(\n",
" processor,\n",
" internalize_messages,\n",
" device,\n",
" video_fps=train_args.video_fps,\n",
" max_frames=train_args.max_frames,\n",
" video_size_longest_edge=train_args.video_size_longest_edge,\n",
" video_load_backend=train_args.video_load_backend\n",
" )\n",
"\n",
" ctx_features, ctx_attn_mask, ctx_position_ids = extract_l2l_fused_text_features(\n",
" raw_model,\n",
" vlm_inputs,\n",
" num_target_layers=model.hypernet.n_layers\n",
" )\n",
"\n",
" generated_loras, _ = model.generate_weights(\n",
" ctx_ids=None,\n",
" ctx_features=ctx_features,\n",
" ctx_attn_mask=ctx_attn_mask,\n",
" ctx_position_ids=ctx_position_ids\n",
" )\n",
"\n",
" generated_loras = combine_lora(\n",
" generated_loras,\n",
" torch.ones(1, dtype=torch.int32, device=device),\n",
" lora_bias=model.hypernet.get_head_bias() if model.hypernet.config.use_bias else None\n",
" )\n",
"\n",
" return generated_loras\n",
"\n",
"# Example run of internalization for child watering plants (index 1)\n",
"print(f\"Running internalization for Example 2: {examples[1]['id']}...\")\n",
"sample_loras = run_internalization(examples[1], model, raw_model, processor, train_args, device)\n",
"print(\"Internalization successful.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 7: Run QA Inference (Base Model vs Video2LoRA)\n",
"\n",
"To perform a fair comparison, we first run inference with the **Base Model** (which requires processing the full visual video tokens in context). Afterwards, we run inference with **Video2LoRA** (which bypasses visual tokens completely and queries the model using only text, utilizing the dynamic LoRA adapter)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def run_inference(example, generated_loras, model, raw_model, processor, tokenizer, train_args, device):\n",
" \"\"\"\n",
" Runs inference comparing base model (using visual tokens) vs Video2LoRA (zero visual tokens).\n",
" \"\"\"\n",
" # === Part A: Base Model Inference (with visual tokens) ===\n",
" base_messages = [\n",
" [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"video\", \"path\": example[\"video_path\"]},\n",
" {\"type\": \"text\", \"text\": example[\"prompt\"]}\n",
" ]\n }\n",
" ]\n",
" ]\n",
"\n",
" base_inputs = prepare_smolvlm_inputs(\n",
" processor,\n",
" base_messages,\n",
" device,\n",
" video_fps=train_args.video_fps,\n",
" max_frames=train_args.max_frames,\n",
" video_size_longest_edge=train_args.video_size_longest_edge,\n",
" video_load_backend=train_args.video_load_backend\n",
" )\n",
"\n",
" base_generated_ids = raw_model.generate(\n",
" **base_inputs,\n",
" max_new_tokens=train_args.generation_max_new_tokens,\n",
" do_sample=False,\n",
" pad_token_id=tokenizer.pad_token_id,\n",
" eos_token_id=tokenizer.eos_token_id\n",
" )\n",
"\n",
" base_prediction = tokenizer.decode(\n",
" base_generated_ids[0][base_inputs[\"input_ids\"].shape[1]:],\n",
" skip_special_tokens=True\n",
" ).strip()\n",
"\n",
" # === Part B: Video2LoRA Inference (zero visual tokens) ===\n",
" apply_lora_to_layers(\n",
" model.base_model,\n",
" model.hypernet.layer_indices,\n",
" generated_loras,\n",
" torch.ones(1, dtype=torch.int32, device=device),\n",
" position_ids=None\n",
" )\n",
"\n",
" prompt_ids = tokenizer.apply_chat_template(\n",
" [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [{\"type\": \"text\", \"text\": example[\"prompt\"]}]\n }\n ],\n",
" tokenize=True,\n",
" add_generation_prompt=True,\n",
" return_tensors=\"pt\"\n",
" ).to(device)\n",
"\n",
" generated_ids = model.base_model.generate(\n",
" input_ids=prompt_ids,\n",
" attention_mask=torch.ones_like(prompt_ids),\n",
" max_new_tokens=train_args.generation_max_new_tokens,\n",
" do_sample=False,\n",
" pad_token_id=tokenizer.pad_token_id,\n",
" eos_token_id=tokenizer.eos_token_id\n",
" )\n",
"\n",
" video2lora_prediction = tokenizer.decode(\n",
" generated_ids[0][prompt_ids.shape[1]:],\n",
" skip_special_tokens=True\n",
" ).strip()\n",
"\n",
" # Reset model LoRA patches\n",
" model.reset()\n",
"\n",
" return base_prediction, video2lora_prediction\n",
"\n",
"# Run comparison inference on the sample example\n",
"print(f\"Running inference for Example 2: {examples[1]['id']}...\")\n",
"base_prediction, video2lora_prediction = run_inference(\n",
" examples[1], sample_loras, model, raw_model, processor, tokenizer, train_args, device\n",
")\n",
"print(f\"\\nPredictions Generated:\\nBase Model: {base_prediction}\\nVideo2LoRA: {video2lora_prediction}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 8: Qualitative Comparison Dashboard\n",
"\n",
"Display the comparison between the Base Model and Video2LoRA using the dynamically generated prediction outputs in a custom IPython HTML layout matching the evaluation layout."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import HTML, display\n",
"\n",
"def display_comparison(\n",
" video_path,\n",
" question_prompt,\n",
" ground_truth,\n",
" base_model_output,\n",
" video2lora_output,\n",
" dataset_name\n",
"):\n",
" \"\"\"\n",
" Renders a beautifully styled comparison board with local HTML5 video player.\n",
" \"\"\"\n",
" html_content = f\"\"\"\n",
" <div style=\"font-family: 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; max-width: 900px; margin: 20px auto; color: #1e293b; background-color: #f8fafc; padding: 24px; border-radius: 16px; box-shadow: 0 4px 20px rgba(0,0,0,0.05);\">\n",
" \n",
" <!-- Header Source Badge -->\n",
" <div style=\"display: flex; justify-content: space-between; align-items: center; margin-bottom: 16px;\">\n",
" <h3 style=\"margin: 0; font-size: 18px; font-weight: 700; color: #0f172a;\">Qualitative Comparison</h3>\n",
" <span style=\"background-color: #e2e8f0; color: #475569; padding: 4px 10px; border-radius: 9999px; font-size: 12px; font-weight: 600;\">{dataset_name}</span>\n",
" </div>\n",
"\n",
" <!-- Local HTML5 Video Player -->\n",
" <div style=\"text-align: center; margin-bottom: 20px; background: #000; border-radius: 12px; overflow: hidden; box-shadow: 0 4px 12px rgba(0,0,0,0.15);\">\n",
" <video src=\"{video_path}\" controls loop muted autoplay style=\"width: 100%; max-height: 400px; object-fit: contain; display: block; margin: 0 auto;\"></video>\n",
" </div>\n",
" \n",
" <!-- Question Prompt Card -->\n",
" <div style=\"background-color: #f0f4ff; border: 1px solid #dbeafe; border-radius: 8px; padding: 16px; margin-bottom: 16px;\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #1e40af; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">QUESTION PROMPT</h5>\n",
" <p style=\"margin: 0; font-size: 15px; line-height: 1.5; color: #1e293b;\">{question_prompt}</p>\n",
" </div>\n",
" \n",
" <!-- Ground Truth Card -->\n",
" <div style=\"background-color: #fefbeb; border: 1px solid #fef3c7; border-radius: 8px; padding: 16px; margin-bottom: 16px;\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #854d0e; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">GROUND TRUTH</h5>\n",
" <p style=\"margin: 0; font-size: 15px; line-height: 1.5; color: #1e293b; font-weight: 500;\">{ground_truth}</p>\n",
" </div>\n",
" \n",
" <!-- Model Comparisons Grid -->\n",
" <div style=\"display: grid; grid-template-columns: 1fr 1fr; gap: 16px;\">\n",
" \n",
" <!-- Base Model Card -->\n",
" <div style=\"background-color: #fff; border: 1.5px solid #ea580c; border-radius: 8px; padding: 16px; box-shadow: 0 2px 4px rgba(0,0,0,0.02);\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #ea580c; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">BASE MODEL (with visual tokens)</h5>\n",
" <p style=\"margin: 0; font-size: 14px; line-height: 1.5; color: #334155;\">{base_model_output}</p>\n",
" </div>\n",
" \n",
" <!-- Video2LoRA Card -->\n",
" <div style=\"background-color: #f0fdf4; border: 1.5px solid #16a34a; border-radius: 8px; padding: 16px; box-shadow: 0 2px 4px rgba(0,0,0,0.02);\">\n",
" <h5 style=\"margin: 0 0 6px 0; color: #16a34a; text-transform: uppercase; font-size: 11px; letter-spacing: 0.05em; font-weight: 700;\">VIDEO2LORA (zero visual tokens)</h5>\n",
" <p style=\"margin: 0; font-size: 14px; line-height: 1.5; color: #1e293b; font-weight: 500;\">{video2lora_output}</p>\n",
" </div>\n",
" \n",
" </div>\n",
" \n",
" </div>\n",
" \"\"\"\n",
" display(HTML(html_content))\n",
"\n",
"# Display the comparison dashboard dynamically for our completed example!\n",
"display_comparison(\n",
" video_path=examples[1][\"video_path\"],\n",
" question_prompt=examples[1][\"prompt\"],\n",
" ground_truth=examples[1][\"target_text\"],\n",
" base_model_output=base_prediction,\n",
" video2lora_output=video2lora_prediction,\n",
" dataset_name=examples[1][\"dataset\"]\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in a new issue View git blame Copy permalink