cover + readme

README.md

@@ -1,20 +1,41 @@
+# `Ctx-to-LoRA`: Compressing Context Information to LoRAs 🧠
+<div align="center">
+    <img height="500px" src="assets/cover.png" />
+</div>
+
+
+---
+
 [Project doc](https://docs.google.com/document/d/1RCQDzlVU7YGoTwR84gLQfhxTv0RFnW6srQlqfCC2bvQ/edit?usp=sharing)
 
-### Finetuning the base model with LoRA adaptor
+<!-- ### Finetuning the base model with LoRA adaptor
 ```bash
 WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=lora --train_ds_name=data/raw_datasets/context_numbers_small
+``` -->
+
+## 🚀 `Ctx-to-LoRA` API
+```python
+from ctx_to_lora.modeling import ModulatedPretrainedModel
+model = ModulatedPretrainedModel.from_state_dict(...)
+
+ctx_info = "..."
+query = "..."
+
+ctx_ids = model.ctx_encoder.tokenize(ctx_info)
+input_ids = model.tokenize(query)
+outputs = model.generate(ctx_ids, input_ids)
+print(model.decode(outputs))
 ```
-
-
-### HyperLoRA w/ context_numbers_10
+## 🏋️ Training
+### 🔢 HyperLoRA w/ context_numbers_10
 ```bash
-WANDB_MODE=disabled run python hyperlora/intx_sft.py configs/context_numbers_10.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=100 --per_device_train_batch_size=64 --per_device_eval_batch_size=64 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj,up_proj
+WANDB_MODE=disabled uv run python intx_sft.py configs/context_numbers_10.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=100 --per_device_train_batch_size=64 --per_device_eval_batch_size=64 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj --per_rank_gen=True --per_layer_processing=True --decoder_depth=2 --seed=1 --gen_lora_l1_reg_coef=0 --use_token_mixing=False --lora_r=8 --bf16=True --tf32=True --dataloader_num_workers=8 --dataloader_prefetch_factor=8
 ```
 
-### HyperLoRA w/ context_numbers_128
+<!-- ### HyperLoRA w/ context_numbers_128
 ```bash
 WANDB_MODE=disabled run python hyperlora/intx_sft.py configs/context_numbers_128.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=10 --per_device_train_batch_size=64 --per_device_eval_batch_size=8 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj
-```
+``` -->
 
 ### Generate fineweb qa
 ```bash
@@ -26,7 +47,7 @@ conda activate vllm;
 vllm_model=mistralai/Mistral-Small-3.1-24B-Instruct-2503 run python generate_fw_edu_qa_vllm.py "0*_*" 3
 ```
 
-
+<!-- 
 ### GSM8k LoRA
 ```bash
 run python intx_sft.py configs/gsm8k.yaml --model_name_or_path=google/gemma-2-2b-it --num_train_epochs=10 --per_device_train_batch_size=16 --gradient_accumulation_steps=1 --per_device_eval_batch_size=32 --exp_setup=lora --target_modules=up_proj,down_proj --eval_steps=5000 --save_steps=5000 --learning_rate=1e-4 --neftune_noise_alpha=5 --load_best_model_at_end=True --metric_for_best_model=gsm8k_loss
@@ -40,7 +61,7 @@ run python intx_sft.py configs/math_and_code.yaml --model_name_or_path=google/ge
 ### HyperLoRA finetune on GSM8K
 ```bash
 run python intx_sft.py configs/gsm8k.yaml --model_name_or_path=google/gemma-2-2b-it --num_train_epochs=5 --per_device_train_batch_size=16 --gradient_accumulation_steps=1 --per_device_eval_batch_size=8 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj --num_blocks=1 --num_self_attends_per_block=16 --self_attention_widening_factor=1 --eval_steps=5000 --save_steps=5000 --learning_rate=2e-5 --neftune_noise_alpha=5 --use_light_weight_lora=True --light_weight_latent_size=512 --load_best_model_at_end=True --metric_for_best_model=pwc_loss --add_negative_prompt=False --add_repeat_prompt=False --ctx_encoder_model_name_or_path=meta-llama/Llama-3.2-11B-Vision-Instruct
-```
+``` -->
 
 ### Continue from a checkpoint
 ```bash
@@ -51,8 +72,16 @@ run python intx_sft.py configs/...yaml ... --from_pretrained_checkpoint=train_ou
 LongBench
 ```bash
 # generative
-run python src/ctx_to_lora/eval.py --checkpoint_path train_outputs/runs/.../pytorch_model.bin --datasets negative_nq triviaqa_retrieved hotpot_qa squad longbench_e -
--split test
+run python src/ctx_to_lora/eval.py --checkpoint_path train_outputs/runs/.../pytorch_model.bin --datasets negative_nq triviaqa_retrieved squad longbench_e --split test
+
+# hypernet checkpoint
+uv run python eval.py --checkpoint_path train_outputs/runs/May08_13-56-31_slurm0-a3nodeset-5_59383_906acb28/checkpoint-105000/pytorch_model.bin --datasets negative_nq triviaqa_retrieved squad longbench_e --split test
+
+# base model
+uv run python eval.py --model_name_or_path google/gemma-2-2b-it --datasets longbench_e --split test
+
+# base model w/o context
+run uv run python eval.py --model_name_or_path google/gemma-2-2b-it --datasets longbench_e --split test --remove_context
 
 # run python src/ctx_to_lora/eval.py --checkpoint_path train_outputs/runs/May08_13-56-31_slurm0-a3nodeset-5_59383_906acb28/checkpoint-105000/pytorch_model.bin