apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
Hypernetworks that update LLMs to remember factual information https://arxiv.org/abs/2602.15902
89 commits 2 branches 0 tags 47 MiB
Find a file
51616 b55b9a5643 workaround to work with deepspeed (avoid dtype cast of hypernet)
2025-01-08 19:33:08 +00:00
chat_templates/meta-llama fix labels pad + chat templates 2024-12-27 12:18:49 +00:00
configs add repeat prompt 2025-01-08 16:17:07 +00:00
data/raw_datasets ctx numbers is now reconstruction prompt 2025-01-08 16:16:43 +00:00
hyperlora workaround to work with deepspeed (avoid dtype cast of hypernet) 2025-01-08 19:33:08 +00:00
icae_v2 formatting 2024-12-20 10:18:30 +00:00
webui display more results 2025-01-08 10:48:20 +00:00
.gitignore line length to 89 2024-12-19 13:08:22 +00:00
.pre-commit-config.yaml init commit 2024-12-18 13:05:19 +00:00
install.sh refactor config + schedule free adam + liger kernel 2024-12-21 12:39:03 +00:00
pyproject.toml line length to 89 2024-12-19 13:08:22 +00:00
README.md cmd for context_numbers_10/128 2025-01-03 09:37:52 +00:00
requirements.txt add webui requirements 2024-12-23 15:17:15 +00:00

README.md

Project doc

Finetuning the base model with LoRA adaptor

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=lora --train_ds_name=data/raw_datasets/context_numbers_small

HyperLoRA w/ context_numbers_10

WANDB_MODE=disabled run python hyperlora/intx_sft.py configs/context_numbers_10.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=100 --per_device_train_batch_size=64 --per_device_eval_batch_size=64 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj,up_proj

HyperLoRA w/ context_numbers_128

WANDB_MODE=disabled run python hyperlora/intx_sft.py configs/context_numbers_128.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=10 --per_device_train_batch_size=64 --per_device_eval_batch_size=8 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj