apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
Hypernetworks that update LLMs to remember factual information https://arxiv.org/abs/2602.15902
52 commits 2 branches 0 tags 47 MiB
Find a file
51616 25bb3864e8 fix rouge metric prefix
2024-12-25 08:50:43 +00:00
configs fix tokenizer pad token generation warning 2024-12-24 13:44:51 +00:00
data/raw_datasets formatting 2024-12-24 07:50:08 +00:00
hyperlora fix rouge metric prefix 2024-12-25 08:50:43 +00:00
icae_v2 formatting 2024-12-20 10:18:30 +00:00
webui formatting 2024-12-24 07:50:08 +00:00
.gitignore line length to 89 2024-12-19 13:08:22 +00:00
.pre-commit-config.yaml init commit 2024-12-18 13:05:19 +00:00
install.sh refactor config + schedule free adam + liger kernel 2024-12-21 12:39:03 +00:00
pyproject.toml line length to 89 2024-12-19 13:08:22 +00:00
README.md train cmds 2024-12-24 17:19:52 +00:00
requirements.txt add webui requirements 2024-12-23 15:17:15 +00:00

README.md

Project doc

Finetuning the base model with LoRA adaptor

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=lora --train_ds_name=data/raw_datasets/context_numbers_small

HyperLoRA w/ context_numbers_small

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=hyper_lora --train_ds_name=data/raw_datasets/context_numbers_small

HyperLoRA w/ Perceiver w/ context_numbers_medium

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=1 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=hyper_lora --train_ds_name=data/raw_datasets/context_numbers_medium --aggregator_type=perceiver