apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
Hypernetworks that update LLMs to remember factual information https://arxiv.org/abs/2602.15902
131 commits 2 branches 0 tags 47 MiB
Find a file
51616 c5fb48c2ae fix liger kernel ctx model
2025-01-20 18:32:39 +00:00
chat_templates add squad and gemma 2025-01-18 12:36:12 +00:00
configs openmathintx ft ds 2025-01-20 09:18:14 +00:00
data/raw_datasets ctx numbers is now reconstruction prompt 2025-01-08 16:16:43 +00:00
icae_v2 formatting 2024-12-20 10:18:30 +00:00
src/ctx_to_lora flash attn for eval 2025-01-20 13:50:22 +00:00
webui display more results 2025-01-08 10:48:20 +00:00
.gitignore everything (before us migrate) 2025-01-20 10:31:08 +00:00
.pre-commit-config.yaml init commit 2024-12-18 13:05:19 +00:00
batch.sh everything (before us migrate) 2025-01-20 10:31:08 +00:00
batch_gemma_2_2b.sh everything (before us migrate) 2025-01-20 10:31:08 +00:00
batch_gemma_llama_vision.sh hyperparam 2025-01-20 17:28:59 +00:00
batch_llama_2_7b.sh everything (before us migrate) 2025-01-20 10:31:08 +00:00
batch_llama_3_8b_1k_latent.sh hyperparam 2025-01-20 17:28:59 +00:00
generate_fw_qa.py everything (before us migrate) 2025-01-20 10:31:08 +00:00
generate_one_big_fw_qa_req.py everything (before us migrate) 2025-01-20 10:31:08 +00:00
generate_qa_parallel.py everything (before us migrate) 2025-01-20 10:31:08 +00:00
hotpot_qa_lengths.png everything (before us migrate) 2025-01-20 10:31:08 +00:00
hotpot_qa_lengths_filtered.png everything (before us migrate) 2025-01-20 10:31:08 +00:00
install.sh install and run cmds 2025-01-20 12:19:10 +00:00
intx_sft.py fix liger kernel ctx model 2025-01-20 18:32:39 +00:00
openphi_prog_lengths.png everything (before us migrate) 2025-01-20 10:31:08 +00:00
openphi_prog_lengths_filtered.png everything (before us migrate) 2025-01-20 10:31:08 +00:00
post_process_fw_qa.py include new data! 2025-01-13 14:21:41 +00:00
post_process_parallel_qa.py everything (before us migrate) 2025-01-20 10:31:08 +00:00
process_automathtext_arxiv.py include new data! 2025-01-13 14:21:41 +00:00
process_automathtext_code_python.py include new data! 2025-01-13 14:21:41 +00:00
process_automathtext_web.py include new data! 2025-01-13 14:21:41 +00:00
process_codeparrot_qa.py include new data! 2025-01-13 14:21:41 +00:00
process_fineweb.py include new data! 2025-01-13 14:21:41 +00:00
process_openphi_prog.py include new data! 2025-01-13 14:21:41 +00:00
pyproject.toml install and run cmds 2025-01-20 12:19:10 +00:00
README.md include new data! 2025-01-13 14:21:41 +00:00
requirements.txt install and run cmds 2025-01-20 12:19:10 +00:00
run.sh everything (before us migrate) 2025-01-20 10:31:08 +00:00
setup.py restructure 2025-01-17 17:41:10 +00:00
watcher.py restructure 2025-01-17 17:41:10 +00:00

README.md

Project doc

Finetuning the base model with LoRA adaptor

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=lora --train_ds_name=data/raw_datasets/context_numbers_small

HyperLoRA w/ context_numbers_10

WANDB_MODE=disabled run python hyperlora/intx_sft.py configs/context_numbers_10.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=100 --per_device_train_batch_size=64 --per_device_eval_batch_size=64 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj,up_proj

HyperLoRA w/ context_numbers_128

WANDB_MODE=disabled run python hyperlora/intx_sft.py configs/context_numbers_128.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=10 --per_device_train_batch_size=64 --per_device_eval_batch_size=8 --exp_setup=hyper_lora --aggregator_type=perceiver --target_modules=down_proj

Generate fineweb qa

# this might take several days...
python process_fineweb.py
python generate_fw_qa.py
python post_process_fw_qa.py