apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
doc-to-lora/README.md
51616 b555f1d07a train cmds
2024-12-24 17:19:52 +00:00

1.2 KiB
Raw Blame History

Project doc

Finetuning the base model with LoRA adaptor

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=lora --train_ds_name=data/raw_datasets/context_numbers_small

HyperLoRA w/ context_numbers_small

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=hyper_lora --train_ds_name=data/raw_datasets/context_numbers_small

HyperLoRA w/ Perceiver w/ context_numbers_medium

WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=1 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=hyper_lora --train_ds_name=data/raw_datasets/context_numbers_medium --aggregator_type=perceiver