apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

train cmds

Browse source
This commit is contained in:
51616 2024-12-24 17:19:52 +00:00
parent a88dbd135f
commit b555f1d07a
1 changed files with 11 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

12
README.md
View file

@ -2,5 +2,15 @@
### Finetuning the base model with LoRA adaptor
```bash
python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=100 --per_device_train_batch_size=8 --per_device_eval_batch_size=32 --overwrite_output_dir=True
WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=lora --train_ds_name=data/raw_datasets/context_numbers_small
```
### HyperLoRA w/ context_numbers_small
```bash
WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=50 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=hyper_lora --train_ds_name=data/raw_datasets/context_numbers_small
```
### HyperLoRA w/ Perceiver w/ context_numbers_medium
```bash
WANDB_MODE=disabled python hyperlora/intx_sft.py configs/default.yaml --model_name_or_path=meta-llama/Llama-3.2-1B-Instruct --num_train_epochs=1 --per_device_train_batch_size=128 --per_device_eval_batch_size=128 --exp_setup=hyper_lora --train_ds_name=data/raw_datasets/context_numbers_medium --aggregator_type=perceiver
```