mirror of
https://github.com/SakanaAI/doc-to-lora.git
synced 2026-07-26 17:11:02 +02:00
891c0bd256
* faster slice * add facts + ctx_qa * new configs * new scripts * intx_sft.py to train.py * add kaggle for downloading facts * max_new_tokens cli for eval * generate negative_nq * scripts + configs * default vals * max_val_samples_per_ds=500 * more efficient layer-to-layer ctx encoder * use_per_ctx_average_loss * faster processing * small exp distill * scripts * more robust watcher * per-module l1_norm avg * per-ctx average loss * clear_gpu
12 lines
266 B
YAML
12 lines
266 B
YAML
# LoRA
|
|
lora_r: 8
|
|
lora_dropout: 0.0
|
|
target_modules:
|
|
- down_proj
|
|
|
|
# data
|
|
train_ds_names:
|
|
- self_gen/google/gemma-2-2b-it_temp_0.0_closed_qa_prob_0.0/context_numbers_2_10
|
|
|
|
val_ds_names:
|
|
- self_gen/google/gemma-2-2b-it_temp_0.0_closed_qa_prob_0.0/context_numbers_2_10
|