apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
395 commits 2 branches 0 tags 47 MiB
Commit graph

87 commits

Author SHA1 Message Date
51616
8093817c34 icml cleaned 2026-02-13 06:02:29 +00:00
51616
2ed418bc22 vlm ctx-encoder + slrum scripts + qwen self-gen 2025-10-27 06:20:26 +00:00
51616
178822b7d3 mistral config + script 2025-10-13 07:45:47 +00:00
51616
3b798c670c python api + remove generate_multi_lora + rename config path + test eval scripts 2025-10-01 13:11:54 +00:00
51616
b6679ba755 iclr cleanup 2025-09-30 14:53:21 +00:00
51616
c21ba2b481 iclr 2026 2025-09-29 00:39:17 +09:00
51616
650395874f toy exp w/ self-gen + use double bos for eval (following vllm 0.8.4) 2025-09-07 17:16:20 +09:00
51616
3505b006ad toy ctx magic num eval upto 32k + remove configs 2025-09-01 05:57:06 +00:00
Rujikorn Charakorn
ee9039295e
LoRA separate bias param + merger + toy ds (#15) 
* separate bias + head weight init

* smaller bias init + sum with 1/sqrt(combined_r) scaling + better  lora merge logic

* ok init

* chunked ctx magic num working

* configs

* no transpose + only sum aggregation

* fix cli

* cli

* 32-256 data

* example cli + smaller dataset

* toy ctx magic num ds size + quantize + remove head bias

* configs removed

* require lora_bias

* Merge remote-tracking branch 'origin/main' into new-arch
 2025-08-29 11:55:24 +09:00
Rujikorn Charakorn
c5e9bc769d
toy ctx nums and multi-lora training (#11) 
* multi-lora trainable toy number repeat dataset

* per rank bias init

* remove head_bias +simplify merge + skip perplexities metric

* ctx_numbers train example

* self-gen ctx numbers example
 2025-08-18 18:38:07 +09:00
51616
178e7165ea per-rank bias + skip interleave if cached+ add bnb 2025-08-09 15:20:37 +00:00
Rujikorn Charakorn
891c0bd256
Refactor_and_improve_data (#7) 
* faster slice

* add facts + ctx_qa

* new configs

* new scripts

* intx_sft.py to train.py

* add kaggle for downloading facts

* max_new_tokens cli for eval

* generate negative_nq

* scripts + configs

* default vals

* max_val_samples_per_ds=500

* more efficient layer-to-layer ctx encoder

* use_per_ctx_average_loss

* faster processing

* small exp distill

* scripts

* more robust watcher

* per-module l1_norm avg

* per-ctx average loss

* clear_gpu
 2025-08-04 20:52:37 +09:00
Rujikorn Charakorn
610750e6fb
refactor after distillation (#6) 2025-07-30 18:38:51 +09:00
ShinnosukeUesakaSakana
d99fa8732b
Add distillation training (#5) 
- work with vllm self-gen data (everything is tokenized during self-gen)
- fix chat template!
- not use liger kernel for distillation (for both training and eval)
 2025-07-29 15:32:06 +09:00
51616
c7ab8d05b3 small configs 2025-07-21 09:03:58 +00:00
51616
9411b81b7f self-gen parquet format 2025-07-10 10:04:45 +00:00
51616
f60d98aa8b self_gen parquet pattern + configs 2025-07-10 09:09:28 +00:00
51616
2bce7d164d new self-gen exp (not using self-gen for qa ds with perfect label + small exp + remove "refusal" prompt) 2025-07-09 16:59:25 +00:00
51616
c9e958f963 self-gen w/ closed_qa format + perceiver scripts + no strip response 2025-07-08 14:21:43 +00:00
51616
43f47d40ad tiny exp 2025-07-03 16:26:26 +00:00
51616
1a03b01a52 small adjustment data gen (remove empty) 2025-07-03 11:02:46 +00:00
51616
f8a81a1eea self_gen temp fix + lv1 temp varying configs 2025-07-01 15:30:28 +00:00
51616
28eb85a2f3 tiny configs + fix dataset gen (remove repeated response + qa template for compact ds) + add ropes and drop eval ds 2025-06-30 16:10:52 +09:00
51616
19417d28a6 self_gen data name 2025-06-27 05:56:08 +00:00
51616
f07b38d5da more configs 2025-06-27 01:57:15 +09:00
51616
1457981348 compact ds + self_gen + update rope + drop multi answer eval 2025-06-26 16:04:33 +00:00
51616
f08831d957 qa data exp 2025-06-26 15:30:26 +09:00
51616
816377b4ac clean up + configs + scripts 2025-06-24 21:51:33 +09:00
51616
0d8e50d218 new trainng data format working non-packing 2025-06-23 23:21:22 +09:00
51616
36d78c827c transformers ver + squad only example 2025-06-20 21:38:26 +09:00
51616
4c1e4062e1 squad_hotpot config 2025-06-19 23:08:14 +09:00
51616
b218df0feb gemma 3 2025-06-17 11:35:12 +00:00
51616
b61086034c data exp + packing 2025-06-17 11:34:16 +00:00
51616
bae2593cd4 fix configs 2025-06-10 15:39:56 +00:00
51616
669186ace6 gemma data exp 2025-06-06 15:29:27 +00:00
51616
6242341099 data scripts + readme 2025-06-06 09:48:42 +00:00
51616
a9a164e2de clean up configs + scripts + chat_templates (+ %generation% tag) 2025-06-06 04:56:45 +00:00
51616
8b15901fb9 self-gen data naming + num_proc to 2 2025-06-03 15:32:38 +00:00
51616
5aa83b96e2 self gen data for all ds both train + val 2025-06-03 14:30:38 +00:00
51616
dcda29bae6 watcher + partial refactor eval + debug train w/ pwc and hotpot 2025-06-02 09:02:20 +00:00
51616
0ff762cdff medium fw qa 3 + fallback to manual cache dataset 2025-05-10 15:31:08 +00:00
51616
dde5f5cb4b load data from disk then convert to iterable 2025-05-09 13:11:34 +00:00
51616
bd61787486 allow streaming data 2025-05-09 10:59:08 +00:00
51616
dfac7a51b3 fw_qa_3_mini 2025-05-09 02:14:26 +00:00
51616
49214d34df add per rank latent query + per layer hyper head 2025-04-19 07:03:11 +00:00
51616
ff7bf56029 add booksum and gov_report 2025-02-18 10:43:02 +00:00
51616
584b2bfb44 fw_qa_xl + modernbert 2025-02-14 14:59:25 +00:00
51616
4bad251877 default warmup + decay 2025-01-28 21:36:19 +00:00
51616
cf742f8fbd everything :) 2025-01-22 12:59:22 +00:00
51616
930a22b2c0 openmathintx ft ds 2025-01-20 09:18:14 +00:00