apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
395 commits 2 branches 0 tags 47 MiB
Commit graph

65 commits

Author SHA1 Message Date
51616
696b45c51b icml rebuttal 2026-06-15 04:31:47 +00:00
51616
22267c7666 Update README 2026-02-13 08:55:01 +00:00
51616
32b5d2a197 video update 2026-02-13 08:49:48 +00:00
51616
8beaa905e7 Update README demo video section for GitHub rendering 2026-02-13 08:43:49 +00:00
51616
ec16f545c1 add bib gif video (waiting for arxiv id) 2026-02-13 08:30:08 +00:00
51616
cddb1238ff self-gen data viewer 2025-10-10 08:09:46 +00:00
51616
a074fd3305 self-gen data misaligned fixed + niah reproducible + vllm-tokenizers version combo 2025-10-08 05:17:50 +00:00
51616
3b798c670c python api + remove generate_multi_lora + rename config path + test eval scripts 2025-10-01 13:11:54 +00:00
51616
b6679ba755 iclr cleanup 2025-09-30 14:53:21 +00:00
51616
7d95db6672 context distillation eval + timer 2025-09-12 16:34:03 +09:00
51616
650395874f toy exp w/ self-gen + use double bos for eval (following vllm 0.8.4) 2025-09-07 17:16:20 +09:00
51616
a4f4dfd522 magic num eval upto 128k 2025-09-06 02:21:15 +09:00
51616
41a1cfb90a AB scaler + use bias for generation 2025-09-04 21:15:42 +09:00
Rujikorn Charakorn
ee9039295e
LoRA separate bias param + merger + toy ds (#15) 
* separate bias + head weight init

* smaller bias init + sum with 1/sqrt(combined_r) scaling + better  lora merge logic

* ok init

* chunked ctx magic num working

* configs

* no transpose + only sum aggregation

* fix cli

* cli

* 32-256 data

* example cli + smaller dataset

* toy ctx magic num ds size + quantize + remove head bias

* configs removed

* require lora_bias

* Merge remote-tracking branch 'origin/main' into new-arch
 2025-08-29 11:55:24 +09:00
51616
9d82014dc2 q gen can take ds_names + self_gen summ/struct/cot prob + ctx qa v3 data 2025-08-28 17:42:43 +00:00
ShinnosukeUesakaSakana
e1a3926e25
Improved training data generation (#14) 
* add catridge prompts

* fix bugs

* add instruction

* add generic prompt + improved prompts and template

* default max len

---------

Co-authored-by: 51616 <rujikorn.ch@gmail.com>
 2025-08-28 21:03:53 +09:00
Rujikorn Charakorn
c5e9bc769d
toy ctx nums and multi-lora training (#11) 
* multi-lora trainable toy number repeat dataset

* per rank bias init

* remove head_bias +simplify merge + skip perplexities metric

* ctx_numbers train example

* self-gen ctx numbers example
 2025-08-18 18:38:07 +09:00
51616
996b89b5a0 chunked eval + backward compat model load 2025-08-12 07:58:36 +00:00
51616
48be13dae3 self-gen script example 2025-08-11 09:47:05 +00:00
Rujikorn Charakorn
ec955a935d
chunked ctx eval (#9) 
* got burned by floating point precision again :)))))))))))

* split-ctx working eval (batch_size=1)

* batched lora aggregation (sum + mean) eval + finegrain eval len bins

* fix ctx_ids assert
 2025-08-11 18:44:26 +09:00
51616
6bdc9f7239 fix gcp bucket watcher 2025-08-11 14:58:45 +09:00
51616
178e7165ea per-rank bias + skip interleave if cached+ add bnb 2025-08-09 15:20:37 +00:00
Rujikorn Charakorn
891c0bd256
Refactor_and_improve_data (#7) 
* faster slice

* add facts + ctx_qa

* new configs

* new scripts

* intx_sft.py to train.py

* add kaggle for downloading facts

* max_new_tokens cli for eval

* generate negative_nq

* scripts + configs

* default vals

* max_val_samples_per_ds=500

* more efficient layer-to-layer ctx encoder

* use_per_ctx_average_loss

* faster processing

* small exp distill

* scripts

* more robust watcher

* per-module l1_norm avg

* per-ctx average loss

* clear_gpu
 2025-08-04 20:52:37 +09:00
Rujikorn Charakorn
610750e6fb
refactor after distillation (#6) 2025-07-30 18:38:51 +09:00
ShinnosukeUesakaSakana
d99fa8732b
Add distillation training (#5) 
- work with vllm self-gen data (everything is tokenized during self-gen)
- fix chat template!
- not use liger kernel for distillation (for both training and eval)
 2025-07-29 15:32:06 +09:00
51616
82e44c72c8 more robust watcher 2025-07-22 00:48:29 +00:00
51616
2bce7d164d new self-gen exp (not using self-gen for qa ds with perfect label + small exp + remove "refusal" prompt) 2025-07-09 16:59:25 +00:00
51616
e4cb1ffa55 add todo for self-gen logits 2025-06-27 17:43:37 +09:00
51616
faac7dc35f readme dataloading + comments cleanup 2025-06-27 14:59:27 +09:00
51616
ec2e717d61 self-gen data 2025-06-27 01:56:54 +09:00
51616
dee7df746d download self gen qa 2025-06-26 16:30:04 +00:00
51616
1457981348 compact ds + self_gen + update rope + drop multi answer eval 2025-06-26 16:04:33 +00:00
51616
4cedfc2514 hf data 2025-06-26 16:23:14 +09:00
51616
816377b4ac clean up + configs + scripts 2025-06-24 21:51:33 +09:00
51616
c884dad363 context number example cmd 2025-06-24 18:17:08 +09:00
51616
d8d2b2fa2e data gen cmd 2025-06-23 13:31:50 +09:00
51616
36d78c827c transformers ver + squad only example 2025-06-20 21:38:26 +09:00
51616
94b8cc931c data gen example + gemma3-1b-it example 2025-06-19 23:08:35 +09:00
51616
bacbe7673e lora r 16 example 2025-06-17 11:36:05 +00:00
51616
0e76489366 more organized eval + viz_utils (llm-comparator) 2025-06-13 08:17:21 +00:00
51616
32efc47845 fix eval errs 2025-06-12 16:04:26 +00:00
51616
eb87612f91 eval ds 2025-06-12 10:00:41 +00:00
51616
76d2b154fd pre-packing runnable 2025-06-10 15:41:18 +00:00
51616
78393db4b6 remove old data gen 2025-06-06 10:37:19 +00:00
51616
6242341099 data scripts + readme 2025-06-06 09:48:42 +00:00
51616
016b47dedc pre-train processing 2025-06-06 05:01:22 +00:00
51616
3eb9fda2bf self-gen cmd 2025-06-03 15:59:51 +00:00
51616
5d793e329b cmd readme 2025-06-03 14:29:04 +00:00
51616
dcda29bae6 watcher + partial refactor eval + debug train w/ pwc and hotpot 2025-06-02 09:02:20 +00:00
51616
f8a7ef4b92 versioning + cli commands 2025-05-30 08:11:00 +00:00