apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
395 commits 2 branches 0 tags 47 MiB
Commit graph

46 commits

Author SHA1 Message Date
51616
696b45c51b icml rebuttal 2026-06-15 04:31:47 +00:00
51616
8093817c34 icml cleaned 2026-02-13 06:02:29 +00:00
51616
2ed418bc22 vlm ctx-encoder + slrum scripts + qwen self-gen 2025-10-27 06:20:26 +00:00
51616
178822b7d3 mistral config + script 2025-10-13 07:45:47 +00:00
51616
cddb1238ff self-gen data viewer 2025-10-10 08:09:46 +00:00
51616
fc086356b4 add qwen3-4b and mistral-7b (niah working) 2025-10-10 06:31:20 +00:00
51616
a074fd3305 self-gen data misaligned fixed + niah reproducible + vllm-tokenizers version combo 2025-10-08 05:17:50 +00:00
51616
3b798c670c python api + remove generate_multi_lora + rename config path + test eval scripts 2025-10-01 13:11:54 +00:00
51616
530fcd46cc permission + remove copilot intx + remove monkey patch 2025-09-30 15:18:09 +00:00
51616
b6679ba755 iclr cleanup 2025-09-30 14:53:21 +00:00
51616
c21ba2b481 iclr 2026 2025-09-29 00:39:17 +09:00
51616
650395874f toy exp w/ self-gen + use double bos for eval (following vllm 0.8.4) 2025-09-07 17:16:20 +09:00
51616
a4f4dfd522 magic num eval upto 128k 2025-09-06 02:21:15 +09:00
51616
3505b006ad toy ctx magic num eval upto 32k + remove configs 2025-09-01 05:57:06 +00:00
51616
f209c8b9a1 sakura scripts 2025-08-11 15:00:30 +09:00
51616
178e7165ea per-rank bias + skip interleave if cached+ add bnb 2025-08-09 15:20:37 +00:00
Rujikorn Charakorn
891c0bd256
Refactor_and_improve_data (#7) 
* faster slice

* add facts + ctx_qa

* new configs

* new scripts

* intx_sft.py to train.py

* add kaggle for downloading facts

* max_new_tokens cli for eval

* generate negative_nq

* scripts + configs

* default vals

* max_val_samples_per_ds=500

* more efficient layer-to-layer ctx encoder

* use_per_ctx_average_loss

* faster processing

* small exp distill

* scripts

* more robust watcher

* per-module l1_norm avg

* per-ctx average loss

* clear_gpu
 2025-08-04 20:52:37 +09:00
Rujikorn Charakorn
610750e6fb
refactor after distillation (#6) 2025-07-30 18:38:51 +09:00
51616
2b824ed813 scripts 2025-07-29 07:00:03 +00:00
ShinnosukeUesakaSakana
d99fa8732b
Add distillation training (#5) 
- work with vllm self-gen data (everything is tokenized during self-gen)
- fix chat template!
- not use liger kernel for distillation (for both training and eval)
 2025-07-29 15:32:06 +09:00
51616
6c5a954188 short ctx exp scripts 2025-07-21 09:05:08 +00:00
51616
f5e2f55ada default script in w/ perceiver impl 2025-07-09 01:52:37 +00:00
51616
c9e958f963 self-gen w/ closed_qa format + perceiver scripts + no strip response 2025-07-08 14:21:43 +00:00
51616
63ecb73fdb bigger perceiver scripts 2025-07-08 06:40:47 +00:00
Rujikorn Charakorn
9613c65c20
Perceiver blocks + layer-to-layer (#4) 2025-07-07 17:31:07 +09:00
51616
43f47d40ad tiny exp 2025-07-03 16:26:26 +00:00
51616
375c50757c eval generative only + scripts 2025-06-30 16:41:26 +09:00
51616
4ef5132c56 self_gen train script 2025-06-27 15:05:17 +09:00
51616
faac7dc35f readme dataloading + comments cleanup 2025-06-27 14:59:27 +09:00
51616
f08831d957 qa data exp 2025-06-26 15:30:26 +09:00
51616
816377b4ac clean up + configs + scripts 2025-06-24 21:51:33 +09:00
51616
462ac33d7b per layer lightweightlora 2025-06-17 11:34:47 +00:00
51616
b61086034c data exp + packing 2025-06-17 11:34:16 +00:00
51616
4a2960cd64 gemma data exp scripts 2025-06-10 15:40:19 +00:00
51616
f30225f3c0 config name 2025-06-06 15:40:36 +00:00
51616
669186ace6 gemma data exp 2025-06-06 15:29:27 +00:00
51616
a9a164e2de clean up configs + scripts + chat_templates (+ %generation% tag) 2025-06-06 04:56:45 +00:00
51616
6c43c6a329 fw_qa_3_mini_self_gen scripts + sleep before load 2025-06-04 14:07:32 +00:00
51616
0e4c1444f9 webui chat + result display + eval datasets + preprocessing eval + refactoring data_utils.py to data_definition.py 2025-05-20 06:06:20 +00:00
51616
8fea5153dc faster processing (num_proc + only main preprocess ds) 2025-05-10 17:06:05 +00:00
51616
0ff762cdff medium fw qa 3 + fallback to manual cache dataset 2025-05-10 15:31:08 +00:00
51616
dde5f5cb4b load data from disk then convert to iterable 2025-05-09 13:11:34 +00:00
51616
dfac7a51b3 fw_qa_3_mini 2025-05-09 02:14:26 +00:00
51616
767dc634b9 per-layer processing working + linear proj before mlp 2025-05-07 15:12:02 +00:00
51616
cd28af25c0 roughly match the base arch (not yet per layer) 2025-05-03 12:22:59 +00:00
51616
49214d34df add per rank latent query + per layer hyper head 2025-04-19 07:03:11 +00:00