51616
8093817c34
icml cleaned
2026-02-13 06:02:29 +00:00
51616
2ed418bc22
vlm ctx-encoder + slrum scripts + qwen self-gen
2025-10-27 06:20:26 +00:00
51616
b6679ba755
iclr cleanup
2025-09-30 14:53:21 +00:00
51616
c21ba2b481
iclr 2026
2025-09-29 00:39:17 +09:00
51616
7d95db6672
context distillation eval + timer
2025-09-12 16:34:03 +09:00
Rujikorn Charakorn
ee9039295e
LoRA separate bias param + merger + toy ds ( #15 )
...
* separate bias + head weight init
* smaller bias init + sum with 1/sqrt(combined_r) scaling + better lora merge logic
* ok init
* chunked ctx magic num working
* configs
* no transpose + only sum aggregation
* fix cli
* cli
* 32-256 data
* example cli + smaller dataset
* toy ctx magic num ds size + quantize + remove head bias
* configs removed
* require lora_bias
* Merge remote-tracking branch 'origin/main' into new-arch
2025-08-29 11:55:24 +09:00
Rujikorn Charakorn
c5e9bc769d
toy ctx nums and multi-lora training ( #11 )
...
* multi-lora trainable toy number repeat dataset
* per rank bias init
* remove head_bias +simplify merge + skip perplexities metric
* ctx_numbers train example
* self-gen ctx numbers example
2025-08-18 18:38:07 +09:00
Rujikorn Charakorn
ec955a935d
chunked ctx eval ( #9 )
...
* got burned by floating point precision again :)))))))))))
* split-ctx working eval (batch_size=1)
* batched lora aggregation (sum + mean) eval + finegrain eval len bins
* fix ctx_ids assert
2025-08-11 18:44:26 +09:00
51616
508928d429
interleave + packed caching (working)
2025-08-10 05:25:29 +00:00
51616
178e7165ea
per-rank bias + skip interleave if cached+ add bnb
2025-08-09 15:20:37 +00:00
Rujikorn Charakorn
891c0bd256
Refactor_and_improve_data ( #7 )
...
* faster slice
* add facts + ctx_qa
* new configs
* new scripts
* intx_sft.py to train.py
* add kaggle for downloading facts
* max_new_tokens cli for eval
* generate negative_nq
* scripts + configs
* default vals
* max_val_samples_per_ds=500
* more efficient layer-to-layer ctx encoder
* use_per_ctx_average_loss
* faster processing
* small exp distill
* scripts
* more robust watcher
* per-module l1_norm avg
* per-ctx average loss
* clear_gpu
2025-08-04 20:52:37 +09:00
