32 commits

Author	SHA1	Message	Date
51616	f209c8b9a1	sakura scripts	2025-08-11 15:00:30 +09:00
51616	178e7165ea	per-rank bias + skip interleave if cached+ add bnb	2025-08-09 15:20:37 +00:00
Rujikorn Charakorn	891c0bd256	Refactor_and_improve_data (#7) ... * faster slice * add facts + ctx_qa * new configs * new scripts * intx_sft.py to train.py * add kaggle for downloading facts * max_new_tokens cli for eval * generate negative_nq * scripts + configs * default vals * max_val_samples_per_ds=500 * more efficient layer-to-layer ctx encoder * use_per_ctx_average_loss * faster processing * small exp distill * scripts * more robust watcher * per-module l1_norm avg * per-ctx average loss * clear_gpu	2025-08-04 20:52:37 +09:00
Rujikorn Charakorn	610750e6fb	refactor after distillation (#6)	2025-07-30 18:38:51 +09:00
51616	2b824ed813	scripts	2025-07-29 07:00:03 +00:00
ShinnosukeUesakaSakana	d99fa8732b	Add distillation training (#5) ... - work with vllm self-gen data (everything is tokenized during self-gen) - fix chat template! - not use liger kernel for distillation (for both training and eval)	2025-07-29 15:32:06 +09:00
51616	6c5a954188	short ctx exp scripts	2025-07-21 09:05:08 +00:00
51616	f5e2f55ada	default script in w/ perceiver impl	2025-07-09 01:52:37 +00:00
51616	c9e958f963	self-gen w/ closed_qa format + perceiver scripts + no strip response	2025-07-08 14:21:43 +00:00
51616	63ecb73fdb	bigger perceiver scripts	2025-07-08 06:40:47 +00:00
Rujikorn Charakorn	9613c65c20	Perceiver blocks + layer-to-layer (#4)	2025-07-07 17:31:07 +09:00
51616	43f47d40ad	tiny exp	2025-07-03 16:26:26 +00:00
51616	375c50757c	eval generative only + scripts	2025-06-30 16:41:26 +09:00
51616	4ef5132c56	self_gen train script	2025-06-27 15:05:17 +09:00
51616	faac7dc35f	readme dataloading + comments cleanup	2025-06-27 14:59:27 +09:00
51616	f08831d957	qa data exp	2025-06-26 15:30:26 +09:00
51616	816377b4ac	clean up + configs + scripts	2025-06-24 21:51:33 +09:00
51616	462ac33d7b	per layer lightweightlora	2025-06-17 11:34:47 +00:00
51616	b61086034c	data exp + packing	2025-06-17 11:34:16 +00:00
51616	4a2960cd64	gemma data exp scripts	2025-06-10 15:40:19 +00:00
51616	f30225f3c0	config name	2025-06-06 15:40:36 +00:00
51616	669186ace6	gemma data exp	2025-06-06 15:29:27 +00:00
51616	a9a164e2de	clean up configs + scripts + chat_templates (+ %generation% tag)	2025-06-06 04:56:45 +00:00
51616	6c43c6a329	fw_qa_3_mini_self_gen scripts + sleep before load	2025-06-04 14:07:32 +00:00
51616	0e4c1444f9	webui chat + result display + eval datasets + preprocessing eval + refactoring data_utils.py to data_definition.py	2025-05-20 06:06:20 +00:00
51616	8fea5153dc	faster processing (num_proc + only main preprocess ds)	2025-05-10 17:06:05 +00:00
51616	0ff762cdff	medium fw qa 3 + fallback to manual cache dataset	2025-05-10 15:31:08 +00:00
51616	dde5f5cb4b	load data from disk then convert to iterable	2025-05-09 13:11:34 +00:00
51616	dfac7a51b3	fw_qa_3_mini	2025-05-09 02:14:26 +00:00
51616	767dc634b9	per-layer processing working + linear proj before mlp	2025-05-07 15:12:02 +00:00
51616	cd28af25c0	roughly match the base arch (not yet per layer)	2025-05-03 12:22:59 +00:00
51616	49214d34df	add per rank latent query + per layer hyper head	2025-04-19 07:03:11 +00:00