57 commits

Author	SHA1	Message	Date
51616	696b45c51b	icml rebuttal	2026-06-15 04:31:47 +00:00
51616	8093817c34	icml cleaned	2026-02-13 06:02:29 +00:00
51616	2ed418bc22	vlm ctx-encoder + slrum scripts + qwen self-gen	2025-10-27 06:20:26 +00:00
51616	a074fd3305	self-gen data misaligned fixed + niah reproducible + vllm-tokenizers version combo	2025-10-08 05:17:50 +00:00
51616	3b798c670c	python api + remove generate_multi_lora + rename config path + test eval scripts	2025-10-01 13:11:54 +00:00
51616	b6679ba755	iclr cleanup	2025-09-30 14:53:21 +00:00
51616	ad18d38374	iclr 2026 (3)	2025-09-28 15:46:09 +00:00
51616	c21ba2b481	iclr 2026	2025-09-29 00:39:17 +09:00
51616	86ba2bc4a5	self-gen data import	2025-09-13 23:52:24 +09:00
51616	f2efe22cba	Merge branch 'main' of https://github.com/SakanaAI/ctx-to-lora	2025-09-07 17:16:40 +09:00
51616	650395874f	toy exp w/ self-gen + use double bos for eval (following vllm 0.8.4)	2025-09-07 17:16:20 +09:00
51616	0c1a8bb3e8	update data gen/def	2025-09-06 08:44:11 +00:00
51616	a4f4dfd522	magic num eval upto 128k	2025-09-06 02:21:15 +09:00
51616	3505b006ad	toy ctx magic num eval upto 32k + remove configs	2025-09-01 05:57:06 +00:00
51616	a269aff51d	fix empty q_list	2025-08-29 04:50:18 +00:00
Rujikorn Charakorn	ee9039295e	LoRA separate bias param + merger + toy ds (#15) ... * separate bias + head weight init * smaller bias init + sum with 1/sqrt(combined_r) scaling + better lora merge logic * ok init * chunked ctx magic num working * configs * no transpose + only sum aggregation * fix cli * cli * 32-256 data * example cli + smaller dataset * toy ctx magic num ds size + quantize + remove head bias * configs removed * require lora_bias * Merge remote-tracking branch 'origin/main' into new-arch	2025-08-29 11:55:24 +09:00
51616	9d82014dc2	q gen can take ds_names + self_gen summ/struct/cot prob + ctx qa v3 data	2025-08-28 17:42:43 +00:00
ShinnosukeUesakaSakana	e1a3926e25	Improved training data generation (#14) ... * add catridge prompts * fix bugs * add instruction * add generic prompt + improved prompts and template * default max len --------- Co-authored-by: 51616 <rujikorn.ch@gmail.com>	2025-08-28 21:03:53 +09:00
Rujikorn Charakorn	c5e9bc769d	toy ctx nums and multi-lora training (#11) ... * multi-lora trainable toy number repeat dataset * per rank bias init * remove head_bias +simplify merge + skip perplexities metric * ctx_numbers train example * self-gen ctx numbers example	2025-08-18 18:38:07 +09:00
51616	8bbb5d91a1	fix compact builder (import)	2025-08-11 14:59:03 +09:00
Rujikorn Charakorn	891c0bd256	Refactor_and_improve_data (#7) ... * faster slice * add facts + ctx_qa * new configs * new scripts * intx_sft.py to train.py * add kaggle for downloading facts * max_new_tokens cli for eval * generate negative_nq * scripts + configs * default vals * max_val_samples_per_ds=500 * more efficient layer-to-layer ctx encoder * use_per_ctx_average_loss * faster processing * small exp distill * scripts * more robust watcher * per-module l1_norm avg * per-ctx average loss * clear_gpu	2025-08-04 20:52:37 +09:00
Rujikorn Charakorn	610750e6fb	refactor after distillation (#6)	2025-07-30 18:38:51 +09:00
ShinnosukeUesakaSakana	d99fa8732b	Add distillation training (#5) ... - work with vllm self-gen data (everything is tokenized during self-gen) - fix chat template! - not use liger kernel for distillation (for both training and eval)	2025-07-29 15:32:06 +09:00
51616	0945adeee5	compact script update	2025-07-21 09:04:53 +00:00
51616	c39e44cc90	self-gen can now take --temp and --closed_qa_prob	2025-07-10 02:22:11 +00:00
51616	2bce7d164d	new self-gen exp (not using self-gen for qa ds with perfect label + small exp + remove "refusal" prompt)	2025-07-09 16:59:25 +00:00
Edoardo Cetin	64f7b3d401	Store top k logits tests and scripts (#2) (to be refactored + integrated with data pipeline)	2025-07-09 13:18:15 +09:00
51616	c9e958f963	self-gen w/ closed_qa format + perceiver scripts + no strip response	2025-07-08 14:21:43 +00:00
51616	1a03b01a52	small adjustment data gen (remove empty)	2025-07-03 11:02:46 +00:00
51616	f8a81a1eea	self_gen temp fix + lv1 temp varying configs	2025-07-01 15:30:28 +00:00
51616	92ba1ca76c	tiny data def	2025-07-01 05:52:47 +00:00
51616	4d544e538d	temp naming	2025-06-30 16:38:10 +09:00
51616	28eb85a2f3	tiny configs + fix dataset gen (remove repeated response + qa template for compact ds) + add ropes and drop eval ds	2025-06-30 16:10:52 +09:00
51616	e4cb1ffa55	add todo for self-gen logits	2025-06-27 17:43:37 +09:00
51616	dee7df746d	download self gen qa	2025-06-26 16:30:04 +00:00
51616	1457981348	compact ds + self_gen + update rope + drop multi answer eval	2025-06-26 16:04:33 +00:00
51616	c9b6346659	download generated fw qa	2025-06-26 16:24:18 +09:00
51616	4cedfc2514	hf data	2025-06-26 16:23:14 +09:00
51616	3732f0860b	data name	2025-06-20 21:37:13 +09:00
51616	165fe1c982	qa datagen v2	2025-06-19 23:09:13 +09:00
51616	4a5773bf69	install + data folder for fav num	2025-06-19 04:21:52 +09:00
51616	6242341099	data scripts + readme	2025-06-06 09:48:42 +00:00
51616	7a889d0d7e	self-gen naming	2025-06-03 15:33:53 +00:00
51616	5aa83b96e2	self gen data for all ds both train + val	2025-06-03 14:30:38 +00:00
51616	31de0779da	data hash + pretrain data gen from fw qa	2025-05-30 12:56:45 +00:00
51616	b1f92663a7	data gen scripts	2025-05-30 03:11:03 +00:00
Rujikorn Charakorn	9f986cf938	rearrange + move to uv (#1)	2025-05-27 21:18:15 +09:00
51616	944d844da8	add len info eval + triviaqa_retrieved	2025-05-26 09:16:35 +00:00
51616	1699668aef	ctx numbers is now reconstruction prompt	2025-01-08 16:16:43 +00:00
51616	2a4e5742ca	data + configs testing model capability	2025-01-07 14:44:51 +00:00