apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

LoRA separate bias param + merger + toy ds (#15)

Browse source 
* separate bias + head weight init

* smaller bias init + sum with 1/sqrt(combined_r) scaling + better  lora merge logic

* ok init

* chunked ctx magic num working

* configs

* no transpose + only sum aggregation

* fix cli

* cli

* 32-256 data

* example cli + smaller dataset

* toy ctx magic num ds size + quantize + remove head bias

* configs removed

* require lora_bias

* Merge remote-tracking branch 'origin/main' into new-arch
This commit is contained in:
Rujikorn Charakorn 2025-08-29 11:55:24 +09:00 • committed by GitHub
parent 9d82014dc2
commit ee9039295e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 400 additions and 455 deletions
Download patch file Download diff file Expand all files Collapse all files

11
run_eval.py
View file

@ -66,12 +66,6 @@ if __name__ == "__main__":
default=-1,
help="Maximum length of context chunk for evaluation",
)
parser.add_argument(
"--lora_aggregation",
choices=["mean", "sum"],
default="sum",
help="LoRA aggregation method",
)
parser.add_argument(
"--max_new_tokens",
type=int,
@ -87,6 +81,11 @@ if __name__ == "__main__":
cli_args = vars(parser.parse_args())
# setup_logging(output_dir, debug=os.getenv("DEBUG", False))
if cli_args["model_name_or_path"]:
assert cli_args["max_ctx_chunk_len"] <= 0, (
f"Evaluating base model shouldn't be used with `max_ctx_chunk_len`"
)
eval_batch_size_gen = cli_args.pop("eval_batch_size_gen")
eval_batch_size = cli_args.pop("eval_batch_size")
# run_eval(