apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

remove old data gen

Browse source
This commit is contained in:
51616 2025-06-06 10:37:19 +00:00
parent 9e8ba28969
commit 78393db4b6
1 changed files with 0 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

7
README.md
View file

@ -82,13 +82,6 @@ uv run python data/generate_fav_num_big.py
### Data Generation
```bash
# this might take several days...
# python process_fineweb.py
# python generate_fw_qa.py
# python post_process_fw_qa.py
conda activate vllm;
# [WIP] download fineweb-edu
vllm_model=mistralai/Mistral-Small-3.1-24B-Instruct-2503 uv run python generate_fw_edu_qa_vllm.py "0*_*" 3
# [WIP] there are other datasets where we used openAI to gen QA pairs
# 0. download fineweb_edu to `data/raw_datasets/fineweb_edu