apunkt/doc-to-lora
1
0
Fork 0
mirror of https://github.com/SakanaAI/doc-to-lora.git synced 2026-07-23 17:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix gcp bucket watcher

Browse source
This commit is contained in:
51616 2025-08-11 14:58:45 +09:00
parent ea2abde3a1
commit 6bdc9f7239
2 changed files with 19 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

16
README.md
View file

@ -77,15 +77,25 @@ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download --repo-type dataset rajpurk
gsutil -m rsync -r data/raw_datasets/self_gen gs://ctx-to-lora/data/raw_datasets/self_gen
# downloading from gcp bucket to login node
gsutil -m rsync -r gs://ctx-to-lora/data/raw_datasets/self_gen data/raw_datasets/self_gen
mkdir -p data/raw_datasets/self_gen
gsutil -m rsync -r gs://ctx-to-lora/data/raw_datasets/self_gen data/raw_datasets/self_gen
```
### Data Generation (v2)
### Upload/download checkpoints
```bash
# upload to bucket
gsutil -m rsync -r train_outputs gs://ctx-to-lora/train_outputs
# download from bucket
gsutil -m rsync -r gs://ctx-to-lora/train_outputs train_outputs
```
<!-- ### Data Generation (v2)
***Download a subset of generated Fineweb-QA directly***
```bash
uv run data/download_generated_fineweb_qa.py
uv run data/download_self_gen_qa.py
```
``` -->
Loading self-generated data
```python

7
gcp_bucket_watcher.py
View file

@ -128,7 +128,12 @@ def sync_cycle(args: argparse.Namespace, client: storage.Client) -> int:
if should_skip(rel, includes, excludes):
continue
if needs_transfer(meta, remote.get(rel), tolerance=2.0):
upload_file(client, args.bucket, args.prefix, meta, args.dest)
try:
upload_file(client, args.bucket, args.prefix, meta, args.dest)
except FileNotFoundError:
# File disappeared between scan and upload attempt; skip gracefully.
print(f"[skipped missing] {rel}")
continue
print(f"[uploaded] {rel} ({meta.size} bytes)")
changed += 1
if changed == 0: