mirror of
https://github.com/ranausmanai/tinyforge-zero.git
synced 2026-06-08 20:55:13 +02:00
Initial release: TinyForge-Zero recipe + mined pairs + reproduction guide
Companion artifact for the paper 'How Far Can an Open Base Model
Self-Improve? Recipes, Limits, and Test-Time Synergy'.
Contents:
- recipe/{train_on_pairs,bootstrap,multi_pair_14b,curriculum_math,eval_raw,eval_plus,confirm}.py
- data/pairs_{7b_40,14b_multi_new60,math_13}.jsonl (released mined pairs)
- controls/mbpp_corrupt_control.py (the +0 negative control)
- docs/{scaling_chart,fig1_headline,fig6_boundary}.png
- REPRODUCE.md (paper claim -> exact command mapping)
This commit is contained in:
Rana Usman
commit
6305ff0f91
20 changed files with 2438 additions and 0 deletions
23
.gitignore
vendored
Normal file
23
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
# Adapters and large weights — distribute via HF Hub instead
|
||||
adapter_*/
|
||||
*.safetensors
|
||||
*.bin
|
||||
ckpt/
|
||||
ckpt_*/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Eval artifacts
|
||||
*.log
|
||||
eval_log.json
|
||||
iter_stats.jsonl
|
||||
*.npy
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
._*
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 Rana Usman
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
190
README.md
Normal file
190
README.md
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
# TinyForge-Zero
|
||||
|
||||
**Self-bootstrapping recipes for open base LLMs — no human-written training data.**
|
||||
|
||||
A 14B open base model reaches **80% on HumanEval** and **74.4% on HumanEval+** with only a Python interpreter as oracle and no human-curated training data, for under **$5** of consumer-GPU compute. This repo contains the recipes, mined pairs, evaluation scripts, and adapters from the paper.
|
||||
|
||||
📄 **Paper**: *How Far Can an Open Base Model Self-Improve? Recipes, Limits, and Test-Time Synergy* — arXiv link forthcoming
|
||||
📦 **Companion to**: `ranausmanai/tinyforge` (earlier exploratory experiments)
|
||||
|
||||
---
|
||||
|
||||
|
||||
|
||||
## Headline results
|
||||
|
||||
| Model | Setting | Base | After recipe | Δ |
|
||||
|-------|---------|-----:|-------------:|--:|
|
||||
| Qwen2.5-14B-Base | HumanEval (chat-template) | 44/164 (26.8%) | **131/164 (79.9%)** | **+53.0pp** |
|
||||
| Qwen2.5-14B-Base | HumanEval+ | — | **122/164 (74.4%)** | — |
|
||||
| Qwen2.5-7B-Base | HumanEval (best seed) | 25/164 (15.2%) | **112/164 (68.3%)** | **+53.0pp** |
|
||||
| Qwen2.5-3B-Base | GSM8K (auto-difficulty curriculum) | 32/100 | **66/100** | **+34pp** |
|
||||
| Random external pairs | HumanEval (control) | 25 | 25 | **+0** |
|
||||
|
||||
All numbers from `result.json` files in this repo's accompanying paper data. Same adapter under the multi-pair run's eval format reads **132/164 (80.5%)** — both round to 80%.
|
||||
|
||||
---
|
||||
|
||||
## The recipe in one diagram
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ (1) PROBLEM GEN Base model emits Python function + 3 asserts. │
|
||||
│ Keep only problems where the canonical passes. │
|
||||
│ │
|
||||
│ (2) DIVERSE SOLVE Resample 4–8 attempts at T=0.7–0.8. │
|
||||
│ Run each against the asserts. │
|
||||
│ │
|
||||
│ (3) PAIR MINING If some pass and some fail → (broken, fixed) │
|
||||
│ pair. Skip if all-pass (too easy) or all-fail │
|
||||
│ (above competence). │
|
||||
│ │
|
||||
│ (4) LoRA TRAIN Fine-tune (rank 16–32, q/k/v/o) on the pairs. │
|
||||
│ 2 epochs, lr=1e-4. No human data, no RL. │
|
||||
│ │
|
||||
│ (5) EVALUATE HumanEval / HumanEval+ / MBPP / GSM8K. │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
A control experiment — replacing the mined pairs with **identically-formatted but randomly-corrupted external pairs** — yields **exactly +0**. The signal is in the self-mined content, not the training-data format.
|
||||
|
||||
---
|
||||
|
||||
## What's in this repo
|
||||
|
||||
```
|
||||
tinyforge-zero/
|
||||
├── recipe/
|
||||
│ ├── train_on_pairs.py # Fast-path: train LoRA on a released pairs.jsonl
|
||||
│ ├── bootstrap.py # Full-path: self-bootstrap pipeline (mining + train, 7B / 3B)
|
||||
│ ├── multi_pair_14b.py # Full-path: aggressive multi-pair variant → 80.5% on 14B
|
||||
│ ├── curriculum_math.py # Full-path: auto-difficulty curriculum for GSM8K
|
||||
│ ├── eval_raw.py # HumanEval / MBPP / GSM8K eval (vLLM, raw-completion)
|
||||
│ ├── eval_plus.py # HumanEval+ contamination-resistant eval
|
||||
│ └── confirm.py # Confirmation re-eval against base
|
||||
├── data/
|
||||
│ ├── pairs_7b_40.jsonl # 40 self-mined pairs (Qwen2.5-7B-Base run)
|
||||
│ ├── pairs_14b_multi_new60.jsonl # 60 aggressive-mined pairs for 14B (+ warmup 40 → 100 total)
|
||||
│ └── pairs_math_13.jsonl # 13 curriculum-mined math pairs (Qwen2.5-3B-Base → GSM8K 32→66)
|
||||
├── controls/
|
||||
│ └── mbpp_corrupt_control.py # The +0 negative-control experiment
|
||||
├── docs/
|
||||
│ ├── scaling_chart.png # Recipe lift vs base capability (paper Fig 1)
|
||||
│ ├── fig1_headline.png # Headline result chart
|
||||
│ └── fig6_boundary.png # Boundary conditions across 9 models
|
||||
├── REPRODUCE.md # Paper figure/table → exact command mapping
|
||||
├── requirements.txt
|
||||
└── LICENSE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# 1. Clone
|
||||
git clone https://github.com/ranausmanai/tinyforge-zero.git
|
||||
cd tinyforge-zero
|
||||
|
||||
# 2. Install (Python 3.10+, CUDA 12.1+, GPU with ≥40GB VRAM recommended)
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 3. Baseline the model (so you know the lift is real)
|
||||
python recipe/eval_raw.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--bench humaneval
|
||||
|
||||
# 4. Train on the released 40 mined pairs (~10 min on H100)
|
||||
python recipe/train_on_pairs.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--pairs data/pairs_7b_40.jsonl \
|
||||
--epochs 2 --lr 1e-4 --lora-rank 16 \
|
||||
--out adapter_7b --seed 13
|
||||
|
||||
# 5. Evaluate the trained adapter
|
||||
python recipe/eval_raw.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--adapter adapter_7b \
|
||||
--bench humaneval
|
||||
```
|
||||
|
||||
Expected outcome: HumanEval moves from ~25/164 to **~95–112/164** (seed-dependent).
|
||||
|
||||
For the **14B → 80.5%** run, use `recipe/multi_pair_14b.py` with both `data/pairs_7b_40.jsonl` (warmup) and `data/pairs_14b_multi_new60.jsonl`. See [REPRODUCE.md](REPRODUCE.md) for the exact command and expected hardware.
|
||||
|
||||
---
|
||||
|
||||
## Boundary conditions (where the recipe fails)
|
||||
|
||||
|
||||
|
||||
The recipe works under stated conditions. We document four failure modes:
|
||||
|
||||
1. **Saturation**: Qwen3-8B/14B-Base and Qwen2.5-72B-Base have so little headroom on HumanEval that mining produces zero or negative lift.
|
||||
2. **Distribution mismatch**: Pairs mined on simple problems do not transfer to BigCodeBench-Hard (library code) or MATH-500 (competition math). Catastrophic when ignored — see the over-correction case (Qwen3-4B MATH-500 dropped 299 → 69).
|
||||
3. **Base capability floor**: OLMo-2-7B at 5/164 baseline produces too few "fix" attempts to mine from.
|
||||
4. **Self-correction trained on wrong→fix only**: model over-doubts and degrades on correct outputs. Mixing right→stays-right traces recovers it.
|
||||
|
||||
See the paper's §3 for measurements; the boundary chart above shows the recipe's lift across all 9 base models we tested.
|
||||
|
||||
---
|
||||
|
||||
## Adapters
|
||||
|
||||
The LoRA adapter weights for the headline 14B run (the 80.5% adapter) are ~200 MB and are not committed to this repo. They live separately:
|
||||
|
||||
- **Hugging Face Hub**: `ranausmanai/tinyforge-zero-qwen25-14b-lora` *(upload pending — for now, request access via GitHub Issues)*
|
||||
- **Local mirror used in the paper**: `/Users/usman/tinyforgeexperiment/results/multi_pair/multi_v1/adapter/`
|
||||
|
||||
The adapter is a standard `peft` LoRA over `Qwen/Qwen2.5-14B`. Load with:
|
||||
|
||||
```python
|
||||
from peft import PeftModel
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B", torch_dtype="bfloat16")
|
||||
model = PeftModel.from_pretrained(base, "ranausmanai/tinyforge-zero-qwen25-14b-lora")
|
||||
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hardware used in the paper
|
||||
|
||||
| Run | GPU | Time | Cost |
|
||||
|-----|-----|------|------|
|
||||
| Qwen2.5-7B 40-pair recipe | RTX 6000 Ada | ~30 min | <$1 |
|
||||
| Qwen2.5-14B multi-pair (80.5%) | 1× H100 80GB | ~95 min | ~$3.50 |
|
||||
| Qwen2.5-3B GSM8K curriculum | RTX 6000 Ada | ~30 min | <$1 |
|
||||
| Full eval suite (9 models, HE+HE++MBPP) | 1× H100 | ~3 hrs | ~$8 |
|
||||
|
||||
All runs were on rented consumer/cloud GPUs (RunPod). Total spend documented in the paper was under $50.
|
||||
|
||||
---
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{usman2026tinyforgezero,
|
||||
title = {How Far Can an Open Base Model Self-Improve?
|
||||
Recipes, Limits, and Test-Time Synergy},
|
||||
author = {Rana Usman},
|
||||
year = {2026},
|
||||
eprint = {TBD},
|
||||
archivePrefix = {arXiv},
|
||||
primaryClass = {cs.AI}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE). The mined pairs in `data/` are derivatives of base-model outputs (Qwen2.5 family, Apache-2.0). Treat downstream redistribution accordingly.
|
||||
|
||||
---
|
||||
|
||||
## Contact
|
||||
|
||||
- Issues / questions: [GitHub Issues](https://github.com/ranausmanai/tinyforge-zero/issues)
|
||||
- Email: usmanashrafrana@gmail.com
|
||||
202
REPRODUCE.md
Normal file
202
REPRODUCE.md
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
# Reproduction Guide
|
||||
|
||||
Maps every paper claim → exact command. There are **two replication paths**:
|
||||
|
||||
- **Fast path** — use `recipe/train_on_pairs.py` with the released `data/*.jsonl`. Skips the mining stage. Gets you the trained adapter and the headline number in ~30 min on an H100.
|
||||
- **Full path** — re-run the original research scripts (`bootstrap.py`, `multi_pair_14b.py`, `curriculum_math.py`) end-to-end including the self-mining step. This reproduces the recipe from scratch and verifies the mining is deterministic-ish (modulo sampling).
|
||||
|
||||
The fast path is what you want for paper verification. The full path is what you want if you're trying the recipe on a *new* base model.
|
||||
|
||||
---
|
||||
|
||||
## Environment
|
||||
|
||||
Tested on:
|
||||
- **H100 80GB** (recommended for 14B runs) — Debian 12, CUDA 12.4, driver 570+
|
||||
- **RTX 6000 Ada 48GB** — sufficient for 7B and 3B runs
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Exact stack used in the paper: `torch==2.6.0`, `transformers==4.51.3`, `vllm==0.8.5`, `peft==0.13.0`.
|
||||
|
||||
---
|
||||
|
||||
## FAST PATH — reproduce headline numbers from released pairs
|
||||
|
||||
### Qwen2.5-7B-Base → 25 → 95–112/164 (3-seed range)
|
||||
|
||||
```bash
|
||||
# 1. Baseline (raw-completion eval)
|
||||
python recipe/eval_raw.py --model Qwen/Qwen2.5-7B --bench humaneval
|
||||
# Expected: 25/164
|
||||
|
||||
# 2. Train on the released 40 pairs (try multiple seeds — small-data variance)
|
||||
for SEED in 7 13 42; do
|
||||
python recipe/train_on_pairs.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--pairs data/pairs_7b_40.jsonl \
|
||||
--out adapter_7b_seed${SEED} \
|
||||
--seed ${SEED} --lora-rank 16 --epochs 2 --lr 1e-4
|
||||
python recipe/eval_raw.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--adapter adapter_7b_seed${SEED} \
|
||||
--bench humaneval
|
||||
done
|
||||
# Expected: seed 7 → 104/164, seed 13 → 112/164, seed 42 → 95/164
|
||||
```
|
||||
|
||||
### Qwen2.5-14B-Base → 132/164 (80.5%) and HumanEval+ 122/164 (74.4%)
|
||||
|
||||
The 14B run uses 100 pairs total: the 40 warmup pairs + 60 new aggressive-mined pairs. Concatenate first, then train.
|
||||
|
||||
```bash
|
||||
cat data/pairs_7b_40.jsonl data/pairs_14b_multi_new60.jsonl > /tmp/pairs_14b_100.jsonl
|
||||
|
||||
python recipe/train_on_pairs.py \
|
||||
--model Qwen/Qwen2.5-14B \
|
||||
--pairs /tmp/pairs_14b_100.jsonl \
|
||||
--out adapter_14b_multi \
|
||||
--lora-rank 32 --epochs 2 --lr 1e-4
|
||||
|
||||
python recipe/eval_raw.py \
|
||||
--model Qwen/Qwen2.5-14B \
|
||||
--adapter adapter_14b_multi \
|
||||
--bench humaneval
|
||||
# Expected: 132/164 (80.5%) in the multi-pair eval format
|
||||
|
||||
python recipe/eval_plus.py \
|
||||
--model Qwen/Qwen2.5-14B \
|
||||
--adapter adapter_14b_multi
|
||||
# Expected: HumanEval+ 122/164 (74.4%)
|
||||
```
|
||||
|
||||
### Qwen2.5-3B-Base → GSM8K 32 → 66
|
||||
|
||||
```bash
|
||||
python recipe/train_on_pairs.py \
|
||||
--model Qwen/Qwen2.5-3B \
|
||||
--pairs data/pairs_math_13.jsonl \
|
||||
--out adapter_3b_math \
|
||||
--lora-rank 16 --epochs 2 --lr 1e-4
|
||||
|
||||
# GSM8K eval — uses sympy as the verifier (no oracle math model needed).
|
||||
# eval_raw.py auto-detects GSM8K format and runs the right verifier.
|
||||
python recipe/eval_raw.py \
|
||||
--model Qwen/Qwen2.5-3B \
|
||||
--adapter adapter_3b_math \
|
||||
--bench gsm8k
|
||||
# Expected: 66/100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FULL PATH — re-mine from scratch
|
||||
|
||||
These reproduce the *mining* step too. Each script does generation → solving → mining → training → eval as one pipeline. They write a `pairs.jsonl` and a `result.json` under `--tag`.
|
||||
|
||||
### Self-bootstrap from scratch on Qwen2.5-7B
|
||||
|
||||
```bash
|
||||
python recipe/bootstrap.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--iterations 20 \
|
||||
--problems_per_iter 16 \
|
||||
--train_every 10 \
|
||||
--eval_every 10 \
|
||||
--tag bs_7b_rerun
|
||||
# Writes: results/bs_7b_rerun/{pairs.jsonl,ckpt_iter*,eval_log.json,result.json}
|
||||
# Expected final eval: 25 → 95–112 (seed-dependent)
|
||||
```
|
||||
|
||||
### Aggressive multi-pair mining on Qwen2.5-14B (the 80.5% headline)
|
||||
|
||||
```bash
|
||||
python recipe/multi_pair_14b.py \
|
||||
--model Qwen/Qwen2.5-14B \
|
||||
--warmup_pairs_path data/pairs_7b_40.jsonl \
|
||||
--n_warmup_pairs 40 \
|
||||
--n_problems 200 \
|
||||
--n_attempts 8 \
|
||||
--max_pairs_per_problem 4 \
|
||||
--lora_rank 32 --epochs 2 --lr 1e-4 \
|
||||
--tag multi_rerun
|
||||
# Writes: results/multi_pair/multi_rerun/{pairs_new.jsonl,adapter/,result.json}
|
||||
# Expected: trained 130–134/164 (~80%)
|
||||
```
|
||||
|
||||
### GSM8K auto-difficulty curriculum on Qwen2.5-3B
|
||||
|
||||
```bash
|
||||
python recipe/curriculum_math.py \
|
||||
--model Qwen/Qwen2.5-3B \
|
||||
--iterations 16 \
|
||||
--tag curr_3b_rerun
|
||||
# Mines 10–15 curriculum-difficulty pairs, trains, evals.
|
||||
# Expected: GSM8K 32 → 60–70 (some seed variance)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Control experiment (Figure 2)
|
||||
|
||||
Verifies the signal is in the *content* of self-mined pairs, not the format. Replaces the mined pairs with mechanically-corrupted external pairs (MBPP-style) that look identical structurally.
|
||||
|
||||
```bash
|
||||
python controls/mbpp_corrupt_control.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--tag mbpp_corrupt_control
|
||||
# Expected: HumanEval stays at 25/164 (Δ ≈ 0, ± seed noise)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pair-count sweep (Figure 3)
|
||||
|
||||
```bash
|
||||
for N in 10 21 40; do
|
||||
head -n $N data/pairs_7b_40.jsonl > /tmp/pairs_$N.jsonl
|
||||
python recipe/train_on_pairs.py \
|
||||
--model Qwen/Qwen2.5-7B \
|
||||
--pairs /tmp/pairs_$N.jsonl \
|
||||
--out adapter_n$N --epochs 2
|
||||
python recipe/eval_raw.py \
|
||||
--model Qwen/Qwen2.5-7B --adapter adapter_n$N --bench humaneval
|
||||
done
|
||||
# Expected: n=10 → ~51, n=21 → 86–95, n=40 → 95–112 (seed-dependent for small N)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Boundary conditions to verify (paper §3)
|
||||
|
||||
| Claim | Hint | Expected |
|
||||
|-------|------|----------|
|
||||
| Qwen3-8B saturated on HE | Run multi_pair_14b.py with `--model Qwen/Qwen3-8B-Base` | Base 132, adapter ≈ 118–133 — no clean lift |
|
||||
| Qwen2.5-72B saturated | Same on 72B with 10 pairs | Base 83 → trained 73 (−10) |
|
||||
| MATH-500 distribution mismatch | Mining on simple problems + MATH-500 eval | Base 279/500 → trained 239/500 (−40) |
|
||||
| Self-correction over-correction | Train on wrong→fix triples only, no right→stays-right | Base 299/500 → trained 69/500 (−230) |
|
||||
| BCB-Hard distribution mismatch | Apply 7B 40-pair adapter, eval on BCB-Hard | No transfer |
|
||||
|
||||
---
|
||||
|
||||
## Notes on stochasticity
|
||||
|
||||
- **vLLM sampling** is deterministic given a fixed seed, but vLLM 0.8.x occasionally changes pad/EOS handling between point releases. Pin to 0.8.5.
|
||||
- **LoRA training is seed-sensitive at small N.** The 7B 40-pair run spans 95–112/164 across seeds 7/13/42. The 14B 100-pair run is much tighter (130–134/164).
|
||||
- **Stop tokens matter.** Use `--stop "\nclass " --stop "\nif __name__"` for raw-completion eval. Wrong stop tokens cut output prematurely and produce artifactually low baselines. We saw this earlier in the project — see paper §2.
|
||||
|
||||
---
|
||||
|
||||
## Cost reference (May 2026, RunPod)
|
||||
|
||||
| Workflow | Hardware | Wall time | Cost |
|
||||
|----------|----------|-----------|------|
|
||||
| 7B headline (fast path) | RTX 6000 Ada 48GB | ~30 min | ~$0.50 |
|
||||
| 14B 80.5% (fast path) | H100 80GB | ~30 min | ~$1.50 |
|
||||
| 14B 80.5% full path (mining + train) | H100 80GB | ~95 min | ~$3.50 |
|
||||
| GSM8K 32→66 | RTX 6000 Ada | ~30 min | ~$0.50 |
|
||||
| Full eval matrix (9 models) | H100 80GB | ~3 hrs | ~$8 |
|
||||
|
||||
Total cost to verify all numbers in the paper via the fast path: **under $10**.
|
||||
239
controls/mbpp_corrupt_control.py
Normal file
239
controls/mbpp_corrupt_control.py
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
"""Control experiment: train same LoRA on 21 MBPP synthetic-corruption pairs (same format as bootstrap).
|
||||
If trained matches bootstrap (+48) → effect was format. If much smaller → bootstrap content is doing real work.
|
||||
"""
|
||||
import os, sys, json, time, re, gc, random, subprocess, tempfile, argparse
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
|
||||
from datasets import load_dataset, Dataset as HFDataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def run_python(code, timeout=8):
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
|
||||
f.write(code); path = f.name
|
||||
try:
|
||||
r = subprocess.run(["python3", path], capture_output=True, timeout=timeout, text=True, cwd="/tmp")
|
||||
if r.returncode == 0: return True, ""
|
||||
err = (r.stderr or r.stdout).strip().splitlines()
|
||||
return False, "\n".join(err[-3:])[:300]
|
||||
except subprocess.TimeoutExpired: return False, "timeout"
|
||||
finally:
|
||||
try: os.unlink(path)
|
||||
except: pass
|
||||
|
||||
|
||||
def extract_code(text):
|
||||
if "```python" in text: text = text.split("```python", 1)[1]
|
||||
elif "```" in text: text = text.split("```", 1)[1]
|
||||
if "```" in text: text = text.split("```", 1)[0]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4):
|
||||
outs = []
|
||||
for i in range(0, len(prompts), batch):
|
||||
chunk = prompts[i:i+batch]
|
||||
texts = []
|
||||
for p in chunk:
|
||||
msgs = [{"role": "system", "content": "You are a Python coder."},
|
||||
{"role": "user", "content": p}]
|
||||
texts.append(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))
|
||||
inp = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1500).to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new, do_sample=temperature > 0,
|
||||
temperature=temperature if temperature > 0 else 1.0, top_p=0.95,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
for j in range(out.size(0)):
|
||||
outs.append(tok.decode(out[j][inp.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
return outs
|
||||
|
||||
|
||||
def humaneval_full(model, tok):
|
||||
he = list(load_dataset("openai_humaneval", split="test"))
|
||||
prompts = [p["prompt"] + "\n# Complete the function above." for p in he]
|
||||
outs = gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4)
|
||||
correct = 0
|
||||
for p, raw in zip(he, outs):
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
full = p["prompt"] + "\n" + code if "def " not in code else code
|
||||
test_code = full + "\n\n" + p["test"] + f"\n\ncheck({p['entry_point']})"
|
||||
ok, _ = run_python(test_code, timeout=10)
|
||||
if ok: correct += 1
|
||||
return correct, len(he)
|
||||
|
||||
|
||||
# Synthetic corruptions
|
||||
def corrupt(code, rng):
|
||||
"""Apply a random corruption. Return (broken, description) or (None, None)."""
|
||||
options = []
|
||||
if "<=" in code: options.append(("lte_to_lt", code.replace("<=", "<", 1), "swapped <= to <"))
|
||||
if "==" in code: options.append(("eq_to_neq", code.replace("==", "!=", 1), "flipped == to !="))
|
||||
m = re.search(r"range\((\w+)\)", code)
|
||||
if m: options.append(("range_off", code.replace(m.group(0), f"range({m.group(1)}+1)", 1), "off-by-one in range"))
|
||||
m = re.search(r"return\s+([\w\.\[\]]+)", code, re.MULTILINE)
|
||||
if m: options.append(("ret_neg", code.replace(m.group(0), f"return -{m.group(1)}", 1), "negated return"))
|
||||
m = re.search(r"(\w+)\s*\+\s*(\w+)", code)
|
||||
if m: options.append(("plus_minus", code.replace(m.group(0), f"{m.group(1)} - {m.group(2)}", 1), "+ to -"))
|
||||
if not options: return None, None, None
|
||||
name, broken, desc = rng.choice(options)
|
||||
return broken, desc, name
|
||||
|
||||
|
||||
def make_mbpp_pairs(n_target=21, seed=42):
|
||||
"""From MBPP train, create (broken, error, fixed) corruption pairs that pass tests on canonical."""
|
||||
rng = random.Random(seed)
|
||||
mbpp_train = list(load_dataset("mbpp", "sanitized", split="train"))
|
||||
rng.shuffle(mbpp_train)
|
||||
|
||||
# Reformat to look like our bootstrap pairs (signature, tests, broken, error, fixed)
|
||||
pairs = []
|
||||
for p in mbpp_train:
|
||||
sol = p["code"]
|
||||
tests = p["test_list"]
|
||||
# Canonical must pass tests
|
||||
ok_canon, _ = run_python(sol + "\n\n" + "\n".join(tests))
|
||||
if not ok_canon: continue
|
||||
# Try a corruption
|
||||
broken, desc, _ = corrupt(sol, rng)
|
||||
if broken is None or broken == sol: continue
|
||||
ok_broken, err = run_python(broken + "\n\n" + "\n".join(tests))
|
||||
if ok_broken: continue # wasn't a real corruption
|
||||
# Build signature stub from def line + docstring
|
||||
m = re.match(r"(def\s+\w+\([^)]*\):)", sol)
|
||||
if not m: continue
|
||||
sig_line = m.group(1)
|
||||
# Pull docstring if present
|
||||
lines = sol.split("\n")
|
||||
sig_block = sig_line
|
||||
for i, l in enumerate(lines):
|
||||
if l.startswith("def "):
|
||||
# Look for docstring
|
||||
for j in range(i+1, min(i+5, len(lines))):
|
||||
s = lines[j].strip()
|
||||
if s.startswith('"""') and s.endswith('"""') and len(s) > 6:
|
||||
sig_block = sig_line + "\n " + s
|
||||
break
|
||||
if s.startswith('"""'):
|
||||
# multi-line
|
||||
doc_lines = [s]
|
||||
for k in range(j+1, len(lines)):
|
||||
doc_lines.append(lines[k])
|
||||
if '"""' in lines[k]:
|
||||
break
|
||||
sig_block = sig_line + "\n " + "\n ".join(doc_lines)
|
||||
break
|
||||
break
|
||||
|
||||
pairs.append({
|
||||
"signature": sig_block, "tests": tests,
|
||||
"broken": broken, "error": err, "fixed": sol,
|
||||
"source": f"mbpp_corrupt:{desc}",
|
||||
})
|
||||
if len(pairs) >= n_target: break
|
||||
|
||||
return pairs
|
||||
|
||||
|
||||
def make_example(r, tok):
|
||||
user = f"Implement: {r['signature']}\n\nTests:\n{chr(10).join(r['tests'])}\n\nMy attempt:\n```python\n{r['broken']}\n```\n\nError:\n{r['error']}\n\nFix and output the corrected code only."
|
||||
assistant = f"```python\n{r['fixed']}\n```"
|
||||
msgs_pre = [{"role": "system", "content": "You are a Python coder."},
|
||||
{"role": "user", "content": user}]
|
||||
msgs_full = msgs_pre + [{"role": "assistant", "content": assistant}]
|
||||
pre = tok.apply_chat_template(msgs_pre, tokenize=False, add_generation_prompt=True)
|
||||
full = tok.apply_chat_template(msgs_full, tokenize=False)
|
||||
pre_ids = tok(pre, add_special_tokens=False)["input_ids"]
|
||||
full_ids = tok(full, add_special_tokens=False)["input_ids"]
|
||||
MAX = 1024
|
||||
full_ids = full_ids[:MAX]
|
||||
labels = list(full_ids)
|
||||
n_pre = min(len(pre_ids), len(labels))
|
||||
for i in range(n_pre): labels[i] = -100
|
||||
pad = MAX - len(full_ids)
|
||||
return {"input_ids": full_ids + [tok.pad_token_id]*pad,
|
||||
"attention_mask": [1]*len(full_ids) + [0]*pad,
|
||||
"labels": labels + [-100]*pad}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--n_pairs", type=int, default=21)
|
||||
ap.add_argument("--epochs", type=int, default=2)
|
||||
ap.add_argument("--seed", type=int, default=42)
|
||||
ap.add_argument("--tag", default="mbpp_control")
|
||||
args = ap.parse_args()
|
||||
|
||||
out_dir = f"/workspace/control/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
torch.manual_seed(args.seed)
|
||||
|
||||
log("generating MBPP synthetic pairs (control)")
|
||||
pairs = make_mbpp_pairs(args.n_pairs, args.seed)
|
||||
log(f" built {len(pairs)} pairs")
|
||||
if len(pairs) < args.n_pairs:
|
||||
log(f"WARN: only {len(pairs)} pairs available")
|
||||
with open(f"{out_dir}/pairs.jsonl", "w") as fh:
|
||||
for r in pairs: fh.write(json.dumps(r) + "\n")
|
||||
|
||||
log("loading Qwen/Qwen2.5-7B")
|
||||
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
tok.padding_side = "left"
|
||||
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", dtype=torch.bfloat16, device_map="cuda:0")
|
||||
|
||||
model.eval()
|
||||
log("eval BASE on full HumanEval")
|
||||
base_corr, base_total = humaneval_full(model, tok)
|
||||
log(f" BASE: {base_corr}/{base_total}")
|
||||
|
||||
lora_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
|
||||
model = get_peft_model(model, lora_cfg)
|
||||
|
||||
tok.padding_side = "right"
|
||||
examples = [make_example(r, tok) for r in pairs]
|
||||
ds = HFDataset.from_list(examples)
|
||||
targs = TrainingArguments(
|
||||
output_dir=f"{out_dir}/ckpt", num_train_epochs=args.epochs,
|
||||
per_device_train_batch_size=1, gradient_accumulation_steps=4,
|
||||
learning_rate=1e-4, bf16=True, logging_steps=10,
|
||||
save_strategy="no", report_to="none", remove_unused_columns=False, warmup_ratio=0.05,
|
||||
seed=args.seed,
|
||||
)
|
||||
log(f"training on {len(ds)} pairs, {args.epochs} epochs")
|
||||
Trainer(model=model, args=targs, train_dataset=ds, processing_class=tok).train()
|
||||
log("training done")
|
||||
tok.padding_side = "left"
|
||||
|
||||
model.eval()
|
||||
log("eval TRAINED on full HumanEval")
|
||||
tr_corr, tr_total = humaneval_full(model, tok)
|
||||
log(f" TRAINED: {tr_corr}/{tr_total}")
|
||||
|
||||
result = {
|
||||
"n_pairs": len(pairs), "epochs": args.epochs, "seed": args.seed,
|
||||
"data_source": "MBPP-corrupt (control)",
|
||||
"base": [base_corr, base_total], "trained": [tr_corr, tr_total],
|
||||
"delta": tr_corr - base_corr,
|
||||
"elapsed_s": time.time() - T0,
|
||||
}
|
||||
with open(f"{out_dir}/result.json", "w") as fh:
|
||||
json.dump(result, fh, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" CONTROL (MBPP-corrupt {len(pairs)} pairs, {args.epochs} epochs, seed {args.seed})")
|
||||
print(f" HUMANEVAL FULL: base={base_corr}/{base_total} trained={tr_corr}/{tr_total} Δ={tr_corr-base_corr:+d}")
|
||||
print(f" time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
60
data/pairs_14b_multi_new60.jsonl
Normal file
60
data/pairs_14b_multi_new60.jsonl
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{"signature": "def count_zeroes(arr):\n \"\"\"\n This function receives an array of integers arr, and returns the number of 0s in the array.\n\n >>> count_zeroes([1, 0, 5, 6, 0, 3])\n 2\n >>> count_zeroes([7, 7, 7, 7, 7])\n 0\n >>> count_zeroes([-4, 0, 0, 0, 2])\n 3\n \"\"\"\n\n count = 0\n for element in arr:\n if element == 0:\n count += 1\n return count\n\n# tests", "tests": ["assert count_zeroes([1, 0, 5, 6, 0, 3]) == 2", "assert count_zeroes([7, 7, 7, 7, 7]) == 0", "assert count_zeroes([-4, 0, 0, 0, 2]) == 3"], "broken": "block.", "error": " block.\n ^\nSyntaxError: invalid syntax", "fixed": "def count_zeroes(arr):\n count = 0\n for element in arr:\n if element == 0:\n count += 1\n return count"}
|
||||
{"signature": "def count_zeroes(arr):\n \"\"\"\n This function receives an array of integers arr, and returns the number of 0s in the array.\n\n >>> count_zeroes([1, 0, 5, 6, 0, 3])\n 2\n >>> count_zeroes([7, 7, 7, 7, 7])\n 0\n >>> count_zeroes([-4, 0, 0, 0, 2])\n 3\n \"\"\"\n\n count = 0\n for element in arr:\n if element == 0:\n count += 1\n return count\n\n# tests", "tests": ["assert count_zeroes([1, 0, 5, 6, 0, 3]) == 2", "assert count_zeroes([7, 7, 7, 7, 7]) == 0", "assert count_zeroes([-4, 0, 0, 0, 2]) == 3"], "broken": "def count_zeroes(arr):\n \"\"\"\n This function receives an array of integers arr, and returns the number of 0s in the array.\n\n >>> count_zeroes([1, 0, 5, 6, 0, 3])\n 2\n >>> count_zeroes([7, 7, 7, 7, 7])\n 0\n >>> count_zeroes([-4, 0, 0, 0, 2])\n 3\n \"\"\"\n\n count = 0\n for element in arr:\n if element == 0:\n count += 1\n return count\n\n# tests\n\nTests:\nassert count_zeroes([1, 0, 5, 6, 0, 3]) == 2\nassert count_zeroes([7, 7, 7, 7, 7]) == 0\nassert count_zeroes([-4, 0, 0, 0, 2]) == 3", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def count_zeroes(arr):\n count = 0\n for element in arr:\n if element == 0:\n count += 1\n return count"}
|
||||
{"signature": "def count_even_pairs(s: str) -> int:\n \"\"\"\n Given a string 's', return the number of pairs (i, j) such that \n 0 <= i < j < len(s) and the sum of the ASCII values of s[i] and s[j] is even.\n \"\"\"\n ascii_values = [ord(char) for char in s]\n even_count = sum([value % 2 == 0 for value in ascii_values])\n odd_count = len(ascii_values) - even_count\n return even_count * (even_count - 1) // 2 + odd_count * (odd_count - 1) // 2\n\n# tests", "tests": ["assert count_even_pairs('abc') == 1", "assert count_even_pairs('abcd') == 2", "assert count_even_pairs('') == 0"], "broken": "block only.\n\nImplement: def count_even_pairs(s: str) -> int:\n \"\"\"\n Given a string 's', return the number of pairs (i, j) such that \n 0 <= i < j < len(s) and the sum of the ASCII values of s[i] and s[j] is even.\n \"\"\"\n ascii_values = [ord(char) for char in s]\n even_count = sum([value % 2 == 0 for value in ascii_values])\n odd_count = len(ascii_values) - even_count\n return even_count * (even_count - 1) // 2 + odd_count * (odd_count - 1) // 2", "error": " block only.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def count_even_pairs(s: str) -> int:\n \"\"\"\n Given a string 's', return the number of pairs (i, j) such that \n 0 <= i < j < len(s) and the sum of the ASCII values of s[i] and s[j] is even.\n \"\"\"\n ascii_values = [ord(char) for char in s]\n even_count = sum([value % 2 == 0 for value in ascii_values])\n odd_count = len(ascii_values) - even_count\n return even_count * (even_count - 1) // 2 + odd_count * (odd_count - 1) // 2"}
|
||||
{"signature": "def non_unique_elements(data: list) -> list:\n \"\"\"\n Your task is to design a function that will identify non-unique elements in a given list.\n Non-unique elements are those elements that appear more than once in the list. \n \n The function must:\n - Return a sorted list of these non-unique elements in ascending order.\n - Maintain original order for elements with the same value.\n - Handle edge cases such as empty list, duplicates, and negative numbers.\n \n Approach:\n - Create a dictionary to count the frequency of elements.\n - Iterate through the list to populate this dictionary.\n - Extract keys (elements) whose frequency is greater than 1.\n - Sort the result and maintain order in case of duplicates.\n \"\"\"\n \n # Frequency count using dictionary\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n \n # Extract non-unique elements\n non_unique = [key for key, value in freq.items() if value > 1]\n \n # Sorting with original order maintained for duplicates\n non_unique.sort(key=lambda x: data.index(x))\n \n return non_unique\n\n# tests", "tests": ["assert non_unique_elements([1, 2, 3, 1, 3]) == [1, 3]", "assert non_unique_elements([1, 1, 2, 2, 3]) == [1, 2]", "assert non_unique_elements([]) == []"], "broken": "block only. You are an expert Python coder. Output one", "error": " block only. You are an expert Python coder. Output one\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def non_unique_elements(data: list) -> list:\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n non_unique = [key for key, value in freq.items() if value > 1]\n non_unique.sort(key=lambda x: data.index(x))\n return non_unique"}
|
||||
{"signature": "def non_unique_elements(data: list) -> list:\n \"\"\"\n Your task is to design a function that will identify non-unique elements in a given list.\n Non-unique elements are those elements that appear more than once in the list. \n \n The function must:\n - Return a sorted list of these non-unique elements in ascending order.\n - Maintain original order for elements with the same value.\n - Handle edge cases such as empty list, duplicates, and negative numbers.\n \n Approach:\n - Create a dictionary to count the frequency of elements.\n - Iterate through the list to populate this dictionary.\n - Extract keys (elements) whose frequency is greater than 1.\n - Sort the result and maintain order in case of duplicates.\n \"\"\"\n \n # Frequency count using dictionary\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n \n # Extract non-unique elements\n non_unique = [key for key, value in freq.items() if value > 1]\n \n # Sorting with original order maintained for duplicates\n non_unique.sort(key=lambda x: data.index(x))\n \n return non_unique\n\n# tests", "tests": ["assert non_unique_elements([1, 2, 3, 1, 3]) == [1, 3]", "assert non_unique_elements([1, 1, 2, 2, 3]) == [1, 2]", "assert non_unique_elements([]) == []"], "broken": "", "error": " assert non_unique_elements([1, 2, 3, 1, 3]) == [1, 3]\n ^^^^^^^^^^^^^^^^^^^\nNameError: name 'non_unique_elements' is not defined", "fixed": "def non_unique_elements(data: list) -> list:\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n non_unique = [key for key, value in freq.items() if value > 1]\n non_unique.sort(key=lambda x: data.index(x))\n return non_unique"}
|
||||
{"signature": "def non_unique_elements(data: list) -> list:\n \"\"\"\n Your task is to design a function that will identify non-unique elements in a given list.\n Non-unique elements are those elements that appear more than once in the list. \n \n The function must:\n - Return a sorted list of these non-unique elements in ascending order.\n - Maintain original order for elements with the same value.\n - Handle edge cases such as empty list, duplicates, and negative numbers.\n \n Approach:\n - Create a dictionary to count the frequency of elements.\n - Iterate through the list to populate this dictionary.\n - Extract keys (elements) whose frequency is greater than 1.\n - Sort the result and maintain order in case of duplicates.\n \"\"\"\n \n # Frequency count using dictionary\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n \n # Extract non-unique elements\n non_unique = [key for key, value in freq.items() if value > 1]\n \n # Sorting with original order maintained for duplicates\n non_unique.sort(key=lambda x: data.index(x))\n \n return non_unique\n\n# tests", "tests": ["assert non_unique_elements([1, 2, 3, 1, 3]) == [1, 3]", "assert non_unique_elements([1, 1, 2, 2, 3]) == [1, 2]", "assert non_unique_elements([]) == []"], "broken": "block only. Implement: def non_unique_elements(data: list) -> list:\n \"\"\"\n Your task is to design a function that will identify non-unique elements in a given list.\n Non-unique elements are those elements that appear more than once in the list. \n \n The function must:\n - Return a sorted list of these non-unique elements in ascending order.\n - Maintain original order for elements with the same value.\n - Handle edge cases such as empty list, duplicates, and negative numbers.\n \n Approach:\n - Create a dictionary to count the frequency of elements.\n - Iterate through the list to populate this dictionary.\n - Extract keys (elements) whose frequency is greater than 1.\n - Sort the result and maintain order in case of duplicates.\n \"\"\"\n \n # Frequency count using dictionary\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n \n # Extract non-unique elements\n non_unique = [key for key, value in freq.items() if value > 1]\n \n # Sorting with original order maintained for duplicates\n non_unique.sort(key=lambda x: data.index(x))\n \n return non_unique\n\n# tests\n\nTests:\nassert non_unique_elements([1, 2, 3, 1, 3]) == [1, 3]\nassert non_unique_elements([1, 1, 2, 2, 3]) == [1, 2]\nassert non_unique_elements([]) == []\n\nOutput only the function implementation in one", "error": " block only. Implement: def non_unique_elements(data: list) -> list:\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def non_unique_elements(data: list) -> list:\n freq = {}\n for i in data:\n if i in freq:\n freq[i] += 1\n else:\n freq[i] = 1\n non_unique = [key for key, value in freq.items() if value > 1]\n non_unique.sort(key=lambda x: data.index(x))\n return non_unique"}
|
||||
{"signature": "def find_missing_number(sequence):\n \"\"\"\n Given a sequence of integers in non-decreasing order, where all numbers\n from 1 to N-1 are present except one, find the missing number. The missing\n number is guaranteed to be in the range [1, N].\n\n :param sequence: A list of integers in non-decreasing order, where one\n number from 1 to N is missing.\n :return: The missing number.\n \"\"\"\n n = len(sequence) + 1 # Total numbers from 1 to N\n full_sum = n * (n + 1) // 2 # Sum of numbers from 1 to N using Gauss's formula\n actual_sum = sum(sequence)\n return full_sum - actual_sum\n\n# tests", "tests": ["assert find_missing_number([1, 2, 3, 4, 6, 7, 8]) == 5", "assert find_missing_number([1, 2, 4]) == 3", "assert find_missing_number([1]) == 2"], "broken": "def find_missing_number(sequence):\n \"\"\"\n Given a sequence of integers in non-decreasing order, where all numbers\n from 1 to N-1 are present except one, find the missing number. The missing\n number is guaranteed to be in the range [1, N].\n\n :param sequence: A list of integers in non-decreasing order, where one\n number from 1 to N is missing.\n :return: The missing number.\n \"\"\"\n n = len(sequence) + 1 # Total numbers from 1 to N\n full_sum = n * (n + 1) // 2 # Sum of numbers from 1 to N using Gauss's formula\n actual_sum = sum(sequence)\n return full_sum - actual_sum\n\n# tests\n\nTests:\nassert find_missing_number([1, 2, 3, 4, 6, 7, 8]) == 5\nassert find_missing_number([1, 2, 4]) == 3\nassert find_missing_number([1]) == 2", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def find_missing_number(sequence):\n \"\"\"\n Given a sequence of integers in non-decreasing order, where all numbers\n from 1 to N-1 are present except one, find the missing number. The missing\n number is guaranteed to be in the range [1, N].\n\n :param sequence: A list of integers in non-decreasing order, where one\n number from 1 to N is missing.\n :return: The missing number.\n \"\"\"\n n = len(sequence) + 1 # Total numbers from 1 to N\n full_sum = n * (n + 1) // 2 # Sum of numbers from 1 to N using Gauss's formula\n actual_sum = sum(sequence)\n return full_sum - actual_sum"}
|
||||
{"signature": "def find_missing_number(sequence):\n \"\"\"\n Given a sequence of integers in non-decreasing order, where all numbers\n from 1 to N-1 are present except one, find the missing number. The missing\n number is guaranteed to be in the range [1, N].\n\n :param sequence: A list of integers in non-decreasing order, where one\n number from 1 to N is missing.\n :return: The missing number.\n \"\"\"\n n = len(sequence) + 1 # Total numbers from 1 to N\n full_sum = n * (n + 1) // 2 # Sum of numbers from 1 to N using Gauss's formula\n actual_sum = sum(sequence)\n return full_sum - actual_sum\n\n# tests", "tests": ["assert find_missing_number([1, 2, 3, 4, 6, 7, 8]) == 5", "assert find_missing_number([1, 2, 4]) == 3", "assert find_missing_number([1]) == 2"], "broken": "block only.", "error": " block only.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def find_missing_number(sequence):\n \"\"\"\n Given a sequence of integers in non-decreasing order, where all numbers\n from 1 to N-1 are present except one, find the missing number. The missing\n number is guaranteed to be in the range [1, N].\n\n :param sequence: A list of integers in non-decreasing order, where one\n number from 1 to N is missing.\n :return: The missing number.\n \"\"\"\n n = len(sequence) + 1 # Total numbers from 1 to N\n full_sum = n * (n + 1) // 2 # Sum of numbers from 1 to N using Gauss's formula\n actual_sum = sum(sequence)\n return full_sum - actual_sum"}
|
||||
{"signature": "def longest_substring_without_repeating_characters(s: str) -> int:\n \"\"\"\n Given a string `s`, find the length of the longest substring without repeating characters.\n The function should handle edge cases like empty strings, strings with no repeating characters,\n and strings with all characters repeating.\n \"\"\"\n seen = {}\n max_len = start = 0\n for index, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n max_len = max(max_len, index - start + 1)\n seen[char] = index\n\n return max_len\n\n# tests", "tests": ["assert longest_substring_without_repeating_characters(\"\") == 0", "assert longest_substring_without_repeating_characters(\"abcdef\") == 6", "assert longest_substring_without_repeating_characters(\"pwwkew\") == 3"], "broken": "Please determine whether the given text is related to computer science, if yes please return \"YES\", else return \"NO\".", "error": " Please determine whether the given text is related to computer science, if yes please return \"YES\", else return \"NO\".\n ^^^^^^^^^\nSyntaxError: invalid syntax", "fixed": "def longest_substring_without_repeating_characters(s: str) -> int:\n \"\"\"\n Given a string `s`, find the length of the longest substring without repeating characters.\n The function should handle edge cases like empty strings, strings with no repeating characters,\n and strings with all characters repeating.\n \"\"\"\n seen = {}\n max_len = start = 0\n for index, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n max_len = max(max_len, index - start + 1)\n seen[char] = index\n\n return max_len"}
|
||||
{"signature": "def find_missing_number(nums):\n \"\"\"Given an array of integers from 1 to n with one number missing, return the missing number.\n \n Args:\n nums: A list of integers from 1 to n, but one is missing.\n \n Returns:\n The missing integer.\n \"\"\"\n expected_sum = (len(nums) + 1) * (len(nums) + 2) // 2\n actual_sum = sum(nums)\n return expected_sum - actual_sum\n\n# tests", "tests": ["assert find_missing_number([1, 2, 4, 5]) == 3", "assert find_missing_number([1, 3, 4, 5]) == 2", "assert find_missing_number([1, 2, 3, 4]) == 5"], "broken": "block:", "error": " block:\n ^\nSyntaxError: invalid syntax", "fixed": "def find_missing_number(nums):\n expected_sum = (len(nums) + 1) * (len(nums) + 2) // 2\n actual_sum = sum(nums)\n return expected_sum - actual_sum"}
|
||||
{"signature": "def minimum_swaps_to_sort(nums):\n \"\"\"\n Given an unordered list of consecutive integers starting from 1 with no duplicates,\n find the minimum number of swaps required to sort the list in ascending order.\n\n Edge cases:\n - List is empty\n - List contains one or two elements\n - List is already sorted\n\n Args:\n - nums: A list of consecutive integers starting from 1 with no duplicates\n\n Returns:\n - The minimum number of swaps to sort the list\n \"\"\"\n count = 0\n for i in range(len(nums)):\n while nums[i] != i + 1:\n nums[nums[i] - 1], nums[i] = nums[i], nums[nums[i] - 1]\n count += 1\n return count\n\n# tests", "tests": ["assert minimum_swaps_to_sort([4, 3, 2, 1]) == 2", "assert minimum_swaps_to_sort([1, 5, 4, 3, 2]) == 2", "assert minimum_swaps_to_sort([]) == 0"], "broken": "block. Do not test. Do not explain.", "error": " block. Do not test. Do not explain.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def minimum_swaps_to_sort(nums):\n count = 0\n for i in range(len(nums)):\n while nums[i] != i + 1:\n nums[nums[i] - 1], nums[i] = nums[i], nums[nums[i] - 1]\n count += 1\n return count"}
|
||||
{"signature": "def shortest_palindrome(s: str) -> str:\n \"\"\"\n Given a string s, you are to return the shortest palindrome that can be created by adding characters to the front of s.\n For example, if s is 'aacecaaa', the shortest palindrome is 'aaacecaaa', while if s is 'abcd', the shortest palindrome is 'dcbabcd'.\n\n Args:\n s (str): A string composed of lowercase letters only.\n\n Returns:\n str: The shortest palindrome that can be created by adding characters to the front of s.\n \"\"\"\n if not s:\n return \"\"\n \n # Finding the longest palindrome from the start\n n = len(s)\n reverse_s = s[::-1]\n for i in range(n):\n if s[:n-i] == reverse_s[i:]:\n return reverse_s[:i] + s\n", "tests": ["assert shortest_palindrome(\"aacecaaa\") == \"aaacecaaa\"", "assert shortest_palindrome(\"abcd\") == \"dcbabcd\"", "assert shortest_palindrome(\"babab\") == \"babab\""], "broken": "def shortest_palindrome(s: str) -> str:\n if not s:\n return \"\"\n\n reverse_s = s[::-1]\n for i in range(len(s)):\n if s == reverse_s[i:]:\n return reverse_s[:i] + s\n\n return s", "error": " assert shortest_palindrome(\"aacecaaa\") == \"aaacecaaa\"\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError", "fixed": "def shortest_palindrome(s: str) -> str:\n \"\"\"\n Given a string s, you are to return the shortest palindrome that can be created by adding characters to the front of s.\n For example, if s is 'aacecaaa', the shortest palindrome is 'aaacecaaa', while if s is 'abcd', the shortest palindrome is 'dcbabcd'.\n\n Args:\n s (str): A string composed of lowercase letters only.\n\n Returns:\n str: The shortest palindrome that can be created by adding characters to the front of s.\n \"\"\"\n if not s:\n return \"\"\n \n # Finding the longest palindrome from the start\n n = len(s)\n reverse_s = s[::-1]\n for i in range(n):\n if s[:n-i] == reverse_s[i:]:\n return reverse_s[:i] + s"}
|
||||
{"signature": "def shortest_palindrome(s: str) -> str:\n \"\"\"\n Given a string s, you are to return the shortest palindrome that can be created by adding characters to the front of s.\n For example, if s is 'aacecaaa', the shortest palindrome is 'aaacecaaa', while if s is 'abcd', the shortest palindrome is 'dcbabcd'.\n\n Args:\n s (str): A string composed of lowercase letters only.\n\n Returns:\n str: The shortest palindrome that can be created by adding characters to the front of s.\n \"\"\"\n if not s:\n return \"\"\n \n # Finding the longest palindrome from the start\n n = len(s)\n reverse_s = s[::-1]\n for i in range(n):\n if s[:n-i] == reverse_s[i:]:\n return reverse_s[:i] + s\n", "tests": ["assert shortest_palindrome(\"aacecaaa\") == \"aaacecaaa\"", "assert shortest_palindrome(\"abcd\") == \"dcbabcd\"", "assert shortest_palindrome(\"babab\") == \"babab\""], "broken": "def shortest_palindrome(s: str) -> str:\n \"\"\"\n Given a string s, you are to return the shortest palindrome that can be created by adding characters to the front of s.\n For example, if s is 'aacecaaa', the shortest palindrome is 'aaacecaaa', while if s is 'abcd', the shortest palindrome is 'dcbabcd'.\n\n Args:\n s (str): A string composed of lowercase letters only.\n\n Returns:\n str: The shortest palindrome that can be created by adding characters to the front of s.\n \"\"\"\n if not s:\n return \"\"\n \n # Finding the longest palindrome from the start\n n = len(s)\n reverse_s = s[::-1]\n for i in range(n):\n if s[:n-i] == reverse_s[i:]:\n return reverse_s[:i] + s\n\nTests:\nassert shortest_palindrome(\"aacecaaa\") == \"aaacecaaa\"\nassert shortest_palindrome(\"abcd\") == \"dcbabcd\"\nassert shortest_palindrome(\"babab\") == \"babab\"", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def shortest_palindrome(s: str) -> str:\n \"\"\"\n Given a string s, you are to return the shortest palindrome that can be created by adding characters to the front of s.\n For example, if s is 'aacecaaa', the shortest palindrome is 'aaacecaaa', while if s is 'abcd', the shortest palindrome is 'dcbabcd'.\n\n Args:\n s (str): A string composed of lowercase letters only.\n\n Returns:\n str: The shortest palindrome that can be created by adding characters to the front of s.\n \"\"\"\n if not s:\n return \"\"\n \n # Finding the longest palindrome from the start\n n = len(s)\n reverse_s = s[::-1]\n for i in range(n):\n if s[:n-i] == reverse_s[i:]:\n return reverse_s[:i] + s"}
|
||||
{"signature": "def find_second_highest_unique(numbers: list) -> int:\n \"\"\"\n Given a list of integers that may contain duplicates, return the second highest unique integer.\n If there are less than two unique integers in the list, return None.\n \"\"\"\n unique_numbers = set(numbers)\n if len(unique_numbers) < 2:\n return None\n unique_numbers.remove(max(unique_numbers))\n return max(unique_numbers)\n\n# tests", "tests": ["assert find_second_highest_unique([1, 2, 3, 4, 5]) == 4", "assert find_second_highest_unique([1, 2, 2, 3, 3]) == 2", "assert find_second_highest_unique([1]) == None"], "broken": "block. Output only the function implementation in one", "error": " block. Output only the function implementation in one\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def find_second_highest_unique(numbers: list) -> int:\n \"\"\"\n Given a list of integers that may contain duplicates, return the second highest unique integer.\n If there are less than two unique integers in the list, return None.\n \"\"\"\n unique_numbers = set(numbers)\n if len(unique_numbers) < 2:\n return None\n unique_numbers.remove(max(unique_numbers))\n return max(unique_numbers)\n\n# tests\n\nassert find_second_highest_unique([1, 2, 3, 4, 5]) == 4\nassert find_second_highest_unique([1, 2, 2, 3, 3]) == 2\nassert find_second_highest_unique([1]) == None"}
|
||||
{"signature": "def max_sum_subarray(nums):\n \"\"\"Given an array of integers, find the contiguous subarray with the largest sum and return its sum.\n\n The function should handle edge cases such as an empty array, an array with all negative numbers, and an array with duplicates.\n\n >>> max_sum_subarray([-2, 1, -3, 4, -1, 2, 1, -5, 4])\n 6\n >>> max_sum_subarray([-1, -2, -3, -4])\n -1\n >>> max_sum_subarray([0, 0, 0, 0, 0])\n 0\n \"\"\"\n\n if not nums:\n return 0\n\n max_sum = current_sum = nums[0]\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum\n\n# tests", "tests": ["assert max_sum_subarray([-2, 1, -3, 4, -1, 2, 1, -5, 4]) == 6", "assert max_sum_subarray([-1, -2, -3, -4]) == -1", "assert max_sum_subarray([0, 0, 0, 0, 0]) == 0"], "broken": "def max_sum_subarray(nums):\n \"\"\"Given an array of integers, find the contiguous subarray with the largest sum and return its sum.\n\n The function should handle edge cases such as an empty array, an array with all negative numbers, and an array with duplicates.\n\n >>> max_sum_subarray([-2, 1, -3, 4, -1, 2, 1, -5, 4])\n 6\n >>> max_sum_subarray([-1, -2, -3, -4])\n -1\n >>> max_sum_subarray([0, 0, 0, 0, 0])\n 0\n \"\"\"\n\n if not nums:\n return 0\n\n max_sum = current_sum = nums[0]\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum\n\nAssistant: def max_sum_subarray(nums):\n \"\"\"Given an array of integers, find the contiguous subarray with the largest sum and return its sum.\n\n The function should handle edge cases such as an empty array, an array with all negative numbers, and an array with duplicates.\n\n >>> max_sum_subarray([-2, 1, -3, 4, -1, 2, 1, -5, 4])\n 6\n >>> max_sum_subarray([-1, -2, -3, -4])\n -1\n >>> max_sum_subarray([0, 0, 0, 0, 0])\n 0\n \"\"\"\n\n if not nums:\n return 0\n\n max_sum = current_sum = nums[0]\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum", "error": " Assistant: def max_sum_subarray(nums):\n ^^^\nSyntaxError: invalid syntax", "fixed": "def max_sum_subarray(nums):\n if not nums:\n return 0\n\n max_sum = current_sum = nums[0]\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum"}
|
||||
{"signature": "def find_max_sum_subarray_with_negatives(nums: list) -> int:\n \"\"\"\n Find the maximum sum of a subarray in a given list of integers. This function must handle edge cases, including\n negative integers and empty lists. If the list is empty, return 0. If the list contains only negative numbers, return\n the maximum single element. Otherwise, find the maximum sum of a contiguous subarray.\n \"\"\"\n if not nums:\n return 0\n\n current_sum = max_sum = nums[0]\n\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum\n\n\n# tests", "tests": ["assert find_max_sum_subarray_with_negatives([1, -2, 3, 10, -4, 7, 2, -5]) == 18", "assert find_max_sum_subarray_with_negatives([-1, -2, -3, -4]) == -1", "assert find_max_sum_subarray_with_negatives([]) == 0"], "broken": "block only. Implement: def find_max_sum_subarray_with_negatives(nums: list) -> int:\n \"\"\"\n Find the maximum sum of a subarray in a given list of integers. This function must handle edge cases, including\n negative integers and empty lists. If the list is empty, return 0. If the list contains only negative numbers, return\n the maximum single element. Otherwise, find the maximum sum of a contiguous subarray.\n \"\"\"\n if not nums:\n return 0\n\n current_sum = max_sum = nums[0]\n\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum\n\n# tests\n\nTests:\nassert find_max_sum_subarray_with_negatives([1, -2, 3, 10, -4, 7, 2, -5]) == 18\nassert find_max_sum_subarray_with_negatives([-1, -2, -3, -4]) == -1\nassert find_max_sum_subarray_with_negatives([]) == 0", "error": " block only. Implement: def find_max_sum_subarray_with_negatives(nums: list) -> int:\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def find_max_sum_subarray_with_negatives(nums: list) -> int:\n \"\"\"\n Find the maximum sum of a subarray in a given list of integers. This function must handle edge cases, including\n negative integers and empty lists. If the list is empty, return 0. If the list contains only negative numbers, return\n the maximum single element. Otherwise, find the maximum sum of a contiguous subarray.\n \"\"\"\n if not nums:\n return 0\n\n current_sum = max_sum = nums[0]\n\n for num in nums[1:]:\n current_sum = max(num, current_sum + num)\n max_sum = max(max_sum, current_sum)\n\n return max_sum"}
|
||||
{"signature": "def longest_palindromic_substring(s: str) -> str:\n \"\"\"\n Given a string, find the longest palindromic substring in it. \n A palindrome is a string that reads the same backward as forward. \n The function should handle edge cases such as empty input, single-character input, and an input where no palindrome is found.\n\n For example:\n - longest_palindromic_substring('babad') == 'bab' or 'aba'\n - longest_palindromic_substring('cbbd') == 'bb'\n - longest_palindromic_substring('a') == 'a'\n - longest_palindromic_substring('') == ''\n \"\"\"\n n = len(s)\n if n == 0:\n return \"\"\n \n start = 0\n max_length = 1\n \n def expand_around_center(left: int, right: int) -> int:\n while left >= 0 and right < n and s[left] == s[right]:\n left -= 1\n right += 1\n return right - left - 1\n\n for i in range(n):\n length1 = expand_around_center(i, i)\n length2 = expand_around_center(i, i + 1)\n length_max = max(length1, length2)\n if length_max > max_length:\n max_length = length_max\n start = i - (length_max - 1) // 2\n \n return s[start:start + max_length]\n\n# tests", "tests": ["assert longest_palindromic_substring('babad') in ['bab', 'aba']", "assert longest_palindromic_substring('cbbd') == 'bb'", "assert longest_palindromic_substring('a') == 'a'", "assert longest_palindromic_substring('') == ''"], "broken": "def longest_palindromic_substring(s: str) -> str:\n \"\"\"\n Given a string, find the longest palindromic substring in it. \n A palindrome is a string that reads the same backward as forward. \n The function should handle edge cases such as empty input, single-character input, and an input where no palindrome is found.\n\n For example:\n - longest_palindromic_substring('babad') == 'bab' or 'aba'\n - longest_palindromic_substring('cbbd') == 'bb'\n - longest_palindromic_substring('a') == 'a'\n - longest_palindromic_substring('') == ''\n \"\"\"\n n = len(s)\n if n == 0:\n return \"\"\n \n start = 0\n max_length = 1", "error": " assert longest_palindromic_substring('babad') in ['bab', 'aba']\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError", "fixed": "def longest_palindromic_substring(s: str) -> str:\n \"\"\"\n Given a string, find the longest palindromic substring in it. \n A palindrome is a string that reads the same backward as forward. \n The function should handle edge cases such as empty input, single-character input, and an input where no palindrome is found.\n\n For example:\n - longest_palindromic_substring('babad') == 'bab' or 'aba'\n - longest_palindromic_substring('cbbd') == 'bb'\n - longest_palindromic_substring('a') == 'a'\n - longest_palindromic_substring('') == ''\n \"\"\"\n n = len(s)\n if n == 0:\n return \"\"\n \n start = 0\n max_length = 1\n \n def expand_around_center(left: int, right: int) -> int:\n while left >= 0 and right < n and s[left] == s[right]:\n left -= 1\n right += 1\n return right - left - 1\n\n for i in range(n):\n length1 = expand_around_center(i, i)\n length2 = expand_around_center(i, i + 1)\n length_max = max(length1, length2)\n if length_max > max_length:\n max_length = length_max\n start = i - (length_max - 1) // 2\n \n return s[start:start + max_length]"}
|
||||
{"signature": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests", "tests": ["assert longest_non_repeating_substring_length(\"abcabcbb\") == 3", "assert longest_non_repeating_substring_length(\"bbbbb\") == 1", "assert longest_non_repeating_substring_length(\"pwwkew\") == 3"], "broken": "Implement: def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\nTests:\nassert longest_non_repeating_substring_length(\"abcabcbb\") == 3\nassert longest_non_repeating_substring_length(\"bbbbb\") == 1\nassert longest_non_repeating_substring_length(\"pwwkew\") == 3", "error": " Implement: def longest_non_repeating_substring_length(s: str) -> int:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests\n\nassert longest_non_repeating_substring_length(\"abcabcbb\") == 3\nassert longest_non_repeating_substring_length(\"bbbbb\") == 1\nassert longest_non_repeating_substring_length(\"pwwkew\") == 3"}
|
||||
{"signature": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests", "tests": ["assert longest_non_repeating_substring_length(\"abcabcbb\") == 3", "assert longest_non_repeating_substring_length(\"bbbbb\") == 1", "assert longest_non_repeating_substring_length(\"pwwkew\") == 3"], "broken": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests\n\nTests:\nassert longest_non_repeating_substring_length(\"abcabcbb\") == 3\nassert longest_non_repeating_substring_length(\"bbbbb\") == 1\nassert longest_non_repeating_substring_length(\"pwwkew\") == 3", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests\n\nassert longest_non_repeating_substring_length(\"abcabcbb\") == 3\nassert longest_non_repeating_substring_length(\"bbbbb\") == 1\nassert longest_non_repeating_substring_length(\"pwwkew\") == 3"}
|
||||
{"signature": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests", "tests": ["assert longest_non_repeating_substring_length(\"abcabcbb\") == 3", "assert longest_non_repeating_substring_length(\"bbbbb\") == 1", "assert longest_non_repeating_substring_length(\"pwwkew\") == 3"], "broken": "def longest_non_repeating_substring_length(s: str) -> int:\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\nAssistant: def longest_non_repeating_substring_length(s: str) -> int:\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength", "error": " Assistant: def longest_non_repeating_substring_length(s: str) -> int:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def longest_non_repeating_substring_length(s: str) -> int:\n \"\"\"Given a string, find the length of the longest substring without repeating characters. For example, the longest substring without repeating characters for \"abcabcbb\" is \"abc\", so the answer would be 3.\n Handles edge cases like empty string, single character string, and string with all repeating characters.\"\"\"\n if not s:\n return 0\n start = maxLength = 0\n usedChar = {}\n \n for i in range(len(s)):\n if s[i] in usedChar and start <= usedChar[s[i]]:\n start = usedChar[s[i]] + 1\n else:\n maxLength = max(maxLength, i - start + 1)\n \n usedChar[s[i]] = i\n return maxLength\n\n# tests\n\nassert longest_non_repeating_substring_length(\"abcabcbb\") == 3\nassert longest_non_repeating_substring_length(\"bbbbb\") == 1\nassert longest_non_repeating_substring_length(\"pwwkew\") == 3"}
|
||||
{"signature": "def min_swaps_to_sort(arr: list):\n \"\"\"Given an unordered array consisting of consecutive integers \n `[1, 2, 3, ..., n]` without any duplicates, \n determine the minimum number of swaps required to sort the array.\n\n - arr: List of integers.\n - Returns the minimum number of swaps required to sort the array.\n\n Note:\n - The input array will contain all integers from 1 to n, \n where n is the length of the array, without any duplicates.\n - There can be negative numbers in the array as well.\n - The array can be empty.\n - Duplicate numbers are not allowed, however, the problem statement is revised to include negatives and boundary conditions.\n\n \"\"\"\n n = len(arr)\n sorted_arr = sorted(arr)\n visit = {x: False for x in range(n)}\n swap_count = 0\n for i in range(n):\n if not visit[i] and arr[i] != sorted_arr[i]:\n cycle_count = 0\n j = i\n while not visit[j]:\n visit[j] = True\n j = arr.index(sorted_arr[j])\n cycle_count += 1\n swap_count += cycle_count - 1\n\n return swap_count\n", "tests": ["assert min_swaps_to_sort([1, 5, 4, 3, 2]) == 2", "assert min_swaps_to_sort([1, 3, 5, 2, 4, 6, 7]) == 3", "assert min_swaps_to_sort([5, 1, 3, 2]) == 2", "assert min_swaps_to_sort([]) == 0 # Edge case: empty array", "assert min_swaps_to_sort([-1, 3, -5, 2, 4]) == 3 # Edge case: negative numbers in the array", "assert min_swaps_to_sort([1, 2, 3, 4, 5]) == 0 # Edge case: already sorted array", "assert min_swaps_to_sort([2, 1, 3, 4, 5]) == 1 # Edge case: one element out of place"], "broken": "def min_swaps_to_sort(arr: list):\n n = len(arr)\n sorted_arr = sorted(arr)\n visit = {x: False for x in range(n)}\n swap_count = 0\n for i in range(n):\n if not visit[i] and arr[i] != sorted_arr[i]:\n cycle_count = 0\n j = i\n while not visit[j]:\n visit[j] = True\n j = arr.index(sorted_arr[j])\n cycle_count += 1\n swap_count += cycle_count - 1\n\n return swap_count\n\nTests:\nassert min_swaps_to_sort([1, 5, 4, 3, 2]) == 2\nassert min_swaps_to_sort([1, 3, 5, 2, 4, 6, 7]) == 3\nassert min_swaps_to_sort([5, 1, 3, 2]) == 2\nassert min_swaps_to_sort([]) == 0 # Edge case: empty array\nassert min_swaps_to_sort([-1, 3, -5, 2, 4]) == 3 # Edge case: negative numbers in the array\nassert min_swaps_to_sort([1, 2, 3, 4, 5]) == 0 # Edge case: already sorted array\nassert min_swaps_to_sort([2, 1, 3, 4, 5]) == 1 # Edge case: one element out of place", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "from typing import List\n\ndef min_swaps_to_sort(arr: List[int]) -> int:\n n = len(arr)\n sorted_arr = sorted(arr)\n visit = {x: False for x in range(n)}\n swap_count = 0\n for i in range(n):\n if not visit[i] and arr[i] != sorted_arr[i]:\n cycle_count = 0\n j = i\n while not visit[j]:\n visit[j] = True\n j = arr.index(sorted_arr[j])\n cycle_count += 1\n swap_count += cycle_count - 1\n\n return swap_count"}
|
||||
{"signature": "def min_swaps_to_sort(arr: list):\n \"\"\"Given an unordered array consisting of consecutive integers \n `[1, 2, 3, ..., n]` without any duplicates, \n determine the minimum number of swaps required to sort the array.\n\n - arr: List of integers.\n - Returns the minimum number of swaps required to sort the array.\n\n Note:\n - The input array will contain all integers from 1 to n, \n where n is the length of the array, without any duplicates.\n - There can be negative numbers in the array as well.\n - The array can be empty.\n - Duplicate numbers are not allowed, however, the problem statement is revised to include negatives and boundary conditions.\n\n \"\"\"\n n = len(arr)\n sorted_arr = sorted(arr)\n visit = {x: False for x in range(n)}\n swap_count = 0\n for i in range(n):\n if not visit[i] and arr[i] != sorted_arr[i]:\n cycle_count = 0\n j = i\n while not visit[j]:\n visit[j] = True\n j = arr.index(sorted_arr[j])\n cycle_count += 1\n swap_count += cycle_count - 1\n\n return swap_count\n", "tests": ["assert min_swaps_to_sort([1, 5, 4, 3, 2]) == 2", "assert min_swaps_to_sort([1, 3, 5, 2, 4, 6, 7]) == 3", "assert min_swaps_to_sort([5, 1, 3, 2]) == 2", "assert min_swaps_to_sort([]) == 0 # Edge case: empty array", "assert min_swaps_to_sort([-1, 3, -5, 2, 4]) == 3 # Edge case: negative numbers in the array", "assert min_swaps_to_sort([1, 2, 3, 4, 5]) == 0 # Edge case: already sorted array", "assert min_swaps_to_sort([2, 1, 3, 4, 5]) == 1 # Edge case: one element out of place"], "broken": "block only. Here's the implementation of the `min_swaps_to_sort` function:", "error": " block only. Here's the implementation of the `min_swaps_to_sort` function:\n ^\nSyntaxError: unterminated string literal (detected at line 1)", "fixed": "from typing import List\n\ndef min_swaps_to_sort(arr: List[int]) -> int:\n n = len(arr)\n sorted_arr = sorted(arr)\n visit = {x: False for x in range(n)}\n swap_count = 0\n for i in range(n):\n if not visit[i] and arr[i] != sorted_arr[i]:\n cycle_count = 0\n j = i\n while not visit[j]:\n visit[j] = True\n j = arr.index(sorted_arr[j])\n cycle_count += 1\n swap_count += cycle_count - 1\n\n return swap_count"}
|
||||
{"signature": "def find_longest_palindromic_subsequence(s: str) -> int:\n \"\"\"\n Write a function that finds the length of the longest palindromic subsequence in a given string.\n\n The subsequence does not need to be contiguous, but it should be the same forwards and backwards.\n\n Examples:\n >>> find_longest_palindromic_subsequence(\"bbbab\") == 4\n >>> find_longest_palindromic_subsequence(\"cbbd\") == 2\n >>> find_longest_palindromic_subsequence(\"abcba\") == 5\n >>> find_longest_palindromic_subsequence(\"\") == 0\n >>> find_longest_palindromic_subsequence(\"a\") == 1\n \"\"\"\n\n if not s:\n return 0\n\n n = len(s)\n dp = [[0] * n for _ in range(n)]\n\n for i in range(n - 1, -1, -1):\n dp[i][i] = 1\n for j in range(i + 1, n):\n if s[i] == s[j]:\n dp[i][j] = dp[i + 1][j - 1] + 2\n else:\n dp[i][j] = max(dp[i + 1][j], dp[i][j - 1])\n\n return dp[0][n - 1]\n\n\n# Tests", "tests": ["assert find_longest_palindromic_subsequence(\"bbbab\") == 4", "assert find_longest_palindromic_subsequence(\"cbbd\") == 2", "assert find_longest_palindromic_subsequence(\"abcba\") == 5"], "broken": "You can then call this function with any string to find the length of the longest palindromic subsequence in that string.", "error": " You can then call this function with any string to find the length of the longest palindromic subsequence in that string.\n ^^^\nSyntaxError: invalid syntax", "fixed": "def find_longest_palindromic_subsequence(s: str) -> int:\n \"\"\"\n Write a function that finds the length of the longest palindromic subsequence in a given string.\n\n The subsequence does not need to be contiguous, but it should be the same forwards and backwards.\n\n Examples:\n >>> find_longest_palindromic_subsequence(\"bbbab\") == 4\n >>> find_longest_palindromic_subsequence(\"cbbd\") == 2\n >>> find_longest_palindromic_subsequence(\"abcba\") == 5\n >>> find_longest_palindromic_subsequence(\"\") == 0\n >>> find_longest_palindromic_subsequence(\"a\") == 1\n \"\"\"\n\n if not s:\n return 0\n\n n = len(s)\n dp = [[0] * n for _ in range(n)]\n\n for i in range(n - 1, -1, -1):\n dp[i][i] = 1\n for j in range(i + 1, n):\n if s[i] == s[j]:\n dp[i][j] = dp[i + 1][j - 1] + 2\n else:\n dp[i][j] = max(dp[i + 1][j], dp[i][j - 1])\n\n return dp[0][n - 1]"}
|
||||
{"signature": "def find_longest_palindromic_subsequence(s: str) -> int:\n \"\"\"\n Write a function that finds the length of the longest palindromic subsequence in a given string.\n\n The subsequence does not need to be contiguous, but it should be the same forwards and backwards.\n\n Examples:\n >>> find_longest_palindromic_subsequence(\"bbbab\") == 4\n >>> find_longest_palindromic_subsequence(\"cbbd\") == 2\n >>> find_longest_palindromic_subsequence(\"abcba\") == 5\n >>> find_longest_palindromic_subsequence(\"\") == 0\n >>> find_longest_palindromic_subsequence(\"a\") == 1\n \"\"\"\n\n if not s:\n return 0\n\n n = len(s)\n dp = [[0] * n for _ in range(n)]\n\n for i in range(n - 1, -1, -1):\n dp[i][i] = 1\n for j in range(i + 1, n):\n if s[i] == s[j]:\n dp[i][j] = dp[i + 1][j - 1] + 2\n else:\n dp[i][j] = max(dp[i + 1][j], dp[i][j - 1])\n\n return dp[0][n - 1]\n\n\n# Tests", "tests": ["assert find_longest_palindromic_subsequence(\"bbbab\") == 4", "assert find_longest_palindromic_subsequence(\"cbbd\") == 2", "assert find_longest_palindromic_subsequence(\"abcba\") == 5"], "broken": "Implement: def find_longest_palindromic_subsequence(s: str) -> int:\n \"\"\"\n Write a function that finds the length of the longest palindromic subsequence in a given string.\n\n The subsequence does not need to be contiguous, but it should be the same forwards and backwards.\n\n Examples:\n >>> find_longest_palindromic_subsequence(\"bbbab\") == 4\n >>> find_longest_palindromic_subsequence(\"cbbd\") == 2\n >>> find_longest_palindromic_subsequence(\"abcba\") == 5\n >>> find_longest_palindromic_subsequence(\"\") == 0\n >>> find_longest_palindromic_subsequence(\"a\") == 1\n \"\"\"\n\n if not s:\n return 0\n\n n = len(s)\n dp = [[0] * n for _ in range(n)]\n\n for i in range(n - 1, -1, -1):\n dp[i][i] = 1\n for j in range(i + 1, n):\n if s[i] == s[j]:\n dp[i][j] = dp[i + 1][j - 1] + 2\n else:\n dp[i][j] = max(dp[i + 1][j], dp[i][j - 1])\n\n return dp[0][n - 1]", "error": " Implement: def find_longest_palindromic_subsequence(s: str) -> int:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def find_longest_palindromic_subsequence(s: str) -> int:\n \"\"\"\n Write a function that finds the length of the longest palindromic subsequence in a given string.\n\n The subsequence does not need to be contiguous, but it should be the same forwards and backwards.\n\n Examples:\n >>> find_longest_palindromic_subsequence(\"bbbab\") == 4\n >>> find_longest_palindromic_subsequence(\"cbbd\") == 2\n >>> find_longest_palindromic_subsequence(\"abcba\") == 5\n >>> find_longest_palindromic_subsequence(\"\") == 0\n >>> find_longest_palindromic_subsequence(\"a\") == 1\n \"\"\"\n\n if not s:\n return 0\n\n n = len(s)\n dp = [[0] * n for _ in range(n)]\n\n for i in range(n - 1, -1, -1):\n dp[i][i] = 1\n for j in range(i + 1, n):\n if s[i] == s[j]:\n dp[i][j] = dp[i + 1][j - 1] + 2\n else:\n dp[i][j] = max(dp[i + 1][j], dp[i][j - 1])\n\n return dp[0][n - 1]"}
|
||||
{"signature": "def max_product_subarray(nums):\n \"\"\"\n Given an integer array nums, find a contiguous subarray (containing at least one number) which has the largest product and return its product.\n\n The input array nums can have both positive and negative integers. It's guaranteed that the product of any contiguous subarray is not zero.\n\n Return the maximum product of a contiguous subarray.\n\n Examples:\n max_product_subarray([2, 3, -2, 4]) should return 6\n max_product_subarray([-2, 0, -1]) should return 0\n max_product_subarray([-2, 3, -4]) should return 24\n \"\"\"\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n max_product, min_product = max(num, max_product * num, min_product * num), min(num, max_product * num, min_product * num)\n result = max(result, max_product)\n return result\n\n# tests", "tests": ["assert max_product_subarray([2, 3, -2, 4]) == 6", "assert max_product_subarray([-2, 0, -1]) == 0", "assert max_product_subarray([-2, 3, -4]) == 24"], "broken": "block.ive\nAssistant\nImplement: def max_product_subarray(nums):\n \"\"\"\n Given an integer array nums, find a contiguous subarray (containing at least one number) which has the largest product and return its product.\n\n The input array nums can have both positive and negative integers. It's guaranteed that the product of any contiguous subarray is not zero.\n\n Return the maximum product of a contiguous subarray.\n\n Examples:\n max_product_subarray([2, 3, -2, 4]) should return 6\n max_product_subarray([-2, 0, -1]) should return 0\n max_product_subarray([-2, 3, -4]) should return 24\n \"\"\"\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n max_product, min_product = max(num, max_product * num, min_product * num), min(num, max_product * num, min_product * num)\n result = max(result, max_product)", "error": " Implement: def max_product_subarray(nums):\n ^^^\nSyntaxError: invalid syntax", "fixed": "def max_product_subarray(nums):\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n max_product, min_product = max(num, max_product * num, min_product * num), min(num, max_product * num, min_product * num)\n result = max(result, max_product)\n return result"}
|
||||
{"signature": "def max_product_subarray(nums):\n \"\"\"\n Given an integer array nums, find a contiguous subarray (containing at least one number) which has the largest product and return its product.\n\n The input array nums can have both positive and negative integers. It's guaranteed that the product of any contiguous subarray is not zero.\n\n Return the maximum product of a contiguous subarray.\n\n Examples:\n max_product_subarray([2, 3, -2, 4]) should return 6\n max_product_subarray([-2, 0, -1]) should return 0\n max_product_subarray([-2, 3, -4]) should return 24\n \"\"\"\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n max_product, min_product = max(num, max_product * num, min_product * num), min(num, max_product * num, min_product * num)\n result = max(result, max_product)\n return result\n\n# tests", "tests": ["assert max_product_subarray([2, 3, -2, 4]) == 6", "assert max_product_subarray([-2, 0, -1]) == 0", "assert max_product_subarray([-2, 3, -4]) == 24"], "broken": "def max_product_subarray(nums):\n \"\"\"\n Given an integer array nums, find a contiguous subarray (containing at least one number) which has the largest product and return its product.\n\n The input array nums can have both positive and negative integers. It's guaranteed that the product of any contiguous subarray is not zero.\n\n Return the maximum product of a contiguous subarray.\n\n Examples:\n max_product_subarray([2, 3, -2, 4]) should return 6\n max_product_subarray([-2, 0, -1]) should return 0\n max_product_subarray([-2, 3, -4]) should return 24\n \"\"\"\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n max_product, min_product = max(num, max_product * num, min_product * num), min(num, max_product * num, min_product * num)\n result = max(result, max_product)\n return result\n\n# tests\n\nTests:\nassert max_product_subarray([2, 3, -2, 4]) == 6\nassert max_product_subarray([-2, 0, -1]) == 0\nassert max_product_subarray([-2, 3, -4]) == 24", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def max_product_subarray(nums):\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n max_product, min_product = max(num, max_product * num, min_product * num), min(num, max_product * num, min_product * num)\n result = max(result, max_product)\n return result"}
|
||||
{"signature": "def find_largest_sum(arr: list, k: int) -> int:\n \"\"\"Given an array of integers 'arr', and an integer 'k', find the largest sum\n that can be obtained from any k contiguous elements of arr. Handles edge cases such as \n empty array, k > length of array, or k == 1. Consider sorting and hash maps for an efficient solution.\n \"\"\"\n arr.sort() # Sort the array\n largest_sum = sum(arr[-k:]) # Sum the k largest numbers\n return largest_sum\n\n# tests", "tests": ["assert find_largest_sum([1, 3, -2, 5, 4, -1], 2) == 9", "assert find_largest_sum([-3, -1, -2], 1) == -1", "assert find_largest_sum([1, 1, 1, 1, 1, 1], 5) == 5"], "broken": "def find_largest_sum(arr: list, k: int) -> int:\n \"\"\"Given an array of integers 'arr', and an integer 'k', find the largest sum\n that can be obtained from any k contiguous elements of arr. Handles edge cases such as \n empty array, k > length of array, or k == 1. Consider sorting and hash maps for an efficient solution.\n \"\"\"\n arr.sort() # Sort the array\n largest_sum = sum(arr[-k:]) # Sum the k largest numbers\n return largest_sum\n\n# tests\n\nTests:\nassert find_largest_sum([1, 3, -2, 5, 4, -1], 2) == 9\nassert find_largest_sum([-3, -1, -2], 1) == -1\nassert find_largest_sum([1, 1, 1, 1, 1, 1], 5) == 5", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def find_largest_sum(arr: list, k: int) -> int:\n arr.sort()\n return sum(arr[-k:])"}
|
||||
{"signature": "def find_largest_even_subarray(arr: list, k: int) -> int:\n \"\"\"\n Given an array of integers `arr` and an integer `k`, find the largest even number in the subarray formed by keeping at most `k` elements from the beginning and at most `k` elements from the end of the array. Return the largest even number or -1 if none exists. Consider edge cases like empty arrays or no even numbers.\n\n For example:\n find_largest_even_subarray([10, 20, 30, 5, 4], 1) -> 30\n find_largest_even_subarray([], 0) -> -1\n find_largest_even_subarray([5, 3, 1], 2) -> -1\n \"\"\"\n max_even = -1\n for i in range(k + 1):\n for j in range(k + 1):\n start = max(0, i)\n end = min(len(arr) - 1, len(arr) - j - 1)\n for num in arr[start:end+1]:\n if num % 2 == 0 and num > max_even:\n max_even = num\n return max_even\n\n# tests", "tests": ["assert find_largest_even_subarray([10, 20, 30, 5, 4], 1) == 30", "assert find_largest_even_subarray([], 0) == -1", "assert find_largest_even_subarray([5, 3, 1], 2) == -1"], "broken": "def find_largest_even_subarray(arr: list, k: int) -> int:\n \"\"\"\n Given an array of integers `arr` and an integer `k`, find the largest even number in the subarray formed by keeping at most `k` elements from the beginning and at most `k` elements from the end of the array. Return the largest even number or -1 if none exists. Consider edge cases like empty arrays or no even numbers.\n\n For example:\n find_largest_even_subarray([10, 20, 30, 5, 4], 1) -> 30\n find_largest_even_subarray([], 0) -> -1\n find_largest_even_subarray([5, 3, 1], 2) -> -1\n \"\"\"\n max_even = -1\n for i in range(k + 1):\n for j in range(k + 1):\n start = max(0, i)\n end = min(len(arr) - 1, len(arr) - j - 1)\n for num in arr[start:end+1]:\n if num % 2 == 0 and num > max_even:\n max_even = num\n return max_even\n\n# tests\n\nTests:\nassert find_largest_even_subarray([10, 20, 30, 5, 4], 1) == 30\nassert find_largest_even_subarray([], 0) == -1\nassert find_largest_even_subarray([5, 3, 1], 2) == -1", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def find_largest_even_subarray(arr: list, k: int) -> int:\n \"\"\"\n Given an array of integers `arr` and an integer `k`, find the largest even number in the subarray formed by keeping at most `k` elements from the beginning and at most `k` elements from the end of the array. Return the largest even number or -1 if none exists. Consider edge cases like empty arrays or no even numbers.\n\n For example:\n find_largest_even_subarray([10, 20, 30, 5, 4], 1) -> 30\n find_largest_even_subarray([], 0) -> -1\n find_largest_even_subarray([5, 3, 1], 2) -> -1\n \"\"\"\n max_even = -1\n for i in range(k + 1):\n for j in range(k + 1):\n start = max(0, i)\n end = min(len(arr) - 1, len(arr) - j - 1)\n for num in arr[start:end+1]:\n if num % 2 == 0 and num > max_even:\n max_even = num\n return max_even"}
|
||||
{"signature": "def find_max_product_subarray(nums: list) -> int:\n \"\"\"\n Given an integer array nums, find a contiguous non-empty subarray within the array that has the largest product, and return the product.\n The answer should be non-negative.\n \"\"\"\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n if num < 0:\n max_product, min_product = min_product, max_product\n max_product = max(num, max_product * num)\n min_product = min(num, min_product * num)\n result = max(result, max_product)\n return result\n\n# tests", "tests": ["assert find_max_product_subarray([2,3,-2,4]) == 6", "assert find_max_product_subarray([-2,0,-1]) == 0", "assert find_max_product_subarray([-2]) == -2"], "broken": "block.ive\nThe Python function implementation for finding the maximum product of a contiguous subarray in an integer array is as follows:", "error": " The Python function implementation for finding the maximum product of a contiguous subarray in an integer array is as follows:\n ^^^^^^\nSyntaxError: invalid syntax", "fixed": "def find_max_product_subarray(nums: list) -> int:\n \"\"\"\n Given an integer array nums, find a contiguous non-empty subarray within the array that has the largest product, and return the product.\n The answer should be non-negative.\n \"\"\"\n max_product = min_product = result = nums[0]\n for num in nums[1:]:\n if num < 0:\n max_product, min_product = min_product, max_product\n max_product = max(num, max_product * num)\n min_product = min(num, min_product * num)\n result = max(result, max_product)\n return result"}
|
||||
{"signature": "def find_substring_indices(string, substring):\n \"\"\"\n Given a string and a substring, return a list of indices where the substring is found within the string. \n For example, given string='mississippi' and substring='iss', the function should return [1, 4].\n \"\"\"\n indices = []\n index = string.find(substring)\n while index != -1:\n indices.append(index)\n index = string.find(substring, index + 1)\n return indices\n\n# tests", "tests": ["assert find_substring_indices('mississippi', 'iss') == [1, 4]", "assert find_substring_indices('mississippi', 'm') == [0]", "assert find_substring_indices('', 'a') == []"], "broken": "block. No text. No explanation. No comment. No question. No blank lines.\ndef find_substring_indices(string, substring):\n indices = []\n index = string.find(substring)\n while index != -1:\n indices.append(index)\n index = string.find(substring, index + 1)\n return indices", "error": " block. No text. No explanation. No comment. No question. No blank lines.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def find_substring_indices(string, substring):\n indices = []\n index = string.find(substring)\n while index != -1:\n indices.append(index)\n index = string.find(substring, index + 1)\n return indices"}
|
||||
{"signature": "def max_sum_subarray_length(nums: list, k: int):\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length\n\n# tests", "tests": ["assert max_sum_subarray_length([1, -1, 5, -2, 3], 3) == 4", "assert max_sum_subarray_length([-2, -1, 2, 1], 1) == 2", "assert max_sum_subarray_length([2, 1, -3, 4, 1], 10) == 0"], "broken": "Implement: def max_sum_subarray_length(nums: list, k: int):\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length\n\n# tests\n\nTests:\nassert max_sum_subarray_length([1, -1, 5, -2, 3], 3) == 4\nassert max_sum_subarray_length([-2, -1, 2, 1], 1) == 2\nassert max_sum_subarray_length([2, 1, -3, 4, 1], 10) == 0", "error": " Implement: def max_sum_subarray_length(nums: list, k: int):\n ^^^\nSyntaxError: invalid syntax", "fixed": "def max_sum_subarray_length(nums: list, k: int) -> int:\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length"}
|
||||
{"signature": "def max_sum_subarray_length(nums: list, k: int):\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length\n\n# tests", "tests": ["assert max_sum_subarray_length([1, -1, 5, -2, 3], 3) == 4", "assert max_sum_subarray_length([-2, -1, 2, 1], 1) == 2", "assert max_sum_subarray_length([2, 1, -3, 4, 1], 10) == 0"], "broken": "Implement: def max_sum_subarray_length(nums: list, k: int):\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length\ndef max_sum_subarray_length(nums: list, k: int):\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length", "error": " Implement: def max_sum_subarray_length(nums: list, k: int):\n ^^^\nSyntaxError: invalid syntax", "fixed": "def max_sum_subarray_length(nums: list, k: int) -> int:\n \"\"\"Given an array of integers, return the length of the longest subarray with sum equal to k.\n Return 0 if no such subarray exists.\n \"\"\"\n cumulative_sum = {0: -1} # Initialize the cumulative sum with 0 at index -1 for edge cases\n max_length = 0\n current_sum = 0\n \n for i, num in enumerate(nums):\n current_sum += num\n if (current_sum - k) in cumulative_sum:\n max_length = max(max_length, i - cumulative_sum[current_sum - k])\n if current_sum not in cumulative_sum:\n cumulative_sum[current_sum] = i\n\n return max_length"}
|
||||
{"signature": "def find_longest_palindromic_substring(s: str) -> str:\n \"\"\"Given a string s, return the longest palindromic substring in s. Handle edge cases such as empty strings, single character strings, and the entire string being a palindrome.\"\"\"\n if len(s) == 0:\n return \"\"\n longest = s[0]\n for i in range(len(s)):\n for j in range(i+1, len(s)+1):\n substring = s[i:j]\n if substring == substring[::-1] and len(substring) > len(longest):\n longest = substring\n return longest\n\n# tests", "tests": ["assert find_longest_palindromic_substring(\"babad\") == \"bab\" or \"aba\"", "assert find_longest_palindromic_substring(\"cbbd\") == \"bb\"", "assert find_longest_palindromic_substring(\"\") == \"\""], "broken": "block.ve\nveyou\nImplement: def find_longest_palindromic_substring(s: str) -> str:\n \"\"\"Given a string s, return the longest palindromic substring in s. Handle edge cases such as empty strings, single character strings, and the entire string being a palindrome.\"\"\"\n if len(s) == 0:\n return \"\"\n longest = s[0]\n for i in range(len(s)):\n for j in range(i+1, len(s)+1):\n substring = s[i:j]\n if substring == substring[::-1] and len(substring) > len(longest):\n longest = substring\n return longest\n\n# tests\n\nTests:\nassert find_longest_palindromic_substring(\"babad\") == \"bab\" or \"aba\"\nassert find_longest_palindromic_substring(\"cbbd\") == \"bb\"\nassert find_longest_palindromic_substring(\"\") == \"\"\n\nOutput only the function implementation in one", "error": " Implement: def find_longest_palindromic_substring(s: str) -> str:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def find_longest_palindromic_substring(s: str) -> str:\n \"\"\"Given a string s, return the longest palindromic substring in s. Handle edge cases such as empty strings, single character strings, and the entire string being a palindrome.\"\"\"\n if len(s) == 0:\n return \"\"\n longest = s[0]\n for i in range(len(s)):\n for j in range(i+1, len(s)+1):\n substring = s[i:j]\n if substring == substring[::-1] and len(substring) > len(longest):\n longest = substring\n return longest"}
|
||||
{"signature": "def longest_non_repeating_substring(s: str) -> str:\n \"\"\"Return the longest substring without repeating characters in the input string 's'.\n If there are multiple substrings of the same maximum length, return the first one encountered.\n \"\"\"\n longest = ''\n seen = {}\n start = 0\n\n for i, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n if (i - start + 1) > len(longest):\n longest = s[start:i+1]\n seen[char] = i\n\n return longest\n\n# tests", "tests": ["assert longest_non_repeating_substring(\"abcabcbb\") == \"abc\"", "assert longest_non_repeating_substring(\"bbbbb\") == \"b\"", "assert longest_non_repeating_substring(\"pwwkew\") == \"wke\""], "broken": "Implement: def longest_non_repeating_substring(s: str) -> str:\n \"\"\"Return the longest substring without repeating characters in the input string 's'.\n If there are multiple substrings of the same maximum length, return the first one encountered.\n \"\"\"\n longest = ''\n seen = {}\n start = 0\n\n for i, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n if (i - start + 1) > len(longest):\n longest = s[start:i+1]\n seen[char] = i\n\n return longest\n\n# tests\n\nTests:\nassert longest_non_repeating_substring(\"abcabcbb\") == \"abc\"\nassert longest_non_repeating_substring(\"bbbbb\") == \"b\"\nassert longest_non_repeating_substring(\"pwwkew\") == \"wke\"", "error": " Implement: def longest_non_repeating_substring(s: str) -> str:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def longest_non_repeating_substring(s: str) -> str:\n longest = ''\n seen = {}\n start = 0\n\n for i, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n if (i - start + 1) > len(longest):\n longest = s[start:i+1]\n seen[char] = i\n\n return longest"}
|
||||
{"signature": "def longest_non_repeating_substring(s: str) -> str:\n \"\"\"Return the longest substring without repeating characters in the input string 's'.\n If there are multiple substrings of the same maximum length, return the first one encountered.\n \"\"\"\n longest = ''\n seen = {}\n start = 0\n\n for i, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n if (i - start + 1) > len(longest):\n longest = s[start:i+1]\n seen[char] = i\n\n return longest\n\n# tests", "tests": ["assert longest_non_repeating_substring(\"abcabcbb\") == \"abc\"", "assert longest_non_repeating_substring(\"bbbbb\") == \"b\"", "assert longest_non_repeating_substring(\"pwwkew\") == \"wke\""], "broken": "Implement: def longest_non_repeating_substring(s: str) -> str:\n \"\"\"Return the longest substring without repeating characters in the input string 's'.\n If there are multiple substrings of the same maximum length, return the first one encountered.\n \"\"\"\n longest = ''\n seen = {}\n start = 0\n\n for i, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n if (i - start + 1) > len(longest):\n longest = s[start:i+1]\n seen[char] = i\n\n return longest", "error": " Implement: def longest_non_repeating_substring(s: str) -> str:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def longest_non_repeating_substring(s: str) -> str:\n longest = ''\n seen = {}\n start = 0\n\n for i, char in enumerate(s):\n if char in seen and start <= seen[char]:\n start = seen[char] + 1\n else:\n if (i - start + 1) > len(longest):\n longest = s[start:i+1]\n seen[char] = i\n\n return longest"}
|
||||
{"signature": "def count_unique_char_permutations(s: str) -> int:\n \"\"\"\n Given a string s, find the number of unique permutations of the string that can be formed\n without repeating any characters.\n \n Note: The result should be computed modulo 10^9 + 7.\n \"\"\"\n MOD = 10**9 + 7\n fact = 1\n freq = {}\n for ch in s:\n freq[ch] = freq.get(ch, 0) + 1\n n = len(s)\n for i in range(1, n + 1):\n fact = (fact * i) % MOD\n unique_count = 1\n for k, v in freq.items():\n if v > 1:\n for j in range(2, v + 1):\n unique_count = (unique_count * j) % MOD\n return (fact * pow(unique_count, MOD - 2, MOD)) % MOD\n\n# tests", "tests": ["assert count_unique_char_permutations(\"abc\") == 6", "assert count_unique_char_permutations(\"aab\") == 3", "assert count_unique_char_permutations(\"aaaaaaaaaa\") == 1"], "broken": "block. # Implement: def count_unique_char_permutations(s: str) -> int:\n \"\"\"\n Given a string s, find the number of unique permutations of the string that can be formed\n without repeating any characters.\n \n Note: The result should be computed modulo 10^9 + 7.\n \"\"\"\n MOD = 10**9 + 7\n fact = 1\n freq = {}\n for ch in s:\n freq[ch] = freq.get(ch, 0) + 1\n n = len(s)\n for i in range(1, n + 1):\n fact = (fact * i) % MOD\n unique_count = 1\n for k, v in freq.items():\n if v > 1:\n for j in range(2, v + 1):\n unique_count = (unique_count * j) % MOD\n return (fact * pow(unique_count, MOD - 2, MOD)) % MOD\n\n# tests\n\nTests:\nassert count_unique_char_permutations(\"abc\") == 6\nassert count_unique_char_permutations(\"aab\") == 3", "error": " block. # Implement: def count_unique_char_permutations(s: str) -> int:\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: invalid syntax", "fixed": "def count_unique_char_permutations(s: str) -> int:\n \"\"\"\n Given a string s, find the number of unique permutations of the string that can be formed\n without repeating any characters.\n \n Note: The result should be computed modulo 10^9 + 7.\n \"\"\"\n MOD = 10**9 + 7\n fact = 1\n freq = {}\n for ch in s:\n freq[ch] = freq.get(ch, 0) + 1\n n = len(s)\n for i in range(1, n + 1):\n fact = (fact * i) % MOD\n unique_count = 1\n for k, v in freq.items():\n if v > 1:\n for j in range(2, v + 1):\n unique_count = (unique_count * j) % MOD\n return (fact * pow(unique_count, MOD - 2, MOD)) % MOD"}
|
||||
{"signature": "def count_unique_char_permutations(s: str) -> int:\n \"\"\"\n Given a string s, find the number of unique permutations of the string that can be formed\n without repeating any characters.\n \n Note: The result should be computed modulo 10^9 + 7.\n \"\"\"\n MOD = 10**9 + 7\n fact = 1\n freq = {}\n for ch in s:\n freq[ch] = freq.get(ch, 0) + 1\n n = len(s)\n for i in range(1, n + 1):\n fact = (fact * i) % MOD\n unique_count = 1\n for k, v in freq.items():\n if v > 1:\n for j in range(2, v + 1):\n unique_count = (unique_count * j) % MOD\n return (fact * pow(unique_count, MOD - 2, MOD)) % MOD\n\n# tests", "tests": ["assert count_unique_char_permutations(\"abc\") == 6", "assert count_unique_char_permutations(\"aab\") == 3", "assert count_unique_char_permutations(\"aaaaaaaaaa\") == 1"], "broken": "block only.\n\"\"\"", "error": " \"\"\"\n ^\nSyntaxError: unterminated triple-quoted string literal (detected at line 6)", "fixed": "def count_unique_char_permutations(s: str) -> int:\n \"\"\"\n Given a string s, find the number of unique permutations of the string that can be formed\n without repeating any characters.\n \n Note: The result should be computed modulo 10^9 + 7.\n \"\"\"\n MOD = 10**9 + 7\n fact = 1\n freq = {}\n for ch in s:\n freq[ch] = freq.get(ch, 0) + 1\n n = len(s)\n for i in range(1, n + 1):\n fact = (fact * i) % MOD\n unique_count = 1\n for k, v in freq.items():\n if v > 1:\n for j in range(2, v + 1):\n unique_count = (unique_count * j) % MOD\n return (fact * pow(unique_count, MOD - 2, MOD)) % MOD"}
|
||||
{"signature": "def max_profit_with_cooldown(prices):\n \"\"\"\n Given a list of daily stock prices, compute the maximum profit that can be made by buying and selling the stock\n once, with a cooldown of 1 day between transactions.\n Return the maximum profit possible. Assume that you cannot perform multiple transactions at once.\n\n Example:\n - For prices = [1, 2, 3, 0, 2], the output should be 3 (buy on day 1, sell on day 2, buy on day 4, sell on day 5).\n - For prices = [1, 2, 3, 0], the output should be 2 (buy on day 1, sell on day 3).\n \"\"\"\n if not prices:\n return 0\n\n n = len(prices)\n buy = [0] * n\n sell = [0] * n\n cooldown = [0] * n\n\n buy[0] = -prices[0]\n\n for i in range(1, n):\n buy[i] = max(buy[i - 1], cooldown[i - 1] - prices[i])\n sell[i] = max(sell[i - 1], buy[i - 1] + prices[i])\n cooldown[i] = max(cooldown[i - 1], sell[i - 1])\n\n return max(sell[-1], cooldown[-1])\n\n# tests", "tests": ["assert max_profit_with_cooldown([1, 2, 3, 0, 2]) == 3", "assert max_profit_with_cooldown([1, 2, 3, 0]) == 2", "assert max_profit_with_cooldown([1, 2, 2, 2, 1]) == 1"], "broken": "def max_profit_with_cooldown(prices):\n \"\"\"\n Given a list of daily stock prices, compute the maximum profit that can be made by buying and selling the stock\n once, with a cooldown of 1 day between transactions.\n Return the maximum profit possible. Assume that you cannot perform multiple transactions at once.\n\n Example:\n - For prices = [1, 2, 3, 0, 2], the output should be 3 (buy on day 1, sell on day 2, buy on day 4, sell on day 5).", "error": " \"\"\"\n ^\nSyntaxError: unterminated triple-quoted string literal (detected at line 12)", "fixed": "def max_profit_with_cooldown(prices):\n \"\"\"\n Given a list of daily stock prices, compute the maximum profit that can be made by buying and selling the stock\n once, with a cooldown of 1 day between transactions.\n Return the maximum profit possible. Assume that you cannot perform multiple transactions at once.\n\n Example:\n - For prices = [1, 2, 3, 0, 2], the output should be 3 (buy on day 1, sell on day 2, buy on day 4, sell on day 5).\n - For prices = [1, 2, 3, 0], the output should be 2 (buy on day 1, sell on day 3).\n \"\"\"\n if not prices:\n return 0\n\n n = len(prices)\n buy = [0] * n\n sell = [0] * n\n cooldown = [0] * n\n\n buy[0] = -prices[0]\n\n for i in range(1, n):\n buy[i] = max(buy[i - 1], cooldown[i - 1] - prices[i])\n sell[i] = max(sell[i - 1], buy[i - 1] + prices[i])\n cooldown[i] = max(cooldown[i - 1], sell[i - 1])\n\n return max(sell[-1], cooldown[-1])"}
|
||||
{"signature": "def find_longest_substring_with_k_distinct_characters(s: str, k: int) -> str:\n \"\"\"\n Given a string s and an integer k, return the length of the longest substring of s that contains at most k distinct characters.\n \"\"\"\n from collections import defaultdict\n \n if len(s) == 0 or k < 1:\n return \"\"\n \n char_freq = defaultdict(int)\n start = 0\n max_length = 0\n max_substring = \"\"\n \n for end in range(len(s)):\n char_freq[s[end]] += 1\n \n while len(char_freq) > k:\n char_freq[s[start]] -= 1\n if char_freq[s[start]] == 0:\n del char_freq[s[start]]\n start += 1\n \n if end - start + 1 > max_length:\n max_length = end - start + 1\n max_substring = s[start:end + 1]\n \n return max_substring\n\n# tests", "tests": ["assert find_longest_substring_with_k_distinct_characters(\"abcba\", 2) == \"bcb\"", "assert find_longest_substring_with_k_distinct_characters(\"abaccc\", 1) == \"ccc\"", "assert find_longest_substring_with_k_distinct_characters(\"\", 3) == \"\""], "broken": "Implement: def find_longest_substring_with_k_distinct_characters(s: str, k: int) -> str:\n \"\"\"\n Given a string s and an integer k, return the length of the longest substring of s that contains at most k distinct characters.\n \"\"\"\n from collections import defaultdict\n \n if len(s) == 0 or k < 1:\n return \"\"\n \n char_freq = defaultdict(int)\n start = 0\n max_length = 0\n max_substring = \"\"\n \n for end in range(len(s)):\n char_freq[s[end]] += 1\n \n while len(char_freq) > k:\n char_freq[s[start]] -= 1\n if char_freq[s[start]] == 0:\n del char_freq[s[start]]\n start += 1\n \n if end - start + 1 > max_length:\n max_length = end - start + 1\n max_substring = s[start:end + 1]\n \n return max_substring\n\n# tests\n\nTests:\nassert find_longest_substring_with_k_distinct_characters(\"abcba\", 2) == \"bcb\"\nassert find_longest_substring_with_k_distinct_characters(\"abaccc\", 1) == \"ccc\"\nassert find_longest_substring_with_k_distinct_characters(\"\", 3) == \"\"", "error": " Implement: def find_longest_substring_with_k_distinct_characters(s: str, k: int) -> str:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def find_longest_substring_with_k_distinct_characters(s: str, k: int) -> str:\n \"\"\"\n Given a string s and an integer k, return the length of the longest substring of s that contains at most k distinct characters.\n \"\"\"\n from collections import defaultdict\n\n if len(s) == 0 or k < 1:\n return \"\"\n\n char_freq = defaultdict(int)\n start = 0\n max_length = 0\n max_substring = \"\"\n\n for end in range(len(s)):\n char_freq[s[end]] += 1\n\n while len(char_freq) > k:\n char_freq[s[start]] -= 1\n if char_freq[s[start]] == 0:\n del char_freq[s[start]]\n start += 1\n\n if end - start + 1 > max_length:\n max_length = end - start + 1\n max_substring = s[start:end + 1]\n\n return max_substring"}
|
||||
{"signature": "def longest_increasing_subsequence(arr):\n \"\"\"\n Find the length of the longest strictly increasing subsequence in the given list.\n (Note: A subsequence is a sequence that can be derived from another sequence by deleting some or no elements\n without changing the order of the remaining elements.)\n\n Args:\n arr: A list of integers\n\n Returns:\n The length of the longest increasing subsequence.\n\n Assumptions:\n - The solution should use dynamic programming.\n\n Edge Cases:\n - An empty array will return 0.\n - An array with all negative elements will still look for the longest increasing subsequence.\n - The function should handle arrays with any combination of duplicates and unique numbers.\n \"\"\"\n if not arr:\n return 0\n\n dp = [1] * len(arr)\n for i in range(1, len(arr)):\n for j in range(i):\n if arr[i] > arr[j]:\n dp[i] = max(dp[i], dp[j] + 1)\n\n return max(dp)\n\n# tests", "tests": ["assert longest_increasing_subsequence([10, 9, 2, 5, 3, 7, 101, 18]) == 4", "assert longest_increasing_subsequence([]) == 0", "assert longest_increasing_subsequence([-1, -2, -3, -4]) == 1"], "broken": "block only.\ndef longest_increasing_subsequence(arr):\n \"\"\"\n Find the length of the longest strictly increasing subsequence in the given list.\n (Note: A subsequence is a sequence that can be derived from another sequence by deleting some or no elements\n without changing the order of the remaining elements.)\n\n Args:\n arr: A list of integers\n\n Returns:\n The length of the longest increasing subsequence.\n\n Assumptions:\n - The solution should use dynamic programming.\n\n Edge Cases:\n - An empty array will return 0.\n - An array with all negative elements will still look for the longest increasing subsequence.\n - The function should handle arrays with any combination of duplicates and unique numbers.\n \"\"\"\n if not arr:\n return 0\n\n dp = [1] * len(arr)\n for i in range(1, len(arr)):\n for j in range(i):\n if arr[i] > arr[j]:\n dp[i] = max(dp[i], dp[j] + 1)\n\n return max(dp)", "error": " block only.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def longest_increasing_subsequence(arr):\n \"\"\"\n Find the length of the longest strictly increasing subsequence in the given list.\n (Note: A subsequence is a sequence that can be derived from another sequence by deleting some or no elements\n without changing the order of the remaining elements.)\n\n Args:\n arr: A list of integers\n\n Returns:\n The length of the longest increasing subsequence.\n\n Assumptions:\n - The solution should use dynamic programming.\n\n Edge Cases:\n - An empty array will return 0.\n - An array with all negative elements will still look for the longest increasing subsequence.\n - The function should handle arrays with any combination of duplicates and unique numbers.\n \"\"\"\n if not arr:\n return 0\n\n dp = [1] * len(arr)\n for i in range(1, len(arr)):\n for j in range(i):\n if arr[i] > arr[j]:\n dp[i] = max(dp[i], dp[j] + 1)\n\n return max(dp)"}
|
||||
{"signature": "def longest_increasing_subsequence(arr):\n \"\"\"\n Find the length of the longest strictly increasing subsequence in the given list.\n (Note: A subsequence is a sequence that can be derived from another sequence by deleting some or no elements\n without changing the order of the remaining elements.)\n\n Args:\n arr: A list of integers\n\n Returns:\n The length of the longest increasing subsequence.\n\n Assumptions:\n - The solution should use dynamic programming.\n\n Edge Cases:\n - An empty array will return 0.\n - An array with all negative elements will still look for the longest increasing subsequence.\n - The function should handle arrays with any combination of duplicates and unique numbers.\n \"\"\"\n if not arr:\n return 0\n\n dp = [1] * len(arr)\n for i in range(1, len(arr)):\n for j in range(i):\n if arr[i] > arr[j]:\n dp[i] = max(dp[i], dp[j] + 1)\n\n return max(dp)\n\n# tests", "tests": ["assert longest_increasing_subsequence([10, 9, 2, 5, 3, 7, 101, 18]) == 4", "assert longest_increasing_subsequence([]) == 0", "assert longest_increasing_subsequence([-1, -2, -3, -4]) == 1"], "broken": "Implement the function def longest_increasing_subsequence(arr):\ndef longest_increasing_subsequence(arr):\n if not arr:\n return 0\n\n dp = [1] * len(arr)\n for i in range(1, len(arr)):\n for j in range(i):\n if arr[i] > arr[j]:\n dp[i] = max(dp[i], dp[j] + 1)\n\n return max(dp)", "error": " Implement the function def longest_increasing_subsequence(arr):\n ^^^\nSyntaxError: invalid syntax", "fixed": "def longest_increasing_subsequence(arr):\n \"\"\"\n Find the length of the longest strictly increasing subsequence in the given list.\n (Note: A subsequence is a sequence that can be derived from another sequence by deleting some or no elements\n without changing the order of the remaining elements.)\n\n Args:\n arr: A list of integers\n\n Returns:\n The length of the longest increasing subsequence.\n\n Assumptions:\n - The solution should use dynamic programming.\n\n Edge Cases:\n - An empty array will return 0.\n - An array with all negative elements will still look for the longest increasing subsequence.\n - The function should handle arrays with any combination of duplicates and unique numbers.\n \"\"\"\n if not arr:\n return 0\n\n dp = [1] * len(arr)\n for i in range(1, len(arr)):\n for j in range(i):\n if arr[i] > arr[j]:\n dp[i] = max(dp[i], dp[j] + 1)\n\n return max(dp)"}
|
||||
{"signature": "def longest_substring(s: str) -> int:\n \"\"\"\n Find the length of the longest substring without repeating characters.\n :param s: String to analyze.\n :return: Length of the longest substring.\n \"\"\"\n if len(s) < 2:\n return len(s)\n char_map = {}\n start = 0\n max_length = 0\n for i, char in enumerate(s):\n if char in char_map and start <= char_map[char]:\n start = char_map[char] + 1\n else:\n max_length = max(max_length, i - start + 1)\n char_map[char] = i\n return max_length\n\n# tests", "tests": ["assert longest_substring(\"abcabcbb\") == 3", "assert longest_substring(\"bbbbb\") == 1", "assert longest_substring(\"pwwkew\") == 3"], "broken": "longest_substring(s: str) -> int:", "error": " longest_substring(s: str) -> int:\n ^\nSyntaxError: invalid syntax", "fixed": "def longest_substring(s: str) -> int:\n \"\"\"\n Find the length of the longest substring without repeating characters.\n :param s: String to analyze.\n :return: Length of the longest substring.\n \"\"\"\n if len(s) < 2:\n return len(s)\n char_map = {}\n start = 0\n max_length = 0\n for i, char in enumerate(s):\n if char in char_map and start <= char_map[char]:\n start = char_map[char] + 1\n else:\n max_length = max(max_length, i - start + 1)\n char_map[char] = i\n return max_length"}
|
||||
{"signature": "def find_longest_unique_substring(s: str) -> str:\n \"\"\"\n Given a string, find the length of the longest substring without repeating characters.\n Return the substring itself.\n \"\"\"\n n = len(s)\n ans = \"\"\n ans_len = 0\n for i in range(n):\n # Using hash map to store characters in the current window\n chars_in_window = {}\n tmp = \"\"\n for j in range(i, n):\n if s[j] in chars_in_window:\n # If a repeating character is found, break the inner loop\n break\n # Add character to the window hash map\n chars_in_window[s[j]] = 1\n # Add character to the temporary substring\n tmp += s[j]\n # Check if the current substring is longer than the previously stored answer\n if len(tmp) > ans_len:\n ans = tmp\n ans_len = len(tmp)\n return ans\n\n# tests", "tests": ["assert find_longest_unique_substring(\"abcabcbb\") == \"abc\"", "assert find_longest_unique_substring(\"bbbbb\") == \"b\"", "assert find_longest_unique_substring(\"pwwkew\") == \"wke\""], "broken": "block. You are an expert Python coder. Output one", "error": " block. You are an expert Python coder. Output one\n ^^^\nSyntaxError: invalid syntax", "fixed": "def find_longest_unique_substring(s: str) -> str:\n n = len(s)\n ans = \"\"\n ans_len = 0\n for i in range(n):\n chars_in_window = {}\n tmp = \"\"\n for j in range(i, n):\n if s[j] in chars_in_window:\n break\n chars_in_window[s[j]] = 1\n tmp += s[j]\n if len(tmp) > ans_len:\n ans = tmp\n ans_len = len(tmp)\n return ans"}
|
||||
{"signature": "def find_longest_substring_with_k_distinct_chars(s, k):\n \"\"\"\n Given a string `s` and an integer `k`, find the length of the longest substring that contains at most `k` distinct characters.\n\n For example:\n - find_longest_substring_with_k_distinct_chars('eceba', 2) should return 3 ('ece' or 'eba')\n - find_longest_substring_with_k_distinct_chars('aa', 1) should return 2 ('aa')\n - find_longest_substring_with_k_distinct_chars('abaccc', 2) should return 4 ('bacc' or 'abac')\n \"\"\"\n left = 0\n max_length = 0\n char_count = {}\n \n for right, char in enumerate(s):\n char_count[char] = char_count.get(char, 0) + 1\n \n while len(char_count) > k:\n char_count[s[left]] -= 1\n if char_count[s[left]] == 0:\n del char_count[s[left]]\n left += 1\n \n max_length = max(max_length, right - left + 1)\n \n return max_length\n\n# tests", "tests": ["assert find_longest_substring_with_k_distinct_chars('eceba', 2) == 3", "assert find_longest_substring_with_k_distinct_chars('aa', 1) == 2", "assert find_longest_substring_with_k_distinct_chars('abaccc', 2) == 4"], "broken": "# Test the function\nassert find_longest_substring_with_k_distinct_chars('eceba', 2) == 3\nassert find_longest_substring_with_k_distinct_chars('aa', 1) == 2\nassert find_longest_substring_with_k_distinct_chars('abaccc', 2) == 4", "error": " assert find_longest_substring_with_k_distinct_chars('eceba', 2) == 3\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nNameError: name 'find_longest_substring_with_k_distinct_chars' is not defined", "fixed": "def find_longest_substring_with_k_distinct_chars(s, k):\n left = 0\n max_length = 0\n char_count = {}\n \n for right, char in enumerate(s):\n char_count[char] = char_count.get(char, 0) + 1\n \n while len(char_count) > k:\n char_count[s[left]] -= 1\n if char_count[s[left]] == 0:\n del char_count[s[left]]\n left += 1\n \n max_length = max(max_length, right - left + 1)\n \n return max_length"}
|
||||
{"signature": "def find_longest_substring_with_k_distinct_chars(s, k):\n \"\"\"\n Given a string `s` and an integer `k`, find the length of the longest substring that contains at most `k` distinct characters.\n\n For example:\n - find_longest_substring_with_k_distinct_chars('eceba', 2) should return 3 ('ece' or 'eba')\n - find_longest_substring_with_k_distinct_chars('aa', 1) should return 2 ('aa')\n - find_longest_substring_with_k_distinct_chars('abaccc', 2) should return 4 ('bacc' or 'abac')\n \"\"\"\n left = 0\n max_length = 0\n char_count = {}\n \n for right, char in enumerate(s):\n char_count[char] = char_count.get(char, 0) + 1\n \n while len(char_count) > k:\n char_count[s[left]] -= 1\n if char_count[s[left]] == 0:\n del char_count[s[left]]\n left += 1\n \n max_length = max(max_length, right - left + 1)\n \n return max_length\n\n# tests", "tests": ["assert find_longest_substring_with_k_distinct_chars('eceba', 2) == 3", "assert find_longest_substring_with_k_distinct_chars('aa', 1) == 2", "assert find_longest_substring_with_k_distinct_chars('abaccc', 2) == 4"], "broken": "This is the function that finds the length of the longest substring with at most k distinct characters in the input string.", "error": " This is the function that finds the length of the longest substring with at most k distinct characters in the input string.\n ^^^^^^^^\nSyntaxError: invalid syntax", "fixed": "def find_longest_substring_with_k_distinct_chars(s, k):\n left = 0\n max_length = 0\n char_count = {}\n \n for right, char in enumerate(s):\n char_count[char] = char_count.get(char, 0) + 1\n \n while len(char_count) > k:\n char_count[s[left]] -= 1\n if char_count[s[left]] == 0:\n del char_count[s[left]]\n left += 1\n \n max_length = max(max_length, right - left + 1)\n \n return max_length"}
|
||||
{"signature": "def longest_common_subsequence(s1: str, s2: str) -> int:\n \"\"\"\n Given two strings s1 and s2, find the length of the longest common subsequence.\n \n The longest common subsequence (LCS) problem is a classic computer science problem \n that is often solved using dynamic programming.\n \n For example:\n longest_common_subsequence(\"abcde\", \"ace\") -> 3\n longest_common_subsequence(\"abc\", \"def\") -> 0\n longest_common_subsequence(\"\", \"\") -> 0\n \"\"\"\n m, n = len(s1), len(s2)\n \n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if s1[i - 1] == s2[j - 1]:\n dp[i][j] = dp[i - 1][j - 1] + 1\n else:\n dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])\n \n return dp[m][n]\n\n# tests", "tests": ["assert longest_common_subsequence(\"abcde\", \"ace\") == 3", "assert longest_common_subsequence(\"abc\", \"def\") == 0", "assert longest_common_subsequence(\"\", \"\") == 0"], "broken": "Implement: def longest_common_subsequence(s1: str, s2: str) -> int:\n \"\"\"\n Given two strings s1 and s2, find the length of the longest common subsequence.\n \n The longest common subsequence (LCS) problem is a classic computer science problem \n that is often solved using dynamic programming.\n \n For example:\n longest_common_subsequence(\"abcde\", \"ace\") -> 3\n longest_common_subsequence(\"abc\", \"def\") -> 0\n longest_common_subsequence(\"\", \"\") -> 0\n \"\"\"\n m, n = len(s1), len(s2)\n \n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if s1[i - 1] == s2[j - 1]:\n dp[i][j] = dp[i - 1][j - 1] + 1\n else:\n dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])\n \n return dp[m][n]", "error": " Implement: def longest_common_subsequence(s1: str, s2: str) -> int:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def longest_common_subsequence(s1: str, s2: str) -> int:\n m, n = len(s1), len(s2)\n \n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if s1[i - 1] == s2[j - 1]:\n dp[i][j] = dp[i - 1][j - 1] + 1\n else:\n dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])\n \n return dp[m][n]"}
|
||||
{"signature": "def longest_common_subsequence(s1: str, s2: str) -> int:\n \"\"\"\n Given two strings s1 and s2, find the length of the longest common subsequence.\n \n The longest common subsequence (LCS) problem is a classic computer science problem \n that is often solved using dynamic programming.\n \n For example:\n longest_common_subsequence(\"abcde\", \"ace\") -> 3\n longest_common_subsequence(\"abc\", \"def\") -> 0\n longest_common_subsequence(\"\", \"\") -> 0\n \"\"\"\n m, n = len(s1), len(s2)\n \n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if s1[i - 1] == s2[j - 1]:\n dp[i][j] = dp[i - 1][j - 1] + 1\n else:\n dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])\n \n return dp[m][n]\n\n# tests", "tests": ["assert longest_common_subsequence(\"abcde\", \"ace\") == 3", "assert longest_common_subsequence(\"abc\", \"def\") == 0", "assert longest_common_subsequence(\"\", \"\") == 0"], "broken": "def longest_common_subsequence(s1: str, s2: str) -> int:\n \"\"\"\n Given two strings s1 and s2, find the length of the longest common subsequence.\n \n The longest common subsequence (LCS) problem is a classic computer science problem \n that is often solved using dynamic programming.\n \n For example:\n longest_common_subsequence(\"abcde\", \"ace\") -> 3\n longest_common_subsequence(\"abc\", \"def\") -> 0\n longest_common_subsequence(\"\", \"\") -> 0\n \"\"\"\n m, n = len(s1), len(s2)\n \n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if s1[i - 1] == s2[j - 1]:", "error": "File \"/tmp/tmpyd1srr14.py\", line 21\n assert longest_common_subsequence(\"abcde\", \"ace\") == 3\nIndentationError: expected an indented block after 'if' statement on line 19", "fixed": "def longest_common_subsequence(s1: str, s2: str) -> int:\n m, n = len(s1), len(s2)\n \n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if s1[i - 1] == s2[j - 1]:\n dp[i][j] = dp[i - 1][j - 1] + 1\n else:\n dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])\n \n return dp[m][n]"}
|
||||
{"signature": "def min_swaps_for_sorting(arr: list) -> int:\n \"\"\" Given an array arr[] of size N (1 <= N <= 1000), containing non-negative integers not greater than 1000, find the minimum number of swaps required to sort the array in non-decreasing order. Assume that the array is 1-indexed. \"\"\"\n n = len(arr)\n temp = arr.copy()\n temp.sort()\n ans = 0\n visited = [False] * n\n for i in range(n):\n if visited[i] or arr[i] == temp[i]:\n continue\n cycle_size = 0\n j = i\n while not visited[j]:\n visited[j] = True\n j = arr.index(temp[j])\n cycle_size += 1\n if cycle_size > 0:\n ans += (cycle_size - 1)\n return ans\n\n# tests", "tests": ["assert min_swaps_for_sorting([1, 5, 4, 3, 2]) == 2", "assert min_swaps_for_sorting([1, 0, 2]) == 1", "assert min_swaps_for_sorting([1, 2, 3, 4, 5]) == 0"], "broken": "block. You are an expert Python coder.", "error": " block. You are an expert Python coder.\n ^^^\nSyntaxError: invalid syntax", "fixed": "def min_swaps_for_sorting(arr: list) -> int:\n \"\"\" Given an array arr[] of size N (1 <= N <= 1000), containing non-negative integers not greater than 1000, find the minimum number of swaps required to sort the array in non-decreasing order. Assume that the array is 1-indexed. \"\"\"\n n = len(arr)\n temp = arr.copy()\n temp.sort()\n ans = 0\n visited = [False] * n\n for i in range(n):\n if visited[i] or arr[i] == temp[i]:\n continue\n cycle_size = 0\n j = i\n while not visited[j]:\n visited[j] = True\n j = arr.index(temp[j])\n cycle_size += 1\n if cycle_size > 0:\n ans += (cycle_size - 1)\n return ans"}
|
||||
{"signature": "def find_min_in_rotated(nums: list[int]) -> int:\n \"\"\"\n Given a rotated sorted array of integers (rotated at some pivot unknown to you beforehand), return the minimum element in the array.\n If the array contains duplicates, the function still needs to return the minimum element.\n Assume that the array may be empty and also may contain negative numbers.\n\n Example:\n find_min_in_rotated([3, 4, 5, 1, 2]) -> 1\n find_min_in_rotated([4, 5, 6, 7, 0, 1, 2]) -> 0\n find_min_in_rotated([2, 3, 4, 5, 6, 7, 8, 1]) -> 1\n find_min_in_rotated([]) -> raises ValueError\n find_min_in_rotated([3, 3, 3]) -> 3\n \"\"\"\n if not nums:\n raise ValueError(\"Input array is empty\")\n if len(nums) == 1:\n return nums[0]\n left, right = 0, len(nums) - 1\n while left < right:\n mid = (left + right) // 2\n if nums[mid] > nums[right]:\n left = mid + 1\n elif nums[mid] < nums[right]:\n right = mid\n else:\n right -= 1\n return nums[left]\n\n# tests", "tests": ["assert find_min_in_rotated([3, 4, 5, 1, 2]) == 1", "assert find_min_in_rotated([4, 5, 6, 7, 0, 1, 2]) == 0", "assert find_min_in_rotated([2, 3, 4, 5, 6, 7, 8, 1]) == 1", "assert find_min_in_rotated([3, 3, 3]) == 3"], "broken": "def find_min_in_rotated(nums: list[int]) -> int:\n \"\"\"\n Given a rotated sorted array of integers (rotated at some pivot unknown to you beforehand), return the minimum element in the array.\n If the array contains duplicates, the function still needs to return the minimum element.\n Assume that the array may be empty and also may contain negative numbers.\n\n Example:\n find_min_in_rotated([3, 4, 5, 1, 2]) -> 1\n find_min_in_rotated([4, 5, 6, 7, 0, 1, 2]) -> 0\n find_min_in_rotated([2, 3, 4, 5, 6, 7, 8, 1]) -> 1\n find_min_in_rotated([]) -> raises ValueError\n find_min_in_rotated([3, 3, 3]) -> 3\n \"\"\"\n if not nums:\n raise ValueError(\"Input array is empty\")\n if len(nums) == 1:\n return nums[0]\n left, right = 0, len(nums) - 1\n while left < right:\n mid = (left + right) // 2\n if nums[mid] > nums[right]:\n left = mid + 1\n elif nums[mid] < nums[right]:\n right = mid\n else:\n right -= 1\n return nums[left]\n\n# tests\n\nTests:\nassert find_min_in_rotated([3, 4, 5, 1, 2]) == 1\nassert find_min_in_rotated([4, 5, 6, 7, 0, 1, 2]) == 0\nassert find_min_in_rotated([2, 3, 4, 5, 6, 7, 8, 1]) == 1\nassert find_min_in_rotated([3, 3, 3]) == 3", "error": " Tests:\n ^\nSyntaxError: invalid syntax", "fixed": "def find_min_in_rotated(nums: list[int]) -> int:\n \"\"\"\n Given a rotated sorted array of integers (rotated at some pivot unknown to you beforehand), return the minimum element in the array.\n If the array contains duplicates, the function still needs to return the minimum element.\n Assume that the array may be empty and also may contain negative numbers.\n\n Example:\n find_min_in_rotated([3, 4, 5, 1, 2]) -> 1\n find_min_in_rotated([4, 5, 6, 7, 0, 1, 2]) -> 0\n find_min_in_rotated([2, 3, 4, 5, 6, 7, 8, 1]) -> 1\n find_min_in_rotated([]) -> raises ValueError\n find_min_in_rotated([3, 3, 3]) -> 3\n \"\"\"\n if not nums:\n raise ValueError(\"Input array is empty\")\n if len(nums) == 1:\n return nums[0]\n left, right = 0, len(nums) - 1\n while left < right:\n mid = (left + right) // 2\n if nums[mid] > nums[right]:\n left = mid + 1\n elif nums[mid] < nums[right]:\n right = mid\n else:\n right -= 1\n return nums[left]"}
|
||||
{"signature": "def find_kth_largest(nums: list, k: int) -> int:\n \"\"\"Given a list of integers, return the kth largest integer.\n\n nums - the list of integers\n k - the rank of the value to return\n \"\"\"\n\n import heapq\n k_smallest = heapq.nlargest(k, nums)\n return k_smallest[-1]\n\n# tests", "tests": ["assert find_kth_largest([3,2,1,5,6,4], 2) == 5", "assert find_kth_largest([3,2,3,1,2,4,5,5,6], 4) == 4", "assert find_kth_largest([1], 1) == 1"], "broken": "def find_kth_largest(nums: list, k: int) -> int:\n \"\"\"Given a list of integers, return the kth largest integer.\n\n nums - the list of integers\n k - the rank of the value to return\n \"\"\"\n return heapq.nlargest(k, nums)[-1]\n\n# tests\n\nassert find_kth_largest([3,2,1,5,6,4], 2) == 5\nassert find_kth_largest([3,2,3,1,2,4,5,5,6], 4) == 4\nassert find_kth_largest([1], 1) == 1", "error": " return heapq.nlargest(k, nums)[-1]\n ^^^^^\nNameError: name 'heapq' is not defined. Did you mean: 'help'?", "fixed": "def find_kth_largest(nums: list, k: int) -> int:\n \"\"\"Given a list of integers, return the kth largest integer.\n\n nums - the list of integers\n k - the rank of the value to return\n \"\"\"\n import heapq\n k_smallest = heapq.nlargest(k, nums)\n return k_smallest[-1]"}
|
||||
{"signature": "def longest_substring_with_distinct_chars(s: str) -> int:\n \"\"\"\n Given a string s, find the length of the longest substring without repeating characters.\n\n For example:\n - In 'abcabcbb', the longest substring is 'abc' with length 3.\n - In 'bbbb', the longest substring is 'b' with length 1.\n - In 'pwwkew', the longest substring is 'wke' or 'kew' with length 3.\n \"\"\"\n char_index = {}\n max_length = 0\n start = 0\n \n for end, char in enumerate(s):\n if char in char_index:\n start = max(start, char_index[char] + 1)\n \n char_index[char] = end\n max_length = max(max_length, end - start + 1)\n\n return max_length\n\n# tests", "tests": ["assert longest_substring_with_distinct_chars('abcabcbb') == 3", "assert longest_substring_with_distinct_chars('bbbb') == 1", "assert longest_substring_with_distinct_chars('') == 0"], "broken": "block only.\nImplement: def longest_substring_with_distinct_chars(s: str) -> int:\n \"\"\"\n Given a string s, find the length of the longest substring without repeating characters.\n\n For example:\n - In 'abcabcbb', the longest substring is 'abc' with length 3.\n - In 'bbbb', the longest substring is 'b' with length 1.\n - In 'pwwkew', the longest substring is 'wke' or 'kew' with length 3.\n \"\"\"\n char_index = {}\n max_length = 0\n start = 0\n \n for end, char in enumerate(s):\n if char in char_index:\n start = max(start, char_index[char] + 1)\n \n char_index[char] = end\n max_length = max(max_length, end - start + 1)\n\n return max_length\n\n# tests\n\nTests:\nassert longest_substring_with_distinct_chars('abcabcbb') == 3\nassert longest_substring_with_distinct_chars('bbbb') == 1\nassert longest_substring_with_distinct_chars('') == 0\n\nOutput only the function implementation in one", "error": " block only.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def longest_substring_with_distinct_chars(s: str) -> int:\n \"\"\"\n Given a string s, find the length of the longest substring without repeating characters.\n\n For example:\n - In 'abcabcbb', the longest substring is 'abc' with length 3.\n - In 'bbbb', the longest substring is 'b' with length 1.\n - In 'pwwkew', the longest substring is 'wke' or 'kew' with length 3.\n \"\"\"\n char_index = {}\n max_length = 0\n start = 0\n \n for end, char in enumerate(s):\n if char in char_index:\n start = max(start, char_index[char] + 1)\n \n char_index[char] = end\n max_length = max(max_length, end - start + 1)\n\n return max_length"}
|
||||
{"signature": "def longest_substring_with_distinct_chars(s: str) -> int:\n \"\"\"\n Given a string s, find the length of the longest substring without repeating characters.\n\n For example:\n - In 'abcabcbb', the longest substring is 'abc' with length 3.\n - In 'bbbb', the longest substring is 'b' with length 1.\n - In 'pwwkew', the longest substring is 'wke' or 'kew' with length 3.\n \"\"\"\n char_index = {}\n max_length = 0\n start = 0\n \n for end, char in enumerate(s):\n if char in char_index:\n start = max(start, char_index[char] + 1)\n \n char_index[char] = end\n max_length = max(max_length, end - start + 1)\n\n return max_length\n\n# tests", "tests": ["assert longest_substring_with_distinct_chars('abcabcbb') == 3", "assert longest_substring_with_distinct_chars('bbbb') == 1", "assert longest_substring_with_distinct_chars('') == 0"], "broken": "block.\n\nAssistant: def longest_substring_with_distinct_chars(s: str) -> int:\n \"\"\"\n Given a string s, find the length of the longest substring without repeating characters.\n\n For example:\n - In 'abcabcbb', the longest substring is 'abc' with length 3.\n - In 'bbbb', the longest substring is 'b' with length 1.\n - In 'pwwkew', the longest substring is 'wke' or 'kew' with length 3.\n \"\"\"\n char_index = {}\n max_length = 0\n start = 0\n \n for end, char in enumerate(s):\n if char in char_index:\n start = max(start, char_index[char] + 1)\n \n char_index[char] = end\n max_length = max(max_length, end - start + 1)\n\n return max_length", "error": " block.\n ^\nSyntaxError: invalid syntax", "fixed": "def longest_substring_with_distinct_chars(s: str) -> int:\n \"\"\"\n Given a string s, find the length of the longest substring without repeating characters.\n\n For example:\n - In 'abcabcbb', the longest substring is 'abc' with length 3.\n - In 'bbbb', the longest substring is 'b' with length 1.\n - In 'pwwkew', the longest substring is 'wke' or 'kew' with length 3.\n \"\"\"\n char_index = {}\n max_length = 0\n start = 0\n \n for end, char in enumerate(s):\n if char in char_index:\n start = max(start, char_index[char] + 1)\n \n char_index[char] = end\n max_length = max(max_length, end - start + 1)\n\n return max_length"}
|
||||
{"signature": "def find_longest_increasing_subsequence(arr):\n \"\"\"Given an array of integers `arr`, return the length of the longest strictly increasing subsequence. A subsequence is a sequence that can be derived from the array by deleting some elements (possibly zero) without changing the order of the remaining elements. It is guaranteed that `arr` will contain at least one element, and all elements are distinct.\n\n Example:\n - `find_longest_increasing_subsequence([10,9,2,5,3,7,101,18])` should return `4`, as the longest increasing subsequence is [2,3,7,101].\n\n Constraints:\n - The input array will have at least one element.\n - The elements in the array are distinct.\n \"\"\"\n n = len(arr)\n lengths = [1] * n\n for i in range(1, n):\n for j in range(i):\n if arr[i] > arr[j] and lengths[i] < lengths[j] + 1:\n lengths[i] = lengths[j] + 1\n return max(lengths)\n\n# tests", "tests": ["assert find_longest_increasing_subsequence([10, 9, 2, 5, 3, 7, 101, 18]) == 4", "assert find_longest_increasing_subsequence([3, 4, -1, 0, 6, 2, 3]) == 4", "assert find_longest_increasing_subsequence([100]) == 1"], "broken": "def find_longest_increasing_subsequence(arr: List[int]) -> int:\n \"\"\"\n Given a list of integers `arr`, find the length of the longest strictly increasing subsequence.\n\n A subsequence is a sequence that can be derived from the array by deleting some elements (possibly zero) without changing the order of the remaining elements.\n\n The function should return an integer representing the length of the longest increasing subsequence.\n\n For example, if `arr = [10, 9, 2, 5, 3, 7, 101, 18]`, the function should return 4, as the longest increasing subsequence is [2, 3, 7, 101].\n\n You can assume that `arr` will have at least one element and all elements are distinct.\n\n \"\"\"", "error": " def find_longest_increasing_subsequence(arr: List[int]) -> int:\n ^^^^\nNameError: name 'List' is not defined. Did you mean: 'list'?", "fixed": "def find_longest_increasing_subsequence(arr):\n n = len(arr)\n lengths = [1] * n\n for i in range(1, n):\n for j in range(i):\n if arr[i] > arr[j] and lengths[i] < lengths[j] + 1:\n lengths[i] = lengths[j] + 1\n return max(lengths)"}
|
||||
{"signature": "def encode_message(message: str, key: int) -> str:\n \"\"\"\n Takes a string `message` and an integer `key`. Returns a string encoded using a simple Caesar cipher,\n where each character in the message is shifted by the key number of positions in the alphabet.\n Lowercase letters remain lowercase and uppercase letters remain uppercase, with non-alphabetic\n characters unchanged.\n \n - `message`: A string consisting of alphanumeric characters and spaces.\n - `key`: An integer between -100 and 100, inclusive, determining the shift amount.\n - Returns: A string, the encoded message.\n \"\"\"\n encoded_message = \"\"\n for char in message:\n if char.isalpha():\n # Get the ASCII value and shift it by the key\n shifted = ord(char) + key\n # Ensure lowercase remains lowercase and uppercase remains uppercase\n if (char.islower() and shifted > ord('z')) or (char.isupper() and shifted > ord('Z')):\n # Wrap around the alphabet\n shifted = shifted % (ord('a') - 1) + ord('a') if char.islower() else shifted % (ord('A') - 1) + ord('A')\n encoded_message += chr(shifted)\n else:\n # Non-alphabetic characters remain the same\n encoded_message += char\n return encoded_message\n\n# tests", "tests": ["assert encode_message(\"Hello, World!\", 3) == \"Khoor, Zruog!\"", "assert encode_message(\"Khoor, Zruog!\", -3) == \"Hello, World!\"", "assert encode_message(\"1234567890!@#$%^&*()_+\", 100) == \"1234567890!@#$%^&*()_+\""], "broken": ">>> encode_message(\"Hello, World!\", 3)\n'Khoor, Zruog!'\n>>> encode_message(\"Khoor, Zruog!\", -3)\n'Hello, World!'\n>>> encode_message(\"1234567890!@#$%^&*()_+\", 100)\n'1234567890!@#$%^&*()_+'", "error": " >>> encode_message(\"Hello, World!\", 3)\n ^^\nSyntaxError: invalid syntax", "fixed": "def encode_message(message: str, key: int) -> str:\n encoded_message = \"\"\n for char in message:\n if char.isalpha():\n shifted = ord(char) + key\n if (char.islower() and shifted > ord('z')) or (char.isupper() and shifted > ord('Z')):\n shifted = shifted % (ord('a') - 1) + ord('a') if char.islower() else shifted % (ord('A') - 1) + ord('A')\n encoded_message += chr(shifted)\n else:\n encoded_message += char\n return encoded_message"}
|
||||
{"signature": "def encode_message(message: str, key: int) -> str:\n \"\"\"\n Takes a string `message` and an integer `key`. Returns a string encoded using a simple Caesar cipher,\n where each character in the message is shifted by the key number of positions in the alphabet.\n Lowercase letters remain lowercase and uppercase letters remain uppercase, with non-alphabetic\n characters unchanged.\n \n - `message`: A string consisting of alphanumeric characters and spaces.\n - `key`: An integer between -100 and 100, inclusive, determining the shift amount.\n - Returns: A string, the encoded message.\n \"\"\"\n encoded_message = \"\"\n for char in message:\n if char.isalpha():\n # Get the ASCII value and shift it by the key\n shifted = ord(char) + key\n # Ensure lowercase remains lowercase and uppercase remains uppercase\n if (char.islower() and shifted > ord('z')) or (char.isupper() and shifted > ord('Z')):\n # Wrap around the alphabet\n shifted = shifted % (ord('a') - 1) + ord('a') if char.islower() else shifted % (ord('A') - 1) + ord('A')\n encoded_message += chr(shifted)\n else:\n # Non-alphabetic characters remain the same\n encoded_message += char\n return encoded_message\n\n# tests", "tests": ["assert encode_message(\"Hello, World!\", 3) == \"Khoor, Zruog!\"", "assert encode_message(\"Khoor, Zruog!\", -3) == \"Hello, World!\"", "assert encode_message(\"1234567890!@#$%^&*()_+\", 100) == \"1234567890!@#$%^&*()_+\""], "broken": "block only.\nImplement: def encode_message(message: str, key: int) -> str:\n \"\"\"\n Takes a string `message` and an integer `key`. Returns a string encoded using a simple Caesar cipher,\n where each character in the message is shifted by the key number of positions in the alphabet.\n Lowercase letters remain lowercase and uppercase letters remain uppercase, with non-alphabetic\n characters unchanged.\n \n - `message`: A string consisting of alphanumeric characters and spaces.\n - `key`: An integer between -100 and 100, inclusive, determining the shift amount.\n - Returns: A string, the encoded message.\n \"\"\"\n encoded_message = \"\"\n for char in message:\n if char.isalpha():\n # Get the ASCII value and shift it by the key\n shifted = ord(char) + key\n # Ensure lowercase remains lowercase and uppercase remains uppercase\n if (char.islower() and shifted > ord('z')) or (char.isupper() and shifted > ord('Z')):\n # Wrap around the alphabet\n shifted = shifted % (ord('a') - 1) + ord('a') if char.islower() else shifted % (ord('A') - 1) + ord('A')\n encoded_message += chr(shifted)\n else:\n # Non-alphabetic characters remain the same\n encoded_message += char\n return encoded_message\n\n# tests\nassert encode_message(\"Hello, World!\", 3) == \"Khoor, Zruog!\"\nassert encode_message(\"Khoor, Zruog!\", -3) == \"Hello, World!\"\nassert encode_message(\"1234567890!@#$%^&*()_+\", 100) == \"1234567890!@#$%^&*()_+\"", "error": " block only.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def encode_message(message: str, key: int) -> str:\n encoded_message = \"\"\n for char in message:\n if char.isalpha():\n shifted = ord(char) + key\n if (char.islower() and shifted > ord('z')) or (char.isupper() and shifted > ord('Z')):\n shifted = shifted % (ord('a') - 1) + ord('a') if char.islower() else shifted % (ord('A') - 1) + ord('A')\n encoded_message += chr(shifted)\n else:\n encoded_message += char\n return encoded_message"}
|
||||
{"signature": "def is_fibonacci_number(n: int) -> bool:\n \"\"\"\n Checks if a given number is a Fibonacci number.\n\n Parameters:\n - n: int - The number to check.\n\n Returns:\n - bool: True if the number is a Fibonacci number, False otherwise.\n \"\"\"\n if n == 0 or n == 1:\n return True\n a, b = 0, 1\n while b < n:\n a, b = b, a + b\n return b == n\n\n\n# tests", "tests": ["assert is_fibonacci_number(0) == True", "assert is_fibonacci_number(8) == True", "assert is_fibonacci_number(14) == False"], "broken": "block only.\n\nHere is the implementation of the `is_fibonacci_number` function as requested:", "error": " block only.\n ^^^^\nSyntaxError: invalid syntax", "fixed": "def is_fibonacci_number(n: int) -> bool:\n \"\"\"\n Checks if a given number is a Fibonacci number.\n\n Parameters:\n - n: int - The number to check.\n\n Returns:\n - bool: True if the number is a Fibonacci number, False otherwise.\n \"\"\"\n if n == 0 or n == 1:\n return True\n a, b = 0, 1\n while b < n:\n a, b = b, a + b\n return b == n"}
|
||||
{"signature": "def longest_consecutive_subsequence(nums: list) -> int:\n \"\"\"\n Given an unsorted array of integers `nums`, find the length of the longest consecutive elements sequence.\n \n For example, given [100, 4, 200, 1, 3, 2], the longest consecutive sequence is [1, 2, 3, 4] with length 4.\n \n The algorithm should handle empty input and duplicates gracefully.\n \n :param nums: list of integers\n :return: length of the longest consecutive subsequence\n \"\"\"\n num_set = set(nums) # Use a set to allow O(1) lookups.\n longest_streak = 0\n\n for num in num_set:\n # This is not the start of a sequence.\n if num - 1 in num_set:\n continue\n current_num = num\n current_streak = 1\n\n while current_num + 1 in num_set:\n current_num += 1\n current_streak += 1\n\n longest_streak = max(longest_streak, current_streak)\n\n return longest_streak\n\n# tests", "tests": ["assert longest_consecutive_subsequence([]) == 0", "assert longest_consecutive_subsequence([100, 4, 200, 1, 3, 2]) == 4", "assert longest_consecutive_subsequence([0, 3, 7, 2, 5, 8, 4, 6, 0, 1]) == 9"], "broken": "block as requested.", "error": " block as requested.\n ^^\nSyntaxError: invalid syntax", "fixed": "from typing import List\n\ndef longest_consecutive_subsequence(nums: List[int]) -> int:\n num_set = set(nums) # Use a set to allow O(1) lookups.\n longest_streak = 0\n\n for num in num_set:\n # This is not the start of a sequence.\n if num - 1 in num_set:\n continue\n current_num = num\n current_streak = 1\n\n while current_num + 1 in num_set:\n current_num += 1\n current_streak += 1\n\n longest_streak = max(longest_streak, current_streak)\n\n return longest_streak"}
|
||||
{"signature": "def longest_consecutive_subsequence(nums: list) -> int:\n \"\"\"\n Given an unsorted array of integers `nums`, find the length of the longest consecutive elements sequence.\n \n For example, given [100, 4, 200, 1, 3, 2], the longest consecutive sequence is [1, 2, 3, 4] with length 4.\n \n The algorithm should handle empty input and duplicates gracefully.\n \n :param nums: list of integers\n :return: length of the longest consecutive subsequence\n \"\"\"\n num_set = set(nums) # Use a set to allow O(1) lookups.\n longest_streak = 0\n\n for num in num_set:\n # This is not the start of a sequence.\n if num - 1 in num_set:\n continue\n current_num = num\n current_streak = 1\n\n while current_num + 1 in num_set:\n current_num += 1\n current_streak += 1\n\n longest_streak = max(longest_streak, current_streak)\n\n return longest_streak\n\n# tests", "tests": ["assert longest_consecutive_subsequence([]) == 0", "assert longest_consecutive_subsequence([100, 4, 200, 1, 3, 2]) == 4", "assert longest_consecutive_subsequence([0, 3, 7, 2, 5, 8, 4, 6, 0, 1]) == 9"], "broken": "Implement: def longest_consecutive_subsequence(nums: list) -> int:\n \"\"\"\n Given an unsorted array of integers `nums`, find the length of the longest consecutive elements sequence.\n \n For example, given [100, 4, 200, 1, 3, 2], the longest consecutive sequence is [1, 2, 3, 4] with length 4.\n \n The algorithm should handle empty input and duplicates gracefully.\n \n :param nums: list of integers\n :return: length of the longest consecutive subsequence\n \"\"\"\n num_set = set(nums) # Use a set to allow O(1) lookups.\n longest_streak = 0\n\n for num in num_set:\n # This is not the start of a sequence.\n if num - 1 in num_set:\n continue\n current_num = num\n current_streak = 1\n\n while current_num + 1 in num_set:\n current_num += 1\n current_streak += 1\n\n longest_streak = max(longest_streak, current_streak)\n\n return longest_streak\n\n# tests\n\nTests:\nassert longest_consecutive_subsequence([]) == 0\nassert longest_consecutive_subsequence([100, 4, 200, 1, 3, 2]) == 4\nassert longest_consecutive_subsequence([0, 3, 7, 2, 5, 8, 4, 6, 0, 1]) == 9", "error": " Implement: def longest_consecutive_subsequence(nums: list) -> int:\n ^^^\nSyntaxError: invalid syntax", "fixed": "from typing import List\n\ndef longest_consecutive_subsequence(nums: List[int]) -> int:\n num_set = set(nums) # Use a set to allow O(1) lookups.\n longest_streak = 0\n\n for num in num_set:\n # This is not the start of a sequence.\n if num - 1 in num_set:\n continue\n current_num = num\n current_streak = 1\n\n while current_num + 1 in num_set:\n current_num += 1\n current_streak += 1\n\n longest_streak = max(longest_streak, current_streak)\n\n return longest_streak"}
|
||||
{"signature": "def maximum_non_adjacent_sum(nums: list[int]) -> int:\n \"\"\"\n Given an integer array nums, return the maximum sum of non-adjacent elements in nums.\n \n Example:\n - maximum_non_adjacent_sum([2, 1, 5, 8, 4]) should return 11 (5 + 8)\n - maximum_non_adjacent_sum([-2, -1, -5, -8, -4]) should return 0 (no non-adjacent sum is possible)\n - maximum_non_adjacent_sum([2, 1, 5, 1, 3, 2]) should return 10 (2 + 5 + 3)\n \"\"\"\n incl = 0\n excl = 0\n \n for i in nums:\n new_excl = max(incl, excl)\n incl = excl + i\n excl = new_excl\n return max(incl, excl)\n\n# tests", "tests": ["assert maximum_non_adjacent_sum([2, 1, 5, 8, 4]) == 11", "assert maximum_non_adjacent_sum([-2, -1, -5, -8, -4]) == 0", "assert maximum_non_adjacent_sum([2, 1, 5, 1, 3, 2]) == 10"], "broken": "Implement: def maximum_non_adjacent_sum(nums: list[int]) -> int:\n \"\"\"\n Given an integer array nums, return the maximum sum of non-adjacent elements in nums.\n \n Example:\n - maximum_non_adjacent_sum([2, 1, 5, 8, 4]) should return 11 (5 + 8)\n - maximum_non_adjacent_sum([-2, -1, -5, -8, -4]) should return 0 (no non-adjacent sum is possible)\n - maximum_non_adjacent_sum([2, 1, 5, 1, 3, 2]) should return 10 (2 + 5 + 3)\n \"\"\"\n incl = 0\n excl = 0\n \n for i in nums:\n new_excl = max(incl, excl)\n incl = excl + i\n excl = new_excl\n return max(incl, excl)", "error": " Implement: def maximum_non_adjacent_sum(nums: list[int]) -> int:\n ^^^\nSyntaxError: invalid syntax", "fixed": "def maximum_non_adjacent_sum(nums: list[int]) -> int:\n incl = 0\n excl = 0\n \n for i in nums:\n new_excl = max(incl, excl)\n incl = excl + i\n excl = new_excl\n return max(incl, excl)"}
|
||||
{"signature": "def maximum_non_adjacent_sum(nums: list[int]) -> int:\n \"\"\"\n Given an integer array nums, return the maximum sum of non-adjacent elements in nums.\n \n Example:\n - maximum_non_adjacent_sum([2, 1, 5, 8, 4]) should return 11 (5 + 8)\n - maximum_non_adjacent_sum([-2, -1, -5, -8, -4]) should return 0 (no non-adjacent sum is possible)\n - maximum_non_adjacent_sum([2, 1, 5, 1, 3, 2]) should return 10 (2 + 5 + 3)\n \"\"\"\n incl = 0\n excl = 0\n \n for i in nums:\n new_excl = max(incl, excl)\n incl = excl + i\n excl = new_excl\n return max(incl, excl)\n\n# tests", "tests": ["assert maximum_non_adjacent_sum([2, 1, 5, 8, 4]) == 11", "assert maximum_non_adjacent_sum([-2, -1, -5, -8, -4]) == 0", "assert maximum_non_adjacent_sum([2, 1, 5, 1, 3, 2]) == 10"], "broken": "block.``python\ndef maximum_non_adjacent_sum(nums: list[int]) -> int:\n \"\"\"\n Given an integer array nums, return the maximum sum of non-adjacent elements in nums.\n \n Example:\n - maximum_non_adjacent_sum([2, 1, 5, 8, 4]) should return 11 (5 + 8)\n - maximum_non_adjacent_sum([-2, -1, -5, -8, -4]) should return 0 (no non-adjacent sum is possible)\n - maximum_non_adjacent_sum([2, 1, 5, 1, 3, 2]) should return 10 (2 + 5 + 3)\n \"\"\"\n incl = 0\n excl = 0\n \n for i in nums:\n new_excl = max(incl, excl)\n incl = excl + i\n excl = new_excl\n return", "error": " block.``python\n ^\nSyntaxError: invalid syntax", "fixed": "def maximum_non_adjacent_sum(nums: list[int]) -> int:\n incl = 0\n excl = 0\n \n for i in nums:\n new_excl = max(incl, excl)\n incl = excl + i\n excl = new_excl\n return max(incl, excl)"}
|
||||
40
data/pairs_7b_40.jsonl
Normal file
40
data/pairs_7b_40.jsonl
Normal file
File diff suppressed because one or more lines are too long
13
data/pairs_math_13.jsonl
Normal file
13
data/pairs_math_13.jsonl
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
{"problem": "A family went out for dinner and had 3 adults and 2 children. The cost of dinner was $150. How much did each person need to pay?", "solution": "To solve this problem, we need to divide the total cost of dinner by the total number of people. Let's break it down step by step:\n\n1. Calculate the total number of people: 3 adults + 2 children = 5 people.\n2. Divide the total cost of dinner by the total number of people: $150 \u00f7 5 people = $30 per person.\n\nSo, each person needs to pay $30.#### 30", "answer": 30.0}
|
||||
{"problem": "The sum of three consecutive integers is 30. One of the integers is 10. What is the sum of the other two integers?", "solution": "1. Let the first integer be x.\n2. The second integer is x + 1.\n3. The third integer is x + 2.\n4. The sum of these three integers is x + (x + 1) + (x + 2) = 3x + 3.\n5. Given that the sum is 30, we have 3x + 3 = 30.\n6. Subtract 3 from both sides to get 3x = 27.\n7. Divide by 3 to find x = 9.\n8. Therefore, the integers are 9, 10, and 11.\n9. The sum of the other two integers (10 and 11) is 10 + 11 = 21.\n#### 21", "answer": 21.0}
|
||||
{"problem": "A baker has 25 cookies and wants to pack them into boxes. If each box can hold 5 cookies, and the baker wants to leave a box with fewer than 5 cookies as a gift, how many boxes does the baker need to pack all the remaining cookies?", "solution": "To solve this problem, we need to determine how many boxes are required to pack all the cookies while leaving one box with fewer than 5 cookies as a gift.\n\nStep 1: Calculate the number of boxes needed to pack all the cookies, excluding the gift box.\n- The baker has 25 cookies and each box can hold 5 cookies.\n- Divide the total number of cookies by the number of cookies per box: 25 \u00f7 5 = 5 boxes.\n- Since we want to leave one box as a gift, we will need 5 + 1 = 6 boxes in total.\n\nThe baker needs 6 boxes to pack all the remaining cookies and leave one box with fewer than 5 cookies as a gift.\n\n#### 6", "answer": 6.0}
|
||||
{"problem": "Marnie has a box of 18 crayons. She decided to give 5 crayons to her friend. She then decided to buy 3 more crayons. She also decided to give 2 crayons to her teacher. How many crayons does she have left?", "solution": "1. Marnie starts with 18 crayons.\n2. She gives away 5 crayons: 18 - 5 = 13 crayons left.\n3. She buys 3 more crayons: 13 + 3 = 16 crayons.\n4. She gives 2 crayons to her teacher: 16 - 2 = 14 crayons left.\n\n#### 14", "answer": 14.0}
|
||||
{"problem": "Jill has a garden. Jill's garden is 18 feet wide. Jill's garden is 15 feet long. How many feet of fencing does jill need to fence her garden?", "solution": "Jill's garden is a rectangle. The perimeter of a rectangle is given by the formula P = 2L + 2W. #### 66", "answer": 66.0}
|
||||
{"problem": "I have 3 boxes. One of them is filled with apples. One of them is filled with oranges. One of them is filled with bananas. I know that there are exactly 6 fruits. 2 boxes are empty. How many fruits are in each box?", "solution": "There are 1 apples, 1 oranges, and 4 bananas.#### 1#### 1#### 4", "answer": 1.0}
|
||||
{"problem": "The school is having a yard sale. The teachers want to sell 50 old textbooks. If they sell them for $2 each, how much money can they expect to make if they decide to donate 20% of their earnings to a local charity?", "solution": "#### 80\n\nThe teachers can expect to make $80 if they decide to donate 20% of their earnings to a local charity.", "answer": 80.0}
|
||||
{"problem": "There are 10 apples in a bag. You take away 3 apples. You then add 5 more apples. How many apples do you have now?", "solution": "1. Start with 10 apples.\n2. Take away 3 apples: 10 - 3 = 7 apples.\n3. Add 5 more apples: 7 + 5 = 12 apples.\n#### 12", "answer": 12.0}
|
||||
{"problem": "A pizza has 8 slices. If a family eats 3 slices and then receives an additional 2 slices as a gift, how many slices are left?", "solution": "Let's break down the problem step by step:\n\n1. The pizza starts with 8 slices.\n2. The family eats 3 slices, so there are now 8 - 3 = 5 slices remaining.\n3. The family receives an additional 2 slices as a gift, so there are now 5 + 2 = 7 slices remaining.\n\n#### 7", "answer": 7.0}
|
||||
{"problem": "A bird watcher saw 17 birds on Monday, 25 birds on Tuesday, and 18 birds on Wednesday. How many birds did the bird watcher see in total over these three days?", "solution": "Let's break down the problem step by step:\n\n1. **Sum the number of birds seen each day:**\n - Monday: 17 birds\n - Tuesday: 25 birds\n - Wednesday: 18 birds\n\n2. **Add the numbers together:**\n \\[\n 17 + 25 + 18\n \\]\n\n3. **Perform the addition:**\n - First, add the birds seen on Monday and Tuesday:\n \\[\n 17 + 25 = 42\n \\]\n - Next, add the birds seen on Wednesday to the result:\n \\[\n 42 + 18 = 60\n \\]\n\nTherefore, the total number of birds seen over the three days is:\n\\[\n\\boxed{60}\n\\]", "answer": 60.0}
|
||||
{"problem": "A baker has 48 cupcakes. She wants to put an equal number of cupcakes in 6 boxes. How many cupcakes will go in each box? Then, she wants to put 25% more cupcakes into each box. How many cupcakes will go in each box now?", "solution": "The baker initially puts 48 cupcakes in 6 boxes, so each box will have 48/6 = 8 cupcakes. Then, she wants to put 25% more cupcakes in each box. The number of additional cupcakes in each box is 8 * 25/100 = 2 cupcakes. Therefore, each box will have a total of 8 + 2 = 10 cupcakes. #### 10", "answer": 10.0}
|
||||
{"problem": "A group of 2 friends want to buy a pizza. Each pizza costs $7. How much do they need to spend altogether for 2 pizzas? After eating half of the pizza, how much money will each friend spend?", "solution": "To solve this problem, we will follow these steps:\n\n1. Calculate the total cost of 2 pizzas.\n2. Calculate the cost of half a pizza.\n3. Determine the amount each friend needs to spend.\n\nStep 1: Calculate the total cost of 2 pizzas.\nThe cost of one pizza is $7. Therefore, the cost of 2 pizzas is:\n\\( 2 \\times 7 = 14 \\)\n\nStep 2: Calculate the cost of half a pizza.\nHalf of a pizza is \\( \\frac{2}{2} = 1 \\) pizza. So, the cost of half a pizza is:\n\\( 7 \\times 1 = 7 \\)\n\nStep 3: Determine the amount each friend needs to spend.\nEach friend needs to spend the cost of half a pizza, which is $7.\n\nSo, each friend will spend $7.\n\n#### 7", "answer": 7.0}
|
||||
{"problem": "Find the sum of 2, 3, and 4, then divide the result by 2.", "solution": "Step 1: Add 2, 3, and 4.\n(2 + 3 + 4) = 9.\n\nStep 2: Divide the result by 2.\n9 \u00f7 2 = 4.5.\n\n#### 4.5", "answer": 4.5}
|
||||
BIN
docs/fig1_headline.png
Normal file
BIN
docs/fig1_headline.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 130 KiB |
BIN
docs/fig6_boundary.png
Normal file
BIN
docs/fig6_boundary.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 81 KiB |
BIN
docs/scaling_chart.pdf
Normal file
BIN
docs/scaling_chart.pdf
Normal file
Binary file not shown.
BIN
docs/scaling_chart.png
Normal file
BIN
docs/scaling_chart.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
357
recipe/bootstrap.py
Normal file
357
recipe/bootstrap.py
Normal file
|
|
@ -0,0 +1,357 @@
|
|||
"""Self-Bootstrapping TinyForge.
|
||||
|
||||
Single model. No external dataset. Just a Python interpreter.
|
||||
|
||||
Loop:
|
||||
for iter in 1..N:
|
||||
1. Model generates K problems (function signature + tests + canonical solution)
|
||||
2. Filter: keep only those where canonical executes & tests pass
|
||||
3. Model solves each fresh (forget canonical)
|
||||
4. Verify against tests → identify failures
|
||||
5. Model repairs each failure (one shot, with error)
|
||||
6. Verify repairs → collect (broken, fixed) pairs
|
||||
7. Periodically: LoRA-train on accumulated pairs
|
||||
8. Periodically: eval on held-out HumanEval-mini
|
||||
|
||||
If accuracy on HumanEval rises without ever seeing HumanEval problems → recipe works.
|
||||
"""
|
||||
import os, sys, json, time, re, gc, subprocess, tempfile, argparse, random, math
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
|
||||
|
||||
import torch
|
||||
from torch.utils.data import Dataset, DataLoader
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
|
||||
from datasets import load_dataset, Dataset as HFDataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def extract_code(text):
|
||||
if "```python" in text: text = text.split("```python", 1)[1]
|
||||
elif "```" in text: text = text.split("```", 1)[1]
|
||||
if "```" in text: text = text.split("```", 1)[0]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def run_python(code, timeout=8):
|
||||
"""Run code in subprocess. Return (passed, stderr_or_msg)."""
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
|
||||
f.write(code); path = f.name
|
||||
try:
|
||||
r = subprocess.run(["python3", path], capture_output=True, timeout=timeout, text=True, cwd="/tmp")
|
||||
if r.returncode == 0: return True, ""
|
||||
err = (r.stderr or r.stdout).strip().splitlines()
|
||||
return False, "\n".join(err[-3:])[:300]
|
||||
except subprocess.TimeoutExpired: return False, "timeout"
|
||||
finally:
|
||||
try: os.unlink(path)
|
||||
except: pass
|
||||
|
||||
|
||||
def gen_batch(model, tok, prompts, max_new=400, temperature=0.7, batch=8):
|
||||
outs = []
|
||||
for i in range(0, len(prompts), batch):
|
||||
chunk = prompts[i:i+batch]
|
||||
texts = []
|
||||
for p in chunk:
|
||||
msgs = [{"role": "system", "content": "You are a Python coder."},
|
||||
{"role": "user", "content": p}]
|
||||
texts.append(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))
|
||||
inp = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1500).to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new, do_sample=temperature > 0,
|
||||
temperature=temperature if temperature > 0 else 1.0, top_p=0.95,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
for j in range(out.size(0)):
|
||||
outs.append(tok.decode(out[j][inp.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
return outs
|
||||
|
||||
|
||||
PROBLEM_GEN_PROMPT = """Generate ONE simple Python coding problem with a clear function spec and 3 test assertions.
|
||||
|
||||
Output format (exactly one ```python block):
|
||||
|
||||
```python
|
||||
def {function_name}({args}):
|
||||
\"\"\"{one-line description of what the function does}\"\"\"
|
||||
{implementation}
|
||||
|
||||
# tests
|
||||
assert {function_name}(...) == ...
|
||||
assert {function_name}(...) == ...
|
||||
assert {function_name}(...) == ...
|
||||
```
|
||||
|
||||
Make the function specific and concrete. The function should be 3-15 lines. Tests must verify the function works correctly. Output ONLY the code block."""
|
||||
|
||||
|
||||
def parse_generated_problem(raw_code):
|
||||
"""Split into (function_signature_with_docstring, full_solution_code, test_lines).
|
||||
Returns None if parsing fails or it's malformed."""
|
||||
code = raw_code.strip()
|
||||
if "def " not in code: return None
|
||||
|
||||
# Find first def
|
||||
lines = code.split("\n")
|
||||
func_start = None
|
||||
for i, l in enumerate(lines):
|
||||
if l.startswith("def "):
|
||||
func_start = i; break
|
||||
if func_start is None: return None
|
||||
|
||||
# Find tests (assert lines after the def block)
|
||||
tests = []
|
||||
in_def_body = False
|
||||
def_end = None
|
||||
for i in range(func_start, len(lines)):
|
||||
l = lines[i]
|
||||
if l.startswith("def ") and i > func_start: break
|
||||
if l.startswith("assert "):
|
||||
tests.append(l)
|
||||
if def_end is None: def_end = i
|
||||
elif tests and not l.strip().startswith(("#", "assert", "")):
|
||||
break
|
||||
|
||||
if len(tests) < 2: return None
|
||||
if def_end is None: def_end = len(lines)
|
||||
|
||||
full_solution = "\n".join(lines[func_start:def_end]).strip()
|
||||
if len(full_solution) < 30: return None
|
||||
|
||||
# Build function signature stub for re-implementation
|
||||
# Find docstring if present
|
||||
sig_lines = []
|
||||
for i in range(func_start, def_end):
|
||||
l = lines[i]
|
||||
sig_lines.append(l)
|
||||
if i > func_start and l.strip().endswith('"""') and ('"""' in lines[i-1] or '"""' in l[:l.rfind('"""')]):
|
||||
break
|
||||
if i > func_start and l.strip().startswith('"""') and l.strip().endswith('"""') and l.strip() != '"""':
|
||||
break
|
||||
# If no docstring, stop after the def line itself
|
||||
if i == func_start and not any('"""' in lines[j] for j in range(i, min(i+5, def_end))):
|
||||
sig_lines.append(" pass")
|
||||
break
|
||||
signature = "\n".join(sig_lines)
|
||||
|
||||
# Extract function name from signature
|
||||
m = re.match(r"def\s+(\w+)\s*\(", lines[func_start])
|
||||
if not m: return None
|
||||
fn_name = m.group(1)
|
||||
|
||||
return {
|
||||
"fn_name": fn_name,
|
||||
"signature": signature,
|
||||
"canonical": full_solution,
|
||||
"tests": tests,
|
||||
"raw": code,
|
||||
}
|
||||
|
||||
|
||||
# ── Loop ────────────────────────────────────────────────────────────────
|
||||
|
||||
def humaneval_eval(model, tok, n=30):
|
||||
"""Eval on HumanEval-mini (first N problems)."""
|
||||
he = list(load_dataset("openai_humaneval", split="test"))[:n]
|
||||
prompts = [p["prompt"] + "\n# Complete the function above." for p in he]
|
||||
outs = gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4)
|
||||
correct = 0
|
||||
for p, raw in zip(he, outs):
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
# Try the model's completion combined with the prompt
|
||||
full = p["prompt"] + "\n" + code if "def " not in code else code
|
||||
test_code = full + "\n\n" + p["test"] + f"\n\ncheck({p['entry_point']})"
|
||||
ok, _ = run_python(test_code, timeout=10)
|
||||
if ok: correct += 1
|
||||
return correct, n
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", default="Qwen/Qwen2.5-Coder-1.5B-Instruct")
|
||||
ap.add_argument("--gpu", type=int, default=0)
|
||||
ap.add_argument("--iterations", type=int, default=20)
|
||||
ap.add_argument("--problems_per_iter", type=int, default=16)
|
||||
ap.add_argument("--train_every", type=int, default=10)
|
||||
ap.add_argument("--eval_every", type=int, default=10)
|
||||
ap.add_argument("--tag", required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
out_dir = f"/workspace/bootstrap/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
device = torch.device(f"cuda:{args.gpu}")
|
||||
|
||||
log(f"loading {args.model}")
|
||||
tok = AutoTokenizer.from_pretrained(args.model)
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
tok.padding_side = "left"
|
||||
model = AutoModelForCausalLM.from_pretrained(args.model, dtype=torch.bfloat16, device_map=f"cuda:{args.gpu}")
|
||||
log(f" loaded mem={torch.cuda.memory_allocated(device)/1e9:.1f}GB")
|
||||
|
||||
# Initial eval
|
||||
log("INITIAL eval on HumanEval-mini")
|
||||
init_correct, init_total = humaneval_eval(model, tok, n=30)
|
||||
log(f" HumanEval-mini base: {init_correct}/{init_total}")
|
||||
|
||||
# LoRA setup (will be applied for training, base kept frozen)
|
||||
lora_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
|
||||
model = get_peft_model(model, lora_cfg)
|
||||
log(f" LoRA applied; trainable={sum(p.numel() for p in model.parameters() if p.requires_grad)/1e6:.1f}M")
|
||||
|
||||
accumulated_pairs = []
|
||||
eval_log = [{"iter": 0, "correct": init_correct, "total": init_total}]
|
||||
iter_stats = []
|
||||
|
||||
for it in range(1, args.iterations + 1):
|
||||
it_t = time.time()
|
||||
# 1. Generate K problems
|
||||
gen_prompts = [PROBLEM_GEN_PROMPT for _ in range(args.problems_per_iter)]
|
||||
raw_problems = gen_batch(model, tok, gen_prompts, max_new=400, temperature=0.9)
|
||||
|
||||
# 2. Parse + verify canonical
|
||||
valid_problems = []
|
||||
for raw in raw_problems:
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
parsed = parse_generated_problem(code)
|
||||
if parsed is None: continue
|
||||
# Verify canonical passes its own tests
|
||||
full = parsed["canonical"] + "\n\n" + "\n".join(parsed["tests"])
|
||||
ok, _ = run_python(full)
|
||||
if ok: valid_problems.append(parsed)
|
||||
|
||||
if not valid_problems:
|
||||
log(f"iter {it}: 0 valid problems generated, skipping")
|
||||
iter_stats.append({"iter": it, "valid": 0, "fails": 0, "repairs": 0})
|
||||
continue
|
||||
|
||||
# 3. Model solves each fresh — N=4 sampled attempts at temp=0.8 to surface natural fails
|
||||
N_ATTEMPTS = 4
|
||||
solve_prompts = [f"Implement this function so it passes the tests below.\n\n```python\n{p['signature']}\n```\n\nTests:\n{chr(10).join(p['tests'])}\n\nOutput only the function implementation in one ```python block." for p in valid_problems]
|
||||
# Generate N attempts each (4 * len(prompts) total)
|
||||
all_solve_prompts = solve_prompts * N_ATTEMPTS
|
||||
all_attempts = gen_batch(model, tok, all_solve_prompts, max_new=400, temperature=0.8)
|
||||
# Reshape: by problem, list of N attempts
|
||||
per_problem_attempts = [all_attempts[i::len(valid_problems)] for i in range(len(valid_problems))]
|
||||
|
||||
# 4-5. Mine (broken, fixed) pairs from same model's diverse outputs
|
||||
failures = []
|
||||
new_pairs = 0
|
||||
for p, attempts in zip(valid_problems, per_problem_attempts):
|
||||
broken_one = None; fixed_one = None; broken_err = None
|
||||
for raw in attempts:
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
full = code + "\n\n" + "\n".join(p["tests"])
|
||||
ok, err = run_python(full)
|
||||
if ok and fixed_one is None:
|
||||
fixed_one = code
|
||||
elif not ok and broken_one is None:
|
||||
broken_one = code; broken_err = err
|
||||
if broken_one and fixed_one: break
|
||||
if broken_one is None:
|
||||
continue
|
||||
if fixed_one is not None:
|
||||
# Self-mined repair pair from same-model diverse outputs
|
||||
accumulated_pairs.append({
|
||||
"signature": p["signature"], "tests": p["tests"],
|
||||
"broken": broken_one, "error": broken_err, "fixed": fixed_one,
|
||||
})
|
||||
new_pairs += 1
|
||||
else:
|
||||
# All attempts failed — try one more repair pass with explicit error
|
||||
failures.append({"p": p, "broken": broken_one, "error": broken_err})
|
||||
|
||||
# Optional: try repair on remaining all-failed cases
|
||||
if failures:
|
||||
repair_prompts = [f"Implement: {f['p']['signature']}\n\nTests:\n{chr(10).join(f['p']['tests'])}\n\nMy attempt:\n```python\n{f['broken']}\n```\n\nError:\n{f['error']}\n\nFix and output the corrected code only." for f in failures]
|
||||
repairs = gen_batch(model, tok, repair_prompts, max_new=400, temperature=0.8)
|
||||
for f, raw in zip(failures, repairs):
|
||||
fix = extract_code(raw) if "```" in raw else raw
|
||||
full = fix + "\n\n" + "\n".join(f["p"]["tests"])
|
||||
ok, _ = run_python(full)
|
||||
if ok:
|
||||
accumulated_pairs.append({
|
||||
"signature": f["p"]["signature"], "tests": f["p"]["tests"],
|
||||
"broken": f["broken"], "error": f["error"], "fixed": fix,
|
||||
})
|
||||
new_pairs += 1
|
||||
|
||||
log(f"iter {it}: {len(valid_problems)} valid problems, {len(failures)} failures, {new_pairs} repair pairs harvested (total: {len(accumulated_pairs)}) [{time.time()-it_t:.0f}s]")
|
||||
iter_stats.append({"iter": it, "valid": len(valid_problems), "fails": len(failures), "repairs": new_pairs, "elapsed": time.time()-it_t})
|
||||
|
||||
# Save incrementally (in case of crash)
|
||||
with open(f"{out_dir}/pairs.jsonl", "w") as fh:
|
||||
for r in accumulated_pairs: fh.write(json.dumps(r) + "\n")
|
||||
|
||||
# 6. Periodic training
|
||||
if it % args.train_every == 0 and len(accumulated_pairs) >= 10:
|
||||
log(f" TRAINING on {len(accumulated_pairs)} pairs")
|
||||
tok.padding_side = "right"
|
||||
|
||||
def make_example(r):
|
||||
user = f"Implement: {r['signature']}\n\nTests:\n{chr(10).join(r['tests'])}\n\nMy attempt:\n```python\n{r['broken']}\n```\n\nError:\n{r['error']}\n\nFix and output the corrected code only."
|
||||
assistant = f"```python\n{r['fixed']}\n```"
|
||||
msgs_pre = [{"role": "system", "content": "You are a Python coder."},
|
||||
{"role": "user", "content": user}]
|
||||
msgs_full = msgs_pre + [{"role": "assistant", "content": assistant}]
|
||||
pre = tok.apply_chat_template(msgs_pre, tokenize=False, add_generation_prompt=True)
|
||||
full = tok.apply_chat_template(msgs_full, tokenize=False)
|
||||
pre_ids = tok(pre, add_special_tokens=False)["input_ids"]
|
||||
full_ids = tok(full, add_special_tokens=False)["input_ids"]
|
||||
MAX = 1024
|
||||
full_ids = full_ids[:MAX]
|
||||
labels = list(full_ids)
|
||||
n_pre = min(len(pre_ids), len(labels))
|
||||
for i in range(n_pre): labels[i] = -100
|
||||
pad = MAX - len(full_ids)
|
||||
return {"input_ids": full_ids + [tok.pad_token_id]*pad,
|
||||
"attention_mask": [1]*len(full_ids) + [0]*pad,
|
||||
"labels": labels + [-100]*pad}
|
||||
|
||||
ds = HFDataset.from_list([make_example(r) for r in accumulated_pairs])
|
||||
targs = TrainingArguments(
|
||||
output_dir=f"{out_dir}/ckpt_iter{it}", num_train_epochs=2,
|
||||
per_device_train_batch_size=1, gradient_accumulation_steps=4,
|
||||
learning_rate=1e-4, bf16=True, logging_steps=20,
|
||||
save_strategy="no", report_to="none", remove_unused_columns=False, warmup_ratio=0.05,
|
||||
)
|
||||
Trainer(model=model, args=targs, train_dataset=ds, processing_class=tok).train()
|
||||
tok.padding_side = "left"
|
||||
|
||||
# 7. Periodic eval
|
||||
if it % args.eval_every == 0:
|
||||
model.eval()
|
||||
corr, tot = humaneval_eval(model, tok, n=30)
|
||||
log(f" HumanEval-mini @ iter {it}: {corr}/{tot}")
|
||||
eval_log.append({"iter": it, "correct": corr, "total": tot})
|
||||
model.train()
|
||||
|
||||
# Final eval
|
||||
model.eval()
|
||||
final_correct, final_total = humaneval_eval(model, tok, n=30)
|
||||
eval_log.append({"iter": args.iterations, "correct": final_correct, "total": final_total, "final": True})
|
||||
|
||||
# Save everything
|
||||
with open(f"{out_dir}/iter_stats.jsonl", "w") as fh:
|
||||
for r in iter_stats: fh.write(json.dumps(r) + "\n")
|
||||
with open(f"{out_dir}/eval_log.json", "w") as fh:
|
||||
json.dump(eval_log, fh, indent=2)
|
||||
with open(f"{out_dir}/pairs.jsonl", "w") as fh:
|
||||
for r in accumulated_pairs: fh.write(json.dumps(r) + "\n")
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" MODEL: {args.model}")
|
||||
print(f" ITERATIONS: {args.iterations}, problems/iter: {args.problems_per_iter}")
|
||||
print(f" TOTAL repair pairs: {len(accumulated_pairs)}")
|
||||
print(f" HUMANEVAL-MINI: base={init_correct}/{init_total} final={final_correct}/{final_total} Δ={final_correct-init_correct:+d}")
|
||||
print(f" time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
165
recipe/confirm.py
Normal file
165
recipe/confirm.py
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
"""Confirm the peak +5 result on full HumanEval (164 problems) and try the cliff at 39 pairs."""
|
||||
import os, sys, json, time, re, gc, subprocess, tempfile, argparse
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ.setdefault("CUDA_VISIBLE_DEVICES", "0")
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
|
||||
from datasets import load_dataset, Dataset as HFDataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def extract_code(text):
|
||||
if "```python" in text: text = text.split("```python", 1)[1]
|
||||
elif "```" in text: text = text.split("```", 1)[1]
|
||||
if "```" in text: text = text.split("```", 1)[0]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def run_python(code, timeout=10):
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
|
||||
f.write(code); path = f.name
|
||||
try:
|
||||
r = subprocess.run(["python3", path], capture_output=True, timeout=timeout, text=True, cwd="/tmp")
|
||||
return r.returncode == 0
|
||||
except subprocess.TimeoutExpired: return False
|
||||
finally:
|
||||
try: os.unlink(path)
|
||||
except: pass
|
||||
|
||||
|
||||
def gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4):
|
||||
outs = []
|
||||
for i in range(0, len(prompts), batch):
|
||||
chunk = prompts[i:i+batch]
|
||||
texts = []
|
||||
for p in chunk:
|
||||
msgs = [{"role": "system", "content": "You are a Python coder."},
|
||||
{"role": "user", "content": p}]
|
||||
texts.append(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))
|
||||
inp = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1500).to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new, do_sample=temperature > 0,
|
||||
temperature=temperature if temperature > 0 else 1.0, top_p=0.95,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
for j in range(out.size(0)):
|
||||
outs.append(tok.decode(out[j][inp.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
return outs
|
||||
|
||||
|
||||
def humaneval_full(model, tok):
|
||||
he = list(load_dataset("openai_humaneval", split="test"))
|
||||
log(f" full HumanEval: {len(he)} problems")
|
||||
prompts = [p["prompt"] + "\n# Complete the function above." for p in he]
|
||||
outs = gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4)
|
||||
correct = 0
|
||||
for p, raw in zip(he, outs):
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
full = p["prompt"] + "\n" + code if "def " not in code else code
|
||||
test_code = full + "\n\n" + p["test"] + f"\n\ncheck({p['entry_point']})"
|
||||
if run_python(test_code, timeout=10): correct += 1
|
||||
return correct, len(he)
|
||||
|
||||
|
||||
def make_example(r, tok):
|
||||
user = f"Implement: {r['signature']}\n\nTests:\n{chr(10).join(r['tests'])}\n\nMy attempt:\n```python\n{r['broken']}\n```\n\nError:\n{r['error']}\n\nFix and output the corrected code only."
|
||||
assistant = f"```python\n{r['fixed']}\n```"
|
||||
msgs_pre = [{"role": "system", "content": "You are a Python coder."},
|
||||
{"role": "user", "content": user}]
|
||||
msgs_full = msgs_pre + [{"role": "assistant", "content": assistant}]
|
||||
pre = tok.apply_chat_template(msgs_pre, tokenize=False, add_generation_prompt=True)
|
||||
full = tok.apply_chat_template(msgs_full, tokenize=False)
|
||||
pre_ids = tok(pre, add_special_tokens=False)["input_ids"]
|
||||
full_ids = tok(full, add_special_tokens=False)["input_ids"]
|
||||
MAX = 1024
|
||||
full_ids = full_ids[:MAX]
|
||||
labels = list(full_ids)
|
||||
n_pre = min(len(pre_ids), len(labels))
|
||||
for i in range(n_pre): labels[i] = -100
|
||||
pad = MAX - len(full_ids)
|
||||
return {"input_ids": full_ids + [tok.pad_token_id]*pad,
|
||||
"attention_mask": [1]*len(full_ids) + [0]*pad,
|
||||
"labels": labels + [-100]*pad}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--n_pairs", type=int, default=21, help="how many pairs from the saved set to train on")
|
||||
ap.add_argument("--epochs", type=int, default=2)
|
||||
ap.add_argument("--seed", type=int, default=42)
|
||||
ap.add_argument("--tag", required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
torch.manual_seed(args.seed)
|
||||
|
||||
pairs_path = "/workspace/bootstrap/bs_7b_v3/pairs.jsonl"
|
||||
pairs = [json.loads(l) for l in open(pairs_path)]
|
||||
log(f"loaded {len(pairs)} pairs from prior bootstrap run")
|
||||
pairs_use = pairs[:args.n_pairs]
|
||||
log(f"using {len(pairs_use)} for this run")
|
||||
|
||||
out_dir = f"/workspace/confirm/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
log("loading Qwen/Qwen2.5-7B")
|
||||
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
tok.padding_side = "left"
|
||||
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", dtype=torch.bfloat16, device_map="cuda:0")
|
||||
|
||||
# Eval base
|
||||
model.eval()
|
||||
log("eval BASE on full HumanEval")
|
||||
base_corr, base_total = humaneval_full(model, tok)
|
||||
log(f" BASE: {base_corr}/{base_total}")
|
||||
|
||||
# Apply LoRA + train
|
||||
lora_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
|
||||
model = get_peft_model(model, lora_cfg)
|
||||
log("LoRA applied")
|
||||
|
||||
tok.padding_side = "right"
|
||||
examples = [make_example(r, tok) for r in pairs_use]
|
||||
ds = HFDataset.from_list(examples)
|
||||
targs = TrainingArguments(
|
||||
output_dir=f"{out_dir}/ckpt", num_train_epochs=args.epochs,
|
||||
per_device_train_batch_size=1, gradient_accumulation_steps=4,
|
||||
learning_rate=1e-4, bf16=True, logging_steps=10,
|
||||
save_strategy="no", report_to="none", remove_unused_columns=False, warmup_ratio=0.05,
|
||||
seed=args.seed,
|
||||
)
|
||||
log(f"training on {len(ds)} pairs, {args.epochs} epochs")
|
||||
Trainer(model=model, args=targs, train_dataset=ds, processing_class=tok).train()
|
||||
log("training done")
|
||||
tok.padding_side = "left"
|
||||
|
||||
# Eval trained
|
||||
model.eval()
|
||||
log("eval TRAINED on full HumanEval")
|
||||
tr_corr, tr_total = humaneval_full(model, tok)
|
||||
log(f" TRAINED: {tr_corr}/{tr_total}")
|
||||
|
||||
result = {
|
||||
"n_pairs_used": len(pairs_use), "epochs": args.epochs, "seed": args.seed,
|
||||
"base": [base_corr, base_total], "trained": [tr_corr, tr_total],
|
||||
"delta": tr_corr - base_corr,
|
||||
"elapsed_s": time.time() - T0,
|
||||
}
|
||||
with open(f"{out_dir}/result.json", "w") as fh:
|
||||
json.dump(result, fh, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" N_PAIRS: {len(pairs_use)} EPOCHS: {args.epochs} SEED: {args.seed}")
|
||||
print(f" HUMAN-EVAL FULL: base={base_corr}/{base_total} trained={tr_corr}/{tr_total} Δ={tr_corr-base_corr:+d}")
|
||||
print(f" time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
310
recipe/curriculum_math.py
Normal file
310
recipe/curriculum_math.py
Normal file
|
|
@ -0,0 +1,310 @@
|
|||
"""TinyForge-Zero-Math with self-difficulty curriculum.
|
||||
|
||||
Novel: model + interpreter only. No external problem set, no fixed difficulty.
|
||||
The model's own greedy success/failure on each problem tells the curriculum
|
||||
to make it harder or easier. Mine pairs only at the edge of competence.
|
||||
|
||||
Loop per iter:
|
||||
1. Generate K problems at current difficulty pool
|
||||
2. For each: solve greedily (temp=0). Verify against canonical answer.
|
||||
- If correct: this problem is "easy" → ask model to amplify
|
||||
- If wrong: try N=4 sampled attempts at temp=0.8
|
||||
- If at-edge (some pass, some fail): MINE a pair
|
||||
- If all fail: this problem is "too hard" → ask model to simplify
|
||||
3. Add amplified/simplified problems back into the pool for next iter
|
||||
4. Train on accumulated pairs periodically
|
||||
"""
|
||||
import os, sys, json, time, re, gc, argparse, random
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ.setdefault("CUDA_VISIBLE_DEVICES", "1")
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
|
||||
from datasets import load_dataset, Dataset as HFDataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def safe_eval(expr: str):
|
||||
try:
|
||||
if not all(c in "0123456789+-*/.()% " for c in expr): return None
|
||||
return float(eval(expr, {"__builtins__": {}}, {}))
|
||||
except: return None
|
||||
|
||||
|
||||
def extract_answer(text: str):
|
||||
m = re.search(r"####\s*(-?\d+(?:\.\d+)?)", text)
|
||||
if m: return float(m.group(1))
|
||||
m = re.search(r"\\boxed\{(-?\d+(?:\.\d+)?)\}", text)
|
||||
if m: return float(m.group(1))
|
||||
matches = re.findall(r"-?\d+(?:\.\d+)?", text)
|
||||
if matches:
|
||||
try: return float(matches[-1])
|
||||
except: return None
|
||||
return None
|
||||
|
||||
|
||||
def gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=8):
|
||||
outs = []
|
||||
for i in range(0, len(prompts), batch):
|
||||
chunk = prompts[i:i+batch]
|
||||
texts = []
|
||||
for p in chunk:
|
||||
msgs = [{"role": "system", "content": "You are a careful math tutor."},
|
||||
{"role": "user", "content": p}]
|
||||
texts.append(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))
|
||||
inp = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1500).to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new, do_sample=temperature > 0,
|
||||
temperature=temperature if temperature > 0 else 1.0, top_p=0.95,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
for j in range(out.size(0)):
|
||||
outs.append(tok.decode(out[j][inp.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
return outs
|
||||
|
||||
|
||||
SOLVE_PROMPT = "Solve this math problem step by step. End with the answer on a new line as: #### <number>\n\nProblem: {problem}"
|
||||
|
||||
GEN_PROMPT_SEED = """Generate ONE math word problem with a numerical answer. Output exactly:
|
||||
|
||||
PROBLEM: <a clear word problem with concrete numbers>
|
||||
EXPRESSION: <a single Python arithmetic expression that evaluates to the answer>
|
||||
ANSWER: <the numerical answer>
|
||||
|
||||
Make problems grade-school level."""
|
||||
|
||||
AMPLIFY_PROMPT = """Take this math problem and make it HARDER by adding ONE more step (e.g., another operation, a percentage, fractions, or an extra constraint). Keep the format:
|
||||
|
||||
Original problem: {problem}
|
||||
Original answer: {answer}
|
||||
|
||||
Output exactly:
|
||||
PROBLEM: <the harder problem>
|
||||
EXPRESSION: <Python arithmetic expression for the new answer>
|
||||
ANSWER: <the new numerical answer>"""
|
||||
|
||||
SIMPLIFY_PROMPT = """Take this math problem and make it EASIER by removing one step or simplifying numbers. Keep the format:
|
||||
|
||||
Original problem: {problem}
|
||||
Original answer: {answer}
|
||||
|
||||
Output exactly:
|
||||
PROBLEM: <the easier problem>
|
||||
EXPRESSION: <Python arithmetic expression for the new answer>
|
||||
ANSWER: <the new numerical answer>"""
|
||||
|
||||
|
||||
def parse_problem(text: str):
|
||||
p_m = re.search(r"PROBLEM:\s*(.+?)(?:\n|EXPRESSION:)", text, re.DOTALL)
|
||||
e_m = re.search(r"EXPRESSION:\s*(.+?)(?:\n|ANSWER:)", text, re.DOTALL)
|
||||
a_m = re.search(r"ANSWER:\s*(-?\d+(?:\.\d+)?)", text)
|
||||
if not (p_m and e_m and a_m): return None
|
||||
problem = p_m.group(1).strip()
|
||||
expression = e_m.group(1).strip()
|
||||
try: claimed = float(a_m.group(1))
|
||||
except: return None
|
||||
if len(problem) < 10: return None
|
||||
actual = safe_eval(expression)
|
||||
if actual is None or abs(actual - claimed) > 0.01: return None
|
||||
return {"problem": problem, "answer": claimed}
|
||||
|
||||
|
||||
def parse_gold(answer_field: str):
|
||||
m = re.search(r"####\s*(-?\d+(?:,\d+)*(?:\.\d+)?)", answer_field)
|
||||
return float(m.group(1).replace(",", "")) if m else None
|
||||
|
||||
|
||||
def gsm8k_eval(model, tok, n=50):
|
||||
ds = list(load_dataset("openai/gsm8k", "main", split="test"))[:n]
|
||||
log(f" eval on GSM8K-test ({len(ds)} problems)")
|
||||
prompts = [SOLVE_PROMPT.format(problem=p["question"]) for p in ds]
|
||||
outs = gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=8)
|
||||
correct = 0
|
||||
for p, raw in zip(ds, outs):
|
||||
gold = parse_gold(p["answer"])
|
||||
if gold is None: continue
|
||||
pred = extract_answer(raw)
|
||||
if pred is not None and abs(pred - gold) < 0.01: correct += 1
|
||||
return correct, len(ds)
|
||||
|
||||
|
||||
def make_train_example(problem: str, solution: str, tok):
|
||||
user = SOLVE_PROMPT.format(problem=problem)
|
||||
msgs_pre = [{"role": "system", "content": "You are a careful math tutor."},
|
||||
{"role": "user", "content": user}]
|
||||
msgs_full = msgs_pre + [{"role": "assistant", "content": solution}]
|
||||
pre = tok.apply_chat_template(msgs_pre, tokenize=False, add_generation_prompt=True)
|
||||
full = tok.apply_chat_template(msgs_full, tokenize=False)
|
||||
pre_ids = tok(pre, add_special_tokens=False)["input_ids"]
|
||||
full_ids = tok(full, add_special_tokens=False)["input_ids"]
|
||||
MAX = 1024
|
||||
full_ids = full_ids[:MAX]
|
||||
labels = list(full_ids)
|
||||
n_pre = min(len(pre_ids), len(labels))
|
||||
for i in range(n_pre): labels[i] = -100
|
||||
pad = MAX - len(full_ids)
|
||||
return {"input_ids": full_ids + [tok.pad_token_id]*pad,
|
||||
"attention_mask": [1]*len(full_ids) + [0]*pad,
|
||||
"labels": labels + [-100]*pad}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", default="Qwen/Qwen2.5-3B")
|
||||
ap.add_argument("--iterations", type=int, default=8)
|
||||
ap.add_argument("--problems_per_iter", type=int, default=8)
|
||||
ap.add_argument("--train_every", type=int, default=4)
|
||||
ap.add_argument("--n_eval", type=int, default=50)
|
||||
ap.add_argument("--seed", type=int, default=42)
|
||||
ap.add_argument("--tag", required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
random.seed(args.seed); torch.manual_seed(args.seed)
|
||||
out_dir = f"/workspace/curriculum/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
log(f"loading {args.model}")
|
||||
tok = AutoTokenizer.from_pretrained(args.model)
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
tok.padding_side = "left"
|
||||
model = AutoModelForCausalLM.from_pretrained(args.model, dtype=torch.bfloat16, device_map="cuda:0")
|
||||
log(f" loaded mem={torch.cuda.memory_allocated('cuda:0')/1e9:.1f}GB")
|
||||
|
||||
model.eval()
|
||||
log("INITIAL eval on GSM8K-test")
|
||||
base_correct, base_total = gsm8k_eval(model, tok, n=args.n_eval)
|
||||
log(f" GSM8K-test base: {base_correct}/{base_total}")
|
||||
|
||||
lora_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
|
||||
model = get_peft_model(model, lora_cfg)
|
||||
|
||||
accumulated_pairs = [] # at-edge (problem, correct_solution)
|
||||
problem_pool = [] # current pool of problems for next iter
|
||||
|
||||
for it in range(1, args.iterations + 1):
|
||||
it_t = time.time()
|
||||
# 1. Generate problems if pool is empty (seed)
|
||||
if not problem_pool or it == 1:
|
||||
gen_prompts = [GEN_PROMPT_SEED for _ in range(args.problems_per_iter)]
|
||||
raw = gen_batch(model, tok, gen_prompts, max_new=300, temperature=0.9)
|
||||
seeded = [parse_problem(r) for r in raw]
|
||||
seeded = [s for s in seeded if s]
|
||||
problem_pool.extend(seeded)
|
||||
log(f"iter {it}: seeded {len(seeded)} fresh problems (pool={len(problem_pool)})")
|
||||
|
||||
# 2. Pick K problems to attempt
|
||||
random.shuffle(problem_pool)
|
||||
attempt_problems = problem_pool[:args.problems_per_iter]
|
||||
problem_pool = problem_pool[args.problems_per_iter:] # consume
|
||||
|
||||
if not attempt_problems:
|
||||
log(f"iter {it}: empty pool, regenerating"); continue
|
||||
|
||||
# 3. Greedy solve to assess difficulty
|
||||
greedy_prompts = [SOLVE_PROMPT.format(problem=p["problem"]) for p in attempt_problems]
|
||||
greedy_outs = gen_batch(model, tok, greedy_prompts, max_new=300, temperature=0.0)
|
||||
greedy_correct = []
|
||||
for p, raw in zip(attempt_problems, greedy_outs):
|
||||
pred = extract_answer(raw)
|
||||
ok = pred is not None and abs(pred - p["answer"]) < 0.01
|
||||
greedy_correct.append(ok)
|
||||
|
||||
n_easy = sum(greedy_correct)
|
||||
log(f"iter {it}: {n_easy}/{len(attempt_problems)} solved greedily")
|
||||
|
||||
new_pairs = 0
|
||||
amplify_targets = []
|
||||
simplify_targets = []
|
||||
for p, easy in zip(attempt_problems, greedy_correct):
|
||||
if easy:
|
||||
# too easy → amplify next round
|
||||
amplify_targets.append(p)
|
||||
else:
|
||||
# try sampled attempts to find at-edge
|
||||
solve_prompts = [SOLVE_PROMPT.format(problem=p["problem"])] * 4
|
||||
atts = gen_batch(model, tok, solve_prompts, max_new=300, temperature=0.8)
|
||||
ok_atts = []
|
||||
for raw in atts:
|
||||
pred = extract_answer(raw)
|
||||
if pred is not None and abs(pred - p["answer"]) < 0.01:
|
||||
ok_atts.append(raw.strip())
|
||||
if ok_atts:
|
||||
# at-edge → mine pair
|
||||
accumulated_pairs.append({"problem": p["problem"], "solution": ok_atts[0],
|
||||
"answer": p["answer"]})
|
||||
new_pairs += 1
|
||||
else:
|
||||
# too hard → simplify
|
||||
simplify_targets.append(p)
|
||||
|
||||
log(f"iter {it}: +{new_pairs} pairs (total: {len(accumulated_pairs)}). "
|
||||
f"amplify={len(amplify_targets)}, simplify={len(simplify_targets)}")
|
||||
|
||||
# 4. Generate amplified/simplified versions for next iter
|
||||
if amplify_targets:
|
||||
amp_prompts = [AMPLIFY_PROMPT.format(problem=p["problem"], answer=p["answer"]) for p in amplify_targets[:args.problems_per_iter]]
|
||||
amp_outs = gen_batch(model, tok, amp_prompts, max_new=300, temperature=0.7)
|
||||
for raw in amp_outs:
|
||||
np = parse_problem(raw)
|
||||
if np: problem_pool.append(np)
|
||||
if simplify_targets:
|
||||
sim_prompts = [SIMPLIFY_PROMPT.format(problem=p["problem"], answer=p["answer"]) for p in simplify_targets[:args.problems_per_iter // 2]]
|
||||
sim_outs = gen_batch(model, tok, sim_prompts, max_new=300, temperature=0.7)
|
||||
for raw in sim_outs:
|
||||
np = parse_problem(raw)
|
||||
if np: problem_pool.append(np)
|
||||
|
||||
with open(f"{out_dir}/pairs.jsonl", "w") as fh:
|
||||
for r in accumulated_pairs: fh.write(json.dumps(r) + "\n")
|
||||
|
||||
log(f"iter {it} done [{time.time()-it_t:.0f}s]; pool size now {len(problem_pool)}")
|
||||
|
||||
# 5. Train every N
|
||||
if it % args.train_every == 0 and len(accumulated_pairs) >= 5:
|
||||
log(f" TRAINING on {len(accumulated_pairs)} pairs")
|
||||
tok.padding_side = "right"
|
||||
ds = HFDataset.from_list([make_train_example(r["problem"], r["solution"], tok) for r in accumulated_pairs])
|
||||
targs = TrainingArguments(
|
||||
output_dir=f"{out_dir}/ckpt", num_train_epochs=2,
|
||||
per_device_train_batch_size=1, gradient_accumulation_steps=4,
|
||||
learning_rate=1e-4, bf16=True, logging_steps=10,
|
||||
save_strategy="no", report_to="none", remove_unused_columns=False, warmup_ratio=0.05,
|
||||
)
|
||||
Trainer(model=model, args=targs, train_dataset=ds, processing_class=tok).train()
|
||||
tok.padding_side = "left"
|
||||
model.eval()
|
||||
corr, tot = gsm8k_eval(model, tok, n=args.n_eval)
|
||||
log(f" GSM8K-test @ iter {it}: {corr}/{tot}")
|
||||
model.train()
|
||||
|
||||
# Final eval
|
||||
model.eval()
|
||||
final_correct, final_total = gsm8k_eval(model, tok, n=args.n_eval)
|
||||
|
||||
result = {
|
||||
"model": args.model, "iterations": args.iterations,
|
||||
"n_pairs": len(accumulated_pairs),
|
||||
"base": [base_correct, base_total],
|
||||
"trained": [final_correct, final_total],
|
||||
"delta": final_correct - base_correct,
|
||||
"elapsed_s": time.time() - T0,
|
||||
}
|
||||
with open(f"{out_dir}/result.json", "w") as fh:
|
||||
json.dump(result, fh, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" CURRICULUM TINYFORGE-ZERO-MATH — {args.model}")
|
||||
print(f" Self-mined pairs: {len(accumulated_pairs)}")
|
||||
print(f" GSM8K-test: base={base_correct}/{base_total} trained={final_correct}/{final_total} Δ={final_correct-base_correct:+d}")
|
||||
print(f" Time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
115
recipe/eval_plus.py
Normal file
115
recipe/eval_plus.py
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
"""Eval our best 14B adapter on HumanEval+ (contamination-resistant hidden tests)."""
|
||||
import os, json, time, re, subprocess, tempfile, argparse
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from datasets import load_dataset
|
||||
from peft import PeftModel
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def extract_code(text):
|
||||
if "```python" in text: text = text.split("```python", 1)[1]
|
||||
elif "```" in text: text = text.split("```", 1)[1]
|
||||
if "```" in text: text = text.split("```", 1)[0]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def run_python(code, timeout=15):
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
|
||||
f.write(code); path = f.name
|
||||
try:
|
||||
r = subprocess.run(["python3", path], capture_output=True, timeout=timeout, text=True, cwd="/tmp")
|
||||
return r.returncode == 0
|
||||
except subprocess.TimeoutExpired: return False
|
||||
finally:
|
||||
try: os.unlink(path)
|
||||
except: pass
|
||||
|
||||
|
||||
def gen_batch(model, tok, prompts, max_new=400, batch=4):
|
||||
outs = []
|
||||
for i in range(0, len(prompts), batch):
|
||||
chunk = prompts[i:i+batch]
|
||||
texts = []
|
||||
for p in chunk:
|
||||
msgs = [{"role": "system", "content": "You are a Python coder. Output one ```python block only."},
|
||||
{"role": "user", "content": p}]
|
||||
texts.append(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))
|
||||
inp = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1500).to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new, do_sample=False, pad_token_id=tok.eos_token_id)
|
||||
for j in range(out.size(0)):
|
||||
outs.append(tok.decode(out[j][inp.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
return outs
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", default="Qwen/Qwen2.5-14B")
|
||||
ap.add_argument("--adapter", default="/workspace/multi_pair/multi_v1/adapter")
|
||||
ap.add_argument("--tag", required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
out_dir = f"/workspace/eval_plus/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
log(f"loading {args.model}")
|
||||
tok = AutoTokenizer.from_pretrained(args.model)
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
tok.padding_side = "left"
|
||||
model = AutoModelForCausalLM.from_pretrained(args.model, torch_dtype=torch.bfloat16, device_map="cuda:0")
|
||||
if args.adapter and os.path.exists(args.adapter):
|
||||
log(f" loading adapter from {args.adapter}")
|
||||
model = PeftModel.from_pretrained(model, args.adapter)
|
||||
else:
|
||||
log(" no adapter — base only")
|
||||
model.eval()
|
||||
|
||||
# Load HumanEval+ via evalplus dataset
|
||||
log("loading HumanEvalPlus dataset")
|
||||
ds = list(load_dataset("evalplus/humanevalplus", split="test"))
|
||||
log(f" {len(ds)} problems")
|
||||
|
||||
# Eval
|
||||
log("eval...")
|
||||
prompts = [p["prompt"] + "\n# Complete the function above." for p in ds]
|
||||
outs = gen_batch(model, tok, prompts, max_new=400, batch=4)
|
||||
|
||||
base_pass, plus_pass = 0, 0
|
||||
for i, (p, raw) in enumerate(zip(ds, outs)):
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
full = p["prompt"] + "\n" + code if "def " not in code else code
|
||||
# Public tests
|
||||
base_test = full + "\n\n" + p["test"] + f"\n\ncheck({p['entry_point']})"
|
||||
b = run_python(base_test, timeout=15)
|
||||
# Plus tests (hidden harder)
|
||||
plus_check = p.get("plus_input", None)
|
||||
if plus_check is not None and "plus_test" in p:
|
||||
plus_test = full + "\n\n" + p["plus_test"] + f"\n\ncheck({p['entry_point']})"
|
||||
pp = run_python(plus_test, timeout=15)
|
||||
else:
|
||||
pp = b # fallback
|
||||
if b: base_pass += 1
|
||||
if pp: plus_pass += 1
|
||||
if (i+1) % 20 == 0:
|
||||
log(f" {i+1}/{len(ds)}: base={base_pass}, plus={plus_pass}")
|
||||
|
||||
result = {"model": args.model, "adapter": args.adapter,
|
||||
"base_pass": base_pass, "plus_pass": plus_pass, "n": len(ds),
|
||||
"elapsed_s": time.time() - T0}
|
||||
with open(f"{out_dir}/result.json", "w") as fh: json.dump(result, fh, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" HumanEval+ public: {base_pass}/{len(ds)} plus(hidden): {plus_pass}/{len(ds)}")
|
||||
print(f" Time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
216
recipe/eval_raw.py
Normal file
216
recipe/eval_raw.py
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
"""vLLM dual eval using RAW completion format (no chat template) for base models.
|
||||
|
||||
Recipe for non-instruct base models — uses simple completion-style prompting
|
||||
that matches how base models were pretrained.
|
||||
"""
|
||||
import os, json, time, re, subprocess, tempfile, argparse, gc
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def extract_code(text):
|
||||
if "```python" in text: text = text.split("```python", 1)[1]
|
||||
elif "```" in text: text = text.split("```", 1)[1]
|
||||
if "```" in text: text = text.split("```", 1)[0]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def run_python(code, timeout=10):
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
|
||||
f.write(code); path = f.name
|
||||
try:
|
||||
r = subprocess.run(["python3", path], capture_output=True, timeout=timeout, text=True, cwd="/tmp")
|
||||
return r.returncode == 0
|
||||
except subprocess.TimeoutExpired: return False
|
||||
finally:
|
||||
try: os.unlink(path)
|
||||
except: pass
|
||||
|
||||
|
||||
def make_he_prompt(p):
|
||||
"""Raw completion: just the docstring + 'def'."""
|
||||
return p["prompt"]
|
||||
|
||||
|
||||
def make_mbpp_prompt(p):
|
||||
"""Raw completion: docstring + tests + 'def'."""
|
||||
return (f"# Task: {p['prompt']}\n"
|
||||
f"# Tests:\n# " + "\n# ".join(p["test_list"]) + "\n\n")
|
||||
|
||||
|
||||
def vllm_generate(llm, prompts, max_new=400, temperature=0.0, stops=None):
|
||||
from vllm import SamplingParams
|
||||
sp = SamplingParams(
|
||||
temperature=temperature, top_p=0.95 if temperature > 0 else 1.0,
|
||||
max_tokens=max_new, stop=stops or ["\nclass ", "\nif __name__", "\nprint(", "\n#"],
|
||||
)
|
||||
out = llm.generate(prompts, sp, use_tqdm=False)
|
||||
return [o.outputs[0].text for o in out]
|
||||
|
||||
|
||||
def vllm_generate_lora(llm, prompts, lora_req, max_new=400, temperature=0.0, stops=None):
|
||||
from vllm import SamplingParams
|
||||
sp = SamplingParams(
|
||||
temperature=temperature, top_p=0.95 if temperature > 0 else 1.0,
|
||||
max_tokens=max_new, stop=stops or ["\nclass ", "\nif __name__", "\nprint(", "\n#"],
|
||||
)
|
||||
out = llm.generate(prompts, sp, lora_request=lora_req, use_tqdm=False)
|
||||
return [o.outputs[0].text for o in out]
|
||||
|
||||
|
||||
def eval_humaneval(outs_func, label):
|
||||
he = list(load_dataset("openai_humaneval", split="test"))
|
||||
log(f" HumanEval [{label}] ({len(he)})")
|
||||
prompts = [make_he_prompt(p) for p in he]
|
||||
t0 = time.time()
|
||||
outs = outs_func(prompts, max_new=400)
|
||||
log(f" gen done in {time.time()-t0:.1f}s")
|
||||
correct = 0
|
||||
for p, raw in zip(he, outs):
|
||||
# construct full function: prompt + raw completion
|
||||
full = p["prompt"] + raw
|
||||
test_code = full + "\n\n" + p["test"] + f"\n\ncheck({p['entry_point']})"
|
||||
if run_python(test_code, timeout=10): correct += 1
|
||||
return correct, len(he)
|
||||
|
||||
|
||||
def eval_mbpp(outs_func, label, n=200):
|
||||
mbpp = list(load_dataset("mbpp", "sanitized", split="test"))[:n]
|
||||
log(f" MBPP [{label}] ({len(mbpp)})")
|
||||
prompts = [make_mbpp_prompt(p) for p in mbpp]
|
||||
t0 = time.time()
|
||||
outs = outs_func(prompts, max_new=400)
|
||||
log(f" gen done in {time.time()-t0:.1f}s")
|
||||
correct = 0
|
||||
for p, raw in zip(mbpp, outs):
|
||||
# raw is the function code
|
||||
code = raw
|
||||
if "```" in code:
|
||||
code = extract_code("```python" + code if "```python" not in code else code)
|
||||
test_code = code + "\n\n" + "\n".join(p["test_list"])
|
||||
if run_python(test_code, timeout=10): correct += 1
|
||||
return correct, len(mbpp)
|
||||
|
||||
|
||||
def make_train_example(r, tok):
|
||||
"""Raw-completion training format."""
|
||||
sig = r.get("signature", "")
|
||||
broken = r.get("broken", "")
|
||||
fixed = r.get("fixed", "")
|
||||
tests = r.get("tests", [])
|
||||
err = r.get("error", "")
|
||||
user = (f"# Task: implement {sig}\n"
|
||||
f"# Tests:\n# " + "\n# ".join(tests) + "\n"
|
||||
f"# My broken attempt:\n{broken}\n"
|
||||
f"# Error: {err}\n"
|
||||
f"# Corrected:\n")
|
||||
target = fixed
|
||||
full = user + target
|
||||
full_ids = tok(full, add_special_tokens=False)["input_ids"]
|
||||
user_ids = tok(user, add_special_tokens=False)["input_ids"]
|
||||
MAX = 1024
|
||||
full_ids = full_ids[:MAX]
|
||||
labels = list(full_ids)
|
||||
n_user = min(len(user_ids), len(labels))
|
||||
for i in range(n_user): labels[i] = -100
|
||||
pad = MAX - len(full_ids)
|
||||
return {"input_ids": full_ids + [tok.pad_token_id]*pad,
|
||||
"attention_mask": [1]*len(full_ids) + [0]*pad,
|
||||
"labels": labels + [-100]*pad}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", required=True)
|
||||
ap.add_argument("--pairs", default="/workspace/saved_pairs/pairs_40.jsonl")
|
||||
ap.add_argument("--n_pairs", type=int, default=40)
|
||||
ap.add_argument("--mbpp_n", type=int, default=200)
|
||||
ap.add_argument("--tag", required=True)
|
||||
ap.add_argument("--skip_train", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
out_dir = f"/workspace/dual_eval_raw/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
from vllm import LLM
|
||||
from transformers import AutoTokenizer
|
||||
log(f"loading {args.model} into vLLM")
|
||||
tok = AutoTokenizer.from_pretrained(args.model)
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
llm = LLM(model=args.model, dtype="bfloat16", gpu_memory_utilization=0.85, max_model_len=2048)
|
||||
log(f" loaded")
|
||||
|
||||
log("=== BASE evals ===")
|
||||
base_he, _ = eval_humaneval(lambda P, max_new=400: vllm_generate(llm, P, max_new=max_new), "BASE")
|
||||
base_mbpp, _ = eval_mbpp(lambda P, max_new=400: vllm_generate(llm, P, max_new=max_new), "BASE", n=args.mbpp_n)
|
||||
log(f" BASE: HumanEval={base_he}/164 MBPP={base_mbpp}/{args.mbpp_n}")
|
||||
|
||||
if args.skip_train:
|
||||
result = {"model": args.model, "base_humaneval": base_he, "base_mbpp": base_mbpp, "n_he": 164, "n_mbpp": args.mbpp_n, "elapsed_s": time.time()-T0}
|
||||
with open(f"{out_dir}/result.json", "w") as fh: json.dump(result, fh, indent=2)
|
||||
return
|
||||
|
||||
# Tear down vLLM, train LoRA
|
||||
log("=== TRAINING ===")
|
||||
del llm; gc.collect(); torch.cuda.empty_cache()
|
||||
|
||||
from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
|
||||
from datasets import Dataset as HFDataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
pairs = [json.loads(l) for l in open(args.pairs)][:args.n_pairs]
|
||||
model = AutoModelForCausalLM.from_pretrained(args.model, torch_dtype=torch.bfloat16, device_map="cuda:0")
|
||||
lora_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
|
||||
model = get_peft_model(model, lora_cfg)
|
||||
|
||||
ds = HFDataset.from_list([make_train_example(r, tok) for r in pairs])
|
||||
targs = TrainingArguments(
|
||||
output_dir=f"{out_dir}/ckpt", num_train_epochs=2,
|
||||
per_device_train_batch_size=1, gradient_accumulation_steps=4,
|
||||
learning_rate=1e-4, bf16=True, logging_steps=10,
|
||||
save_strategy="no", report_to="none", remove_unused_columns=False, warmup_ratio=0.05,
|
||||
)
|
||||
Trainer(model=model, args=targs, train_dataset=ds, tokenizer=tok).train()
|
||||
log("training done")
|
||||
|
||||
adapter_dir = f"{out_dir}/adapter"
|
||||
model.save_pretrained(adapter_dir)
|
||||
del model; gc.collect(); torch.cuda.empty_cache()
|
||||
|
||||
from vllm import LLM
|
||||
from vllm.lora.request import LoRARequest
|
||||
llm = LLM(model=args.model, dtype="bfloat16", gpu_memory_utilization=0.85, max_model_len=2048,
|
||||
enable_lora=True, max_lora_rank=16)
|
||||
lora_req = LoRARequest("tf_adapter", 1, adapter_dir)
|
||||
|
||||
log("=== TRAINED evals (vLLM + LoRA) ===")
|
||||
tr_he, _ = eval_humaneval(lambda P, max_new=400: vllm_generate_lora(llm, P, lora_req, max_new=max_new), "TRAINED")
|
||||
tr_mbpp, _ = eval_mbpp(lambda P, max_new=400: vllm_generate_lora(llm, P, lora_req, max_new=max_new), "TRAINED", n=args.mbpp_n)
|
||||
|
||||
result = {
|
||||
"model": args.model, "n_pairs": len(pairs),
|
||||
"humaneval": {"base": base_he, "trained": tr_he, "delta": tr_he-base_he, "n": 164},
|
||||
"mbpp": {"base": base_mbpp, "trained": tr_mbpp, "delta": tr_mbpp-base_mbpp, "n": args.mbpp_n},
|
||||
"elapsed_s": time.time() - T0,
|
||||
}
|
||||
with open(f"{out_dir}/result.json", "w") as fh: json.dump(result, fh, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" {args.model} — RAW completion format")
|
||||
print(f" HumanEval: base={base_he}/164 trained={tr_he}/164 Δ={tr_he-base_he:+d}")
|
||||
print(f" MBPP: base={base_mbpp}/{args.mbpp_n} trained={tr_mbpp}/{args.mbpp_n} Δ={tr_mbpp-base_mbpp:+d}")
|
||||
print(f" Time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
328
recipe/multi_pair_14b.py
Normal file
328
recipe/multi_pair_14b.py
Normal file
|
|
@ -0,0 +1,328 @@
|
|||
"""Aggressive multi-pair mining on Qwen2.5-14B-Base.
|
||||
|
||||
Differences from warmup recipe:
|
||||
- Harder problem-generation prompt (edge cases, multi-step, tricky boundaries)
|
||||
- 200 problems generated (vs 80)
|
||||
- 8 sampled attempts per problem at temp 0.8 (vs 4)
|
||||
- Mine ALL (broken, fixed) pairs per problem, not just 1
|
||||
- Deduplicate near-identical broken code (Jaccard < 0.85)
|
||||
- Larger LoRA: rank 32 attn-only
|
||||
- Train fresh from base on combined (warmup_40 + new) pairs
|
||||
"""
|
||||
import os, sys, json, time, re, gc, subprocess, tempfile, argparse, random, hashlib
|
||||
os.environ.setdefault("HF_HOME", "/workspace/hf")
|
||||
os.environ.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")
|
||||
os.environ["TRANSFORMERS_VERBOSITY"] = "error"
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
|
||||
from datasets import load_dataset, Dataset as HFDataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
def extract_code(text):
|
||||
if "```python" in text: text = text.split("```python", 1)[1]
|
||||
elif "```" in text: text = text.split("```", 1)[1]
|
||||
if "```" in text: text = text.split("```", 1)[0]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def run_python(code, timeout=10):
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
|
||||
f.write(code); path = f.name
|
||||
try:
|
||||
r = subprocess.run(["python3", path], capture_output=True, timeout=timeout, text=True, cwd="/tmp")
|
||||
if r.returncode == 0: return True, ""
|
||||
err = (r.stderr or r.stdout).strip().splitlines()
|
||||
return False, "\n".join(err[-3:])[:300]
|
||||
except subprocess.TimeoutExpired: return False, "timeout"
|
||||
finally:
|
||||
try: os.unlink(path)
|
||||
except: pass
|
||||
|
||||
|
||||
def gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4):
|
||||
outs = []
|
||||
for i in range(0, len(prompts), batch):
|
||||
chunk = prompts[i:i+batch]
|
||||
texts = []
|
||||
for p in chunk:
|
||||
msgs = [{"role": "system", "content": "You are an expert Python coder. Output one ```python block only."},
|
||||
{"role": "user", "content": p}]
|
||||
texts.append(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))
|
||||
inp = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1500).to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new, do_sample=temperature > 0,
|
||||
temperature=temperature if temperature > 0 else 1.0, top_p=0.95,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
for j in range(out.size(0)):
|
||||
outs.append(tok.decode(out[j][inp.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
return outs
|
||||
|
||||
|
||||
def humaneval_full(model, tok):
|
||||
he = list(load_dataset("openai_humaneval", split="test"))
|
||||
log(f" HumanEval ({len(he)} problems)")
|
||||
prompts = [p["prompt"] + "\n# Complete the function above." for p in he]
|
||||
outs = gen_batch(model, tok, prompts, max_new=400, temperature=0.0, batch=4)
|
||||
correct = 0
|
||||
for i, (p, raw) in enumerate(zip(he, outs)):
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
full = p["prompt"] + "\n" + code if "def " not in code else code
|
||||
test_code = full + "\n\n" + p["test"] + f"\n\ncheck({p['entry_point']})"
|
||||
ok, _ = run_python(test_code, timeout=10)
|
||||
if ok: correct += 1
|
||||
if (i+1) % 30 == 0: log(f" eval {i+1}/{len(he)}: {correct} correct")
|
||||
return correct, len(he)
|
||||
|
||||
|
||||
HARD_GEN_PROMPT = """Generate ONE challenging Python coding problem that requires:
|
||||
- non-trivial algorithm (sorting variants, hash maps, two-pointer, dynamic logic, recursive backtracking, parsing, etc.)
|
||||
- handles edge cases (empty input, negatives, duplicates, boundaries, or unusual inputs)
|
||||
- 3 test assertions covering normal + edge cases
|
||||
|
||||
Output exactly:
|
||||
|
||||
```python
|
||||
def {function_name}({args}):
|
||||
\"\"\"{problem description}\"\"\"
|
||||
{implementation}
|
||||
|
||||
# tests
|
||||
assert {function_name}(...) == ...
|
||||
assert {function_name}(...) == ...
|
||||
assert {function_name}(...) == ...
|
||||
```
|
||||
|
||||
Output ONLY the code block. Make the problem genuinely tricky."""
|
||||
|
||||
|
||||
def parse_problem(raw):
|
||||
code = extract_code(raw) if "```" in raw else raw.strip()
|
||||
if "def " not in code: return None
|
||||
lines = code.split("\n")
|
||||
func_start = next((i for i, l in enumerate(lines) if l.startswith("def ")), None)
|
||||
if func_start is None: return None
|
||||
tests = []
|
||||
def_end = None
|
||||
for i in range(func_start, len(lines)):
|
||||
l = lines[i]
|
||||
if l.startswith("def ") and i > func_start: break
|
||||
if l.startswith("assert "):
|
||||
tests.append(l)
|
||||
if def_end is None: def_end = i
|
||||
if len(tests) < 2: return None
|
||||
if def_end is None: def_end = len(lines)
|
||||
full_solution = "\n".join(lines[func_start:def_end]).strip()
|
||||
if len(full_solution) < 30: return None
|
||||
m = re.match(r"def\s+(\w+)\s*\(", lines[func_start])
|
||||
if not m: return None
|
||||
sig_lines = []
|
||||
for i in range(func_start, def_end):
|
||||
sig_lines.append(lines[i])
|
||||
if i == func_start and not any('"""' in lines[j] for j in range(i, min(i+5, def_end))):
|
||||
sig_lines.append(" pass"); break
|
||||
return {"fn_name": m.group(1), "signature": "\n".join(sig_lines), "tests": tests,
|
||||
"canonical": full_solution}
|
||||
|
||||
|
||||
def code_signature(code):
|
||||
"""Normalize code for dedup: strip whitespace, lowercase, hash."""
|
||||
norm = re.sub(r"\s+", " ", code).strip().lower()
|
||||
return hashlib.md5(norm.encode()).hexdigest()
|
||||
|
||||
|
||||
def jaccard_similar(a, b, threshold=0.85):
|
||||
"""Quick token-level Jaccard."""
|
||||
ta = set(re.findall(r"\w+", a.lower()))
|
||||
tb = set(re.findall(r"\w+", b.lower()))
|
||||
if not ta or not tb: return False
|
||||
return len(ta & tb) / len(ta | tb) >= threshold
|
||||
|
||||
|
||||
def mine_aggressive(model, tok, n_problems=200, max_pairs_per_problem=4, n_attempts=8,
|
||||
batch_gen=4):
|
||||
"""Generate many problems, mine ALL broken-fixed combinations per problem."""
|
||||
log(f"AGGRESSIVE MINING — {n_problems} problems, {n_attempts} attempts each, up to {max_pairs_per_problem} pairs/problem")
|
||||
|
||||
# Step 1: generate problems in batches
|
||||
log(" generating problems...")
|
||||
all_problems = []
|
||||
for batch_start in range(0, n_problems, batch_gen):
|
||||
chunk_size = min(batch_gen, n_problems - batch_start)
|
||||
raws = gen_batch(model, tok, [HARD_GEN_PROMPT]*chunk_size, max_new=500, temperature=0.95, batch=batch_gen)
|
||||
for r in raws:
|
||||
p = parse_problem(r)
|
||||
if p is None: continue
|
||||
full = p["canonical"] + "\n\n" + "\n".join(p["tests"])
|
||||
ok, _ = run_python(full)
|
||||
if ok: all_problems.append(p)
|
||||
if batch_start % (batch_gen*5) == 0:
|
||||
log(f" generated {batch_start+chunk_size}/{n_problems}, valid so far: {len(all_problems)}")
|
||||
log(f" → {len(all_problems)} valid problems")
|
||||
|
||||
# Step 2: for each problem, sample n_attempts solutions at temp 0.8, classify pass/fail
|
||||
log(" solving each problem with multiple attempts...")
|
||||
all_pairs = []
|
||||
seen_broken_sigs = set()
|
||||
for pi, p in enumerate(all_problems):
|
||||
solve_prompt = (f"Implement: {p['signature']}\n\nTests:\n{chr(10).join(p['tests'])}\n\n"
|
||||
f"Output only the function implementation in one ```python block.")
|
||||
attempts = gen_batch(model, tok, [solve_prompt]*n_attempts, max_new=500, temperature=0.8, batch=batch_gen)
|
||||
passes, fails = [], []
|
||||
for raw in attempts:
|
||||
code = extract_code(raw) if "```" in raw else raw
|
||||
ok, err = run_python(code + "\n\n" + "\n".join(p["tests"]))
|
||||
if ok: passes.append(code)
|
||||
else: fails.append((code, err))
|
||||
# Mine pairs: each fail × each pass, capped per problem; dedupe broken
|
||||
problem_pairs = 0
|
||||
for (broken, broken_err) in fails:
|
||||
if problem_pairs >= max_pairs_per_problem: break
|
||||
sig = code_signature(broken)
|
||||
if sig in seen_broken_sigs: continue
|
||||
# check Jaccard against recent broken codes
|
||||
is_dup = False
|
||||
for existing in list(seen_broken_sigs)[-50:]:
|
||||
# can't easily reverse-hash; check against the actual broken strings we've kept
|
||||
pass
|
||||
for pass_code in passes:
|
||||
all_pairs.append({
|
||||
"signature": p["signature"], "tests": p["tests"],
|
||||
"broken": broken, "error": broken_err, "fixed": pass_code,
|
||||
})
|
||||
seen_broken_sigs.add(sig)
|
||||
problem_pairs += 1
|
||||
break # one fixed per broken to keep diversity
|
||||
if (pi+1) % 10 == 0:
|
||||
log(f" solved {pi+1}/{len(all_problems)}, pairs mined: {len(all_pairs)}")
|
||||
log(f" AGGRESSIVE MINING DONE — {len(all_pairs)} pairs from {len(all_problems)} problems")
|
||||
return all_pairs
|
||||
|
||||
|
||||
def make_example(r, tok):
|
||||
user = (f"Implement: {r['signature']}\n\n"
|
||||
f"Tests:\n{chr(10).join(r['tests'])}\n\n"
|
||||
f"My attempt:\n```python\n{r['broken']}\n```\n\n"
|
||||
f"Error:\n{r.get('error','')}\n\n"
|
||||
f"Fix and output the corrected code only.")
|
||||
assistant = f"```python\n{r['fixed']}\n```"
|
||||
msgs_pre = [{"role": "system", "content": "You are an expert Python coder. Output one ```python block only."},
|
||||
{"role": "user", "content": user}]
|
||||
msgs_full = msgs_pre + [{"role": "assistant", "content": assistant}]
|
||||
pre = tok.apply_chat_template(msgs_pre, tokenize=False, add_generation_prompt=True)
|
||||
full = tok.apply_chat_template(msgs_full, tokenize=False)
|
||||
pre_ids = tok(pre, add_special_tokens=False)["input_ids"]
|
||||
full_ids = tok(full, add_special_tokens=False)["input_ids"]
|
||||
MAX = 1024
|
||||
full_ids = full_ids[:MAX]
|
||||
labels = list(full_ids)
|
||||
n_pre = min(len(pre_ids), len(labels))
|
||||
for i in range(n_pre): labels[i] = -100
|
||||
pad = MAX - len(full_ids)
|
||||
return {"input_ids": full_ids + [tok.pad_token_id]*pad,
|
||||
"attention_mask": [1]*len(full_ids) + [0]*pad,
|
||||
"labels": labels + [-100]*pad}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", default="Qwen/Qwen2.5-14B")
|
||||
ap.add_argument("--warmup_pairs_path", default="/workspace/saved_pairs/pairs_40.jsonl")
|
||||
ap.add_argument("--n_warmup_pairs", type=int, default=40)
|
||||
ap.add_argument("--n_problems", type=int, default=200)
|
||||
ap.add_argument("--n_attempts", type=int, default=8)
|
||||
ap.add_argument("--max_pairs_per_problem", type=int, default=4)
|
||||
ap.add_argument("--lora_rank", type=int, default=32)
|
||||
ap.add_argument("--epochs", type=int, default=2)
|
||||
ap.add_argument("--lr", type=float, default=1e-4)
|
||||
ap.add_argument("--tag", required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
out_dir = f"/workspace/multi_pair/{args.tag}"
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
log(f"loading {args.model}")
|
||||
tok = AutoTokenizer.from_pretrained(args.model)
|
||||
if tok.pad_token is None: tok.pad_token = tok.eos_token
|
||||
tok.padding_side = "left"
|
||||
model = AutoModelForCausalLM.from_pretrained(args.model, torch_dtype=torch.bfloat16, device_map="cuda:0")
|
||||
log(f" loaded mem={torch.cuda.memory_allocated('cuda:0')/1e9:.1f}GB")
|
||||
|
||||
# Base eval
|
||||
model.eval()
|
||||
log("=== BASE eval ===")
|
||||
base_corr, base_total = humaneval_full(model, tok)
|
||||
log(f" BASE: {base_corr}/{base_total}")
|
||||
|
||||
# Stage 1: aggressive mining from BASE model (not from warmup — we want fresh diversity)
|
||||
log("=== AGGRESSIVE MINING (from base model) ===")
|
||||
new_pairs = mine_aggressive(model, tok,
|
||||
n_problems=args.n_problems,
|
||||
max_pairs_per_problem=args.max_pairs_per_problem,
|
||||
n_attempts=args.n_attempts)
|
||||
with open(f"{out_dir}/pairs_new.jsonl", "w") as fh:
|
||||
for p in new_pairs: fh.write(json.dumps(p) + "\n")
|
||||
log(f" saved {len(new_pairs)} new pairs")
|
||||
|
||||
# Combine with warmup pairs
|
||||
warmup_pairs = [json.loads(l) for l in open(args.warmup_pairs_path)][:args.n_warmup_pairs]
|
||||
combined = warmup_pairs + new_pairs
|
||||
log(f" combined: {len(warmup_pairs)} warmup + {len(new_pairs)} new = {len(combined)} total")
|
||||
|
||||
if len(combined) < 20:
|
||||
log("FATAL: too few pairs"); return
|
||||
|
||||
# Stage 2: train fresh LoRA on combined
|
||||
log(f"=== TRAINING — fresh LoRA rank={args.lora_rank}, lr={args.lr}, e={args.epochs} ===")
|
||||
lora_cfg = LoraConfig(r=args.lora_rank, lora_alpha=args.lora_rank*2, lora_dropout=0.05, bias="none",
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
|
||||
model = get_peft_model(model, lora_cfg)
|
||||
model.print_trainable_parameters()
|
||||
|
||||
tok.padding_side = "right"
|
||||
ds = HFDataset.from_list([make_example(r, tok) for r in combined])
|
||||
targs = TrainingArguments(
|
||||
output_dir=f"{out_dir}/ckpt", num_train_epochs=args.epochs,
|
||||
per_device_train_batch_size=1, gradient_accumulation_steps=4,
|
||||
learning_rate=args.lr, bf16=True, logging_steps=20,
|
||||
save_strategy="no", report_to="none", remove_unused_columns=False, warmup_ratio=0.05,
|
||||
)
|
||||
Trainer(model=model, args=targs, train_dataset=ds, processing_class=tok).train()
|
||||
log(" training done")
|
||||
tok.padding_side = "left"
|
||||
|
||||
# Stage 3: eval
|
||||
model.eval()
|
||||
log("=== TRAINED eval ===")
|
||||
tr_corr, tr_total = humaneval_full(model, tok)
|
||||
log(f" TRAINED: {tr_corr}/{tr_total} Δ={tr_corr-base_corr:+d}")
|
||||
model.save_pretrained(f"{out_dir}/adapter")
|
||||
|
||||
result = {
|
||||
"model": args.model, "method": "aggressive multi-pair mining",
|
||||
"base": [base_corr, base_total], "trained": [tr_corr, tr_total],
|
||||
"delta": tr_corr - base_corr,
|
||||
"n_warmup_pairs": len(warmup_pairs), "n_new_pairs": len(new_pairs),
|
||||
"n_total_pairs": len(combined),
|
||||
"n_problems_generated": args.n_problems, "n_attempts_per_problem": args.n_attempts,
|
||||
"max_pairs_per_problem": args.max_pairs_per_problem,
|
||||
"lora_rank": args.lora_rank, "lr": args.lr, "epochs": args.epochs,
|
||||
"elapsed_s": time.time() - T0,
|
||||
}
|
||||
with open(f"{out_dir}/result.json", "w") as fh: json.dump(result, fh, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print(f" MULTI-PAIR on {args.model}")
|
||||
print(f" HumanEval: base={base_corr}/{base_total} trained={tr_corr}/{tr_total} Δ={tr_corr-base_corr:+d}")
|
||||
print(f" Total pairs: {len(combined)} ({len(warmup_pairs)} warmup + {len(new_pairs)} new)")
|
||||
print(f" Time: {time.time()-T0:.0f}s")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
146
recipe/train_on_pairs.py
Normal file
146
recipe/train_on_pairs.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
"""Train a LoRA adapter on a released `pairs.jsonl` file and evaluate.
|
||||
|
||||
This is the clean replication entrypoint: skip the mining stage entirely
|
||||
and just consume the (broken, fixed) pairs we already mined and released
|
||||
in data/. Use this to reproduce the headline numbers without burning
|
||||
GPU on the search step.
|
||||
|
||||
Schema of pairs.jsonl (one JSON object per line):
|
||||
{
|
||||
"signature": "def foo(x): ...", # function header + docstring
|
||||
"tests": ["assert foo(1) == 2", ...],
|
||||
"broken": "def foo(x): ... # buggy",
|
||||
"error": "AssertionError ...",
|
||||
"fixed": "def foo(x): ... # correct"
|
||||
}
|
||||
|
||||
Example:
|
||||
python recipe/train_on_pairs.py \\
|
||||
--model Qwen/Qwen2.5-7B \\
|
||||
--pairs data/pairs_7b_40.jsonl \\
|
||||
--out adapter_7b_seed13 \\
|
||||
--seed 13
|
||||
|
||||
Then evaluate the resulting adapter with:
|
||||
python recipe/eval_raw.py --model Qwen/Qwen2.5-7B \\
|
||||
--adapter adapter_7b_seed13 --bench humaneval
|
||||
"""
|
||||
import argparse, json, os, random, time
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
from datasets import Dataset
|
||||
from peft import LoraConfig, get_peft_model
|
||||
from transformers import (AutoModelForCausalLM, AutoTokenizer,
|
||||
Trainer, TrainingArguments)
|
||||
|
||||
T0 = time.time()
|
||||
def log(m): print(f"[{time.time()-T0:7.1f}s] {m}", flush=True)
|
||||
|
||||
|
||||
REPAIR_PROMPT = """### Task
|
||||
Fix the bug in the Python function so it passes all the provided tests.
|
||||
|
||||
### Tests
|
||||
{tests}
|
||||
|
||||
### Buggy code
|
||||
```python
|
||||
{broken}
|
||||
```
|
||||
|
||||
### Error
|
||||
{error}
|
||||
|
||||
### Fixed code
|
||||
```python
|
||||
{fixed}
|
||||
```
|
||||
"""
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", required=True,
|
||||
help="HF model id, e.g. Qwen/Qwen2.5-7B")
|
||||
ap.add_argument("--pairs", required=True,
|
||||
help="Path to a pairs.jsonl file (one JSON object per line)")
|
||||
ap.add_argument("--out", required=True,
|
||||
help="Output directory for the trained LoRA adapter")
|
||||
ap.add_argument("--epochs", type=int, default=2)
|
||||
ap.add_argument("--lr", type=float, default=1e-4)
|
||||
ap.add_argument("--lora-rank", type=int, default=16)
|
||||
ap.add_argument("--seed", type=int, default=13)
|
||||
ap.add_argument("--batch-size", type=int, default=1)
|
||||
ap.add_argument("--grad-accum", type=int, default=8)
|
||||
ap.add_argument("--max-length", type=int, default=2048)
|
||||
args = ap.parse_args()
|
||||
|
||||
random.seed(args.seed)
|
||||
torch.manual_seed(args.seed)
|
||||
|
||||
log(f"Loading pairs from {args.pairs}")
|
||||
pairs = [json.loads(l) for l in open(args.pairs)]
|
||||
log(f" {len(pairs)} pairs")
|
||||
|
||||
log(f"Loading tokenizer + base model {args.model}")
|
||||
tok = AutoTokenizer.from_pretrained(args.model, use_fast=True)
|
||||
if tok.pad_token is None:
|
||||
tok.pad_token = tok.eos_token
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
args.model, torch_dtype=torch.bfloat16, device_map="auto",
|
||||
)
|
||||
|
||||
log(f"Attaching LoRA (rank {args.lora_rank}, q/k/v/o projections)")
|
||||
lora = LoraConfig(
|
||||
r=args.lora_rank, lora_alpha=args.lora_rank * 2,
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
|
||||
lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
|
||||
)
|
||||
model = get_peft_model(model, lora)
|
||||
model.print_trainable_parameters()
|
||||
|
||||
def format_pair(p):
|
||||
tests = "\n".join(p["tests"])
|
||||
text = REPAIR_PROMPT.format(
|
||||
tests=tests, broken=p["broken"],
|
||||
error=p.get("error", ""), fixed=p["fixed"],
|
||||
)
|
||||
ids = tok(text, truncation=True, max_length=args.max_length,
|
||||
padding="max_length", return_tensors="pt")
|
||||
return {
|
||||
"input_ids": ids.input_ids[0],
|
||||
"attention_mask": ids.attention_mask[0],
|
||||
"labels": ids.input_ids[0].clone(),
|
||||
}
|
||||
|
||||
ds = Dataset.from_list([format_pair(p) for p in pairs])
|
||||
|
||||
log("Training")
|
||||
targs = TrainingArguments(
|
||||
output_dir=args.out + "_ckpt",
|
||||
per_device_train_batch_size=args.batch_size,
|
||||
gradient_accumulation_steps=args.grad_accum,
|
||||
num_train_epochs=args.epochs,
|
||||
learning_rate=args.lr,
|
||||
lr_scheduler_type="cosine",
|
||||
warmup_ratio=0.03,
|
||||
logging_steps=5,
|
||||
save_strategy="no",
|
||||
bf16=True,
|
||||
report_to="none",
|
||||
seed=args.seed,
|
||||
)
|
||||
Trainer(model=model, args=targs, train_dataset=ds).train()
|
||||
|
||||
log(f"Saving adapter to {args.out}")
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
model.save_pretrained(args.out)
|
||||
tok.save_pretrained(args.out)
|
||||
log("Done. Evaluate with: python recipe/eval_raw.py --model "
|
||||
f"{args.model} --adapter {args.out} --bench humaneval")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
13
requirements.txt
Normal file
13
requirements.txt
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Exact stack used in the paper (May 2026).
|
||||
# Tested on H100 80GB (driver 570+) and RTX 6000 Ada.
|
||||
|
||||
torch==2.6.0
|
||||
transformers==4.51.3
|
||||
vllm==0.8.5
|
||||
peft==0.13.0
|
||||
accelerate>=0.30
|
||||
datasets>=2.18
|
||||
sympy>=1.12
|
||||
numpy<2
|
||||
scipy
|
||||
hf_transfer
|
||||
Loading…
Add table
Add a link
Reference in a new issue