I replicated Ng's RYS method and found that duplicating 3 specific layers in Qwen2.5-32B boosts reasoning by 17% and duplicating layers 12-14 in Devstral-24B improves logical deduction from 0.22→0.76 on BBH — no training, no weight changes, just routing hidden states through the same circuit twice. Tools included. Two AMD GPUs, one evening.
This toolkit finds and exploits "reasoning circuits" hidden inside transformer models. The idea: certain contiguous blocks of layers act as indivisible cognitive units. Duplicate them in the forward pass — same weights, no training, no merging — and the model gets measurably smarter on specific capabilities.
Built on [David Ng's RYS method](https://dnhkng.github.io/posts/rys/) and extended with new findings. Everything here was discovered on two AMD consumer GPUs (RX 7900 XT + RX 6950 XT) in one evening.
## Results
### Devstral-Small-2-24B: Layers 12, 13, 14 duplicated once
I ran the full tests on a H200 instance on Vast.ai and compare devstral base against the surgery model, and the results are in: So the surgery is doing something real and specific: it's boosting mathematical reasoning and causal reasoning but at the cost of instruction following and code generation. The model thinks harder but follows directions less precisely.
Transformers organize themselves during training into functional circuits — multi-layer processing units that perform complete cognitive operations. These circuits are indivisible: duplicating a single layer does almost nothing, but duplicating the right *block* of 3-4 layers gives the model a second pass through its reasoning pipeline.
Different models have different circuits in different places:
- **Devstral-24B** (40 layers): reasoning circuit at layers **12-14**
- **Qwen2.5-32B** (64 layers): reasoning circuit at layers **7-9**
The boundaries are sharp. Shift the block by one layer in either direction and the improvement disappears or inverts.
## The "modes" discovery
Different duplication patterns create distinct cognitive profiles from the same weights:
| Pattern | Math | EQ | Character |
|---------|------|----|-----------|
| Double-pass 13-16 | ↑↑ | ↑ | Math specialist |
| Triple-pass 13-16 | ↑ | ↑↑ | EQ specialist |
| Interleaved 13,13,14,14,15,15,16 | ↑↑↑ | ↓ | Pure math mode |
| `reasoning_probe.py` | BBH-derived causal/logical/navigation/math word problems |
| `compare_eval.py` | Compare lm-evaluation-harness results across runs |
| `visualize.py` | Text and PNG heatmaps of sweep results |
## How the sweep works
1. For each layer configuration (i, j):
- GGUF surgery creates a model where layers i..j-1 are physically duplicated
- The new forward pass: `layers 0..j-1 → layers i..j-1 again → layers j..N-1`
- llama-server loads the modified model
- Three probe suites run: math, EQ, reasoning (BBH-derived)
- Scores are compared to baseline, results printed live
- Server killed, modified GGUF deleted, next config
2. The search strategy:
- **Pass 1**: Large blocks (8 layers), wide stride → find the hot zone
- **Pass 2**: Small blocks (3-5 layers), stride 1 within hot zone → find exact boundaries
- **Pass 3**: Try multi-pass, interleaved, and compound configs
Modified GGUFs are written to tmpfs (`/dev/shm`) and deleted after each test. The base model weights stay on disk.
## Requirements
- Linux with llama.cpp built (CPU, CUDA, Vulkan, or Metal)
- Python 3.10+ with `gguf`, `requests`, `tqdm`
- Enough VRAM/RAM to run the model + a few extra duplicated layers
- Optional: `lm-eval` for benchmark validation, `matplotlib` for heatmap plots
## FAQ
**Does this use more VRAM?**
Yes, the duplicated layers are physical copies in the GGUF. For 3 extra layers on a 24B model, expect ~1.5 GiB additional. A llama.cpp forward-pass patch (using pointers instead of copies) would eliminate this — contributions welcome.
**Does this slow down inference?**
Yes, proportionally to the number of extra layers. 3 extra layers on a 40-layer model = ~7.5% slower. The reasoning improvement is worth it.
**Will this work on my model?**
Probably. We've tested on Mistral-architecture (Devstral) and Qwen2 architecture. Ng's original work was on Qwen2-72B. The circuits exist in all transformer models — the question is where they are and how big they are. Run the sweep and find out.
**Why not fine-tune instead?**
This is orthogonal to fine-tuning. You can do both. In fact, Ng's RYS models were later fine-tuned by others and topped the HuggingFace leaderboard. Layer duplication changes the architecture; fine-tuning changes the weights. Stack them.
## Credits
- [David Ng](https://dnhkng.github.io/posts/rys/) for the RYS method and the insight that transformers have functional neuroanatomy
- [llama.cpp](https://github.com/ggml-org/llama.cpp) for making local inference practical
- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) for benchmark validation
