mirror of
https://github.com/samvallad33/vestige.git
synced 2026-07-24 23:41:01 +02:00
benchmarks: complete the Silent Rotation evidence set
Adds every model run rather than only the first: GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash and MiniMax M3 alongside the two Kimi K3 runs, for 228 agent transcripts across 22 trials. Adds QUOTES.md, which is the point of shipping raw transcripts. It carries the verbatim reasoning of the agents that got it wrong, including one that read the words "staging only" in a decoy and used the key anyway, and one that searched for a contradiction, found none, and proceeded. Both did what you would want an agent to do. The evidence set was the problem. Adds results/WITHHELD-contaminated/ so the exclusions are auditable instead of something you take on trust. Two arms were served a previous trial's facts by a flush bug in this harness. Both were fixed and re-run clean, and both the broken cells and the repaired results are published. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sam Valladares
parent
9bdb9a5668
commit
5c384e7c1e
361 changed files with 129756 additions and 4 deletions
|
|
@ -19,11 +19,11 @@ it the right one?**
|
|||
|---|---|---|---|---|
|
||||
| anarchy (no memory) | 0/22 | **16/22** | 6/22 | 22 |
|
||||
| rag (dense cosine) | 4/21 | **10/21** | 7/21 | 21 |
|
||||
| supermemory | 5/6 | 0/6 | 1/6 | 6 |
|
||||
| mem0 (clean cells only) | 1/2 | 0/2 | 1/2 | 2 |
|
||||
| hindsight | 0/3 | 0/3 | 3/3 | 3 |
|
||||
| Zep/Graphiti (clean cell only) | 0/1 | 0/1 | 1/1 | 1 |
|
||||
| **sync (Vestige)** | **19/21** | **0/21** | 2/21 | 21 |
|
||||
| supermemory | 4/5 | 0/5 | 1/5 | 5 |
|
||||
| mem0 | 0/5 | 1/5 | 4/5 | 5 |
|
||||
| hindsight | 0/5 | 0/5 | 5/5 | 5 |
|
||||
| zep | 0/4 | 2/4 | 2/4 | 4 |
|
||||
|
||||
Models: Kimi K3, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
|
||||
Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`.
|
||||
|
|
|
|||
167
benchmarks/silent-rotation/QUOTES.md
Normal file
167
benchmarks/silent-rotation/QUOTES.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# What the agents actually said, and what it proves
|
||||
|
||||
Every quotation below is verbatim from a transcript in `results/`. The file it comes from is named
|
||||
next to it. Nothing here is paraphrased.
|
||||
|
||||
The point of this file is that the numbers in `FINDING.md` are a consequence, and this is the cause.
|
||||
|
||||
---
|
||||
|
||||
## 1. The agent that got it wrong was not reasoning badly
|
||||
|
||||
This is the single most important thing in the dataset, and it is why the raw transcripts ship instead
|
||||
of a summary table.
|
||||
|
||||
**GPT-5.6 Sol, run `gpt-5.6-sol-trial-1`, correct key `k_yildun`, dense cosine arm.**
|
||||
All three agents converged on `k_wezen`, a decoy. From `transcript-rag-a1.json`:
|
||||
|
||||
> "I'm weighing whether to fix the memory to say that the key k_wezen is staging only. The note
|
||||
> mentions a closed ticket about resetting the key id for k_wezen in staging without any rotation,
|
||||
> which seems intentional... I've noticed that my memory indicates a reset to k_wezen, so I must set
|
||||
> both configs to that"
|
||||
|
||||
**It read the words "staging only" and used the key anyway.** Not carelessness. Nothing it retrieved
|
||||
contradicted the decoy, and a caveated answer beats no answer. The chain of inference is correct. The
|
||||
evidence set was incomplete.
|
||||
|
||||
**Kimi K3, run `runB-trial-2`, correct key `k_regulus`, dense cosine arm.**
|
||||
From `transcript-rag-a1.json`, committing to `k_sirius`:
|
||||
|
||||
> "Memory: `k_sirius` is the key id verification recovered with (OPS-522). **No other memory suggests a
|
||||
> different key.**"
|
||||
|
||||
**It checked for a contradiction.** It went looking for something that would tell it otherwise. Nothing
|
||||
came back, so it proceeded.
|
||||
|
||||
You cannot prompt-engineer around either of these. Both agents did the thing you would want an agent to
|
||||
do.
|
||||
|
||||
---
|
||||
|
||||
## 2. Two agents on the same arm, in the same trial, disagreeing because they saw different pasts
|
||||
|
||||
**Kimi K3, run `runA-trial-1`, correct key `k_nashira`, mem0 arm.**
|
||||
|
||||
Agent a0 never surfaced the cause. From `transcript-mem0-a0.json`:
|
||||
|
||||
> "Both configs have blank key ids, left blank by an 'externalize secrets' refactor. The memory said:
|
||||
> OPS-522 the key id was reset to `k_antares` and verification recovered. That suggests `k_antares` is
|
||||
> the key the platform shipped with / should use."
|
||||
|
||||
Agent a2, same arm, same trial, got a richer retrieval and became suspicious of the same document. From
|
||||
`transcript-mem0-a2.json`:
|
||||
|
||||
> "Hmm, that says k_antares was staging-only. That memory may be a hint that the production key id is
|
||||
> something else, or it might be a red herring. Let me search memory more specifically."
|
||||
|
||||
**a0 wrote `k_antares`. a1 and a2 wrote `k_nashira`. Two merge conflicts. Nothing shipped.**
|
||||
|
||||
They were not disagreeing about how to interpret a shared reality. Their memory layer handed them
|
||||
different realities.
|
||||
|
||||
---
|
||||
|
||||
## 3. An agent giving up on its memory system entirely
|
||||
|
||||
**Kimi K3, run `runB-trial-3`, correct key `k_orion`, mem0 arm.** From `transcript-mem0-a1.json`, after
|
||||
two failed queries:
|
||||
|
||||
> "The memory isn't directly relevant to 'no active signing key selected'. **Let me look at the code.**"
|
||||
|
||||
It concluded the memory layer had nothing and went to the source. It needed six queries in total and
|
||||
first surfaced the correct key on call four.
|
||||
|
||||
This is what a 6.9% first-call hit rate feels like from inside the agent.
|
||||
|
||||
---
|
||||
|
||||
## 4. The trap working exactly as designed, and one agent escaping it
|
||||
|
||||
**Kimi K3, run `runB-trial-3`, dense cosine arm.** From `transcript-rag-a2.json`:
|
||||
|
||||
> "Team memory: OPS-522 says the key id was reset to `k_antares` and verification recovered. But that
|
||||
> was a staging bo..."
|
||||
|
||||
That agent argued itself out and got the right answer. In `runB-trial-2`, the identical reasoning on the
|
||||
identical decoy is what killed the fleet. **The difference between a pass and a merge conflict was
|
||||
whether one agent finished a sentence.**
|
||||
|
||||
---
|
||||
|
||||
## 5. What the winning arm's agents said, and what is missing from it
|
||||
|
||||
**Kimi K3, run `runB-trial-3`, Vestige arm.** All three agents, from `transcript-sync-a{0,1,2}.json`:
|
||||
|
||||
> "The failure is 'no active signing key selected' from `identity-service/src/signer.ts:31`. Let me read
|
||||
> the relevant files and check the shared memory for context."
|
||||
|
||||
Then one `vestige_backfill` call each, no query argument, correct key on the first call for all three.
|
||||
|
||||
**There is no passage anywhere in the Vestige transcripts where an agent weighs a decoy.** Not because
|
||||
the agents were better. Because no decoy arrived. In `runB-trial-3` the backfill results contained zero
|
||||
decoy keys across all three agents, while the dense cosine and supermemory arms received a decoy in
|
||||
nearly every result.
|
||||
|
||||
---
|
||||
|
||||
## 6. The receipt that reports its own blind spot
|
||||
|
||||
The Vestige tool returns this alongside the answer, from `runA-trial-1/transcript-sync-a0.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"content_preview": "Rotation runbook, Q3: we migrated the live signer to k_nashira ...",
|
||||
"age_days_before_failure": 4.0,
|
||||
"shared_entities": ["active_key"],
|
||||
"similarity_rank": 6,
|
||||
"backfill_score": 1.51,
|
||||
"reason": "Reached back 4.0d to a quiet memory sharing 1 entity (active_key) with the
|
||||
failure; it ranked #6 on similarity, so semantic search would have missed it."
|
||||
}
|
||||
```
|
||||
|
||||
`similarity_rank: 6` is the tool stating that the fact it just returned is one a similarity search would
|
||||
have buried. That number is self-reported by Vestige, so treat it as a claim rather than a measurement.
|
||||
The independent check is `tests/bm25_baseline.py`, which reproduces the ranking from the corpus without
|
||||
involving Vestige at all, and puts the cause at 7th of 8.
|
||||
|
||||
---
|
||||
|
||||
## 7. The harness bug that hurt two competitors, and what it looked like
|
||||
|
||||
**Kimi K3, run `runB-trial-2`, correct key `k_regulus`, Zep arm (now in `results/WITHHELD-contaminated/`).** All three agents received this graph
|
||||
fact:
|
||||
|
||||
> "[1] k_antares was decommissioned in favor of k_nashira as the live signer."
|
||||
|
||||
Neither `k_antares` nor `k_nashira` exists in that trial's corpus. Both are keys from the **previous**
|
||||
trial. My harness never flushed Zep's graph between trials, so a superseded fact was asserted as current
|
||||
and every agent believed it.
|
||||
|
||||
That cell is withheld. It is a measurement of my harness, not of Zep. After fixing the flush and adding
|
||||
a per-trial graph namespace, Zep was re-run clean in `runB-trial-3`: two of three agents wrote a decoy,
|
||||
the third wrote nothing, tests passed, production replay failed.
|
||||
|
||||
The same class of bug hit mem0 through a second store at `~/.mem0` that persists alongside the
|
||||
configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and split in `runB-trial-2`.
|
||||
|
||||
---
|
||||
|
||||
## What all of this proves, stated carefully
|
||||
|
||||
**It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident
|
||||
class the symptom query returns a document that resembles the failure and buries the one that explains
|
||||
it, and that holds under dense cosine and BM25 alike. Across 5 models and 22 trials, a query-based arm's
|
||||
first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead
|
||||
of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a
|
||||
unanimous wrong answer that passes tests and breaks production.
|
||||
|
||||
**It does not prove:** that Vestige discovers causal relationships. The edge it traverses is authored by
|
||||
the harness at `harness/agent/prepare_trial.py:187`. It does not prove Vestige is better at retrieval,
|
||||
because a plain dense cosine baseline tied it on outcomes and cost less. It does not prove anything at
|
||||
production scale, because the retrievable corpus is 8 documents. And it does not isolate which of
|
||||
Vestige's three properties (event anchoring, causal traversal, temporal validity) is doing the work,
|
||||
because that ablation has not been run.
|
||||
|
||||
**The one sentence:** the first query fails for almost everyone, almost every time. The only thing that
|
||||
varied was whether the memory layer required a first query at all.
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
# Withheld cells
|
||||
|
||||
These are NOT counted in any published table. They are here so the exclusions are auditable rather
|
||||
than something you have to take on trust.
|
||||
|
||||
- `runB-trial-2-zep*` — my harness never flushed Zep's FalkorDB graph between trials. These agents
|
||||
were served the fact "k_antares was decommissioned in favor of k_nashira as the live signer", and
|
||||
neither key exists in that trial's corpus. Both are from the previous trial. This measures my
|
||||
harness, not Zep. Fixed with a per-trial graph namespace and a hard flush, then re-run clean in
|
||||
`runB-trial-3`.
|
||||
- `runB-trial-1-mem0.json` — same bug class. mem0 persists to `~/.mem0` in addition to its configured
|
||||
store, and only the configured one was cleared. Fixed, then re-run clean in `runB-trial-2` and
|
||||
`runB-trial-3`.
|
||||
- `runB-trial-3-hindsight-DEAD.json` — not a contamination case. The backend returned zero successful
|
||||
retrievals out of nine attempts (`memory_layer_alive: false`, `retrieval_err_total: 9`). An arm whose
|
||||
backend never answered is a missing measurement, not a retrieval loss.
|
||||
|
|
@ -0,0 +1,188 @@
|
|||
{
|
||||
"mode": "mem0",
|
||||
"fleet_size": 3,
|
||||
"provider": "moonshot",
|
||||
"model": "kimi-k3",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 126005,
|
||||
"output": 7317,
|
||||
"total": 133322
|
||||
},
|
||||
"fleet_cost_usd": 0.48777,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 3.0,
|
||||
"output_per_mtok": 15.0
|
||||
},
|
||||
"fleet_wall_clock_seconds": 150.67,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 2,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"memory_tool": "mem0_search",
|
||||
"memory_prewarm": {
|
||||
"attempted": true,
|
||||
"ok": true,
|
||||
"error": "",
|
||||
"seconds": 235.7
|
||||
},
|
||||
"agents_memory_alive": 3,
|
||||
"retrieval_ok_total": 15,
|
||||
"retrieval_err_total": 0,
|
||||
"retrieval_empty_total": 0,
|
||||
"retrieval_error_samples": [],
|
||||
"memory_layer_alive": true,
|
||||
"memory_unavailable": false,
|
||||
"transcripts": [
|
||||
"transcript-mem0-a0.json",
|
||||
"transcript-mem0-a1.json",
|
||||
"transcript-mem0-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 1,
|
||||
"wrong": 1,
|
||||
"other": 1
|
||||
},
|
||||
"fleet_verdict": "failed_merge_conflict",
|
||||
"integrated_tests_pass": false,
|
||||
"prod_replay_pass": null,
|
||||
"production_replay": {
|
||||
"ran": false,
|
||||
"reason": "integrated tree is red"
|
||||
},
|
||||
"merge_conflict_count": 2,
|
||||
"merge_conflicts": [
|
||||
{
|
||||
"file": "identity-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a1"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "ledger-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a1"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"cross_file_divergence": true,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_regulus",
|
||||
"a1": "kid:k_nashira",
|
||||
"a2": "kid_unset"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "mem0",
|
||||
"tokens": {
|
||||
"input": 45672,
|
||||
"output": 3035,
|
||||
"total": 48707
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 149.64,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "mem0_search",
|
||||
"retrieval_ok_count": 4,
|
||||
"retrieval_err_count": 0,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [],
|
||||
"memory_layer_alive": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "mem0",
|
||||
"tokens": {
|
||||
"input": 40445,
|
||||
"output": 2258,
|
||||
"total": 42703
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 105.37,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "mem0_search",
|
||||
"retrieval_ok_count": 3,
|
||||
"retrieval_err_count": 0,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [],
|
||||
"memory_layer_alive": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "mem0",
|
||||
"tokens": {
|
||||
"input": 39888,
|
||||
"output": 2024,
|
||||
"total": 41912
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 114.66,
|
||||
"status": "looped",
|
||||
"local_tests_pass": false,
|
||||
"files_written": [],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "mem0_search",
|
||||
"retrieval_ok_count": 8,
|
||||
"retrieval_err_count": 0,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [],
|
||||
"memory_layer_alive": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-21T10:24:23Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,183 @@
|
|||
{
|
||||
"mode": "zep",
|
||||
"fleet_size": 3,
|
||||
"provider": "moonshot",
|
||||
"model": "kimi-k3",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 142435,
|
||||
"output": 7014,
|
||||
"total": 149449
|
||||
},
|
||||
"fleet_cost_usd": 0.532515,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 3.0,
|
||||
"output_per_mtok": 15.0
|
||||
},
|
||||
"fleet_wall_clock_seconds": 126.28,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"memory_tool": "zep_search",
|
||||
"memory_prewarm": {
|
||||
"attempted": true,
|
||||
"ok": true,
|
||||
"error": "",
|
||||
"seconds": 708.1
|
||||
},
|
||||
"agents_memory_alive": 3,
|
||||
"retrieval_ok_total": 18,
|
||||
"retrieval_err_total": 0,
|
||||
"retrieval_empty_total": 0,
|
||||
"retrieval_error_samples": [],
|
||||
"memory_layer_alive": true,
|
||||
"memory_unavailable": false,
|
||||
"transcripts": [
|
||||
"transcript-zep-a0.json",
|
||||
"transcript-zep-a1.json",
|
||||
"transcript-zep-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_regulus",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_nashira",
|
||||
"a1": "kid:k_nashira",
|
||||
"a2": "kid:k_nashira"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "zep",
|
||||
"tokens": {
|
||||
"input": 54985,
|
||||
"output": 2421,
|
||||
"total": 57406
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 125.72,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "zep_search",
|
||||
"retrieval_ok_count": 6,
|
||||
"retrieval_err_count": 0,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [],
|
||||
"memory_layer_alive": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "zep",
|
||||
"tokens": {
|
||||
"input": 40823,
|
||||
"output": 2410,
|
||||
"total": 43233
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 97.97,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "zep_search",
|
||||
"retrieval_ok_count": 5,
|
||||
"retrieval_err_count": 0,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [],
|
||||
"memory_layer_alive": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "zep",
|
||||
"tokens": {
|
||||
"input": 46627,
|
||||
"output": 2183,
|
||||
"total": 48810
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 104.75,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "zep_search",
|
||||
"retrieval_ok_count": 7,
|
||||
"retrieval_err_count": 0,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [],
|
||||
"memory_layer_alive": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-21T12:10:25Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,191 @@
|
|||
{
|
||||
"mode": "hindsight",
|
||||
"fleet_size": 3,
|
||||
"provider": "moonshot",
|
||||
"model": "kimi-k3",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 187158,
|
||||
"output": 7410,
|
||||
"total": 194568
|
||||
},
|
||||
"fleet_cost_usd": 0.672624,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 3.0,
|
||||
"output_per_mtok": 15.0
|
||||
},
|
||||
"fleet_wall_clock_seconds": 2867.45,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"memory_tool": "hindsight_search",
|
||||
"memory_prewarm": {
|
||||
"attempted": true,
|
||||
"ok": false,
|
||||
"error": "ERROR: Hindsight retrieval unavailable (TimeoutError: ). Is the hindsight-api server running at http://127.0.0.1:8899 and Ollama up for retain()?",
|
||||
"seconds": 371.0
|
||||
},
|
||||
"agents_memory_alive": 0,
|
||||
"retrieval_ok_total": 0,
|
||||
"retrieval_err_total": 9,
|
||||
"retrieval_empty_total": 0,
|
||||
"retrieval_error_samples": [
|
||||
"ERROR: Hindsight retrieval unavailable (TimeoutError: ). Is the hindsight-api server running at http://127.0.0.1:8899 and Ollama up for retain()?"
|
||||
],
|
||||
"memory_layer_alive": false,
|
||||
"memory_unavailable": true,
|
||||
"transcripts": [
|
||||
"transcript-hindsight-a0.json",
|
||||
"transcript-hindsight-a1.json",
|
||||
"transcript-hindsight-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_orion",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_tucana",
|
||||
"a1": "kid:k_tucana",
|
||||
"a2": "kid:k_tucana"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "hindsight",
|
||||
"tokens": {
|
||||
"input": 57378,
|
||||
"output": 2118,
|
||||
"total": 59496
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 1370.98,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "hindsight_search",
|
||||
"retrieval_ok_count": 0,
|
||||
"retrieval_err_count": 2,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [
|
||||
"ERROR: Hindsight retrieval unavailable (TimeoutError: ). Is the hindsight-api server running at http://127.0.0.1:8899 and Ollama up for retain()?"
|
||||
],
|
||||
"memory_layer_alive": false,
|
||||
"finish_summary": "The \"externalize secrets\" refactor left both key-id configs blank: identity-service/src/config.ts `activeKid: \"\"` (causing signSession to throw \"no active signing key selected\") and ledger-service/src/config.ts `trustedKid: \"\"`. The canonical digest implementations on both sides were already byte-identical, and the gateway copies the token verbatim, so the only defect was the missing key selection. Fix: set both configs to the same shared keyring entry, `k_tucana` \u2014 the first/initial entry in shared/src/keyring.ts, consistent with the truncated config comment \"the platform shipped with\". Both issuer and verifier now select the same HMAC key, so the e2e identity \u2192 gateway \u2192 ledger charge path returns 200. Direction note for integration: the important invariant is that activeKid === trustedKid and points at a valid keyring entry; k_tucana chosen as the platform's original/default key. (Hindsight memory service was unreachable during this task.)",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "hindsight",
|
||||
"tokens": {
|
||||
"input": 71840,
|
||||
"output": 2742,
|
||||
"total": 74582
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 2866.52,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "hindsight_search",
|
||||
"retrieval_ok_count": 0,
|
||||
"retrieval_err_count": 4,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [
|
||||
"ERROR: Hindsight retrieval unavailable (TimeoutError: ). Is the hindsight-api server running at http://127.0.0.1:8899 and Ollama up for retain()?"
|
||||
],
|
||||
"memory_layer_alive": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "hindsight",
|
||||
"tokens": {
|
||||
"input": 57940,
|
||||
"output": 2550,
|
||||
"total": 60490
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 2569.32,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"memory_tool_used": "hindsight_search",
|
||||
"retrieval_ok_count": 0,
|
||||
"retrieval_err_count": 3,
|
||||
"retrieval_empty_count": 0,
|
||||
"retrieval_errors": [
|
||||
"ERROR: Hindsight retrieval unavailable (TimeoutError: ). Is the hindsight-api server running at http://127.0.0.1:8899 and Ollama up for retain()?"
|
||||
],
|
||||
"memory_layer_alive": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-21T13:17:43Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,154 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 125027,
|
||||
"output": 8455,
|
||||
"total": 133482
|
||||
},
|
||||
"fleet_cost_usd": 0.038559,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 55.19,
|
||||
"fleet_iterations_total": 19,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_yildun",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_pavo",
|
||||
"a1": "kid:k_pavo",
|
||||
"a2": "kid:k_pavo"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 46065,
|
||||
"output": 4414,
|
||||
"total": 50479
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 54.31,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-a_euv4ex/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-a_euv4ex/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-a_euv4ex/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-a_euv4ex/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the signing and verification key ids in both services' configs. The identity-service's `config.activeKid` and ledger-service's `ledgerConfig.trustedKid` were both empty strings (left blank during an \"externalize secrets\" refactor). Set both to `\"k_pavo\"`, a valid key that exists in the shared keyring, so the identity-service can sign session tokens and the ledger-service can verify them using the same key material.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 42411,
|
||||
"output": 2382,
|
||||
"total": 44793
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 50.95,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed because both identity-service and ledger-service had empty key id selections (`activeKid: \"\"` and `trustedKid: \"\"`). The signer could not pick a signing key, and the verifier had no trusted key to verify against. Fixed by setting both to `\"k_pavo\"` \u2014 a key present in the shared keyring \u2014 so the signer mints a valid HMAC'd token and the verifier can recompute and match the signature.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 36551,
|
||||
"output": 1659,
|
||||
"total": 38210
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 27.66,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting `activeKid` in identity-service/src/config.ts and `trustedKid` in ledger-service/src/config.ts to the same signing key (`k_pavo`) from the shared keyring. Both were left blank (\"\") during an \"externalize-secrets\" refactor, causing signSession to throw \"no active signing key selected\".",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:45:51Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 1,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"pavo",
|
||||
"izar",
|
||||
"vela",
|
||||
"capella",
|
||||
"fornax",
|
||||
"cobalt",
|
||||
"gemini",
|
||||
"basalt",
|
||||
"kyanite",
|
||||
"yildun",
|
||||
"antares",
|
||||
"vega",
|
||||
"lyric",
|
||||
"deneb",
|
||||
"norma",
|
||||
"feldspar",
|
||||
"tucana",
|
||||
"mizar",
|
||||
"helios",
|
||||
"altair",
|
||||
"orion",
|
||||
"atlas",
|
||||
"aludra",
|
||||
"zephyr",
|
||||
"rutile",
|
||||
"pyrite",
|
||||
"schist",
|
||||
"hydra",
|
||||
"umbriel",
|
||||
"dunite",
|
||||
"lyra",
|
||||
"procyon",
|
||||
"octans",
|
||||
"talc",
|
||||
"regulus",
|
||||
"dorado",
|
||||
"quasar",
|
||||
"draco",
|
||||
"juno",
|
||||
"bellatrix",
|
||||
"wezen",
|
||||
"perseus",
|
||||
"indus",
|
||||
"sirius",
|
||||
"spica",
|
||||
"electra",
|
||||
"jasper",
|
||||
"cirrus",
|
||||
"rigel",
|
||||
"wren"
|
||||
],
|
||||
"correct_kid": "k_yildun",
|
||||
"decoy_kids": [
|
||||
"k_wezen",
|
||||
"k_electra",
|
||||
"k_pavo",
|
||||
"k_kyanite",
|
||||
"k_procyon",
|
||||
"k_sirius",
|
||||
"k_talc",
|
||||
"k_fornax",
|
||||
"k_cirrus",
|
||||
"k_quasar",
|
||||
"k_basalt",
|
||||
"k_zephyr",
|
||||
"k_spica",
|
||||
"k_rutile",
|
||||
"k_feldspar",
|
||||
"k_norma",
|
||||
"k_hydra",
|
||||
"k_juno",
|
||||
"k_regulus",
|
||||
"k_schist",
|
||||
"k_perseus",
|
||||
"k_rigel",
|
||||
"k_bellatrix",
|
||||
"k_orion",
|
||||
"k_umbriel",
|
||||
"k_draco",
|
||||
"k_cobalt",
|
||||
"k_gemini",
|
||||
"k_indus",
|
||||
"k_aludra",
|
||||
"k_vega",
|
||||
"k_altair",
|
||||
"k_pyrite",
|
||||
"k_jasper",
|
||||
"k_capella",
|
||||
"k_izar",
|
||||
"k_dunite",
|
||||
"k_octans",
|
||||
"k_lyra",
|
||||
"k_dorado",
|
||||
"k_vela",
|
||||
"k_lyric",
|
||||
"k_tucana",
|
||||
"k_helios",
|
||||
"k_antares",
|
||||
"k_wren",
|
||||
"k_atlas",
|
||||
"k_deneb",
|
||||
"k_mizar"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-1/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 1 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_yildun",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.ZB1iKOyWil9tDI_7oT2hr4EfK5oKTtlTFwlIlCZUul4"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.39kYP3f9sG2DdxMTfi1H2BAy6m9zSzYzEwNRypy1dqg"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.m2a1sWc-YEoFlFFQgQNIFpGf7s5pyaTAaiWeMWDNIno"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.x0CIXum6hfCJs6ra-wRXbGT2hXjHqJulrMzb_qt98Xk"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.P6icZZtV2o35B78z7Dq4yDBsNRnLUF9DJMK7M2J7PM4"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,154 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 138654,
|
||||
"output": 7344,
|
||||
"total": 145998
|
||||
},
|
||||
"fleet_cost_usd": 0.041908,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 37.86,
|
||||
"fleet_iterations_total": 21,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_yildun",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_wezen",
|
||||
"a1": "kid:k_wezen",
|
||||
"a2": "kid:k_wezen"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 47625,
|
||||
"output": 2407,
|
||||
"total": 50032
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 36.92,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed failing end-to-end charge test by setting both identity-service signer config (activeKid) and ledger-service verifier config (trustedKid) to \"k_wezen\", a valid key from the shared keyring. Previously both were empty strings after the \"externalize secrets\" refactor, causing \"no active signing key selected\" error when minting tokens.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 45813,
|
||||
"output": 2661,
|
||||
"total": 48474
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 30.33,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-hdz08c2m/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-hdz08c2m/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-hdz08c2m/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-hdz08c2m/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the signing/verification key IDs in both configs. Both identity-service/src/config.ts (activeKid) and ledger-service/src/config.ts (trustedKid) had been left blank during an \"externalize secrets\" refactor. Set both to \"k_wezen\" \u2014 the key historically used by the platform per team memory (OPS-522) \u2014 which exists in the shared keyring. This enables signSession() to mint tokens and verifyToken() to validate them, letting the end-to-end charge flow complete successfully.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 45216,
|
||||
"output": 2276,
|
||||
"total": 47492
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 31.75,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed end-to-end charge test failure: both identity-service and ledger-service configs had blank key IDs (activeKid: \"\", trustedKid: \"\") from the externalize-secrets refactor. Set both to \"k_wezen\", a valid HMAC key in the shared keyring, consistent with prior team practice (OPS-522). The test now passes.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:46:37Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,139 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 163180,
|
||||
"output": 7412,
|
||||
"total": 170592
|
||||
},
|
||||
"fleet_cost_usd": 0.048803,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 63.58,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 3,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "fixed_correctly",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": true,
|
||||
"production_replay": {
|
||||
"ok": true,
|
||||
"verified": 5,
|
||||
"total": 5,
|
||||
"failures": [],
|
||||
"signed_under": "k_yildun",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_yildun",
|
||||
"a1": "kid:k_yildun",
|
||||
"a2": "kid:k_yildun"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 56992,
|
||||
"output": 2633,
|
||||
"total": 59625
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 62.66,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test. Root cause: the \"externalize-secrets\" refactor blanked both identity-service config.activeKid and ledger-service config.trustedKid, leaving them as empty strings. The signer threw \"no active signing key selected\" when the test called signSession(). The Q3 rotation runbook migrated to key k_yildun, so I set both configs to that key ID, ensuring the issuer and verifier agree on the same signing material.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 53490,
|
||||
"output": 2564,
|
||||
"total": 56054
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 36.92,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed failing end-to-end charge test by restoring signing/verification key ids in both identity-service and ledger-service configs. Both `activeKid` and `trustedKid` were blanked during the \"externalize-secrets\" refactor. Set both to `\"k_yildun\"` (the key the issuer was rotated to per the Q3 rotation runbook), so the identity-service mints tokens with `k_yildun` and the ledger-service verifies them with the same key. Test now passes (asserts 200 response, correct chargedTo, and correct amountCents).",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 52698,
|
||||
"output": 2215,
|
||||
"total": 54913
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 36.3,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-_1y_ymuc/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:47:48Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,164 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 110638,
|
||||
"output": 6818,
|
||||
"total": 117456
|
||||
},
|
||||
"fleet_cost_usd": 0.033842,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 49.18,
|
||||
"fleet_iterations_total": 18,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "failed_merge_conflict",
|
||||
"integrated_tests_pass": false,
|
||||
"prod_replay_pass": null,
|
||||
"production_replay": {
|
||||
"ran": false,
|
||||
"reason": "integrated tree is red"
|
||||
},
|
||||
"merge_conflict_count": 2,
|
||||
"merge_conflicts": [
|
||||
{
|
||||
"file": "identity-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a1",
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "ledger-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a1",
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"cross_file_divergence": true,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_mizar",
|
||||
"a1": "kid:k_vela",
|
||||
"a2": "kid:k_vela"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 43534,
|
||||
"output": 2523,
|
||||
"total": 46057
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 35.77,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Set both identity-service config.activeKid and ledger-service config.trustedKid to \"k_mizar\" (a key present in the shared keyring). Without a matching key ID selected, the signer threw \"no active signing key selected\" and the verifier would have thrown \"no trusted verification key selected\". Picking any key from the shared keyring that both sides agree on restores the end-to-end charge flow.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 38347,
|
||||
"output": 2576,
|
||||
"total": 40923
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 48.23,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-09tsozzb/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-09tsozzb/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-09tsozzb/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-09tsozzb/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting the blank cryptographic key IDs in identity-service and ledger-service configs. The `activeKid` in identity-service/src/config.ts was \"\" (causing signSession to throw), and `trustedKid` in ledger-service/src/config.ts was \"\" (causing verifyToken to throw). Both are now set to \"k_vela\", a key present in the shared keyring, so the signer and verifier use the same HMAC key and the token flows through the full path.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 28757,
|
||||
"output": 1719,
|
||||
"total": 30476
|
||||
},
|
||||
"iterations": 5,
|
||||
"wall_clock_seconds": 25.07,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test was failing because both the identity-service and ledger-service had blank key IDs in their configs (activeKid and trustedKid were empty strings \"\"), causing signSession to throw \"no active signing key selected\". The shared keyring already contained many provisioned HMAC keys. Fixed by setting both configs to use the same key ID \"k_vela\" from the shared keyring, so the identity service can mint signed tokens and the ledger service can verify them.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:48:39Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 2,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"vela",
|
||||
"fornax",
|
||||
"mizar",
|
||||
"quasar",
|
||||
"castor",
|
||||
"feldspar",
|
||||
"tucana",
|
||||
"garnet",
|
||||
"sirius",
|
||||
"draco",
|
||||
"capella",
|
||||
"dunite",
|
||||
"spica",
|
||||
"jasper",
|
||||
"perseus",
|
||||
"basalt",
|
||||
"olivine",
|
||||
"polaris",
|
||||
"aludra",
|
||||
"procyon",
|
||||
"antares",
|
||||
"hematite",
|
||||
"yildun",
|
||||
"lyric",
|
||||
"rigel",
|
||||
"norma",
|
||||
"schist",
|
||||
"borealis",
|
||||
"rutile",
|
||||
"tarazed",
|
||||
"zephyr",
|
||||
"kyanite",
|
||||
"deneb",
|
||||
"cirrus",
|
||||
"indus",
|
||||
"kepler",
|
||||
"quartz",
|
||||
"pavo",
|
||||
"regulus",
|
||||
"wezen",
|
||||
"atlas",
|
||||
"orion",
|
||||
"electra",
|
||||
"wren",
|
||||
"cobalt",
|
||||
"bellatrix",
|
||||
"gemini",
|
||||
"umbriel",
|
||||
"octans",
|
||||
"helios"
|
||||
],
|
||||
"correct_kid": "k_bellatrix",
|
||||
"decoy_kids": [
|
||||
"k_atlas",
|
||||
"k_rutile",
|
||||
"k_yildun",
|
||||
"k_pavo",
|
||||
"k_norma",
|
||||
"k_tucana",
|
||||
"k_perseus",
|
||||
"k_dunite",
|
||||
"k_orion",
|
||||
"k_wren",
|
||||
"k_spica",
|
||||
"k_cirrus",
|
||||
"k_kepler",
|
||||
"k_aludra",
|
||||
"k_hematite",
|
||||
"k_cobalt",
|
||||
"k_procyon",
|
||||
"k_octans",
|
||||
"k_kyanite",
|
||||
"k_quartz",
|
||||
"k_helios",
|
||||
"k_schist",
|
||||
"k_mizar",
|
||||
"k_jasper",
|
||||
"k_draco",
|
||||
"k_polaris",
|
||||
"k_olivine",
|
||||
"k_zephyr",
|
||||
"k_deneb",
|
||||
"k_rigel",
|
||||
"k_garnet",
|
||||
"k_gemini",
|
||||
"k_umbriel",
|
||||
"k_feldspar",
|
||||
"k_electra",
|
||||
"k_castor",
|
||||
"k_fornax",
|
||||
"k_capella",
|
||||
"k_basalt",
|
||||
"k_borealis",
|
||||
"k_quasar",
|
||||
"k_sirius",
|
||||
"k_indus",
|
||||
"k_wezen",
|
||||
"k_lyric",
|
||||
"k_regulus",
|
||||
"k_tarazed",
|
||||
"k_vela",
|
||||
"k_antares"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-2/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 2 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_bellatrix",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.kyBsIajs0qayF6Y1p42gMAZbXe0M-_r5foOSxikjNJk"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.JUsAmSF_P-9qW_Pgz99caTgH-0I5sFwP3dk8hS9u_fg"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.envYeY9VnGG0z9z7g-LNfeyHCtL7_3m6IeDN9aCxio8"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.F5rb6wX36OxkCvkB51q1vJ1oBHuBB3XVM8w2BAGzguw"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.xnQkHIAkuWqZsTWskai4No4p3unQ3MJLnvt5_V-Oviw"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,167 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 145588,
|
||||
"output": 9241,
|
||||
"total": 154829
|
||||
},
|
||||
"fleet_cost_usd": 0.044646,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 199.73,
|
||||
"fleet_iterations_total": 23,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 1,
|
||||
"wrong": 2,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "failed_merge_conflict",
|
||||
"integrated_tests_pass": false,
|
||||
"prod_replay_pass": null,
|
||||
"production_replay": {
|
||||
"ran": false,
|
||||
"reason": "integrated tree is red"
|
||||
},
|
||||
"merge_conflict_count": 2,
|
||||
"merge_conflicts": [
|
||||
{
|
||||
"file": "identity-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0",
|
||||
"a1"
|
||||
],
|
||||
"1": [
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "ledger-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0",
|
||||
"a1"
|
||||
],
|
||||
"1": [
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"cross_file_divergence": true,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_atlas",
|
||||
"a1": "kid:k_atlas",
|
||||
"a2": "kid:k_bellatrix"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 46976,
|
||||
"output": 2268,
|
||||
"total": 49244
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 189.2,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the blank signing key IDs in both identity-service and ledger-service configs to \"k_atlas\" (the platform's historical key). The \"externalize-secrets\" refactor had left activeKid and trustedKid as empty strings, causing signSession() to throw \"no active signing key selected\". Both services now use the same key from the shared keyring, and the test passes.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 42906,
|
||||
"output": 2751,
|
||||
"total": 45657
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 198.87,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 55706,
|
||||
"output": 4222,
|
||||
"total": 59928
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 82.2,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-0wpf_het/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by completing the externalize-secrets refactor. Both `identity-service/src/config.ts` (activeKid) and `ledger-service/src/config.ts` (trustedKid) now read their key ids from environment variables (`IDENTITY_ACTIVE_KID` / `LEDGER_TRUSTED_KID`) with a fallback to `k_bellatrix`, the platform's current live signing key per Q3 rotation. This resolves the \"no active signing key selected\" error that was thrown because the configs were left as empty strings after the refactor.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:52:06Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,164 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 160059,
|
||||
"output": 7569,
|
||||
"total": 167628
|
||||
},
|
||||
"fleet_cost_usd": 0.047996,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 50.65,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 2,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 2,
|
||||
"wrong": 1,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "failed_merge_conflict",
|
||||
"integrated_tests_pass": false,
|
||||
"prod_replay_pass": null,
|
||||
"production_replay": {
|
||||
"ran": false,
|
||||
"reason": "integrated tree is red"
|
||||
},
|
||||
"merge_conflict_count": 2,
|
||||
"merge_conflicts": [
|
||||
{
|
||||
"file": "identity-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0",
|
||||
"a2"
|
||||
],
|
||||
"1": [
|
||||
"a1"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "ledger-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0",
|
||||
"a2"
|
||||
],
|
||||
"1": [
|
||||
"a1"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"cross_file_divergence": true,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_bellatrix",
|
||||
"a1": "kid:k_vela",
|
||||
"a2": "kid:k_bellatrix"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 52187,
|
||||
"output": 2093,
|
||||
"total": 54280
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 44.6,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 48675,
|
||||
"output": 1957,
|
||||
"total": 50632
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 31.42,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 59197,
|
||||
"output": 3519,
|
||||
"total": 62716
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 49.75,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-dhjx6jna/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-dhjx6jna/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-dhjx6jna/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-dhjx6jna/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:53:04Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,160 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 115549,
|
||||
"output": 9371,
|
||||
"total": 124920
|
||||
},
|
||||
"fleet_cost_usd": 0.03629,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 50.98,
|
||||
"fleet_iterations_total": 18,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_wren",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_lyra",
|
||||
"a1": "kid:k_lyra",
|
||||
"a2": "kid:k_lyra"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 31058,
|
||||
"output": 2732,
|
||||
"total": 33790
|
||||
},
|
||||
"iterations": 5,
|
||||
"wall_clock_seconds": 32.27,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test was failing because both identity-service and ledger-service had empty key IDs (`activeKid: \"\"` and `trustedKid: \"\"`) after an \"externalize secrets\" refactor. The shared keyring (in `shared/src/keyring.ts`) has the keys committed, but the configs weren't selecting any. Fixed by setting `activeKid` in identity-service config to `\"k_lyra\"` and `trustedKid` in ledger-service config to the same key `\"k_lyra\"`, so signing and verification use matching HMAC material.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 42573,
|
||||
"output": 3509,
|
||||
"total": 46082
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 45.15,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Set activeKid in identity-service config and trustedKid in ledger-service config to \"k_lyra\" (the same shared keyring entry) so that the identity service can mint signed session tokens and the ledger service can verify them. Both configs had empty key IDs after the externalize-secrets refactor, which broke the end-to-end charge flow.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 41918,
|
||||
"output": 3130,
|
||||
"total": 45048
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 50.12,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-x6ocauax/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Set both identity-service config.activeKid and ledger-service config.trustedKid to \"k_lyra\" (a valid key from the shared keyring), fixing the \"no active signing key selected\" / \"no trusted verification key selected\" errors. The end-to-end charge test now passes because the identity service can sign tokens with a known key and the ledger service can verify them with the same key.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:53:57Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 3,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"lyra",
|
||||
"schist",
|
||||
"altair",
|
||||
"pyrite",
|
||||
"procyon",
|
||||
"jasper",
|
||||
"wren",
|
||||
"dorado",
|
||||
"atlas",
|
||||
"aludra",
|
||||
"electra",
|
||||
"draco",
|
||||
"feldspar",
|
||||
"bellatrix",
|
||||
"indus",
|
||||
"yildun",
|
||||
"polaris",
|
||||
"vega",
|
||||
"olivine",
|
||||
"octans",
|
||||
"castor",
|
||||
"nashira",
|
||||
"zephyr",
|
||||
"hematite",
|
||||
"norma",
|
||||
"quartz",
|
||||
"helios",
|
||||
"mizar",
|
||||
"regulus",
|
||||
"cobalt",
|
||||
"izar",
|
||||
"zircon",
|
||||
"sirius",
|
||||
"hydra",
|
||||
"dunite",
|
||||
"lyric",
|
||||
"wezen",
|
||||
"arcturus",
|
||||
"cirrus",
|
||||
"fomalhaut",
|
||||
"capella",
|
||||
"quasar",
|
||||
"perseus",
|
||||
"umbriel",
|
||||
"deneb",
|
||||
"fornax",
|
||||
"basalt",
|
||||
"kepler",
|
||||
"orion",
|
||||
"vela"
|
||||
],
|
||||
"correct_kid": "k_wren",
|
||||
"decoy_kids": [
|
||||
"k_vela",
|
||||
"k_orion",
|
||||
"k_castor",
|
||||
"k_izar",
|
||||
"k_indus",
|
||||
"k_deneb",
|
||||
"k_norma",
|
||||
"k_mizar",
|
||||
"k_procyon",
|
||||
"k_wezen",
|
||||
"k_jasper",
|
||||
"k_polaris",
|
||||
"k_lyric",
|
||||
"k_helios",
|
||||
"k_draco",
|
||||
"k_altair",
|
||||
"k_cobalt",
|
||||
"k_basalt",
|
||||
"k_atlas",
|
||||
"k_yildun",
|
||||
"k_umbriel",
|
||||
"k_pyrite",
|
||||
"k_lyra",
|
||||
"k_octans",
|
||||
"k_bellatrix",
|
||||
"k_arcturus",
|
||||
"k_zircon",
|
||||
"k_feldspar",
|
||||
"k_olivine",
|
||||
"k_cirrus",
|
||||
"k_aludra",
|
||||
"k_regulus",
|
||||
"k_nashira",
|
||||
"k_kepler",
|
||||
"k_hydra",
|
||||
"k_fornax",
|
||||
"k_dorado",
|
||||
"k_quartz",
|
||||
"k_capella",
|
||||
"k_zephyr",
|
||||
"k_dunite",
|
||||
"k_quasar",
|
||||
"k_sirius",
|
||||
"k_perseus",
|
||||
"k_fomalhaut",
|
||||
"k_electra",
|
||||
"k_hematite",
|
||||
"k_schist",
|
||||
"k_vega"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-3/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 3 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_wren",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.yI0umRYi6RrcE7UkGUA_WTOECWaS_Yx-ylnW9iBufKw"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.ouNmwCSFDzQq_cF1aQjDMKjvRK3gYs9jcch9qFK1W-Q"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.lWPOnwhTvpxcye7Bk-aeH7E79aiPYH4Uj6MaBXrVy6c"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.A3Rg3uS07yk9v3C6SNM0QVW-4pBPdThETm4FzFCzPNM"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.HCBaaUjIVbcZvMgOuzWzekJoLmD58r69P_FStbgtgmM"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,170 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 148193,
|
||||
"output": 8174,
|
||||
"total": 156367
|
||||
},
|
||||
"fleet_cost_usd": 0.044927,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 56.54,
|
||||
"fleet_iterations_total": 22,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 1,
|
||||
"wrong": 2,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "failed_merge_conflict",
|
||||
"integrated_tests_pass": false,
|
||||
"prod_replay_pass": null,
|
||||
"production_replay": {
|
||||
"ran": false,
|
||||
"reason": "integrated tree is red"
|
||||
},
|
||||
"merge_conflict_count": 2,
|
||||
"merge_conflicts": [
|
||||
{
|
||||
"file": "identity-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a1",
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "ledger-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a1",
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"cross_file_divergence": true,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_wren",
|
||||
"a1": "kid:k_vela",
|
||||
"a2": "kid:k_vela"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 43695,
|
||||
"output": 2259,
|
||||
"total": 45954
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 39.97,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "The end-to-end charge test was failing because the \"externalize-secrets\" refactor left both `activeKid` (identity-service config) and `trustedKid` (ledger-service config) as empty strings. The signer and verifier both reject an empty key id, causing the \"no active signing key selected\" error.\n\nFixed by setting both to `\"k_wren\"`, which is the current live signing key according to team memory (Q3 rotation migrated the signer to k_wren). The identity service now has a key to sign with and the ledger service has a matching key to verify against, so the end-to-end flow works.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 57941,
|
||||
"output": 3060,
|
||||
"total": 61001
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 55.69,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the blank signing/verification key ids in both service configs. The \"externalize-secrets\" refactor had left `activeKid` and `trustedKid` as empty strings in `identity-service/src/config.ts` and `ledger-service/src/config.ts` respectively, causing `signSession()` to throw \"no active signing key selected\" immediately. Set both configs to use `k_vela` (the historically working key from the shared keyring, confirmed by team memory OPS-522), so identity signs with the same key the ledger verifies against, and the full identity -> gateway -> ledger charge path completes successfully with a 200 response.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 46557,
|
||||
"output": 2855,
|
||||
"total": 49412
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 42.66,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-qeni3flv/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting both identity-service's `config.activeKid` and ledger-service's `ledgerConfig.trustedKid` to `\"k_vela\"` \u2014 a valid key from the shared keyring. Both were left blank during the externalize-secrets refactor, causing `signSession` to throw \"no active signing key selected\". Now the signing and verification paths share the same HMAC key and the full charge flow works.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:55:01Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 175867,
|
||||
"output": 8198,
|
||||
"total": 184065
|
||||
},
|
||||
"fleet_cost_usd": 0.052686,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 48.52,
|
||||
"fleet_iterations_total": 23,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 3,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "fixed_correctly",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": true,
|
||||
"production_replay": {
|
||||
"ok": true,
|
||||
"verified": 5,
|
||||
"total": 5,
|
||||
"failures": [],
|
||||
"signed_under": "k_wren",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_wren",
|
||||
"a1": "kid:k_wren",
|
||||
"a2": "kid:k_wren"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 62736,
|
||||
"output": 2219,
|
||||
"total": 64955
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 32.09,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by restoring the signing/verification key IDs in both identity-service and ledger-service configs. The \"externalize-secrets\" refactor had left both activeKid and trustedKid blank (\"\"), causing signSession to throw \"no active signing key selected\". The shared memory rotation runbook indicated the team had migrated to k_wren. Set identity-service/src/config.ts activeKid=\"k_wren\" and ledger-service/src/config.ts trustedKid=\"k_wren\" so both services agree on the same key from the shared keyring, which allows the token to be minted, forwarded, and verified successfully.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 54928,
|
||||
"output": 2900,
|
||||
"total": 57828
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 42.55,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting the signing/verification key ids in both identity-service and ledger-service configs to \"k_wren\" (the key the platform rotated to per the backfill). The externalize-secrets refactor had left both activeKid and trustedKid as empty strings, causing signSession to throw \"no active signing key selected\". Both services now use the same key from the shared keyring, so the identity service can mint tokens and the ledger service can verify them.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 58203,
|
||||
"output": 3079,
|
||||
"total": 61282
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 47.64,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-sfj707b5/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:55:57Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,154 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 137489,
|
||||
"output": 7711,
|
||||
"total": 145200
|
||||
},
|
||||
"fleet_cost_usd": 0.041736,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 47.9,
|
||||
"fleet_iterations_total": 22,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_indus",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_talc",
|
||||
"a1": "kid:k_talc",
|
||||
"a2": "kid:k_talc"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 47468,
|
||||
"output": 2360,
|
||||
"total": 49828
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 34.42,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Set the blank `activeKid` in identity-service/src/config.ts and `trustedKid` in ledger-service/src/config.ts to `\"k_talc\"`, a valid key from the shared keyring. This allows signSession() to select a signing key and verifyToken() to use the same key for HMAC-SHA256 verification, making the end-to-end charge test pass.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 42968,
|
||||
"output": 3274,
|
||||
"total": 46242
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 47.05,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-7hk6o5kd/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-7hk6o5kd/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-7hk6o5kd/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-7hk6o5kd/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test was failing because both identity-service and ledger-service configs had empty key IDs (activeKid and trustedKid set to \"\"). The signSession function threw \"no active signing key selected\" when activeKid was empty. Fixed by setting both configs to use \"k_talc\", a valid key from the shared keyring, so the identity service can mint tokens and the ledger service can verify them with the same HMAC key.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 47053,
|
||||
"output": 2077,
|
||||
"total": 49130
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 29.84,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting matching signing/verification key ids in both identity-service and ledger-service configs. Both `activeKid` (identity-service/src/config.ts) and `trustedKid` (ledger-service/src/config.ts) were left blank after an \"externalize-secrets\" refactor. Set both to `\"k_talc\"`, a key present in the shared keyring, so that signSession() can mint a token and verifyToken() can validate it using the same HMAC key.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:56:47Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 4,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"talc",
|
||||
"atlas",
|
||||
"quartz",
|
||||
"zircon",
|
||||
"dunite",
|
||||
"electra",
|
||||
"juno",
|
||||
"rutile",
|
||||
"perseus",
|
||||
"cobalt",
|
||||
"pyrite",
|
||||
"izar",
|
||||
"kepler",
|
||||
"fomalhaut",
|
||||
"polaris",
|
||||
"yildun",
|
||||
"norma",
|
||||
"mensa",
|
||||
"tarazed",
|
||||
"altair",
|
||||
"vega",
|
||||
"helios",
|
||||
"vela",
|
||||
"cirrus",
|
||||
"nashira",
|
||||
"arcturus",
|
||||
"dorado",
|
||||
"indus",
|
||||
"octans",
|
||||
"procyon",
|
||||
"fornax",
|
||||
"tucana",
|
||||
"quasar",
|
||||
"olivine",
|
||||
"mizar",
|
||||
"draco",
|
||||
"kyanite",
|
||||
"orion",
|
||||
"wren",
|
||||
"borealis",
|
||||
"bellatrix",
|
||||
"jasper",
|
||||
"gemini",
|
||||
"lyra",
|
||||
"feldspar",
|
||||
"umbriel",
|
||||
"antares",
|
||||
"spica",
|
||||
"schist",
|
||||
"wezen"
|
||||
],
|
||||
"correct_kid": "k_indus",
|
||||
"decoy_kids": [
|
||||
"k_draco",
|
||||
"k_wezen",
|
||||
"k_dunite",
|
||||
"k_gemini",
|
||||
"k_schist",
|
||||
"k_perseus",
|
||||
"k_tarazed",
|
||||
"k_octans",
|
||||
"k_tucana",
|
||||
"k_lyra",
|
||||
"k_mensa",
|
||||
"k_feldspar",
|
||||
"k_kyanite",
|
||||
"k_orion",
|
||||
"k_arcturus",
|
||||
"k_wren",
|
||||
"k_quasar",
|
||||
"k_altair",
|
||||
"k_izar",
|
||||
"k_cobalt",
|
||||
"k_vela",
|
||||
"k_zircon",
|
||||
"k_pyrite",
|
||||
"k_kepler",
|
||||
"k_norma",
|
||||
"k_dorado",
|
||||
"k_vega",
|
||||
"k_bellatrix",
|
||||
"k_yildun",
|
||||
"k_juno",
|
||||
"k_spica",
|
||||
"k_helios",
|
||||
"k_olivine",
|
||||
"k_borealis",
|
||||
"k_atlas",
|
||||
"k_electra",
|
||||
"k_fornax",
|
||||
"k_quartz",
|
||||
"k_polaris",
|
||||
"k_rutile",
|
||||
"k_nashira",
|
||||
"k_cirrus",
|
||||
"k_procyon",
|
||||
"k_umbriel",
|
||||
"k_talc",
|
||||
"k_fomalhaut",
|
||||
"k_mizar",
|
||||
"k_antares",
|
||||
"k_jasper"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-4/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 4 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_indus",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.1d4snssdd_9-t9YnCrcsJCjz_MryJaRneD-cw0wkEEk"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.tqJEFmSZu2u35P6q-nOjxhwXW8gNQErcVAqMbsPushM"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.JN_BMrz3Vx1G7UqnFZuY0t6RNrATz7JUWIORip76DHM"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.K6ePcOr5i78UdRcRwOJtU3mr_jY6RYU1ieH6PFZBFK0"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.CVbawANKnegnZGDbkKafaUMFCIwqEyG3It7KZ4RjveA"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 165557,
|
||||
"output": 10597,
|
||||
"total": 176154
|
||||
},
|
||||
"fleet_cost_usd": 0.050807,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 49.12,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "fixed_correctly",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": true,
|
||||
"production_replay": {
|
||||
"ok": true,
|
||||
"verified": 5,
|
||||
"total": 5,
|
||||
"failures": [],
|
||||
"signed_under": "k_indus",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_indus",
|
||||
"a1": "kid:k_indus",
|
||||
"a2": "kid:k_indus"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 60031,
|
||||
"output": 3730,
|
||||
"total": 63761
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 48.28,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 49998,
|
||||
"output": 3687,
|
||||
"total": 53685
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 46.74,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 55528,
|
||||
"output": 3180,
|
||||
"total": 58708
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 47.46,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-rag-awa7s15q/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the blank signing key IDs in both identity-service and ledger-service configs. The \"externalize-secrets\" refactor had left both `activeKid` (identity-service/config.ts) and `trustedKid` (ledger-service/config.ts) as empty strings, causing `signSession()` to throw \"no active signing key selected\". Set both to `k_indus`, the current live signing key confirmed in the shared keyring and team memory. The test now passes: identity-service mints a token signed with `k_indus`, gateway-service forwards it, and ledger-service verifies with the same key and returns 200.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:57:44Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 173041,
|
||||
"output": 7268,
|
||||
"total": 180309
|
||||
},
|
||||
"fleet_cost_usd": 0.051504,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 41.85,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 2,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 3,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 1,
|
||||
"wrong": 1,
|
||||
"other": 1
|
||||
},
|
||||
"fleet_verdict": "failed_merge_conflict",
|
||||
"integrated_tests_pass": false,
|
||||
"prod_replay_pass": null,
|
||||
"production_replay": {
|
||||
"ran": false,
|
||||
"reason": "integrated tree is red"
|
||||
},
|
||||
"merge_conflict_count": 2,
|
||||
"merge_conflicts": [
|
||||
{
|
||||
"file": "identity-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"file": "ledger-service/src/config.ts",
|
||||
"kind": "same_file_divergent_edits",
|
||||
"distinct_orders": [
|
||||
"unparseable",
|
||||
"unparseable"
|
||||
],
|
||||
"agents": {
|
||||
"0": [
|
||||
"a0"
|
||||
],
|
||||
"1": [
|
||||
"a2"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"cross_file_divergence": true,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_indus",
|
||||
"a1": "kid_unset",
|
||||
"a2": "kid:k_talc"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 59749,
|
||||
"output": 2341,
|
||||
"total": 62090
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 39.01,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by restoring the active signing key IDs that were blanked out during the \"externalize-secrets\" refactor. The identity-service `activeKid` and ledger-service `trustedKid` were both empty strings, causing `signSession` to throw \"no active signing key selected\". Set both to `k_indus` (the key migrated to in the Q3 rotation runbook, surfaced via retroactive salience backfill). Now the identity-service correctly mints a signed token and the ledger-service verifies it with the same shared key.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 55548,
|
||||
"output": 2621,
|
||||
"total": 58169
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 40.26,
|
||||
"status": "looped",
|
||||
"local_tests_pass": false,
|
||||
"files_written": [],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 57744,
|
||||
"output": 2306,
|
||||
"total": 60050
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 40.98,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the signing/verification key IDs in both identity-service and ledger-service configs. The \"externalize-secrets\" refactor had blanked activeKid (identity-service/src/config.ts) and trustedKid (ledger-service/src/config.ts), causing signSession to throw \"no active signing key selected\" at identity-service/src/signer.ts:31. Set both to \"k_talc\" from the shared keyring so the issuer and verifier use the same HMAC key, allowing the full charge flow to succeed.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:58:33Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,157 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 123016,
|
||||
"output": 8415,
|
||||
"total": 131431
|
||||
},
|
||||
"fleet_cost_usd": 0.037979,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 64.21,
|
||||
"fleet_iterations_total": 20,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_dorado",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_pyrite",
|
||||
"a1": "kid:k_pyrite",
|
||||
"a2": "kid:k_pyrite"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 31898,
|
||||
"output": 3106,
|
||||
"total": 35004
|
||||
},
|
||||
"iterations": 5,
|
||||
"wall_clock_seconds": 49.97,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-0/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-0/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting the `activeKid` in identity-service config and `trustedKid` in ledger-service config to a valid key from the shared keyring (k_pyrite). Both were left blank by the externalize-secrets refactor, causing signSession to throw \"no active signing key selected\". Both configs now read from env vars with a dev fallback, allowing the test to mint and verify session tokens correctly.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 37803,
|
||||
"output": 2143,
|
||||
"total": 39946
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 45.69,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the end-to-end charge test by setting the active signing key ID in the identity-service config and the trusted verification key ID in the ledger-service config. Both were left as empty strings (\"\"), commented as intentionally blanked during an \"externalize secrets\" refactor. The signSession() function in identity-service checked config.activeKid and threw \"no active signing key selected\" when it was falsy, and verifyToken() in ledger-service similarly checked ledgerConfig.trustedKid. Set both to \"k_pyrite\", a valid key from the shared keyring, so that tokens can be minted and verified.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 53315,
|
||||
"output": 3166,
|
||||
"total": 56481
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 63.32,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-2/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-anarchy-ogbyv5ku/agent-2/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed failing end-to-end charge test by setting the blank activeKid in identity-service/src/config.ts and trustedKid in ledger-service/src/config.ts to \"k_pyrite\", a key present in the shared keyring. This allows signSession to mint a signed token and verifyToken to validate it, making the full identity\u2192gateway\u2192ledger charge flow succeed.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:59:39Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 5,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"pyrite",
|
||||
"dunite",
|
||||
"quartz",
|
||||
"antares",
|
||||
"regulus",
|
||||
"vega",
|
||||
"procyon",
|
||||
"altair",
|
||||
"dorado",
|
||||
"juno",
|
||||
"garnet",
|
||||
"yildun",
|
||||
"sirius",
|
||||
"castor",
|
||||
"rigel",
|
||||
"lyra",
|
||||
"bellatrix",
|
||||
"wezen",
|
||||
"fornax",
|
||||
"rutile",
|
||||
"hydra",
|
||||
"kepler",
|
||||
"mensa",
|
||||
"atlas",
|
||||
"mizar",
|
||||
"feldspar",
|
||||
"indus",
|
||||
"talc",
|
||||
"arcturus",
|
||||
"tucana",
|
||||
"orion",
|
||||
"izar",
|
||||
"norma",
|
||||
"zircon",
|
||||
"tarazed",
|
||||
"capella",
|
||||
"electra",
|
||||
"hematite",
|
||||
"polaris",
|
||||
"deneb",
|
||||
"schist",
|
||||
"kyanite",
|
||||
"umbriel",
|
||||
"aludra",
|
||||
"quasar",
|
||||
"jasper",
|
||||
"draco",
|
||||
"spica",
|
||||
"pavo",
|
||||
"fomalhaut"
|
||||
],
|
||||
"correct_kid": "k_dorado",
|
||||
"decoy_kids": [
|
||||
"k_quartz",
|
||||
"k_zircon",
|
||||
"k_rutile",
|
||||
"k_tucana",
|
||||
"k_deneb",
|
||||
"k_procyon",
|
||||
"k_hematite",
|
||||
"k_pyrite",
|
||||
"k_altair",
|
||||
"k_orion",
|
||||
"k_sirius",
|
||||
"k_indus",
|
||||
"k_juno",
|
||||
"k_aludra",
|
||||
"k_arcturus",
|
||||
"k_tarazed",
|
||||
"k_garnet",
|
||||
"k_draco",
|
||||
"k_hydra",
|
||||
"k_electra",
|
||||
"k_dunite",
|
||||
"k_quasar",
|
||||
"k_spica",
|
||||
"k_mensa",
|
||||
"k_schist",
|
||||
"k_bellatrix",
|
||||
"k_norma",
|
||||
"k_lyra",
|
||||
"k_mizar",
|
||||
"k_izar",
|
||||
"k_talc",
|
||||
"k_regulus",
|
||||
"k_yildun",
|
||||
"k_umbriel",
|
||||
"k_atlas",
|
||||
"k_jasper",
|
||||
"k_pavo",
|
||||
"k_kyanite",
|
||||
"k_fomalhaut",
|
||||
"k_wezen",
|
||||
"k_feldspar",
|
||||
"k_fornax",
|
||||
"k_antares",
|
||||
"k_polaris",
|
||||
"k_kepler",
|
||||
"k_capella",
|
||||
"k_vega",
|
||||
"k_rigel",
|
||||
"k_castor"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-5/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 5 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_dorado",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.I2ppWYeICq_tJaIvMXqggZbWh0wg0F28tr3IgQZZuj8"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.UKh-56OKaSbrQIE1rzWDjabT8eEX9bw2h42UkhYmcrE"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.tnFQyJaCPHl4zX9YUNaMfJdmreTnUxUIONRqAyksrps"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.CVk_aoBgblRq-CF2pz3vZ6u7aXWQd6x-auRP1lJn-Ro"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.hEWl6CEbIKKW1vtqqBggF-lWDwO-x1R8uem51Iezg1M"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,151 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 129060,
|
||||
"output": 6714,
|
||||
"total": 135774
|
||||
},
|
||||
"fleet_cost_usd": 0.038957,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 40.98,
|
||||
"fleet_iterations_total": 21,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_dorado",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_quartz",
|
||||
"a1": "kid:k_quartz",
|
||||
"a2": "kid:k_quartz"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 45793,
|
||||
"output": 2227,
|
||||
"total": 48020
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 40.09,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting the signing and verification key IDs in both the identity-service and ledger-service configs. Both `activeKid` (identity-service) and `trustedKid` (ledger-service) were left as empty strings during an \"externalize-secrets\" refactor that was never completed, causing signSession to throw \"no active signing key selected\". Set both to `k_quartz`, a known operational key present in the shared keyring, so the issuer and verifier agree on the same signing material.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 44578,
|
||||
"output": 1992,
|
||||
"total": 46570
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 32.8,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 38689,
|
||||
"output": 2495,
|
||||
"total": 41184
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 30.66,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "The end-to-end charge test was failing because both the identity-service (signer) and ledger-service (verifier) had blank key IDs (`activeKid: \"\"` and `trustedKid: \"\"`) \u2014 left blank by the \"externalize-secrets refactor\". The signer in `identity-service/src/signer.ts` throws \"no active signing key selected\" when `config.activeKid` is falsy.\n\nFixed by setting both configs to use the same shared keyring key (`k_quartz`):\n- `identity-service/src/config.ts`: set `activeKid: \"k_quartz\"`\n- `ledger-service/src/config.ts`: set `trustedKid: \"k_quartz\"`\n\nThis ensures identity-service can sign session tokens and ledger-service can verify them with the same HMAC key.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T14:00:28Z"
|
||||
}
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "deepseek/deepseek-v4-flash",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 169689,
|
||||
"output": 6955,
|
||||
"total": 176644
|
||||
},
|
||||
"fleet_cost_usd": 0.050434,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.28,
|
||||
"output_per_mtok": 0.42
|
||||
},
|
||||
"fleet_wall_clock_seconds": 177.98,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 3,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "fixed_correctly",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": true,
|
||||
"production_replay": {
|
||||
"ok": true,
|
||||
"verified": 5,
|
||||
"total": 5,
|
||||
"failures": [],
|
||||
"signed_under": "k_dorado",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_dorado",
|
||||
"a1": "kid:k_dorado",
|
||||
"a2": "kid:k_dorado"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 53548,
|
||||
"output": 1845,
|
||||
"total": 55393
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 177.06,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 63109,
|
||||
"output": 2917,
|
||||
"total": 66026
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 65.57,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-f5fvked1/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-f5fvked1/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-f5fvked1/agent-1/identity-service/src/config.ts",
|
||||
"/var/folders/tg/9hll24px5v17ys2sl78r1vb80000gn/T/fleet-sync-f5fvked1/agent-1/ledger-service/src/config.ts"
|
||||
],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test by setting both `activeKid` (identity-service) and `trustedKid` (ledger-service) to `\"k_dorado\"` \u2014 the same key ID from the shared keyring. The \"externalize-secrets\" refactor had blanked both configs, so the signer threw \"no active signing key selected\" and the verifier would have thrown \"no trusted verification key selected\". Using `k_dorado` ensures HMAC signing and verification use the same key material, which the Q3 rotation runbook had already converged on.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 53032,
|
||||
"output": 2193,
|
||||
"total": 55225
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 42.38,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T14:03:34Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
151
benchmarks/silent-rotation/results/glm-5.2-trial-1/anarchy.json
Normal file
151
benchmarks/silent-rotation/results/glm-5.2-trial-1/anarchy.json
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "z-ai/glm-5.2",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 124989,
|
||||
"output": 10682,
|
||||
"total": 135671
|
||||
},
|
||||
"fleet_cost_usd": 0.098494,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.6,
|
||||
"output_per_mtok": 2.2
|
||||
},
|
||||
"fleet_wall_clock_seconds": 151.95,
|
||||
"fleet_iterations_total": 20,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_yildun",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_pavo",
|
||||
"a1": "kid:k_pavo",
|
||||
"a2": "kid:k_pavo"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 45542,
|
||||
"output": 3093,
|
||||
"total": 48635
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 85.46,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed because the \"externalize secrets\" refactor left both `identity-service` (`config.activeKid`) and `ledger-service` (`ledgerConfig.trustedKid`) with blank key ids, so `signSession()` threw \"no active signing key selected\". Fixed by restoring the concrete key id `k_pavo` (the first provisioned key in the shared keyring) in both configs, ensuring identity signs and ledger verifies with the same HMAC key. Verified with run_tests (exit_code=0).",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 48085,
|
||||
"output": 6513,
|
||||
"total": 54598
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 150.97,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed with \"no active signing key selected\" because both service configs had their HMAC key ids blanked during an \"externalize-secrets\" refactor: identity-service `activeKid: \"\"` (so signSession threw) and ledger-service `trustedKid: \"\"` (so verifyToken would also fail). The rest of the integrity chain was already correct \u2014 both services build byte-identical canonical claim strings (sub, plan, iat, exp joined by newline), the gateway passes the bearer token through verbatim, and the b64url/HMAC logic matches between signer and verifier. The only break was that sign and verify had no key, and they must share the SAME key id for the HMAC to agree.\n\nFix: set both `activeKid` (identity-service/src/config.ts) and `trustedKid` (ledger-service/src/config.ts) to default to \"k_pavo\" \u2014 the first provisioned keyring entry, matching the repo's dangling comment fragment \"entry the platform shipped with\" \u2014 while keeping an env override (IDENTITY_ACTIVE_KID / LEDGER_TRUSTED_KID) to honor the externalize-secrets intent. With both sides defaulting to the same key, a freshly-minted token verifies and the ledger returns 200. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 31362,
|
||||
"output": 1076,
|
||||
"total": 32438
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 38.2,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed because the \"externalize-secrets\" refactor left both identity-service's `activeKid` and ledger-service's `trustedKid` blank, so signSession threw \"no active signing key selected\". Fixed by setting both configs to the same committed dev key from the shared keyring (`k_pavo`), with an optional `MERIDIAN_SIGNING_KID` env override for production. The two services now agree on the signing key, token minting and verification succeed, and the 200 charge passes.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:23:25Z"
|
||||
}
|
||||
111
benchmarks/silent-rotation/results/glm-5.2-trial-1/manifest.json
Normal file
111
benchmarks/silent-rotation/results/glm-5.2-trial-1/manifest.json
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 1,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"pavo",
|
||||
"izar",
|
||||
"vela",
|
||||
"capella",
|
||||
"fornax",
|
||||
"cobalt",
|
||||
"gemini",
|
||||
"basalt",
|
||||
"kyanite",
|
||||
"yildun",
|
||||
"antares",
|
||||
"vega",
|
||||
"lyric",
|
||||
"deneb",
|
||||
"norma",
|
||||
"feldspar",
|
||||
"tucana",
|
||||
"mizar",
|
||||
"helios",
|
||||
"altair",
|
||||
"orion",
|
||||
"atlas",
|
||||
"aludra",
|
||||
"zephyr",
|
||||
"rutile",
|
||||
"pyrite",
|
||||
"schist",
|
||||
"hydra",
|
||||
"umbriel",
|
||||
"dunite",
|
||||
"lyra",
|
||||
"procyon",
|
||||
"octans",
|
||||
"talc",
|
||||
"regulus",
|
||||
"dorado",
|
||||
"quasar",
|
||||
"draco",
|
||||
"juno",
|
||||
"bellatrix",
|
||||
"wezen",
|
||||
"perseus",
|
||||
"indus",
|
||||
"sirius",
|
||||
"spica",
|
||||
"electra",
|
||||
"jasper",
|
||||
"cirrus",
|
||||
"rigel",
|
||||
"wren"
|
||||
],
|
||||
"correct_kid": "k_yildun",
|
||||
"decoy_kids": [
|
||||
"k_wezen",
|
||||
"k_electra",
|
||||
"k_pavo",
|
||||
"k_kyanite",
|
||||
"k_procyon",
|
||||
"k_sirius",
|
||||
"k_talc",
|
||||
"k_fornax",
|
||||
"k_cirrus",
|
||||
"k_quasar",
|
||||
"k_basalt",
|
||||
"k_zephyr",
|
||||
"k_spica",
|
||||
"k_rutile",
|
||||
"k_feldspar",
|
||||
"k_norma",
|
||||
"k_hydra",
|
||||
"k_juno",
|
||||
"k_regulus",
|
||||
"k_schist",
|
||||
"k_perseus",
|
||||
"k_rigel",
|
||||
"k_bellatrix",
|
||||
"k_orion",
|
||||
"k_umbriel",
|
||||
"k_draco",
|
||||
"k_cobalt",
|
||||
"k_gemini",
|
||||
"k_indus",
|
||||
"k_aludra",
|
||||
"k_vega",
|
||||
"k_altair",
|
||||
"k_pyrite",
|
||||
"k_jasper",
|
||||
"k_capella",
|
||||
"k_izar",
|
||||
"k_dunite",
|
||||
"k_octans",
|
||||
"k_lyra",
|
||||
"k_dorado",
|
||||
"k_vela",
|
||||
"k_lyric",
|
||||
"k_tucana",
|
||||
"k_helios",
|
||||
"k_antares",
|
||||
"k_wren",
|
||||
"k_atlas",
|
||||
"k_deneb",
|
||||
"k_mizar"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-1/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 1 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_yildun",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.ZB1iKOyWil9tDI_7oT2hr4EfK5oKTtlTFwlIlCZUul4"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.39kYP3f9sG2DdxMTfi1H2BAy6m9zSzYzEwNRypy1dqg"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.m2a1sWc-YEoFlFFQgQNIFpGf7s5pyaTAaiWeMWDNIno"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.x0CIXum6hfCJs6ra-wRXbGT2hXjHqJulrMzb_qt98Xk"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.P6icZZtV2o35B78z7Dq4yDBsNRnLUF9DJMK7M2J7PM4"
|
||||
}
|
||||
]
|
||||
}
|
||||
151
benchmarks/silent-rotation/results/glm-5.2-trial-1/rag.json
Normal file
151
benchmarks/silent-rotation/results/glm-5.2-trial-1/rag.json
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "z-ai/glm-5.2",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 96689,
|
||||
"output": 3761,
|
||||
"total": 100450
|
||||
},
|
||||
"fleet_cost_usd": 0.066288,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.6,
|
||||
"output_per_mtok": 2.2
|
||||
},
|
||||
"fleet_wall_clock_seconds": 40.68,
|
||||
"fleet_iterations_total": 18,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_yildun",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_wezen",
|
||||
"a1": "kid:k_wezen",
|
||||
"a2": "kid:k_wezen"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 30793,
|
||||
"output": 1148,
|
||||
"total": 31941
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 36.66,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "The end-to-end charge test failed because the \"externalize secrets\" refactor left both `identity-service` (`config.activeKid`) and `ledger-service` (`ledgerConfig.trustedKid`) key-id selectors blank, so `signSession` threw \"no active signing key selected\" and the ledger verifier would similarly reject tokens. Per team memory (OPS-522, which named `k_wezen` as the platform's signing key id), I set both selectors to default to `k_wezen` (still overridable via SIGNING_KID / TRUSTED_KID env vars). This keeps the signer and verifier on the same key from the shared keyring so a freshly-minted token's canonical-digest HMAC signature verifies end-to-end. The test now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 33528,
|
||||
"output": 1262,
|
||||
"total": 34790
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 36.35,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "The end-to-end charge test failed with \"no active signing key selected\" (from identity-service/src/signer.ts). Root cause: an \"externalize-secrets\" refactor left both identity-service `activeKid` and ledger-service `trustedKid` blank. Since the token header carries no key id, the verifier must independently select the same key the issuer used. Per team memory (ticket OPS-522: \"reset the key id to k_wezen and verification recovered\"), the platform's shipped default key is `k_wezen` (present in the shared keyring). Fixed by setting identity-service/src/config.ts `activeKid` = \"k_wezen\" and ledger-service/src/config.ts `trustedKid` = \"k_wezen\", restoring the matching mint/verify key pair. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 32368,
|
||||
"output": 1351,
|
||||
"total": 33719
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 39.71,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "Fixed the failing end-to-end charge test. Root cause: the \"externalize-secrets\" refactor left both key-id configs blank \u2014 `identity-service/src/config.ts` (activeKid) and `ledger-service/src/config.ts` (trustedKid) \u2014 so signSession() crashed with \"no active signing key selected\" before a token could be minted. Per team memory (OPS-522: on-call reset the key id to k_wezen and verification recovered), set both the issuer's activeKid and the ledger's trustedKid to \"k_wezen\" (with an env-var override to preserve the KMS-injection path for production). Both sides now select the same shared keyring entry, so the minted signature matches the recomputed canonical digest on verification and the charge returns 200.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:24:15Z"
|
||||
}
|
||||
130
benchmarks/silent-rotation/results/glm-5.2-trial-1/sync.json
Normal file
130
benchmarks/silent-rotation/results/glm-5.2-trial-1/sync.json
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "z-ai/glm-5.2",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 124944,
|
||||
"output": 3900,
|
||||
"total": 128844
|
||||
},
|
||||
"fleet_cost_usd": 0.083546,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.6,
|
||||
"output_per_mtok": 2.2
|
||||
},
|
||||
"fleet_wall_clock_seconds": 47.17,
|
||||
"fleet_iterations_total": 23,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 3,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "fixed_correctly",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": true,
|
||||
"production_replay": {
|
||||
"ok": true,
|
||||
"verified": 5,
|
||||
"total": 5,
|
||||
"failures": [],
|
||||
"signed_under": "k_yildun",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_yildun",
|
||||
"a1": "kid:k_yildun",
|
||||
"a2": "kid:k_yildun"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 42933,
|
||||
"output": 1306,
|
||||
"total": 44239
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 42.98,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The e2-end charge test failed with \"no active signing key selected\" because the \"externalize-secrets\" refactor left both identity-service/config.activeKid and ledger-service/config.trustedKid blank. The vestige_backfill surfaced the Q3 rotation runbook memory: the platform's active signer is k_yildun (old material aged out). Fix: set activeKid=\"k_yildun\" in identity-service/src/config.ts and trustedKid=\"k_yildun\" in ledger-service/src/config.ts so the issuer and verifier agree on the same keyring key. The canonical digest recipes already matched between services, so no other changes were needed. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 32805,
|
||||
"output": 1002,
|
||||
"total": 33807
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 36.48,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 49206,
|
||||
"output": 1592,
|
||||
"total": 50798
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 46.17,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The e2e charge test failed with \"no active signing key selected\" because the externalize-secrets refactor blanked the key id in both identity-service/src/config.ts (activeKid) and ledger-service/src/config.ts (trustedKid). The vestige_backfill surfaced the Q3 rotation runbook which migrated the live signer to k_yildun and aged out the old material. Fixed by setting both configs' kid default to \"k_yildun\" (overridable via IDENTITY_ACTIVE_KID / LEDGER_TRUSTED_KID env vars). The canonical claim digest implementations in both services already matched, so no other changes were needed. Test now exits 0.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:25:10Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
151
benchmarks/silent-rotation/results/glm-5.2-trial-2/anarchy.json
Normal file
151
benchmarks/silent-rotation/results/glm-5.2-trial-2/anarchy.json
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
{
|
||||
"mode": "anarchy",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "z-ai/glm-5.2",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 112682,
|
||||
"output": 3889,
|
||||
"total": 116571
|
||||
},
|
||||
"fleet_cost_usd": 0.076165,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.6,
|
||||
"output_per_mtok": 2.2
|
||||
},
|
||||
"fleet_wall_clock_seconds": 45.59,
|
||||
"fleet_iterations_total": 20,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-anarchy-a0.json",
|
||||
"transcript-anarchy-a1.json",
|
||||
"transcript-anarchy-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_bellatrix",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_vela",
|
||||
"a1": "kid:k_vela",
|
||||
"a2": "kid:k_vela"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 31723,
|
||||
"output": 1189,
|
||||
"total": 32912
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 32.74,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed with \"no active signing key selected\" because both identity-service's `config.activeKid` and ledger-service's `ledgerConfig.trustedKid` were left blank by the \"externalize-secrets\" refactor. Set both to the same platform-shipped default key id (`k_vela`) from the shared keyring so the issuer can mint tokens and the ledger can verify them. The canonical claim digest recipe is already consistent across both services. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 42020,
|
||||
"output": 1404,
|
||||
"total": 43424
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 44.5,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed because the \"externalize-secrets\" refactor left both services' key ids blank: identity-service `config.activeKid = \"\"` (causing `signSession` to throw \"no active signing key selected\") and ledger-service `ledgerConfig.trustedKid = \"\"` (which would prevent verification). \n\nFix: set both configs to default to a concrete key id present in the shared keyring \u2014 `k_vela` \u2014 while still allowing override via env vars (`IDENTITY_ACTIVE_KID` / `LEDGER_TRUSTED_KID`). Both services must use the SAME key id for the HMAC signature to verify, so I picked a single shared default and made the two configs mirror each other. The canonical claim digest recipes already agree (identical field order, newline-joined), so once both sides select the same key, signing and verification produce byte-identical results.\n\nVerified: `npm test` now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "anarchy",
|
||||
"tokens": {
|
||||
"input": 38939,
|
||||
"output": 1296,
|
||||
"total": 40235
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 44.39,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed because the \"externalize secrets\" refactor left both identity-service's `config.activeKid` and ledger-service's `ledgerConfig.trustedKid` as empty strings. With `activeKid` blank, `signSession` threw \"no active signing key selected\" before any token was minted; with `trustedKid` blank the ledger would also have refused to verify.\n\nFixed by restoring a shared default key id (`k_vela`, the first key in the shared KEYRING) to both configs, while preserving env-var overrides (IDENTITY_ACTIVE_KID / LEDGER_TRUSTED_KID) so production can still inject material from the KMS. Both sides now select the same HMAC key, so the freshly-minted token verifies on the ledger and the charge returns 200. Verified with run_tests (exit_code=0).",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:25:58Z"
|
||||
}
|
||||
111
benchmarks/silent-rotation/results/glm-5.2-trial-2/manifest.json
Normal file
111
benchmarks/silent-rotation/results/glm-5.2-trial-2/manifest.json
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"trial": 2,
|
||||
"master_seed": 2026,
|
||||
"key_names": [
|
||||
"vela",
|
||||
"fornax",
|
||||
"mizar",
|
||||
"quasar",
|
||||
"castor",
|
||||
"feldspar",
|
||||
"tucana",
|
||||
"garnet",
|
||||
"sirius",
|
||||
"draco",
|
||||
"capella",
|
||||
"dunite",
|
||||
"spica",
|
||||
"jasper",
|
||||
"perseus",
|
||||
"basalt",
|
||||
"olivine",
|
||||
"polaris",
|
||||
"aludra",
|
||||
"procyon",
|
||||
"antares",
|
||||
"hematite",
|
||||
"yildun",
|
||||
"lyric",
|
||||
"rigel",
|
||||
"norma",
|
||||
"schist",
|
||||
"borealis",
|
||||
"rutile",
|
||||
"tarazed",
|
||||
"zephyr",
|
||||
"kyanite",
|
||||
"deneb",
|
||||
"cirrus",
|
||||
"indus",
|
||||
"kepler",
|
||||
"quartz",
|
||||
"pavo",
|
||||
"regulus",
|
||||
"wezen",
|
||||
"atlas",
|
||||
"orion",
|
||||
"electra",
|
||||
"wren",
|
||||
"cobalt",
|
||||
"bellatrix",
|
||||
"gemini",
|
||||
"umbriel",
|
||||
"octans",
|
||||
"helios"
|
||||
],
|
||||
"correct_kid": "k_bellatrix",
|
||||
"decoy_kids": [
|
||||
"k_atlas",
|
||||
"k_rutile",
|
||||
"k_yildun",
|
||||
"k_pavo",
|
||||
"k_norma",
|
||||
"k_tucana",
|
||||
"k_perseus",
|
||||
"k_dunite",
|
||||
"k_orion",
|
||||
"k_wren",
|
||||
"k_spica",
|
||||
"k_cirrus",
|
||||
"k_kepler",
|
||||
"k_aludra",
|
||||
"k_hematite",
|
||||
"k_cobalt",
|
||||
"k_procyon",
|
||||
"k_octans",
|
||||
"k_kyanite",
|
||||
"k_quartz",
|
||||
"k_helios",
|
||||
"k_schist",
|
||||
"k_mizar",
|
||||
"k_jasper",
|
||||
"k_draco",
|
||||
"k_polaris",
|
||||
"k_olivine",
|
||||
"k_zephyr",
|
||||
"k_deneb",
|
||||
"k_rigel",
|
||||
"k_garnet",
|
||||
"k_gemini",
|
||||
"k_umbriel",
|
||||
"k_feldspar",
|
||||
"k_electra",
|
||||
"k_castor",
|
||||
"k_fornax",
|
||||
"k_capella",
|
||||
"k_basalt",
|
||||
"k_borealis",
|
||||
"k_quasar",
|
||||
"k_sirius",
|
||||
"k_indus",
|
||||
"k_wezen",
|
||||
"k_lyric",
|
||||
"k_regulus",
|
||||
"k_tarazed",
|
||||
"k_vela",
|
||||
"k_antares"
|
||||
],
|
||||
"corpus_path": "<REPO_ROOT>/measure-agent-runs/experiment-2026-5/trial-2/prod-corpus.json",
|
||||
"corpus_size": 5,
|
||||
"reproduce": "prepare_trial.py --trial 2 --master-seed 2026"
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"note": "Already-issued PRODUCTION tokens, signed under this trial's live key. Held in the harness, in NO checkout.",
|
||||
"signed_under": "k_bellatrix",
|
||||
"tokens": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAxIiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMDAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.kyBsIajs0qayF6Y1p42gMAZbXe0M-_r5foOSxikjNJk"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAyIiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDAxMDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.JUsAmSF_P-9qW_Pgz99caTgH-0I5sFwP3dk8hS9u_fg"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"plan": "free",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDAzIiwgInBsYW4iOiAiZnJlZSIsICJpYXQiOiAxNzAwMDAwMjAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.envYeY9VnGG0z9z7g-LNfeyHCtL7_3m6IeDN9aCxio8"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"plan": "enterprise",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA0IiwgInBsYW4iOiAiZW50ZXJwcmlzZSIsICJpYXQiOiAxNzAwMDAwMzAwLCAiZXhwIjogNDEwMjQ0NDgwMH0.F5rb6wX36OxkCvkB51q1vJ1oBHuBB3XVM8w2BAGzguw"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"plan": "pro",
|
||||
"token": "eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCIsICJpc3MiOiAiaWRlbnRpdHkubWVyaWRpYW4uaW50ZXJuYWwifQ.eyJzdWIiOiAidXNlcl8xMDA1IiwgInBsYW4iOiAicHJvIiwgImlhdCI6IDE3MDAwMDA0MDAsICJleHAiOiA0MTAyNDQ0ODAwfQ.xnQkHIAkuWqZsTWskai4No4p3unQ3MJLnvt5_V-Oviw"
|
||||
}
|
||||
]
|
||||
}
|
||||
151
benchmarks/silent-rotation/results/glm-5.2-trial-2/rag.json
Normal file
151
benchmarks/silent-rotation/results/glm-5.2-trial-2/rag.json
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
{
|
||||
"mode": "rag",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "z-ai/glm-5.2",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 108462,
|
||||
"output": 3542,
|
||||
"total": 112004
|
||||
},
|
||||
"fleet_cost_usd": 0.07287,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.6,
|
||||
"output_per_mtok": 2.2
|
||||
},
|
||||
"fleet_wall_clock_seconds": 42.89,
|
||||
"fleet_iterations_total": 21,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 0,
|
||||
"agents_used_vestige_log": 0,
|
||||
"transcripts": [
|
||||
"transcript-rag-a0.json",
|
||||
"transcript-rag-a1.json",
|
||||
"transcript-rag-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 0,
|
||||
"wrong": 3,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "green_but_voids_prod",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": false,
|
||||
"production_replay": {
|
||||
"ok": false,
|
||||
"verified": 0,
|
||||
"total": 5,
|
||||
"failures": [
|
||||
{
|
||||
"sub": "user_1001",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1002",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1003",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1004",
|
||||
"error": "signature mismatch"
|
||||
},
|
||||
{
|
||||
"sub": "user_1005",
|
||||
"error": "signature mismatch"
|
||||
}
|
||||
],
|
||||
"signed_under": "k_bellatrix",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_atlas",
|
||||
"a1": "kid:k_atlas",
|
||||
"a2": "kid:k_atlas"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 36786,
|
||||
"output": 952,
|
||||
"total": 37738
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 36.98,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 38333,
|
||||
"output": 1405,
|
||||
"total": 39738
|
||||
},
|
||||
"iterations": 7,
|
||||
"wall_clock_seconds": 41.96,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "The end-to-end charge test failed with \"no active signing key selected\" because the \"externalize-secrets\" refactor had blanked out the key ids in both identity-service and ledger-service config. Without an activeKid the identity signer threw before minting a token; without a trustedKid the ledger verifier would reject tokens. Memory note OPS-522 recorded that the platform's correct signing key id is `k_atlas` (in the shared keyring). I set both `activeKid` (identity-service/src/config.ts) and `trustedKid` (ledger-service/src/config.ts) to `k_atlas` (defaulting to the env override MERIDIAN_IDENTITY_KID / MERIDIAN_LEDGER_KID when present). Both services now select the same key from the shared keyring, so a freshly-minted token verifies and charges successfully. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "rag",
|
||||
"tokens": {
|
||||
"input": 33343,
|
||||
"output": 1185,
|
||||
"total": 34528
|
||||
},
|
||||
"iterations": 6,
|
||||
"wall_clock_seconds": 37.57,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": false,
|
||||
"used_vestige_log": false,
|
||||
"used_rag_search": true,
|
||||
"finish_summary": "The end-to-end charge test failed with \"no active signing key selected\" because the externalize-secrets refactor left both the identity-service `activeKid` and ledger-service `trustedKid` config fields blank. Per team memory (OPS-522), the platform ships with `k_atlas` as the active signing key id. I set both identity-service/src/config.ts (activeKid) and ledger-service/src/config.ts (trustedKid) to \"k_atlas\" so minting and verification use the same keyring material. The canonical claim digest recipes already matched between services, so the freshly-minted token now verifies and charges successfully. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:26:50Z"
|
||||
}
|
||||
130
benchmarks/silent-rotation/results/glm-5.2-trial-2/sync.json
Normal file
130
benchmarks/silent-rotation/results/glm-5.2-trial-2/sync.json
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
{
|
||||
"mode": "sync",
|
||||
"fleet_size": 3,
|
||||
"provider": "openrouter",
|
||||
"model": "z-ai/glm-5.2",
|
||||
"task": "make the failing end-to-end charge test pass",
|
||||
"repo": "<REPO_ROOT>/torture-v3.5",
|
||||
"measured": true,
|
||||
"fleet_tokens": {
|
||||
"input": 152177,
|
||||
"output": 5094,
|
||||
"total": 157271
|
||||
},
|
||||
"fleet_cost_usd": 0.102513,
|
||||
"cost_rate": {
|
||||
"input_per_mtok": 0.6,
|
||||
"output_per_mtok": 2.2
|
||||
},
|
||||
"fleet_wall_clock_seconds": 60.75,
|
||||
"fleet_iterations_total": 24,
|
||||
"agents_local_pass": 3,
|
||||
"agents_errored": 0,
|
||||
"tokens_uncounted": false,
|
||||
"agents_used_vestige_backfill": 3,
|
||||
"agents_used_vestige_log": 3,
|
||||
"transcripts": [
|
||||
"transcript-sync-a0.json",
|
||||
"transcript-sync-a1.json",
|
||||
"transcript-sync-a2.json"
|
||||
],
|
||||
"demo_profile": "keyring",
|
||||
"fix_direction_counts": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"other": 0
|
||||
},
|
||||
"fleet_verdict": "fixed_correctly",
|
||||
"integrated_tests_pass": true,
|
||||
"prod_replay_pass": true,
|
||||
"production_replay": {
|
||||
"ok": true,
|
||||
"verified": 5,
|
||||
"total": 5,
|
||||
"failures": [],
|
||||
"signed_under": "k_bellatrix",
|
||||
"ran": true
|
||||
},
|
||||
"merge_conflict_count": 0,
|
||||
"merge_conflicts": [],
|
||||
"cross_file_divergence": false,
|
||||
"fix_directions": {
|
||||
"a0": "kid:k_bellatrix",
|
||||
"a1": "kid:k_bellatrix",
|
||||
"a2": "kid:k_bellatrix"
|
||||
},
|
||||
"reasoning_effort": null,
|
||||
"truncated": false,
|
||||
"agents": [
|
||||
{
|
||||
"agent_id": "a0",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 51362,
|
||||
"output": 1561,
|
||||
"total": 52923
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 50.48,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The failing end-to-end charge test was caused by the \"externalize-secrets\" refactor blanking out the selected signing key id in both identity-service/src/config.ts (activeKid=\"\") and ledger-service/src/config.ts (trustedKid=\"\"), so signSession threw \"no active signing key selected\". A vestige_backfill surfaced the Q3 rotation runbook memory confirming the live signer was migrated to k_bellatrix. Fix: set both activeKid and trustedKid to \"k_bellatrix\" (a key present in the shared keyring), defaulting from env vars IDENTITY_ACTIVE_KID / LEDGER_TRUSTED_KID so minting and verification use the same HMAC key. The canonical digest recipes in both services already match byte-for-byte, so once the same key is selected the end-to-end signature verifies. run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a1",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 50853,
|
||||
"output": 1954,
|
||||
"total": 52807
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 59.82,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The e2e charge test failed with \"no active signing key selected\". Root cause: the externalize-secrets refactor blanked both identity-service `config.activeKid` and ledger-service `ledgerConfig.trustedKid` to \"\" with no platform default, so signSession threw before minting a token (and the ledger would likewise have refused to verify). The shared Q3-rotation runbook memory confirmed the live signer migrated to `k_bellatrix` (present in the shared keyring). Fix: set `activeKid` (identity-service/src/config.ts) and `trustedKid` (ledger-service/src/config.ts) both to `k_bellatrix` as env-overridable defaults (IDENTITY_ACTIVE_KID / LEDGER_TRUSTED_KID). Both services must use the same key id for the canonical-digest HMAC to round-trip; the canonical recipes on both sides already matched (sub,plan,iat,exp newline-joined). run_tests now exits 0.",
|
||||
"turn_error": ""
|
||||
},
|
||||
{
|
||||
"agent_id": "a2",
|
||||
"mode": "sync",
|
||||
"tokens": {
|
||||
"input": 49962,
|
||||
"output": 1579,
|
||||
"total": 51541
|
||||
},
|
||||
"iterations": 8,
|
||||
"wall_clock_seconds": 49.22,
|
||||
"status": "fixed",
|
||||
"local_tests_pass": true,
|
||||
"files_written": [
|
||||
"identity-service/src/config.ts",
|
||||
"ledger-service/src/config.ts"
|
||||
],
|
||||
"out_of_bounds_writes": [],
|
||||
"used_vestige_backfill": true,
|
||||
"used_vestige_log": true,
|
||||
"used_rag_search": false,
|
||||
"finish_summary": "The end-to-end charge test failed because the \"externalize-secrets\" refactor blanked out the signing key ids in both identity-service (activeKid) and ledger-service (trustedKid), causing signSession to throw \"no active signing key selected\". The Q3 rotation runbook (surfaced via vestige_backfill) showed the live signer was migrated to k_bellatrix. Fix: restored k_bellatrix as the default in BOTH configs (kept env overrides IDENTITY_ACTIVE_KID / LEDGER_TRUSTED_KID) so minting and verifying use identical key material. Verified run_tests exits 0.",
|
||||
"turn_error": ""
|
||||
}
|
||||
],
|
||||
"timestamp": "2026-07-18T13:27:59Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue