feat(analytics-skill): cover multiple prose phrasings and parse a missing join key

Root-cause correction for the deps_dev_v1 fork-count failure: the dominant
cause is not comma-stripping but (1) the same fact phrased several ways in the
narrative, so a single regex drops every other form to NULL, and (2) a table
with no structured key whose identifier lives only in prose. Reframe the
extract rule to sample distinct phrasings and COALESCE one pattern per
phrasing; add a schema-linking rule that parses a missing join key out of the
prose; note a second phrasing as a cause in the parse-coverage check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Q2q8GTZBn98eg6zUjDxvR

packages/cli/src/skills/analytics/SKILL.md

@@ -47,9 +47,10 @@ Heuristics for writing *correct* (not merely runnable) SQL. Each is a default pl
 **Schema discovery before writing SQL**
 - **Sample before you compose.** Inspect representative rows of every table you will touch (`entity_details` plus a small `sql_execution` sample) to confirm date/time encoding (`YYYYMMDD` integer vs ISO text vs epoch), null prevalence in join/filter keys, and the real set of categorical/enum values. Assumptions about encoding and nullability are the most common source of silently-wrong filters.
 - **Cast to the real type before comparing.** Compare a column against a literal of its actual type in `WHERE`/`JOIN`. A string column compared to a numeric literal (or the reverse) can silently match nothing instead of raising an error.
+- **A missing structured key means the identifier lives in the prose — parse it into the join.** When the table the question names has no id/name column to join on (only free-text/narrative columns), the entity identifier is embedded in that prose, and extracting it *is* the join path: parse the name/key into a CTE (see *Extract the number from prose* for the sample-the-phrasings, `COALESCE`-the-patterns habit — it applies to a key just as to a metric) and join downstream tables on the parsed value, rather than concluding the tables cannot be linked. Confirm the parsed key matches the target column's real form (case, separators, owner/name shape) on a sample before trusting the join. *Why:* a table with no key column still participates in joins through its prose — treating "no key column" as "not joinable" silently drops that whole table's rows from the answer's universe.
 - **Parse text-encoded numerics before doing math on them.** When a column the question treats as a number is stored as text, sample its **distinct** values (the *Sample before you compose* habit) to learn the encodings actually present — unit suffixes (`K`/`M`/`B`), currency symbols, thousands separators, percent signs, and non-numeric sentinels (`-`, `N/A`, empty) — and never infer the format from the column name. *Why:* aggregated or compared as-is the text sorts lexically (`'100' < '9'`) and a naive cast collapses formatted values to `0`/NULL, so the query runs but the number is silently wrong instead of erroring.
 - **Strip, scale, and cast in one early CTE.** Strip currency/separator/percent characters, multiply by the suffix scale (`K`=10^3, `M`=10^6, `B`=10^9), map sentinels to `0` **or** `NULL` (by the *Default by additivity* rule below), then cast to a numeric type — all in a single early CTE so every layer above sees clean numbers. This is the *meaning-is-numeric* complement to *Cast to the real type before comparing*. *Why:* one clean conversion at the base keeps the lexical-sort-and-cast-to-0 failure out of every downstream layer.
-- **Extract the number from prose before parsing it.** When the value to rank or aggregate by lives inside a free-text/narrative column rather than its own numeric column, first extract the numeric token *with its separators* from the surrounding prose (get the engine's regex/substring-extract idiom from `sql_dialect_notes`), then apply *Strip, scale, and cast* and the failure-detecting cast to that token. Never `CAST` a raw prose substring directly, and never extract only the bare digit-run — a `,`/`_`/space thousands-separator dropped at extraction time silently truncates `"30,522"` to `30`, so the largest entities sink out of a top-N while smaller cleanly-formatted values float up. *Why:* the number embedded in narrative text is not yet isolated, so the existing parse rules cannot see it until extraction lifts it out intact.
+- **Extract the number from prose by covering every phrasing, not one pattern.** When the value to rank or aggregate by lives inside a free-text/narrative column rather than its own numeric column, first **sample the distinct phrasings** the same fact takes — generated/scraped narratives routinely state one fact several ways (`"30,522 forks"` number-before *and* `"forks count of 7201"` number-after) — then extract with a `COALESCE` over one regex alternative **per phrasing** (get each engine's regex/substring-extract idiom from `sql_dialect_notes`), keeping each token *with its separators* so a later *Strip, scale, and cast* can clean it. A single pattern silently parses the forms it matches and drops the rest to `NULL`, so entities described the other way vanish from the ranking and a top-N loses a true leader with no error; a `,`/`_`/space thousands-separator dropped at extraction time (`"30,522"`→`30`) is the same failure in miniature. *Why:* the number is not isolated until extraction lifts it out, and one pattern only isolates one phrasing.
 - **Confirm the parse covered every value.** After parsing, count the non-sentinel rows that failed to parse — a failed parse should surface as `NULL`, visible only with a **failure-detecting cast** from `sql_dialect_notes` (a plain `CAST` errors on some engines and on sqlite silently returns `0`/partial, so an `IS NULL` check is meaningless there). *Why:* an encoding the sample missed would otherwise vanish into `0`/NULL instead of being caught.
 - **Parse code/dependency text by its real grammar, not one broad regex.** When a question extracts imported/required/loaded packages or modules from stored source text or dependency manifests, parse by the *language or format*, not a single pattern: Java `import`/`import static` — drop the terminal class/member, keep the package path, and allow valid identifier segments with underscores and mixed case (e.g. com.planet_ink.coffee_mud); Python — handle both `import a, b as c` and `from a.b import c`, stripping aliases; R — handle `library(...)` and `require(...)`; notebooks (`.ipynb`) — parse the JSON and read each cell's `source` lines *before* applying the language rules (never regex the raw notebook file, whose prose contains the words "import"/"from"); JSON/manifest files — `PARSE_JSON` and flatten the dependency object's keys (e.g. `require`). Strip comments/prose lines first and split multi-import lines so each declared dependency is counted once. *Why:* a single lowercase-segment regex silently drops real identifiers and matches prose, so the ranking is wrong though the query runs.
 - **Decide the counting population explicitly when a table is deduplicated.** If the source table is de-duplicated and carries a documented copy/occurrence count (e.g. a `copies` column = "repositories sharing this exact content"), the count grain is a real modeling choice: weight by that column only when the question's population is clearly the represented files/repositories; otherwise count the distinct stored rows. State which population the question names and match it — do not default to one silently. *Why:* on a deduplicated table `COUNT(*)` and `SUM(copies)` give different rankings, so the right metric depends on the population the question asks about, not on which is larger.
@@ -186,7 +187,7 @@ FROM account_txns;
 - **A comparison BETWEEN two specific extremes is one wide row.** When the question asks for a single value derived by comparing two named extremes — "the **difference between** the highest and the lowest month", "the ratio of the best to the worst" — present BOTH extremes side by side in ONE row: each extreme's attributes as their own columns (e.g. `highest_month`, `highest_value`, `lowest_month`, `lowest_value`) plus the comparison as a column (`difference`). The comparison is a single fact about the pair, so the answer is one wide row — NOT one row per extreme with the comparison repeated. (Contrast: "report a metric **for each** group/category" — e.g. "a percentage for each helmet group", "the top player for each outcome" — has no cross-item comparison and stays long, one row per group.)
 - **Project BOTH identity and label.** When the result is per-entity, project the entity's **identifier and its human-readable name together** — whichever you grouped by, add the other. The id disambiguates duplicate names, and a consumer may legitimately expect either; supplying both is the safe, complete choice (a per-entity answer that gives only one is a frequent cause of an otherwise-correct result not matching).
 - **Diagnose empty results.** When a result is unexpectedly empty, relax filters one at a time to find which predicate removed the rows instead of guessing.
-- **A top-N built on a parsed-from-text metric must pass the parse-coverage check before you answer.** After ranking by a value parsed out of text, run the *Confirm the parse covered every value* count (non-sentinel rows that are `NULL` after the failure-detecting cast). If any high-looking rows failed to parse, the top-N may be missing its true leaders — re-inspect the raw narrative of the unparsed rows (a comma/suffix/locale format the strip missed is the usual cause) and fix the extraction before answering. This grounds "does the ceiling look too small?" in an observable parse-failure count, not a guessed threshold — do not invent a min-magnitude filter. *Why:* a dropped high-magnitude value does not error, it vanishes from the ranking, so a confidently-wrong top-N is indistinguishable from a correct one without the coverage check.
+- **A top-N built on a parsed-from-text metric must pass the parse-coverage check before you answer.** After ranking by a value parsed out of text, run the *Confirm the parse covered every value* count (non-sentinel rows that are `NULL` after the failure-detecting cast). If any high-looking rows failed to parse, the top-N may be missing its true leaders — re-inspect the raw narrative of the unparsed rows (a second phrasing your pattern didn't cover, or a comma/suffix/locale format the strip missed, is the usual cause) and fix the extraction before answering. This grounds "does the ceiling look too small?" in an observable parse-failure count, not a guessed threshold — do not invent a min-magnitude filter. *Why:* a dropped high-magnitude value does not error, it vanishes from the ranking, so a confidently-wrong top-N is indistinguishable from a correct one without the coverage check.
 - **Spatial predicates ("within area / within N meters / inside this polygon / nearest").** When a question filters or relates rows by geography, use the engine's geospatial functions — get the exact ones from `sql_dialect_notes` — rather than hand-rolling latitude/longitude `BETWEEN` boxes (which are wrong off the equator and ignore polygon shape). Recipe: (1) turn each location into a geography point with the point constructor — **mind argument order, most take longitude before latitude**; (2) for an area of interest build a polygon from its boundary/corner coordinates, closing the ring (first point repeated last); (3) test the relation with the engine's containment (`contains`/`within`), proximity (`dwithin(g1,g2,meters)`), or overlap (`intersects`) predicate. For "the features within the same area as entity X", first resolve X's own geometry in a CTE, then join candidates on the spatial predicate against it. *Why:* spatial relationships are not axis-aligned ranges; the geodesic predicates are both correct and index-assisted, while a raw coordinate box silently includes/excludes the wrong rows.
 - **Collapse a multi-valued attribute to one representative per entity before counting classes or a concentration metric.** When an entity carries a multi-valued classification array (IPC/CPC codes, tags, categories) and the methodology counts *entities per class* or computes a concentration/diversity measure (HHI, originality, a share), pick exactly **one representative value per entity** in a CTE first — use the array's `main`/`primary`/`first` flag when present, else a defined fallback (e.g. the most-frequent value) — then aggregate. Equally, when a metric's denominator is defined as a count of **entities** ("the number of patents cited"), use `COUNT(DISTINCT entity)`, not the count of exploded array rows. *Why:* `LATERAL FLATTEN`/unnest of the array multiplies an entity's weight by how many codes it has, inflating per-class frequencies and skewing any concentration metric — the query runs but the ranking/score is wrong. (Take the representative rule from the methodology/`external_knowledge` doc when it specifies one; do not invent a selection the source does not state.)
 - **Final completeness check.** Before emitting the final SQL, re-read the question and confirm the projection covers: (1) every named **metric / attribute** asked for (→ *answer every requested output*); (2) the **identifier** of each grouped or named entity (→ *expose identity*); (3) every **input** to each derived value (→ *keep the inputs*); (4) all at the **grain** the question specifies (→ *for each X* / *complete the panel*). Run this on every query, not only when a result looks off. **Don't over-project:** anything outside that set — a column the question never asked for, added "to be safe" — adds noise, misleads the reader into thinking it matters, and makes the result harder to consume. Match the request exactly: neither short nor padded.

packages/cli/test/skills/analytics-skill-content.test.ts

@@ -48,7 +48,8 @@ describe('analytics SKILL.md SQL craft', () => {
       'Parse text-encoded numerics before doing math on them', // detect text-encoded numbers (spec 12)
       'Strip, scale, and cast in one early CTE', // parse/scale early (spec 12)
       'Confirm the parse covered every value', // failure-detecting cast coverage (spec 12)
-      'Extract the number from prose before parsing it', // lift the numeric token out of a narrative column first (spec 07)
+      'Extract the number from prose by covering every phrasing', // COALESCE over one regex per phrasing (spec 07)
+      'A missing structured key means the identifier lives in the prose', // parse the join key out of narrative text (spec 07)
       'must pass the parse-coverage check before you answer', // parsed-from-text ranking re-checks parse coverage (spec 07)
       'Answer every requested output', // multi-part/multi-output umbrella over identity+inputs (spec 14)
       'Final completeness check', // re-read the question, confirm the projection covers all four facets (spec 14)