diff --git a/packages/cli/src/context/sql-analysis/dialects/athena.md b/packages/cli/src/context/sql-analysis/dialects/athena.md index d7a377cf..8ed4cdce 100644 --- a/packages/cli/src/context/sql-analysis/dialects/athena.md +++ b/packages/cli/src/context/sql-analysis/dialects/athena.md @@ -8,5 +8,6 @@ - **Approximate aggregates:** `approx_distinct(x)` for cardinality and `approx_percentile(x, 0.5)` for quantiles are far cheaper than exact `COUNT(DISTINCT ...)` on large scans. - **Arrays & maps:** explode with `CROSS JOIN UNNEST(arr) AS t(x)` (add `WITH ORDINALITY` for an index); build with `array_agg(x)`, join with `array_join(arr, ',')`, index 1-based (`arr[1]`), and read a map with `element_at(m, key)`. - **Safe cast:** `TRY_CAST(x AS DOUBLE)` yields `NULL` for a value that does not parse instead of raising, so counting residual `NULL`s catches an encoding the sample missed; `TRY(expr)` swallows other runtime errors. +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `regexp_extract(txt, '[0-9][0-9.,_ ]*')` — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Integer division:** `/` between integers truncates (`5 / 2` → `2`); cast an operand (`x / CAST(y AS DOUBLE)`) to keep the fraction, and round only in the final projection. - **JSON:** `json_extract_scalar(col, '$.a.b')` returns varchar, `json_extract(col, '$.a')` returns json; cast a JSON string with `CAST(json_parse(col) AS ...)`. diff --git a/packages/cli/src/context/sql-analysis/dialects/bigquery.md b/packages/cli/src/context/sql-analysis/dialects/bigquery.md index 4d469d2c..d819c00e 100644 --- a/packages/cli/src/context/sql-analysis/dialects/bigquery.md +++ b/packages/cli/src/context/sql-analysis/dialects/bigquery.md @@ -5,6 +5,7 @@ - **Series:** build a spine with `UNNEST(GENERATE_DATE_ARRAY('2023-01-01', '2023-12-01', INTERVAL 1 MONTH))` for dates (or `GENERATE_ARRAY(1, n)` for integers), then `LEFT JOIN` the aggregated facts onto it so empty periods still appear. - **Rolling window over time:** `RANGE` frames are numeric, so range over an integer day key — `AVG(amount) OVER (ORDER BY UNIX_DATE(day) RANGE BETWEEN 29 PRECEDING AND CURRENT ROW)` is a trailing 30-day average that tolerates gaps; or build a spine (see **Series**) and use a `ROWS` frame. - **Safe cast:** `SAFE_CAST(x AS FLOAT64)` (or `SAFE_CAST(x AS NUMERIC)`) returns `NULL` instead of erroring on a value that does not parse, so counting residual `NULL`s among non-sentinel rows catches an encoding the sample missed. +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `REGEXP_EXTRACT(txt, r'[0-9][0-9.,_ ]*')` — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Safe divide:** `SAFE_DIVIDE(num, den)` returns `NULL` instead of erroring when the denominator is `0`, so a rate/ratio/share is one expression with no `CASE den = 0` guard; multiply by `100` for a percentage. Prefer it over `num / den` for any computed measure whose denominator can be zero. - **Top-N / windows:** `QUALIFY` filters on a window result, e.g. `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) = 1`. - **JSON:** `JSON_VALUE(col, '$.k')` returns a scalar STRING, `JSON_QUERY(col, '$.k')` returns a subtree. diff --git a/packages/cli/src/context/sql-analysis/dialects/clickhouse.md b/packages/cli/src/context/sql-analysis/dialects/clickhouse.md index 17a02356..8446d040 100644 --- a/packages/cli/src/context/sql-analysis/dialects/clickhouse.md +++ b/packages/cli/src/context/sql-analysis/dialects/clickhouse.md @@ -5,5 +5,6 @@ - **Series:** `numbers(n)` / `range(n)` generate an integer sequence; offset a start date with `addMonths(toDate('2023-01-01'), number)` (or `arrayJoin`) to form a spine, then `LEFT JOIN` the aggregated facts onto it so empty periods still appear. - **Rolling window over time:** a numeric range frame over a `Date` column counts in days and tolerates gaps — `AVG(amount) OVER (ORDER BY day RANGE BETWEEN 29 PRECEDING AND CURRENT ROW)` is a trailing 30-day average (use seconds for a `DateTime` key; the `INTERVAL` form is unsupported); or build a spine (see **Series**) and use a `ROWS` frame. - **Safe cast:** `toFloat64OrNull(x)` / `toDecimal64OrNull(x, s)` returns `NULL` on a value that does not parse (the `...OrZero` variants return `0` instead), so counting residual `NULL`s among non-sentinel rows catches an encoding the sample missed. +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `extract(txt, '[0-9][0-9.,_ ]*')` — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Top-N / windows:** use the `LIMIT n BY key` clause for n rows per key, or rank in a CTE with `ROW_NUMBER() OVER (...)` and filter outside it. - **JSON:** extract from a String column with `JSONExtractString(col, 'k')`, `JSONExtractInt(col, 'k')`, etc.; a native `JSON`-typed column is traversed by dot path `col.k`. diff --git a/packages/cli/src/context/sql-analysis/dialects/duckdb.md b/packages/cli/src/context/sql-analysis/dialects/duckdb.md index 8569b0a0..cb9b0476 100644 --- a/packages/cli/src/context/sql-analysis/dialects/duckdb.md +++ b/packages/cli/src/context/sql-analysis/dialects/duckdb.md @@ -6,5 +6,6 @@ - **Rolling window over time:** a native calendar-range frame spans real dates and tolerates gaps — `AVG(amount) OVER (ORDER BY day RANGE BETWEEN INTERVAL 29 DAYS PRECEDING AND CURRENT ROW)` is a trailing 30-day average without a spine; guard minimum periods with `COUNT(*) OVER ()`. - **Integer division:** unlike postgres, `/` is true division (`5 / 2` → `2.5`), so a ratio keeps its fraction; use `//` for floor division (`5 // 2` → `2`) when you want the integer quotient, and round only in the final projection. - **Safe cast:** duckdb has `TRY_CAST` — `TRY_CAST(x AS DOUBLE)` yields `NULL` for a value that does not parse instead of raising, so counting residual `NULL`s among non-sentinel rows catches an encoding the sample missed without a regex guard. +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `regexp_extract(txt, '[0-9][0-9.,_ ]*')` — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Top-N / windows:** filter a window inline with `QUALIFY` — `SELECT ... QUALIFY ROW_NUMBER() OVER (PARTITION BY key ORDER BY x DESC) = 1` returns one row per key without a wrapping CTE; use `ORDER BY ... LIMIT n` for a global top-N. - **JSON / semi-structured:** `col->'k'` returns JSON, `col->>'k'` returns text, deep path `json_extract(col, '$.a.b')`; duckdb also has native `STRUCT`, `LIST`, and `MAP` — read a struct field with `col.field` and a list element with `col[1]` (1-indexed). diff --git a/packages/cli/src/context/sql-analysis/dialects/mysql.md b/packages/cli/src/context/sql-analysis/dialects/mysql.md index e4e2fc42..79627e25 100644 --- a/packages/cli/src/context/sql-analysis/dialects/mysql.md +++ b/packages/cli/src/context/sql-analysis/dialects/mysql.md @@ -5,5 +5,6 @@ - **Series:** no series function — build a spine with a recursive CTE, e.g. `WITH RECURSIVE months(d) AS (SELECT '2023-01-01' UNION ALL SELECT DATE_ADD(d, INTERVAL 1 MONTH) FROM months WHERE d < '2023-12-01')`, then `LEFT JOIN` the aggregated facts onto it so empty periods still appear. - **Rolling window over time:** a native interval range frame over a temporal order key tolerates gaps — `AVG(amount) OVER (ORDER BY day RANGE BETWEEN INTERVAL 29 DAY PRECEDING AND CURRENT ROW)` is a trailing 30-day average without a spine; guard minimum periods with `COUNT(*) OVER ()`. - **Safe cast:** MySQL has no `TRY_CAST`, and `CAST('abc' AS DECIMAL)` returns `0` with a warning rather than erroring — guard with a pattern test first: `CASE WHEN x REGEXP '^-?[0-9.]+$' THEN CAST(x AS DECIMAL(18,4)) END` makes a value that does not parse `NULL`, so a residual-`NULL` count catches an encoding the sample missed (`REGEXP_REPLACE` can strip symbols). +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `REGEXP_SUBSTR(txt, '[0-9][0-9.,_ ]*')` (MySQL 8.0+) — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Top-N / windows:** rank in a CTE with `ROW_NUMBER() OVER (...)` and filter outside it; use `ORDER BY ... LIMIT n` for a global top-N. - **JSON:** `JSON_EXTRACT(col, '$.k')`, or the `col->'$.k'` / `col->>'$.k'` shortcuts (`->>` unquotes to text). diff --git a/packages/cli/src/context/sql-analysis/dialects/postgres.md b/packages/cli/src/context/sql-analysis/dialects/postgres.md index 0b69a282..be96104f 100644 --- a/packages/cli/src/context/sql-analysis/dialects/postgres.md +++ b/packages/cli/src/context/sql-analysis/dialects/postgres.md @@ -6,5 +6,6 @@ - **Rolling window over time:** a native calendar-range frame spans real dates and tolerates gaps — `AVG(amount) OVER (ORDER BY day RANGE BETWEEN INTERVAL '29 days' PRECEDING AND CURRENT ROW)` is a trailing 30-day average without a spine; guard minimum periods with `COUNT(*) OVER ()`. - **Integer division:** `/` between two integers truncates (`5 / 2` → `2`), so a rate or `SUM(a) / COUNT(*)` silently floors to an integer; cast one operand first — `a::numeric / b` or `a * 1.0 / b` — and round only in the final projection. - **Safe cast:** postgres has no `TRY_CAST`; guard a text-encoded number with a pattern test before casting — `CASE WHEN x ~ '^-?[0-9.]+$' THEN x::numeric END` yields `NULL` for a value that does not parse, so counting residual `NULL`s among non-sentinel rows catches an encoding the sample missed (`regexp_replace` can strip symbols, but chained `REPLACE` is the portable default). +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `(regexp_match(txt, '[0-9][0-9.,_ ]*'))[1]` (or `substring(txt from '[0-9][0-9.,_ ]*')`) — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Top-N / windows:** rank in a CTE with `ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...)` and filter in the outer query, or use `DISTINCT ON (key) ... ORDER BY key, ...` for one row per key. - **JSON:** `col->'k'` returns json, `col->>'k'` returns text, deep path `col#>>'{a,b}'`; prefer `jsonb` operators on `jsonb` columns. diff --git a/packages/cli/src/context/sql-analysis/dialects/snowflake.md b/packages/cli/src/context/sql-analysis/dialects/snowflake.md index c7375f9e..3db3cf11 100644 --- a/packages/cli/src/context/sql-analysis/dialects/snowflake.md +++ b/packages/cli/src/context/sql-analysis/dialects/snowflake.md @@ -5,6 +5,7 @@ - **Series:** generate rows with `TABLE(GENERATOR(ROWCOUNT => n))` and offset a start date via `DATEADD('month', SEQ4(), '2023-01-01')` (or a recursive CTE) to form a spine, then `LEFT JOIN` the aggregated facts onto it so empty periods still appear. - **Rolling window over time:** a native interval range frame over a date/timestamp order key tolerates gaps — `AVG(amount) OVER (ORDER BY day RANGE BETWEEN INTERVAL '29 days' PRECEDING AND CURRENT ROW)` is a trailing 30-day average without a spine; guard minimum periods with `COUNT(*) OVER ()`. - **Safe cast:** `TRY_TO_NUMBER(x)` / `TRY_TO_DECIMAL(x, p, s)` (or `TRY_CAST(x AS NUMBER)`) returns `NULL` instead of erroring on a value that does not parse, so a residual-`NULL` count among non-sentinel rows catches an encoding the sample missed. +- **Extract from text:** to rank/aggregate by a number buried in a narrative column, pull the token *with its separators* first — `REGEXP_SUBSTR(txt, '[0-9][0-9.,_ ]*')` — then apply the strip/scale/cast and **Safe cast** above. The class keeps `,`/`_`/space so `'30,522 forks'` yields `'30,522'`, not `'30'`; extracting a bare `[0-9]+` would silently truncate at the first separator. - **Top-N / windows:** `QUALIFY` filters on a window result without a subquery, e.g. `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) = 1`. - **Semi-structured (VARIANT):** traverse with a colon path and cast with `::`, e.g. `src:vehicle[0].make::string`, `payload:events.date::date`; expand arrays with `LATERAL FLATTEN`. - **Geospatial (GEOGRAPHY):** build a point with `ST_MAKEPOINT(longitude, latitude)` — **longitude first** — or `TO_GEOGRAPHY(wkt_or_geojson)`; an area polygon from a closed ring of corner points with `ST_MAKEPOLYGON(ST_MAKELINE(ARRAY_CONSTRUCT(p1, p2, …, p1)))` (repeat the first point last to close). Predicates: proximity `ST_DWITHIN(g1, g2, meters)` (geodesic) and distance `ST_DISTANCE(g1, g2)` (meters); containment `ST_CONTAINS(area, pt)` / `ST_WITHIN(pt, area)` where `ST_WITHIN(a,b)=ST_CONTAINS(b,a)`; overlap `ST_INTERSECTS`. Prefer these predicates over hand-rolled lat/lon `BETWEEN` boxes. diff --git a/packages/cli/src/context/sql-analysis/dialects/sqlite.md b/packages/cli/src/context/sql-analysis/dialects/sqlite.md index 8c5a2e6e..7fcc8d5c 100644 --- a/packages/cli/src/context/sql-analysis/dialects/sqlite.md +++ b/packages/cli/src/context/sql-analysis/dialects/sqlite.md @@ -7,5 +7,6 @@ - **Integer division:** `/` between two integers truncates (`5 / 2` → `2`), so a rate or `SUM(a) / COUNT(*)` silently floors to an integer; force real division with `a * 1.0 / b` (or `CAST(a AS REAL) / b`) and round only in the final projection. - **Safe cast:** sqlite has no failure-signaling cast — `CAST('abc' AS REAL)` returns `0.0` and `CAST('12abc' AS REAL)` returns `12.0` (no error, no `NULL`), so an `IS NULL` coverage check silently passes. Detect a value that did not parse with a pattern guard before casting, e.g. `CASE WHEN cleaned NOT GLOB '*[^0-9.]*' THEN CAST(cleaned AS REAL) END` (strip any leading sign first), then count the residual `NULL`s. - **Rounding (exact half-up at `.5` boundaries):** `ROUND(x, n)` rounds half-away-from-zero, but binary floating-point stores an exact half-way value just *below* it, so the round goes the wrong way — `ROUND(6.475, 2)` returns `6.47`, not `6.48`. When a rounded measure must match exact half-up (a displayed average, rate, or price), nudge by a tiny epsilon below display precision before rounding: `ROUND(x + 1e-9, n)` lifts `6.4749999…` back to `6.475` so it rounds to `6.48` (it leaves non-boundary values unchanged). Round once, at full precision, in the final projection — never in intermediate CTEs. +- **Extract from text:** sqlite's build here registers no regex function, so a number embedded in a narrative column has no clean extract idiom — scan for the numeric run with `INSTR`/`SUBSTR` (find the first digit, keep the following digit/`,`/`_`/space/`.` characters), then strip/scale/cast with **Safe cast** above. Keep the separators in the kept run so `'30,522'` is not truncated to `'30'`. When the numeric field can be isolated upstream (its own column) prefer that — positional extraction in sqlite is brittle. - **Top-N / windows:** rank in a CTE with `ROW_NUMBER() OVER (...)` and filter in the outer query; use `ORDER BY ... LIMIT n` for a global top-N. - **JSON:** `json_extract(col, '$.k')`, or the `col->'$.k'` / `col->>'$.k'` operators (`->>` returns text). diff --git a/packages/cli/src/context/sql-analysis/dialects/tsql.md b/packages/cli/src/context/sql-analysis/dialects/tsql.md index 355b8d5c..927fa8ae 100644 --- a/packages/cli/src/context/sql-analysis/dialects/tsql.md +++ b/packages/cli/src/context/sql-analysis/dialects/tsql.md @@ -6,5 +6,6 @@ - **Rolling window over time:** `RANGE` supports only `UNBOUNDED`/`CURRENT ROW` (no offset frame), so build a gap-free date spine (see **Series**) and use a row frame — `AVG(amount) OVER (ORDER BY day ROWS BETWEEN 29 PRECEDING AND CURRENT ROW)` — or a date-keyed self-join on `f.day BETWEEN DATEADD(day, -29, d.day) AND d.day`; guard minimum periods with `COUNT(*) OVER ()`. - **Integer division:** `/` between two `int`s truncates (`5 / 2` → `2`), so a rate or `SUM(a) / COUNT(*)` silently floors to an integer; cast one operand first — `CAST(a AS decimal(18,4)) / b` or `a * 1.0 / b` — and round only in the final projection. - **Safe cast:** `TRY_CAST(x AS DECIMAL(18,4))` (or `TRY_CONVERT(decimal(18,4), x)`) returns `NULL` instead of erroring on a value that does not parse, so a residual-`NULL` count among non-sentinel rows catches an encoding the sample missed. +- **Extract from text:** SQL Server has no regex-extract function, so pull a number out of a narrative column positionally — `SUBSTRING(txt, PATINDEX('%[0-9]%', txt), )` to reach the first digit, then keep following digit/separator characters — and strip/scale/cast with **Safe cast** above. Keep `,`/`_`/space in the kept run so `'30,522'` is not truncated to `'30'`; a `PATINDEX('%[^0-9,._ ]%', rest)` locates where the numeric run ends. - **Top-N / windows:** `SELECT TOP (n) ... ORDER BY ...` for a global top-N; for per-group, rank in a CTE with `ROW_NUMBER() OVER (...)` and filter in the outer query. - **JSON:** `JSON_VALUE(col, '$.k')` returns a scalar, `JSON_QUERY(col, '$.k')` returns an object/array, and `OPENJSON(col)` shreds JSON into rows.