Profile per-column cardinality and extent; add data-aware chart warnings (M4, A3/A4)

This commit is contained in:
2026-06-07 23:03:17 +03:00
parent dcb7037a71
commit 39555322e4
13 changed files with 552 additions and 90 deletions
+19 -20
View File
@@ -214,30 +214,29 @@ as of 2026-06-06.
un-aggregated case. A general "long labels → go horizontal" suggestion on _any_ vertical
bar is still deferred (needs a label-length / cardinality signal); a blanket warning was
rejected — it would fire on every ordinary vertical bar.
- **A3 · Crowded-axis & high-cardinality warnings** _(partly done)_
- **A3 · Crowded-axis & high-cardinality warnings** _(done)_
- _Done:_ the **un-aggregated crowded axis** — a bar/line/area with a category axis and a
**raw** measure draws one mark (and one label) per row, so over `CROWDED_CATEGORY_ROWS`
(30) rows it warns and points to aggregating, or a horizontal bar. Row-count-based:
`builderWarnings(config, rowCount)`, with `rowCount` from the loaded dataset; detects
exactly the mark-count == row-count case (URL/non-tabular → `rowCount` null → skipped).
- _Remaining (needs the profiling extension below):_ an **aggregated** axis that still has
many distinct **categories**, an unreadable **Color legend** (>10/>20 categories), and
**number-typed-Nominal**. All need per-column **cardinality**, which the profile lacks.
- **A4 · Data-aware Size guard** _(deferred — needs the profiling extension)_ — exclude
**negative**-valued columns from Size (Draco `hard.lp:56`; size implies positive
magnitude). Today only the type-level Size discipline is enforced.
detects exactly the mark-count == row-count case (URL/non-tabular → `rowCount` null → skipped).
- _Done (via the profiling extension below):_ an **aggregated** axis that still has many
distinct **categories** (`CROWDED_CATEGORY_DISTINCT` = 30) and an unreadable **Color
legend** (`CROWDED_LEGEND_DISTINCT` = 12, discrete colour only) now warn from per-column
cardinality. (Number-typed-Nominal remains a possible future nudge; not yet flagged.)
- **A4 · Data-aware Size guard** _(done)_ — Size mapped to a field whose profiled numeric
**extent** goes negative warns (Draco `hard.lp:56`; size implies positive magnitude),
on top of the type-level Size discipline.
> **TODO — profiling extension (the A3-remaining + A4 enabler).** `profile.ts` computes only
> `rowCount` / `columnCount` / `columnTypes`. Extend it, in the **same sample pass** that
> already feeds `inferColumnType` (so it's nearly free), to also derive per column: a
> **capped distinct count** (cardinality — cap at ~50; a sampled count is enough for a
> ">N categories" threshold, no full scan) and a **numeric extent** (min/max → sign).
> Surface them on `DatasetProfile` (alongside `columnTypes`). Then: `builderWarnings` consumes
> cardinality (legend/axis crowding) and the Size gate consumes sign (A4). **Caveats:**
> URL / non-tabular datasets have no rows at profile time → these fields are null and the
> dependent warnings simply skip; and stored datasets predate the field, so this needs a
> recompute-on-read or a `dataset-migrations` bump (see architecture 02 / 06). Keep
> thresholds in `chart-builder.ts` constants like `CROWDED_CATEGORY_ROWS`.
> **Done — profiling extension (the A3 / A4 enabler).** `profile.ts` now derives, in the
> **same sample pass** that feeds `inferColumnType`, a per-column **capped distinct count**
> (`DISTINCT_CAP` = 50, with a `distinctCapped` overflow flag) and a **numeric extent**
> (min/max, numeric columns only), surfaced on `DatasetProfile.columnStats` and the `Dataset`
> record. `builderWarnings(config, rowCount, columns)` consumes them for the legend/axis
> crowding hints and the negative-value Size guard. **Caveats handled:** URL / non-tabular
> data has no rows → `columnStats` is `[]` and the dependent hints skip; datasets stored
> before the field default to `[]` via `dataset-migrations` / import normalization (no forced
> re-profile on read — they pick up stats on next save). Thresholds live in `chart-builder.ts`
> constants alongside `CROWDED_CATEGORY_ROWS`.
**B · Transform-enabled coverage (new core capability + §06 extension)** _(done)_