Profile per-column cardinality and extent; add data-aware chart warnings (M4, A3/A4)

2026-08-08 02:02:33 +00:00 · 2026-06-07 23:03:17 +03:00
parent dcb7037a71
commit 39555322e4
13 changed files with 552 additions and 90 deletions
@@ -169,16 +169,17 @@ A **profile** is the set of derived summary fields stored on a dataset record so
 the UI can describe it without re-parsing the payload. Per the data model, a
 profiled dataset carries:

-| Field         | Type                    | Meaning                            |
-| ------------- | ----------------------- | ---------------------------------- |
-| `rowCount`    | `number \| null`        | Data rows, or `null` when N/A.     |
-| `columnCount` | `number \| null`        | Columns, or `null` when N/A.       |
-| `columns`     | `string[]`              | Column names, in order.            |
-| `columnTypes` | `Array<{ name; type }>` | Per-column inferred type (see §2). |
-| `size`        | `number`                | Approximate payload size in bytes. |
+| Field         | Type                                                       | Meaning                                                            |
+| ------------- | ---------------------------------------------------------- | ------------------------------------------------------------------ |
+| `rowCount`    | `number \| null`                                           | Data rows, or `null` when N/A.                                     |
+| `columnCount` | `number \| null`                                           | Columns, or `null` when N/A.                                       |
+| `columns`     | `string[]`                                                 | Column names, in order.                                            |
+| `columnTypes` | `Array<{ name; type }>`                                    | Per-column inferred type (see §2).                                 |
+| `columnStats` | `Array<{ name; distinct; distinctCapped; numericExtent }>` | Per-column cardinality + numeric range (see §3.3). Empty when N/A. |
+| `size`        | `number`                                                   | Approximate payload size in bytes.                                 |

-`null` row/column counts and an empty `columns`/`columnTypes` are how the UI
-shows **"N/A"** — see §3.2.
+`null` row/column counts and an empty `columns`/`columnTypes`/`columnStats` are
+how the UI shows **"N/A"** — see §3.2.

 ### 3.1 What gets profiled

@@ -208,12 +209,14 @@ of the stored payload), but `rowCount` and `columnCount` are `null`, and
   - `json` that is a non-empty **array of objects** → use it directly.
   - anything else (`topojson`, a lone JSON object, an empty array) → not
     tabular; return the N/A profile (`rowCount: null`, `columnCount: null`,
-     `columns: []`, `columnTypes: []`, plus `size`).
+     `columns: []`, `columnTypes: []`, `columnStats: []`, plus `size`).
 3. **Derive columns** from the union of keys across the rows (or the CSV/TSV
   header), preserving first-seen order.
-4. **Infer each column's type** by collecting that column's values across the
-   rows and calling `inferColumnType` (§2), sampling per §4.
-5. **Assemble** `rowCount`, `columnCount`, `columns`, `columnTypes`, `size`.
+4. **Infer each column's type and stats** by collecting that column's values across
+   the rows (sampling per §4) and calling `inferColumnType` (§2) plus deriving its
+   cardinality + numeric extent (§3.3) — one pass per column over the same sample.
+5. **Assemble** `rowCount`, `columnCount`, `columns`, `columnTypes`, `columnStats`,
+   `size`.

 ### Sketch

@@ -226,6 +229,12 @@ export interface DatasetProfile {
  columnCount: number | null;
  columns: string[];
  columnTypes: Array<{ name: string; type: ColumnType }>;
+  columnStats: Array<{
+    name: string;
+    distinct: number; // capped at DISTINCT_CAP
+    distinctCapped: boolean; // true ⇒ real cardinality ≥ DISTINCT_CAP
+    numericExtent: { min: number; max: number } | null; // numeric columns only
+  }>;
  size: number;
 }

@@ -234,6 +243,7 @@ const NA = (size: number): DatasetProfile => ({
  columnCount: null,
  columns: [],
  columnTypes: [],
+  columnStats: [],
  size,
 });

@@ -259,16 +269,22 @@ export function profileData(
  if (columns.length === 0) return NA(size);

  const sample = sampleRows(rows);
-  const columnTypes = columns.map((name) => ({
-    name,
-    type: inferColumnType(sample.map((r) => r[name])),
-  }));
+  // One pass per column over the sample: type + stats (cardinality, numeric extent).
+  const columnTypes = [];
+  const columnStats = [];
+  for (const name of columns) {
+    const values = sample.map((r) => r[name]);
+    const type = inferColumnType(values);
+    columnTypes.push({ name, type });
+    columnStats.push(columnStatsFor(name, values, type)); // see §3.3
+  }

  return {
    rowCount: rows.length,
    columnCount: columns.length,
    columns,
    columnTypes,
+    columnStats,
    size,
  };
 }
@@ -278,17 +294,41 @@ Parsing CSV/TSV text and detecting the payload shape happen _upstream_ of
 `profileData`; this function takes already-parsed rows so it stays pure and
 trivially testable. The caller passes `null` for URL and non-tabular datasets.

+### 3.3 Column stats: cardinality + numeric extent
+
+Alongside the display type, each column carries the two data-shape signals the
+**Chart Builder** needs for its data-aware Tier-B hints (spec §06; see
+`chart-builder.ts` `builderWarnings`):
+
+- **`distinct`** — the count of distinct non-empty values **in the sample**,
+  counted only up to `DISTINCT_CAP` (50). Past the cap the exact number stops
+  mattering — a legend or category axis with 50 entries is already unreadable — so
+  counting stops and **`distinctCapped`** flags that the real cardinality is at
+  least the cap. Powers the crowded-legend (discrete colour) and crowded-category
+  warnings.
+- **`numericExtent`** — `{ min, max }` over the numeric values, for `number`
+  columns only (`null` otherwise), using the **same numeric test** as
+  `inferColumnType` so the two never disagree. Powers the negative-value **Size**
+  guard (size encodes magnitude → negatives mislead; Draco `hard.lp:56`).
+
+Both are derived in the same sample pass as type inference, so they are nearly
+free, and both are **estimates** (sampled from the head, like the type) — fine for
+soft advisories, not guarantees. URL / non-tabular data and datasets stored before
+this field carry empty `columnStats`, and the dependent hints simply skip.
+
 ---

 ## 4. Sampling vs. full scan

 `rowCount`/`columnCount`/`size` always reflect the **whole** dataset — they're
-cheap (a length and a byte count). Only **type inference** has a per-value cost,
-and it's the one place a huge dataset could hurt.
+cheap (a length and a byte count). Only the **per-value** work — type inference
+and the §3.3 column stats — has a per-row cost, and it's the one place a huge
+dataset could hurt.

-So: infer types from a **bounded sample** of rows, not the full column. A fixed
-cap (e.g. the first ~200 rows) keeps profiling fast and predictable on large
-pasted datasets while still being more than enough signal to classify a column.
+So: derive types **and** stats from a **bounded sample** of rows, not the full
+column. A fixed cap (e.g. the first ~200 rows) keeps profiling fast and
+predictable on large pasted datasets while still being more than enough signal to
+classify a column and estimate its cardinality / range.

 ```ts
 const SAMPLE_SIZE = 200;
@@ -306,7 +346,8 @@ the same paste profiles the same way twice.
 ### Do / Don't

 - **Do** count rows/columns and size over the full payload.
- **Do** cap type-inference sampling at a fixed head slice for determinism.
+- **Do** cap type-inference **and column-stats** sampling at a fixed head slice for
+  determinism (so `distinct` / `numericExtent` are estimates, like the type).
 - **Don't** randomly sample — non-deterministic profiles break tests and confuse
  users.
 - **Don't** scan every value of a million-row paste to guess a type.
@@ -214,30 +214,29 @@ as of 2026-06-06.
  un-aggregated case. A general "long labels → go horizontal" suggestion on _any_ vertical
  bar is still deferred (needs a label-length / cardinality signal); a blanket warning was
  rejected — it would fire on every ordinary vertical bar.
- **A3 · Crowded-axis & high-cardinality warnings** _(partly done)_ —
+- **A3 · Crowded-axis & high-cardinality warnings** _(done)_ —
  - _Done:_ the **un-aggregated crowded axis** — a bar/line/area with a category axis and a
    **raw** measure draws one mark (and one label) per row, so over `CROWDED_CATEGORY_ROWS`
    (30) rows it warns and points to aggregating, or a horizontal bar. Row-count-based:
-    `builderWarnings(config, rowCount)`, with `rowCount` from the loaded dataset; detects
-    exactly the mark-count == row-count case (URL/non-tabular → `rowCount` null → skipped).
-  - _Remaining (needs the profiling extension below):_ an **aggregated** axis that still has
-    many distinct **categories**, an unreadable **Color legend** (>10/>20 categories), and
-    **number-typed-Nominal**. All need per-column **cardinality**, which the profile lacks.
- **A4 · Data-aware Size guard** _(deferred — needs the profiling extension)_ — exclude
-  **negative**-valued columns from Size (Draco `hard.lp:56`; size implies positive
-  magnitude). Today only the type-level Size discipline is enforced.
+    detects exactly the mark-count == row-count case (URL/non-tabular → `rowCount` null → skipped).
+  - _Done (via the profiling extension below):_ an **aggregated** axis that still has many
+    distinct **categories** (`CROWDED_CATEGORY_DISTINCT` = 30) and an unreadable **Color
+    legend** (`CROWDED_LEGEND_DISTINCT` = 12, discrete colour only) now warn from per-column
+    cardinality. (Number-typed-Nominal remains a possible future nudge; not yet flagged.)
+- **A4 · Data-aware Size guard** _(done)_ — Size mapped to a field whose profiled numeric
+  **extent** goes negative warns (Draco `hard.lp:56`; size implies positive magnitude),
+  on top of the type-level Size discipline.

-> **TODO — profiling extension (the A3-remaining + A4 enabler).** `profile.ts` computes only
-> `rowCount` / `columnCount` / `columnTypes`. Extend it, in the **same sample pass** that
-> already feeds `inferColumnType` (so it's nearly free), to also derive per column: a
-> **capped distinct count** (cardinality — cap at ~50; a sampled count is enough for a
-> ">N categories" threshold, no full scan) and a **numeric extent** (min/max → sign).
-> Surface them on `DatasetProfile` (alongside `columnTypes`). Then: `builderWarnings` consumes
-> cardinality (legend/axis crowding) and the Size gate consumes sign (A4). **Caveats:**
-> URL / non-tabular datasets have no rows at profile time → these fields are null and the
-> dependent warnings simply skip; and stored datasets predate the field, so this needs a
-> recompute-on-read or a `dataset-migrations` bump (see architecture 02 / 06). Keep
-> thresholds in `chart-builder.ts` constants like `CROWDED_CATEGORY_ROWS`.
+> **Done — profiling extension (the A3 / A4 enabler).** `profile.ts` now derives, in the
+> **same sample pass** that feeds `inferColumnType`, a per-column **capped distinct count**
+> (`DISTINCT_CAP` = 50, with a `distinctCapped` overflow flag) and a **numeric extent**
+> (min/max, numeric columns only), surfaced on `DatasetProfile.columnStats` and the `Dataset`
+> record. `builderWarnings(config, rowCount, columns)` consumes them for the legend/axis
+> crowding hints and the negative-value Size guard. **Caveats handled:** URL / non-tabular
+> data has no rows → `columnStats` is `[]` and the dependent hints skip; datasets stored
+> before the field default to `[]` via `dataset-migrations` / import normalization (no forced
+> re-profile on read — they pick up stats on next save). Thresholds live in `chart-builder.ts`
+> constants alongside `CROWDED_CATEGORY_ROWS`.

 **B · Transform-enabled coverage (new core capability + §06 extension)** _(done)_