Add Chart Builder: no-JSON Vega-Lite composer from a dataset (M4)

docs/chart-builder-research.md

@@ -0,0 +1,206 @@
+# Chart Builder — Design Research (M4)
+
+> **Status:** research complete; informs the M4 build (spec §06).
+> **Decision:** build **Tier B — "smart + guarded"** (mark-first, still §06-shaped).
+> **Why this doc exists:** the Chart Builder is the point where Astrolabe stops being
+> a pass-through JSON editor and starts making chart-shaped suggestions/defaults.
+> "Which chart, and why" becomes a decision the app owns, so we researched it
+> deliberately before building. This is the record of what we studied and what we
+> took from each source — the citations behind every default and guardrail in
+> `src/core/chart-builder.ts`.
+
+---
+
+## 1. Scope of the builder (the constraint everything maps into)
+
+Spec §06: compose a Vega-Lite chart from a dataset with **one mark** ∈
+{Bar, Line, Point, Area, Circle}, mapping columns to **four channels** (X, Y, Color,
+Size), each carrying a **field type** ∈ {Quantitative, Nominal, Ordinal, Temporal},
+plus optional pixel width/height → a complete spec saved as a snippet that
+references the dataset by name. Column types are inferred upstream as
+`number | string | date | boolean` (`src/core/type-inference.ts`).
+
+No transforms (no binning, aggregation, stacking, regression), no second axis, no
+geo. That narrow surface is the lens through which every source below was read:
+"what does this canon tell us to do **within Bar/Line/Point/Area/Circle and
+X/Y/Color/Size?**"
+
+## 2. The sources
+
+Two kinds: **formal CS** (how recommendation engines actually rank charts) and
+**chart-choice canon** (how practitioners pick). They were chosen for being
+**cloneable/grep-able offline** (the council's working model) and authoritative for
+"which chart," which our other seats (Carbon/GOV.UK/APG/NN/g) don't cover.
+
+| Source                                     | What it is                                                                                                                                                                                                                   | Local path                                  |
+| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
+| **Draco** (uwdata)                         | Visualization design knowledge as ASP constraints — the formal "what makes a good chart," with hard (validity) + soft (preference) rules and weights, some learned from human perception experiments (Kim 2018, Saket 2018). | `reference/draco`                           |
+| **Voyager** (vega)                         | UW IDL's recommendation/exploration tool on CompassQL — the _interaction_ model (field shelves, auto-add, type chips) and effectiveness-ranked encoding suggestions.                                                         | `reference/voyager`                         |
+| **FT Visual Vocabulary** (Financial Times) | A poster/taxonomy mapping _what you want to show_ (9 data-relationship categories) → chart types. **Seated** in the council.                                                                                                 | `reference/chart-doctor/visual-vocabulary/` |
+| **Datawrapper**                            | Practitioner chart-choice in plain language; intent-first ("the chart's main statement becomes a compass"). **Seated** (distilled).                                                                                          | `reference/principles/datawrapper.md`       |
+
+**Theoretical basis, not seated (deliberately):** **Munzner**, _Visualization Analysis
+and Design_ (marks & channels; the channel-effectiveness rankings — magnitude:
+position → length → angle → area …; identity: spatial region → hue → shape;
+expressiveness & effectiveness principles) and **Wilke**, _Fundamentals of Data
+Visualization_ (`clauswilke/dataviz`; example directory by intent + "ugly/bad/wrong"
+pedagogy). They are the _why_ beneath Draco and Voyager — Draco's soft weights are an
+operationalization of exactly these Mackinlay/APT/Munzner effectiveness rankings — but
+they restate the same rules the seated sources already give us, so seating them would
+add overlap, not coverage. Cited here as grounding; revisit if we ever build the
+intent-first "Tier C" front door, where Munzner's typology and Wilke's directory
+would earn their place.
+
+## 3. What we take from each source
+
+### From Draco — validity guardrails + a preference ranking (the rigorous core)
+
+Draco models a chart as ASP facts and rejects/ranks them with **hard** (∞ cost) and
+**soft** (weighted) constraints (`asp/optimize.lp`). We can't ship an ASP solver in a
+browser, but the rules are a lookup table. The portable subset:
+
+- **Hard validity (block in the UI):** `reference/draco/asp/hard.lp`
+  - Quantitative on a string/boolean column — illegal (`:6`). Temporal only on a
+    datetime column (`:7`).
+  - **Size encoding a Nominal field — illegal** ("size implies order; nominal is
+    misleading", `:53`). Size cannot encode **negative** values (`:56`). Size only on
+    point/text marks (`:110`).
+  - Bar/Area must include a **zero baseline** on the measure axis (`:103-104`).
+  - Bar needs a categorical axis — both x and y continuous on a bar is malformed
+    (`:97`); Line/Area need **both** x and y, and not both discrete (`:91,:94`).
+  - Same field on x and y — illegal (`:122`). >20 categorical colors — illegal (`:172`).
+- **Soft preference (the weights, `asp/weights.lp` + `asp/soft.lp`):**
+  - Channel-by-type appropriateness (lower = better): continuous data is free on x/y,
+    costs to put on color (10) or size (1); nominal cheapest on y then x then color;
+    ordered data expensive on size. → **fill X/Y before Color/Size.**
+  - Mark by data shape: continuous×continuous → **point** (line/area heavily
+    penalized); continuous×discrete aggregated → **bar**; discrete×discrete → point/rect.
+  - Prefer time on x (`temporal_y`, `:147`); never type a number as nominal
+    (`number_nominal`, weight 10); the loudest nudge is an all-discrete chart with no
+    measure (`only_discrete`, weight 30).
+
+The hand-tuned `weights.lp` is the portable "common-sense" set; the learned
+`weights_learned.lp` corroborates direction, not magnitude.
+
+### From Voyager — the interaction model + the valid-type table
+
+- **`getValidTypes` (`src/components/data-pane/field-list.tsx:140-155`) — adopted
+  almost verbatim:** number→{quantitative, nominal}, integer→{quantitative, nominal},
+  datetime→{temporal}, string→{nominal}, boolean→{nominal}. The type toggle shows only
+  when ≥2 valid types exist. (We extend slightly — see §4 — to also offer Ordinal,
+  which Voyager deliberately omits, `encoding.ts:131-134`.)
+- **Auto-add / "auto" mark (`models/shelf/index.ts:72-81`):** Voyager lets a field be
+  added with `channel:'?'` and asks CompassQL to place it by `effectiveness`. The small
+  builder analogue is a **non-empty smart default** (`defaultBuilderConfig`) so the
+  preview is never blank.
+- **Type chips + swap:** per-field type indicator with a click-to-change popover, and a
+  cheap x↔y swap (Voyager's `SPEC_FIELD_MOVE` is remove-both + re-add).
+- **Out of scope (Voyager scope creep we reject):** wildcard shelves, the full Related
+  Views gallery, faceting (row/column), and embedding CompassQL/`compassql@0.20.2`
+  itself. We hand-roll a small decision table in `src/core/` instead of pulling the
+  engine.
+
+### From FT Visual Vocabulary — the intent→chart taxonomy (and our coverage gaps)
+
+`reference/chart-doctor/visual-vocabulary/README.md` (taxonomy is prose). Nine
+categories; mapped to **our five marks**:
+
+| FT category          | What it shows               | Our expression                                                                |
+| -------------------- | --------------------------- | ----------------------------------------------------------------------------- |
+| **Magnitude**        | size comparisons            | **Bar** (x=N, y=Q; horizontal x=Q, y=N for long labels) — primary             |
+| **Ranking**          | position in an ordered list | **Bar, sorted** by value (the sort _is_ the feature)                          |
+| **Change over Time** | trends                      | **Line** (x=T, y=Q; color=N for series); Bar/Area alternatives, single series |
+| **Correlation**      | relationship of 2+ measures | **Point** (x=Q, y=Q); **Circle/bubble** + size=Q for a third measure          |
+| **Deviation**        | +/− from a reference        | **Bar** with signed Q (diverging bar only)                                    |
+| **Distribution**     | spread/frequency            | weak: raw **Point** strip, or **Bar** of pre-binned counts (no bin transform) |
+| **Part-to-whole**    | component shares            | **none well** — redirect to Magnitude/Bar; we can't show true proportions     |
+| **Spatial**          | geography                   | **none** — exclude                                                            |
+| **Flow**             | movement between states     | **none** — exclude                                                            |
+
+**Coverage:** strong on Magnitude, Ranking, Change-over-Time, Correlation; partial on
+Deviation/Distribution; none on Part-to-whole/Spatial/Flow. Honest gaps, not silent
+degradation.
+
+### From Datawrapper — plain-language rules + intent labels
+
+`reference/principles/datawrapper.md`. Corroborates the same default-mark-by-intent
+table (comparison→Bar, time→Line, correlation→Point/bubble) and supplies friendlier
+intent words (Developments over time / Shares / Comparison / Correlation). Bindable
+rules: bar is the safe default; bar over column on small screens; line for continuous
+time, columns for a few points; circles are hard to compare precisely; size encodes a
+quantity; area = single total (warn on multi-series).
+
+## 4. The convergent rules — what all four agree on (high-confidence)
+
+These are not a judgment call; the formal engines and the practitioner canon land on
+the same place. They are the spec for `src/core/chart-builder.ts`:
+
+1. **Column type → valid field types** (Voyager `getValidTypes`; Draco `hard.lp:6-7`):
+   `number`→{Quantitative (default), Ordinal, Nominal}; `date`→{Temporal only};
+   `string`→{Nominal (default), Ordinal}; `boolean`→{Nominal}. Never offer Q for
+   string/boolean, never Temporal for a non-date. (We add Ordinal where it's a defensible
+   user assertion of order; Voyager omits it for UX simplicity — our deliberate superset.)
+2. **Default mark from the (X, Y) shape** (Draco mark-by-shape; Voyager effectiveness;
+   FT; Datawrapper): temporal × quantitative → **Line**; quantitative × quantitative →
+   **Point**; (nominal/ordinal) × quantitative → **Bar**; both-discrete → **Point**
+   (Bar/Line/Area are invalid with no continuous axis); single axis or unknown → Bar.
+3. **Channel priority + Size discipline** (Draco `hard.lp:53,56,110` + non-positional
+   pref): fill X/Y before Color/Size; Color before Size. **Size is only valid for
+   Quantitative/Ordinal positive measures on Point/Circle marks** — disabled for Nominal,
+   Temporal, and negative data (not merely discouraged).
+4. **Bar/Area zero-baseline; Line exempt** (Draco `hard.lp:103-104`; FT; ONS/Vox sources
+   FT links). We expose no axis-truncation control, so Vega-Lite's own defaults already
+   give zero-baseline bars and free-baseline lines — the rule is satisfied by _not adding_
+   an override, nothing to emit.
+5. **Chart-choice polish** (FT; Datawrapper): sort bars when ranking; horizontal bar for
+   long category labels; Size encodes a quantity, Color a category; Area is for a single
+   series (warn against color-splitting into many).
+
+## 5. The decision: Tier B — "smart + guarded"
+
+Three tiers were on the table. **Tier B** was chosen (2026-06-05).
+
+- **Tier A — spec-literal:** Bar default, four channel dropdowns, type override, smart
+  pre-population. Matches §06 verbatim but uses almost none of the research; stays a
+  "dumb" composer.
+- **Tier B — smart + guarded (chosen):** Tier A **+** default _mark_ from the (X, Y) type
+  shape (not always Bar) **+** valid-type-only menus **+** inline non-blocking warnings
+  from the Draco rules **+** swap-X/Y **+** Size disabled for Nominal/Temporal/negative.
+  Still mark-first and §06-shaped, but genuinely intelligent. Requires a small §06
+  amendment (documented in the spec).
+- **Tier C — intent-first aid:** Tier B **+** a "what do you want to show?" front door
+  (FT/Datawrapper intents → recommended mark + channel layout from intent × column
+  types). Highest "which chart & why" value; biggest UI; clearly extends §06. Deferred —
+  if revisited, this is where Munzner's typology and Wilke's directory would be seated.
+
+## 6. How it maps to implementation
+
+The convergent rules become pure functions in `src/core/chart-builder.ts`
+(tested in `chart-builder.test.ts`), consumed by the builder store/modal:
+
+- `validFieldTypes(columnType)` → the type menu (rule 1); `defaultFieldType` = its head.
+- `defaultMark(xType, yType)` → smart default mark (rule 2); used by
+  `defaultBuilderConfig`.
+- `isChannelTypeAllowed(channel, type)` → Size discipline gate (rule 3).
+- `builderWarnings(config)` → inline non-blocking hints (rules 3–5: line/area need both
+  axes, area + many series, two measures better as a scatter, etc.).
+- `buildChartSpec` / `buildSnippetSpecText` → assemble the final spec; zero-baseline is
+  Vega-Lite-default (rule 4), so nothing is emitted for it.
+
+## 7. Anti-recommendations (what a naive builder would happily produce, and we don't)
+
+The highest-value guardrails — encodings a naive UI emits that the canon rejects:
+
+- A categorical column on **Size** (Draco hard `:53`) — blocked, not warned.
+- A **truncated-axis bar** — prevented by never exposing an axis override (Draco `:103`).
+- A high-cardinality category on **Color** → unreadable legend (soft w=10; >20 hard).
+- A **Line between two raw measures** instead of a scatter (Draco soft w=20) — warned.
+- An **all-categorical chart with no measure** (Draco soft w=30, the loudest) — warned.
+- A **number typed Nominal** (Draco soft w=10) — discouraged via default = Quantitative.
+
+---
+
+_Citations are to files under `/Users/oleh/code/reference/`. The seated chart-choice
+canon (FT clone + Datawrapper distill) lives in the council roster
+(`.claude/skills/council/SKILL.md`); Draco/Voyager are reference clones, not council
+seats — they're engineering sources, not user-facing design authorities._