astrolabe/docs/chart-builder-research.md

# Chart Builder — Design Research (M4)

> **Status:** research complete; informs the M4 build (spec §06).
> **Decision:** build **Tier B — "smart + guarded"** (mark-first, still §06-shaped).
> **Why this doc exists:** the Chart Builder is the point where Astrolabe stops being
> a pass-through JSON editor and starts making chart-shaped suggestions/defaults.
> "Which chart, and why" becomes a decision the app owns, so we researched it
> deliberately before building. This is the record of what we studied and what we
> took from each source — the citations behind every default and guardrail in
> `src/core/chart-builder.ts`.

---

## 1. Scope of the builder (the constraint everything maps into)

Spec §06: compose a Vega-Lite chart from a dataset with **one mark** ∈
{Bar, Line, Point, Area, Circle}, mapping columns to **four channels** (X, Y, Color,
Size), each carrying a **field type** ∈ {Quantitative, Nominal, Ordinal, Temporal},
plus optional pixel width/height → a complete spec saved as a snippet that
references the dataset by name. Column types are inferred upstream as
`number | string | date | boolean` (`src/core/type-inference.ts`).

No transforms (no binning, aggregation, stacking, regression), no second axis, no
geo. That narrow surface is the lens through which every source below was read:
"what does this canon tell us to do **within Bar/Line/Point/Area/Circle and
X/Y/Color/Size?**"

## 2. The sources

Two kinds: **formal CS** (how recommendation engines actually rank charts) and
**chart-choice canon** (how practitioners pick). They were chosen for being
**cloneable/grep-able offline** (the council's working model) and authoritative for
"which chart," which our other seats (Carbon/GOV.UK/APG/NN/g) don't cover.

| Source                                     | What it is                                                                                                                                                                                                                   | Local path                                  |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| **Draco** (uwdata)                         | Visualization design knowledge as ASP constraints — the formal "what makes a good chart," with hard (validity) + soft (preference) rules and weights, some learned from human perception experiments (Kim 2018, Saket 2018). | `reference/draco`                           |
| **Voyager** (vega)                         | UW IDL's recommendation/exploration tool on CompassQL — the _interaction_ model (field shelves, auto-add, type chips) and effectiveness-ranked encoding suggestions.                                                         | `reference/voyager`                         |
| **FT Visual Vocabulary** (Financial Times) | A poster/taxonomy mapping _what you want to show_ (9 data-relationship categories) → chart types. **Seated** in the council.                                                                                                 | `reference/chart-doctor/visual-vocabulary/` |
| **Datawrapper**                            | Practitioner chart-choice in plain language; intent-first ("the chart's main statement becomes a compass"). **Seated** (distilled).                                                                                          | `reference/principles/datawrapper.md`       |

**Theoretical basis, not seated (deliberately):** **Munzner**, _Visualization Analysis
and Design_ (marks & channels; the channel-effectiveness rankings — magnitude:
position → length → angle → area …; identity: spatial region → hue → shape;
expressiveness & effectiveness principles) and **Wilke**, _Fundamentals of Data
Visualization_ (`clauswilke/dataviz`; example directory by intent + "ugly/bad/wrong"
pedagogy). They are the _why_ beneath Draco and Voyager — Draco's soft weights are an
operationalization of exactly these Mackinlay/APT/Munzner effectiveness rankings — but
they restate the same rules the seated sources already give us, so seating them would
add overlap, not coverage. Cited here as grounding; revisit if we ever build the
intent-first "Tier C" front door, where Munzner's typology and Wilke's directory
would earn their place.

## 3. What we take from each source

### From Draco — validity guardrails + a preference ranking (the rigorous core)

Draco models a chart as ASP facts and rejects/ranks them with **hard** (∞ cost) and
**soft** (weighted) constraints (`asp/optimize.lp`). We can't ship an ASP solver in a
browser, but the rules are a lookup table. The portable subset:

- **Hard validity (block in the UI):** `reference/draco/asp/hard.lp`
  - Quantitative on a string/boolean column — illegal (`:6`). Temporal only on a
    datetime column (`:7`).
  - **Size encoding a Nominal field — illegal** ("size implies order; nominal is
    misleading", `:53`). Size cannot encode **negative** values (`:56`). Size only on
    point/text marks (`:110`).
  - Bar/Area must include a **zero baseline** on the measure axis (`:103-104`).
  - Bar needs a categorical axis — both x and y continuous on a bar is malformed
    (`:97`); Line/Area need **both** x and y, and not both discrete (`:91,:94`).
  - Same field on x and y — illegal (`:122`). >20 categorical colors — illegal (`:172`).
- **Soft preference (the weights, `asp/weights.lp` + `asp/soft.lp`):**
  - Channel-by-type appropriateness (lower = better): continuous data is free on x/y,
    costs to put on color (10) or size (1); nominal cheapest on y then x then color;
    ordered data expensive on size. → **fill X/Y before Color/Size.**
  - Mark by data shape: continuous×continuous → **point** (line/area heavily
    penalized); continuous×discrete aggregated → **bar**; discrete×discrete → point/rect.
  - Prefer time on x (`temporal_y`, `:147`); never type a number as nominal
    (`number_nominal`, weight 10); the loudest nudge is an all-discrete chart with no
    measure (`only_discrete`, weight 30).

The hand-tuned `weights.lp` is the portable "common-sense" set; the learned
`weights_learned.lp` corroborates direction, not magnitude.

### From Voyager — the interaction model + the valid-type table

- **`getValidTypes` (`src/components/data-pane/field-list.tsx:140-155`) — adopted
  almost verbatim:** number→{quantitative, nominal}, integer→{quantitative, nominal},
  datetime→{temporal}, string→{nominal}, boolean→{nominal}. The type toggle shows only
  when ≥2 valid types exist. (We extend slightly — see §4 — to also offer Ordinal,
  which Voyager deliberately omits, `encoding.ts:131-134`.)
- **Auto-add / "auto" mark (`models/shelf/index.ts:72-81`):** Voyager lets a field be
  added with `channel:'?'` and asks CompassQL to place it by `effectiveness`. The small
  builder analogue is a **non-empty smart default** (`defaultBuilderConfig`) so the
  preview is never blank.
- **Type chips + swap:** per-field type indicator with a click-to-change popover, and a
  cheap x↔y swap (Voyager's `SPEC_FIELD_MOVE` is remove-both + re-add).
- **Out of scope (Voyager scope creep we reject):** wildcard shelves, the full Related
  Views gallery, faceting (row/column), and embedding CompassQL/`compassql@0.20.2`
  itself. We hand-roll a small decision table in `src/core/` instead of pulling the
  engine.

### From FT Visual Vocabulary — the intent→chart taxonomy (and our coverage gaps)

`reference/chart-doctor/visual-vocabulary/README.md` (taxonomy is prose). Nine
categories; mapped to **our five marks**:

| FT category          | What it shows               | Our expression                                                                |
| -------------------- | --------------------------- | ----------------------------------------------------------------------------- |
| **Magnitude**        | size comparisons            | **Bar** (x=N, y=Q; horizontal x=Q, y=N for long labels) — primary             |
| **Ranking**          | position in an ordered list | **Bar, sorted** by value (the sort _is_ the feature)                          |
| **Change over Time** | trends                      | **Line** (x=T, y=Q; color=N for series); Bar/Area alternatives, single series |
| **Correlation**      | relationship of 2+ measures | **Point** (x=Q, y=Q); **Circle/bubble** + size=Q for a third measure          |
| **Deviation**        | +/− from a reference        | **Bar** with signed Q (diverging bar only)                                    |
| **Distribution**     | spread/frequency            | weak: raw **Point** strip, or **Bar** of pre-binned counts (no bin transform) |
| **Part-to-whole**    | component shares            | **none well** — redirect to Magnitude/Bar; we can't show true proportions     |
| **Spatial**          | geography                   | **none** — exclude                                                            |
| **Flow**             | movement between states     | **none** — exclude                                                            |

**Coverage:** strong on Magnitude, Ranking, Change-over-Time, Correlation; partial on
Deviation/Distribution; none on Part-to-whole/Spatial/Flow. Honest gaps, not silent
degradation.

### From Datawrapper — plain-language rules + intent labels

`reference/principles/datawrapper.md`. Corroborates the same default-mark-by-intent
table (comparison→Bar, time→Line, correlation→Point/bubble) and supplies friendlier
intent words (Developments over time / Shares / Comparison / Correlation). Bindable
rules: bar is the safe default; bar over column on small screens; line for continuous
time, columns for a few points; circles are hard to compare precisely; size encodes a
quantity; area = single total (warn on multi-series).

## 4. The convergent rules — what all four agree on (high-confidence)

These are not a judgment call; the formal engines and the practitioner canon land on
the same place. They are the spec for `src/core/chart-builder.ts`:

1. **Column type → valid field types** (Voyager `getValidTypes`; Draco `hard.lp:6-7`):
   `number`→{Quantitative (default), Ordinal, Nominal}; `date`→{Temporal only};
   `string`→{Nominal (default), Ordinal}; `boolean`→{Nominal}. Never offer Q for
   string/boolean, never Temporal for a non-date. (We add Ordinal where it's a defensible
   user assertion of order; Voyager omits it for UX simplicity — our deliberate superset.)
2. **Default mark from the (X, Y) shape** (Draco mark-by-shape; Voyager effectiveness;
   FT; Datawrapper): temporal × quantitative → **Line**; quantitative × quantitative →
   **Point**; (nominal/ordinal) × quantitative → **Bar**; both-discrete → **Point**
   (Bar/Line/Area are invalid with no continuous axis); single axis or unknown → Bar.
3. **Channel priority + Size discipline** (Draco `hard.lp:53,56,110` + non-positional
   pref): fill X/Y before Color/Size; Color before Size. **Size is only valid for
   Quantitative/Ordinal positive measures on Point/Circle marks** — disabled for Nominal,
   Temporal, and negative data (not merely discouraged).
4. **Bar/Area zero-baseline; Line exempt** (Draco `hard.lp:103-104`; FT; ONS/Vox sources
   FT links). We expose no axis-truncation control, so Vega-Lite's own defaults already
   give zero-baseline bars and free-baseline lines — the rule is satisfied by _not adding_
   an override, nothing to emit.
5. **Chart-choice polish** (FT; Datawrapper): sort bars when ranking; horizontal bar for
   long category labels; Size encodes a quantity, Color a category; Area is for a single
   series (warn against color-splitting into many).

## 5. The decision: Tier B — "smart + guarded"

Three tiers were on the table. **Tier B** was chosen (2026-06-05).

- **Tier A — spec-literal:** Bar default, four channel dropdowns, type override, smart
  pre-population. Matches §06 verbatim but uses almost none of the research; stays a
  "dumb" composer.
- **Tier B — smart + guarded (chosen):** Tier A **+** default _mark_ from the (X, Y) type
  shape (not always Bar) **+** valid-type-only menus **+** inline non-blocking warnings
  from the Draco rules **+** swap-X/Y **+** Size disabled for Nominal/Temporal/negative.
  Still mark-first and §06-shaped, but genuinely intelligent. Requires a small §06
  amendment (documented in the spec).
- **Tier C — intent-first aid:** Tier B **+** a "what do you want to show?" front door
  (FT/Datawrapper intents → recommended mark + channel layout from intent × column
  types). Highest "which chart & why" value; biggest UI; clearly extends §06. Deferred —
  if revisited, this is where Munzner's typology and Wilke's directory would be seated.

## 6. How it maps to implementation

The convergent rules become pure functions in `src/core/chart-builder.ts`
(tested in `chart-builder.test.ts`), consumed by the builder store/modal:

- `validFieldTypes(columnType)` → the type menu (rule 1); `defaultFieldType` = its head.
- `defaultMark(xType, yType)` → smart default mark (rule 2); used by
  `defaultBuilderConfig`.
- `isChannelTypeAllowed(channel, type)` → Size discipline gate (rule 3).
- `builderWarnings(config)` → inline non-blocking hints (rules 3–5: line/area need both
  axes, area + many series, two measures better as a scatter, etc.).
- `buildChartSpec` / `buildSnippetSpecText` → assemble the final spec; zero-baseline is
  Vega-Lite-default (rule 4), so nothing is emitted for it.

## 7. Anti-recommendations (what a naive builder would happily produce, and we don't)

The highest-value guardrails — encodings a naive UI emits that the canon rejects:

- A categorical column on **Size** (Draco hard `:53`) — blocked, not warned.
- A **truncated-axis bar** — prevented by never exposing an axis override (Draco `:103`).
- A high-cardinality category on **Color** → unreadable legend (soft w=10; >20 hard).
- A **Line between two raw measures** instead of a scatter (Draco soft w=20) — warned.
- An **all-categorical chart with no measure** (Draco soft w=30, the loudest) — warned.
- A **number typed Nominal** (Draco soft w=10) — discouraged via default = Quantitative.

## 8. Future enhancements (backlog)

The Tier-B build is the floor, not the ceiling. The enhancements below were surfaced by
the research. Status as of 2026-06-06.

> **Forward plan moved (2026-06-10):** the _prioritized, sequenced_ enhancement plan now
> lives in [`chart-builder-enhancement-scope.md`](./chart-builder-enhancement-scope.md),
> which merges this backlog with the [`lyra-review.md`](./lyra-review.md) §5 ideas and sets
> the **Tier-C** target. This section remains the research _record_ (the citations behind
> each item); consult the scope doc for _what to build next and in what order_.

**A · Cheap wins inside the current 5-mark / 4-channel scope**

- **A1 · Sort-on-ranking** _(done)_ — chart-level Sort control (Asc/Desc/None) sorts the
  categorical axis by the measure (FT: "bars display ranks much more easily when sorted").
  Appears only for a category-vs-measure pair.
- **A2 · Bar orientation** _(partly done)_ — the **Swap X/Y** control is the manual path to
  a horizontal bar, and the crowded-axis hint (A3, below) now auto-suggests it for the
  un-aggregated case. A general "long labels → go horizontal" suggestion on _any_ vertical
  bar is still deferred (needs a label-length / cardinality signal); a blanket warning was
  rejected — it would fire on every ordinary vertical bar.
- **A3 · Crowded-axis & high-cardinality warnings** _(done)_ —
  - _Done:_ the **un-aggregated crowded axis** — a bar/line/area with a category axis and a
    **raw** measure draws one mark (and one label) per row, so over `CROWDED_CATEGORY_ROWS`
    (30) rows it warns and points to aggregating, or a horizontal bar. Row-count-based:
    detects exactly the mark-count == row-count case (URL/non-tabular → `rowCount` null → skipped).
  - _Done (via the profiling extension below):_ an **aggregated** axis that still has many
    distinct **categories** (`CROWDED_CATEGORY_DISTINCT` = 30) and an unreadable **Color
    legend** (`CROWDED_LEGEND_DISTINCT` = 12, discrete colour only) now warn from per-column
    cardinality. (Number-typed-Nominal remains a possible future nudge; not yet flagged.)
- **A4 · Data-aware Size guard** _(done)_ — Size mapped to a field whose profiled numeric
  **extent** goes negative warns (Draco `hard.lp:56`; size implies positive magnitude),
  on top of the type-level Size discipline.

> **Done — profiling extension (the A3 / A4 enabler).** `profile.ts` now derives, in the
> **same sample pass** that feeds `inferColumnType`, a per-column **capped distinct count**
> (`DISTINCT_CAP` = 50, with a `distinctCapped` overflow flag) and a **numeric extent**
> (min/max, numeric columns only), surfaced on `DatasetProfile.columnStats` and the `Dataset`
> record. `builderWarnings(config, rowCount, columns)` consumes them for the legend/axis
> crowding hints and the negative-value Size guard. **Caveats handled:** URL / non-tabular
> data has no rows → `columnStats` is `[]` and the dependent hints skip; datasets stored
> before the field default to `[]` via `dataset-migrations` / import normalization (no forced
> re-profile on read — they pick up stats on next save). Thresholds live in `chart-builder.ts`
> constants alongside `CROWDED_CATEGORY_ROWS`.

**B · Transform-enabled coverage (new core capability + §06 extension)** _(done)_

- **B5 · Aggregation** _(done)_ — per-channel `sum` / `mean` / `median` / `min` / `max`, plus
  a field-less "Count of records" measure (Voyager's `count(*)`). The priority item.
- **B6 · Binning** _(done)_ — `bin` on a quantitative field → true histograms (closes the
  Distribution gap); mutually exclusive with aggregate on the same field.
- **B7 · Stacking** _(done)_ — `stack` (`zero` / `normalize`) for bar/area + a Color series →
  part-to-whole (closes that gap; enables 100%-stacked).
- **Temporal granularity** _(done)_ — Vega-Lite `timeUnit` (Year / Quarter / Month / Week /
  Day / Hour, plus combined units) on a Temporal field; defaults to None (raw).
- **B8 · Faceting (Row / Column → small multiples)** _(next increment, after A+B + UI land)_
  — two more channels that multiply the chart into a trellis, the clean way to compare many
  categories (Voyager has it; FT/Datawrapper recommend small multiples; we currently can't
  express them). Still mark-first, so a Tier-B extension. **Axis alignment** is the design
  crux: Vega-Lite facets default to **shared scales** (aligned axes) — keep that as the
  default; expose an "independent axes" toggle (`resolve.scale`) only as an advanced option.
  **Verify** faceting against the preview's `"container"` fit modes before trusting it
  (per-cell sizing on facets is finicky). Sequenced as additive after the current build.

**C · Intent-first front door (Tier C)** _(deferred)_ — see §5. "What do you want to
show?" → recommend mark + channels from the FT/Datawrapper taxonomy × column types; where
Munzner + Wilke would be seated.

**D · Plumbing** — **D9** URL hash routing for the open builder (owned by **M6**, spec
§01E); **D10** a GOV.UK/NN-g copy pass over the guidance-hint wording (the M4 council
seating's residual one-off debt).

---

_Citations are to files under `/Users/oleh/code/reference/`. The seated chart-choice
canon (FT clone + Datawrapper distill) lives in the council roster
(`.claude/skills/council/SKILL.md`); Draco/Voyager are reference clones, not council
seats — they're engineering sources, not user-facing design authorities._