astrolabe/docs/exploration/chart-builder-enhancement-scope.md

Chart Builder — Enhancement Scope

Status: scope consolidated 2026-06-10. This is the single forward-looking home for chart-builder enhancement work — it merges the research backlog from chart-builder-research.md §8 (the M4 decision and its deferred items) with the interaction ideas from lyra-review.md §5, read against the current spec (spec/06-chart-builder.md) and the shipped code (src/core/chart-builder.ts).

Goal (the brief): a rapid, intuitive GUI for building Vega-Lite specs, with guidance and recommendations on the fly, moderately capable — not a full visual-design IDE.

Decision (2026-06-10): push the builder from its shipped Tier B ("smart + guarded") up to Tier C ("intent-first aid"). "Moderately capable" is the ceiling: we add the controls that are both common and awkward in JSON and stop there; the long tail of styling/scale/axis breadth stays in Monaco. The two source docs remain the research record (the why and the citations); this doc is the plan (the what next and the order).

---

Status log

Newest first. The at-a-glance build-order tracker is §4; per-item detail is §3. This log is the quick "where are we" — read it first.

• 2026-06-13 (guidance: reason over role, not raw type) — closed a false-positive class in builderWarnings (eng-council + council consult). A histogram (bar, binned-Q X, count Y) tripped "two measures → scatter" because isMeasureMapping/effectiveType ignored bin. Fix: isMeasureMapping is now bin-aware (a binned field is a discretized dimension — mirrors Vega-Lite's own isDiscrete(fieldDef)); a new isReorderableCategory keeps Sort from being offered on a histogram's binned axis (the council-flagged regression); the scatter rule routes through the role predicate + a positive mark list (deletes the mark !== 'rect' bolt-on); stackMeasureChannel excludes binned. Pinned with regression tests (no scatter/Sort on a histogram; real bar still sorts; two raw measures still nudge). Promoted to /alignment check #15 ("reason over role, not raw type"). Not done (deliberately): intent-gating the taste warnings — verified no current intent layout trips one, so the gate would be dead code; recorded as a standing principle in builderWarnings instead. The broader posture (curate good via the front door > enumerate bad via warnings) is the council's recommended bottom for the combinatorial-warning worry. Verified: typecheck + test (913) + eslint + build.

• 2026-06-13 (3A + heatmap mark) — the intent-first front door shipped (Tier C), and the mark set gained rect (Heatmap).

  • Heatmap mark — rect added to MARK_TYPES (six marks; labelled Heatmap in the picker, which now wraps to two rows in the 320–360px pane). Guidance rewired: both-axes rule covers heatmaps; a new hint nudges a Colour measure on a two-axis heatmap with a one-click Colour by count; rect is exempt from the two-measures→scatter nudge (a binned 2-D histogram is valid). Never the auto-default mark; names read "Heatmap of …". Spec §06 mark-type + guidance updated.
  • 3A intent front door — a persistent strip at the top of the config pane (the interaction model chosen with the user over a replace-on-open screen — "shows all controls + a do-it-for-me", the Tableau Show Me parallel). Core (chart-builder.ts): CHART_INTENTS (Compare/Ranking/Change-over-time/Correlation/Distribution/Part-to-whole/ Heatmap), intentLayout (intent × column-roles → mark + channels), intentApplicable (Show-Me gating), applyIntent (reshape, keep dataset/transforms/title), and activeIntent (structural match → the live chart's intent highlights with no stored state; lights the smart default on open). Store: setIntent. UI: an APG toolbar of toggle chips (roving tabindex, arrows move / Enter applies, accent-ring selection, aria-disabled+reason on inapplicable intents).
  • Council run on the front-door copy/flow → toolbar (not radiogroup — select-on-arrow would reshape the chart), disabled-focusable-with-reason, intent-framing (Heatmap kept as the one chart-type label, mirroring the mark). Recorded in architecture/10 §5; spec §06 gained an Intent (the front door) subsection.
  • Verified: typecheck + test (909, +17: intent core/store, heatmap guidance/naming)
    • eslint + build. Owed: a visual pass on the live strip (chips, disabled states, keyboard) + a heatmap rendered from real data. Open (user's call): whether the Heatmap chip should read as an intent phrase instead (council's intent-framing point).

• 2026-06-12 (3D) — entry points & discoverability shipped. Up next: 3A (the front door now has somewhere to be found).

  • Library creation surface forked: primary Build Chart (accent, takes the slack) + ghost New JSON snippet (the old instant create) — two plain buttons, no menu widget; labels collapse in two stages as the pane narrows (the long ghost label first). New chart icon (rising columns, pane-icon rect style).
  • The builder opens un-targeted: init(null) picks the most recently modified dataset; a Dataset picker heads the config pane (replaces the "Building from" line). Switching with an untouched config re-derives smart defaults; a built-on config is rebased via the new core rebaseBuilderConfig (mark/title/size/ sort/stack/calculates/expression-filters kept; encodings + predicate filters bound to columns the new dataset lacks shed; same-schema switch keeps everything).
  • No-datasets state (Carbon no-data): what the builder does + primary "Add a dataset" → Datasets create form. A defensive "Choose a dataset" chooser covers the datasets-exist-but-none-loaded case. New #build hash serializes the no-dataset builder (spec §01E table).
  • Onboarding gains the data-first door ("Build a chart from your data", ghost beside the primary; applies the workspace split up front since the builder can create the first snippet). Dataset-row "Build Chart" unchanged (the contextual shortcut).
  • Contract updated: spec §02 (creation surface, onboarding), §06 (Opening rewritten: doors, dataset picker, no-datasets state), §01E (#build); council resolutions recorded in architecture/10 §5 ("builder-forward, Monaco-intact").
  • Verified: typecheck + test (873, +10: core rebase, store init(null)/ switchDataset, url-hash #build, modal empty states) + eslint + prettier. Owed: a visual pass on the forked creation surface + builder picker/empty states (user-driven, batched with the 2B visual debt).

• 2026-06-12 (polish batch) — the 2B visual-pass findings fixed; the parked council batch resolved and applied; two capability gaps closed.

  • Aggregation for any field type. validAggregateOps replaces the quantitative-only gate: Count distinct on anything (a nominal Colour/Y now measures unique values — emitted as a quantitative { aggregate: 'distinct' }), Min/Max on temporal/ordinal, arithmetic still quantitative-only. Retyping keeps a still-valid aggregate.
  • Title/subtitle (pulled forward from the placement discussion): top-level title emission; a user title becomes the snippet name; subtitle gated on a title.
  • Council batch applied: the type chip and every native <select> in the builder are now SelectControl — a reusable value-picker disclosure (the SortControl primitive generalized; replaces native selects app-wide where the control is part of a designed surface). Field chips open an explicit channel chooser (occupied channels say what they'd replace); an armed channel short-circuits it and is now visible (accent ring + status line + Esc disarms). "or constant" → "Use a constant" ghost button. Chart properties (Title · Subtitle · W · H) moved to a strip under the preview; Sort/Stack stay by the encodings. Resolutions recorded in architecture/10 §5; ux-second-pass.md cleared (drag stays deferred).
  • Visual-pass fixes: pill ✕ flush right (pills hug content), Swap X/Y beside the Axes heading, field shelf scrolls (280px viewport, sticky group heads), per-channel transforms inline beside the pill, preview honors explicit width/height (fit-mode only while auto).
  • Verified: typecheck + test (774, +10) + eslint + build. Owed: a fresh visual look at the reworked surfaces (user-driven).

• 2026-06-12 (scope) — 3D added to Phase 3: builder entry points & discoverability (council-reviewed: NN/g #6 recognition + #7 expert accelerators, Carbon empty-states). Today the builder's only door is Datasets → row "Build Chart" while the primary "Create New Snippet" lands in blank Monaco — the data model's shape, not the user's intent. Decided: fork the library's creation surface (primary Build Chart, ghost New JSON snippet), give the builder an internal dataset picker + a no-datasets empty state, add an onboarding tile. 3D lands with or just before 3A — the Build-Chart button opens onto the intent front door, making 3A the app's guided creation flow. Detail in §3 · 3D.

• 2026-06-11 (scope) — 3C added to Phase 3: open-in-builder (strict spec hydration). A snippet-list button, enabled exactly when the spec round-trips losslessly through the builder dialect (checked by re-assemble + deep-compare, not feature enumeration). Strict only — no lossy/residue modes (§5's round-trip trap stays closed). Detail in §3 · 3C.

• 2026-06-11 (Phase 2) — 2A + 2B shipped: the field-first interaction substrate. Up next: 3A intent-first front door.

  • 2A · Value-or-field channels (the Property model). ChannelMapping gained a constant value arm; the assembler emits { value } and the measure/stack/area-split/prune logic all treat a constant Colour as "no series." New pure helpers (isValueMapping, channelAcceptsValue, defaultChannelValue, coerceChannelValue, isColumnAllowedOnChannel). type stays required (a constant carries a preserved-but- ignored type, so a field↔constant toggle round-trips). Tested.
  • 2B · Field-first builder. The channel-first dropdown rows are replaced by a field shelf (columns as type-glyphed chips, auto Dimensions/Measures split past a column threshold), click-to-assign (armed channel, else first empty that accepts it — assignField/focusChannel), Tableau-style pills (a type chip that cycles the field type in-place, transforms beneath), and the on-chart Columns/Rows shelves above the preview (X/Y as a property of the chart) with a reserved faceting placeholder in each. Colour/Size live in a left Marks card, each switchable to a constant (2A).
  • Decisions taken (overridable): drag deferred (click/keyboard-first, fully tested); field grouping auto; aggregate stays a control under the pill (no pill context-menu yet).
  • Spec §06 rewritten for the field-first model (Layout, Encoding channels, faceting placeholder, constants, validation/output).
  • Verified: typecheck + test (762, +20 core/store/modal) + eslint + build. Owed: a manual/visual pass against the live builder; a /council look at the type-cycle chip (a cycling button gives no direct type pick — explicit menu vs cycle is a real a11y trade-off) and the constant affordance.

• 2026-06-11 (later) — Owed debts on the Data section closed. Up next: Phase 2 (2A value-or-field channels, then 2B field shelf).

  • Council pass on the new error/disclosure copy (the previously-deferred auto-fire surface). Three a11y conformance gaps against architecture/10 were fixed: the inline expression feedback now carries a status glyph (round error / triangle warning), not colour alone (§3, WCAG 1.4.1); the parse error is a polite role="status", not a per-keystroke assertive alert (APG Alert / WCAG 2.2.4); and the message is linked to its input via aria-describedby (GOV.UK error-message). Unknown-field copy clarified to "…— not a column in this dataset." Resolution recorded in architecture/10 §5.
  • Manual/visual pass run via a headless-Chrome (Playwright) walk-through of the live builder — filter shelf (predicate + is between), field↔expression toggle, expression error/unknown-field glyphs (filter and calc inputs), calculated field, data-preview table with type chips, smart-default chart. All surfaces render as intended.
  • Bug found + fixed (mid-edit preview resilience). filterTransformObject / calculateTransformObject emitted any non-empty expression — including a half-typed, unparseable one — so the preview blanked with a raw render error while the user typed. They now drop a syntactically-invalid expression like an empty/incomplete entry (guarded by core validateExpression), matching buildTransforms' own "a config mid-edit still renders" contract; the inline feedback still flags the typo. Pure core, tested (+2 cases). Visually confirmed: an invalid filter/calc now keeps the last-good chart instead of breaking the preview.
  • Verified: typecheck + test (741 passing) + eslint clean.
  • Still open (copy judgment, user's call): the preview's catch-all "Couldn't render this chart: {raw Vega message}" puts a diagnostic in the headline (arch 10 says diagnostics go in a disclosure) — correct in the editor, debatable in the builder; and the builder's terse "Dataset «X» not found." drops the next-step the contract mandates (near-unreachable in the builder). Both noted, not changed.

• 2026-06-11 — 1C + 1D + 1E shipped (the Data section).

  • A new Data section at the top of the builder's left pane — "here are your rows; shape them, then encode them" — emits the spec's top-level transform array.
  • 1C · Filters — a guarded field + operator + value predicate shelf. Operators narrow by field type (validFilterOps): a measure/temporal field offers ordering (< ≤ > ≥) + is between; a category offers is / is not / is one of. Values coerce by type (quantitative → number; others → string, so ISO dates sort right). notEqual emits a { not: { …equal } } wrapper. A reversible expression power-mode takes a raw datum.… predicate. Incomplete filters are skipped so the preview keeps rendering. Multiple filters AND together. Pure core (buildTransforms, validFilterOps, filterOpArity), tested.
  • 1C · Calculated fields — { calculate, as } derived columns. A named field appears in the channel dropdowns via effectiveColumns (defaults Quantitative); emitted before filters (a row-wise calculate is order-independent, so calc-first is equivalent and lets filters reference derived fields). Removing/renaming a referenced field clears the dangling channel (pruneEncodings, in the store on calc edit/remove).
  • 1E · Expression validation — new pure core expr-validate.ts using Vega's own parseExpression (already in the vega chunk, so ~zero bundle cost): inline syntax errors on both expression inputs, plus a soft unknown-field warning when a datum.<field> reference doesn't match a column (referencedFields walks the AST). Field discoverability is served by dataset-derived placeholder examples (e.g. datum.revenue * 2); a full Monaco-style completion popup is deferred (a bare <input> doesn't warrant it — noted, not built).
  • 1D · Data preview — a collapsible, read-only first-N-rows table with a per-column type chip in each header, to sanity-check inferred types before building (reuses core tabularRows). Default collapsed; the scroll region is keyboard-reachable (tabIndex=0 + labelled group — avoids the Datasets-manager a11y gap).
  • Spec §06 gained a "Data (filters, calculated fields, preview)" section; the Layout and Output blocks cross-reference it.
  • Council not yet run on the new error/disclosure copy (the soft auto-fire surface: expression-error + unknown-field copy, the preview disclosure). Conventions were matched to the existing warnings region (arch 10 §5) and SettingsPopover disclosure; flag for a council pass on review if desired.
  • Expression reference: a contextual link to the Vega expression-language docs is shown when an expression input is in play (a calculated field, or a filter in expression mode) — the place the user needs to know the available functions/operators.
  • Verified: typecheck + test (full suite green; +103 new core/store cases, +4 modal smoke tests) + eslint + build (PWA, 45 precache entries). Owed: a manual/visual pass against the live builder (filter shelf, calc → channel, expr errors, preview table) — tests don't cover what the surface looks/feels like.
  • Surfaced direction (now parked): the 1D preview shows raw source rows; a transform-aware data inspector (resolved rows, à la vega-editor, in the builder and below the main Live Preview) is the wanted evolution — documented in docs/data-inspector-exploration.md, deferred.

• 2026-06-10 — Up next: 1C (filter + calculate transforms), paired with 1D (data preview).

  • 1B · Per-chart export shipped: an Export disclosure in the Live Preview header (distinct from the workspace Export) — Copy spec + Download JSON (.vl.json) of the shown text, and Download PNG/SVG of the live chart. Image output rides a new RenderHandle.toImageURL(format, { scale, background }) (PNG via view.toCanvas → blob: URL; SVG via view.toSVG → data: URL), so no component touches the Vega view. Filenames derive from the snippet name — filesystem-safe, script-preserving (pure core/chart-export.ts, tested). Home: the preview header, not the plan's "library row / editor toolbar" — image export needs the live view; the disclosure-of-controls (not an ARIA menu) mirrors SettingsPopover. - Export options (from first-round feedback): PNG Resolution 1×/2×/3× is a multiplier of devicePixelRatio, so the default 1× is Retina-crisp — the soft-1× export was a dpr bug (raw toImageURL scaleFactor ignores dpr). Background Theme(default)/White/None fixes transparent PNGs (the chart config is transparent so the on-screen pane colour shows; export composites the chosen colour under the PNG / adds an SVG <rect>). Referenced data Inline(default)/Keep refs (shown only when the spec references saved datasets) inlines dataset values so the exported spec renders standalone (inlineReferencedDatasets, tested). - Spec §08 gained a Per-chart export section (with the options); §04 cross-references it; architecture/05 §2 records the handle's dpr-aware scale + background compositing.
  • 1A · Actionable hints shipped: one-click fixes on guidance warnings (BuilderWarning.fixes + applyWarningFix), council-reviewed, with focus/announce a11y.
  • Builder UX/perf batch (from dogfooding the Superstore dataset) shipped: near-fullscreen xlarge modal tier; panes scroll internally (preview in its own viewport); backdrop-dismiss guard (dismissOnBackdrop: false); render-timing diagnostics; canvas preview renderer (SVG stays default elsewhere) + canvas max-dimension guard (ChartTooLargeError via a headless probe); data-aware default pre-population (smartDefaultEncodings).
  • Scope consolidated and Tier-C target set (this doc created); the research/Lyra forward sequences are superseded by §4 here.

---

1. Where the builder is today (the floor — don't rebuild)

Tier B is shipped and tested (M4 done). The intelligence that decides which chart and why already exists in src/core/chart-builder.ts:

• Smart default mark from the (X, Y) field-type shape — not unconditionally Bar (defaultMark).
• Valid-type-only field-type menus per column + Size discipline (Nominal / Temporal / negative-extent columns are blocked on Size, not merely warned) (validFieldTypes, isChannelTypeAllowed).
• Non-blocking guidance — builderWarnings (6 rules: line/area needs both axes, all-categorical, two-measures-want-scatter, area-split-many-series, crowded category axis, negative-size guard).
• Per-channel transforms — Aggregate / Bin / timeUnit, chart-level Sort / Stack, and a field-less "Count of records" measure.
• Swap X/Y, debounced live preview, validation gate (≥1 channel mapped).

Everything in §3 below is backlog — verified not yet built: BuilderWarning has no fix field, there is no per-chart export, no top-level transform, no data-table preview, no channels beyond X/Y/Color/Size, no styling/scale controls.

---

2. The target experience (Tier C, concretely)

Reading the brief's four words against the research:

Brief word	What it means here	The levers (from §3)
Rapid	shortest path from "a dataset" to "a chart I'd keep"	smart defaults (have), intent front door, starter examples, field shelf
Intuitive	matches how people think ("I have fields; what shows my point?")	intent front door, field shelf, data preview, value-or-field channels
Guidance & recommendations on the fly	the app proposes and corrects, not just validates	actionable hints, intent front door (the defining Tier-C feature)
Moderately capable	covers the common data-shaping + a few high-value encodings; not every knob	filter / calculate, a small set of promoted controls — and a hard stop short of Lyra's everything-inspector

The defining shift from B→C is the intent-first front door: a "what do you want to show?" entry (FT / Datawrapper intent categories) that recommends a mark + channel layout from intent × column-types, instead of starting the user at a blank mark picker. It is the feature the brief most directly asks for, and it is the one piece the M4 research deferred. Choosing Tier C is choosing to pull it forward.

Tier C extends §06 — it does not replace the mark-first builder. The front door is an on-ramp; the user can still ignore it and drive the channels directly, and can always drop to Monaco. This preserves Astrolabe's core invariant: the JSON spec is the source of truth; the builder is a view that emits it (the Lyra anti-lesson — never let the GUI become the document).

---

3. The consolidated enhancement set

Organized into build phases by dependency and value. Each item carries its source, value/effort, code home, and spec impact. Phases 1–3 are the committed Tier-C scope; Phase 4 is explicitly beyond "moderately capable" and gated on a later decision.

Phase 1 — Guidance + I/O (no new interaction model) — do first

High value-to-effort, mostly pure-core + thin UI, no architectural change. These make the current builder dramatically better and de-risk the bigger phases.

1A · Actionable hints — done (2026-06-10) Turn advisory warnings into one-click fixes. Source: Lyra §3.1 (Hints carries an action). Several existing warnings have an obvious remedy:

• "draws one mark per row → " [Aggregate as Sum] (set the measure's aggregate)
• "long labels → " [Swap X/Y] (the action already exists — just wire it)
• "two measures usually read as a scatter → " [Switch to Point]
• "area split into many series → " [Stack] or [Remove colour]

Implementation: extended BuilderWarning with fixes?: BuilderWarningFix[] ({ label, apply }, pure, unit-tested in chart-builder.test.ts); the modal renders each as a ghost button wired to a new applyWarningFix store action. Wired fixes: [Aggregate as Sum] + [Swap X/Y] (one-mark-per-row), [Swap X/Y] (high-cardinality axis), [Switch to Point] (two measures), [Stack] + [Remove colour] (area split — and a stacked area is no longer flagged, so [Stack] resolves it). Council run (Carbon Actionable notification + APG Alert): ghost buttons, remedy-in-button, polite "Applied: …" announcement, focus moved off the removed button — resolution recorded in architecture/10 §5. §06 "Guidance" amended to document the one-click fixes.

1B · Per-chart export — done (2026-06-10) Today export was workspace-backup only (§08); there was no way to get one chart out. Source: Lyra §3.8. Shipped as an Export disclosure in the Live Preview header:

• Copy spec (clipboard) + Download .vl.json of the currently-shown text.
• Download PNG / SVG of the live chart via a new RenderHandle.toImageURL wrapping view.toImageURL (PNG at 2× → blob: URL, revoked after download; SVG → data: URL), so the embedding boundary holds — no component touches the raw view.
• Filenames from the snippet name, filesystem-safe and script-preserving (pure core/chart-export.ts, tested). Standalone HTML left out (the deferred optional).

Home decision: the preview header, not the plan's original "library row / editor toolbar" suggestion — the image formats need the live rendered view, and "export this chart" reads best beside the chart. The widget is a disclosure-of-action-buttons (not an ARIA menu), mirroring SettingsPopover and sharing its single-open registry. Spec impact: new §08 "Per-chart export" section; §04 cross-reference; architecture/05 §2 handle note.

1C · Filter (+ Calculate) dataset transforms — done (2026-06-11; see status log) The transform layer the builder doesn't touch: top-level transform: []. Source: Lyra §3.12. The builder already tells users to filter in three warnings (chart-builder.ts:475,476,511) while offering no way to do it.

• Filter (highest) — a guarded field + operator + value predicate shelf (Voyager-style, no expression needed for the common case); power form is a raw datum.… expression validated with Vega's parseExpr (see 1E). VL applies top-level transforms before encoding aggregation, so "filter raw rows, then aggregate" is the natural default; filtering on an aggregated value (HAVING) is the advanced case — defer.
• Calculate / derived field (second) — transform: [{calculate, as}]; the new field then appears in column dropdowns like any other.
• Lookup / join a second dataset — larger data-model change (one dataset per snippet today) → defer to Phase 4.

Home: a new "Data" section in the builder's left pane, above the channels (here are your rows [+ Filter] [+ Calculate] → now encode them), paired with 1D. Spec impact: new §06 "Data / transforms" subsection — this is genuinely new behaviour, write it.

1D · Data-table preview — done (2026-06-11; see status log) Neither the builder nor the Datasets manager ever shows the actual rows. Source: Lyra §3.2. A compact, read-only first-N-rows grid with a per-column type chip in each header lets users sanity-check inferred types before building — exactly when inference is most likely to surprise. Type + cardinality + extent already come from profile.ts; we only need the row sample. Homes: a collapsible "Data" strip in the builder's left pane (under the dataset name) and/or the Datasets manager. Keep it read-only (editing data is out of scope). Spec impact: §06 + §05 (Datasets) additions.

1E · Inline expression validation + field autocomplete — done (2026-06-11; completion popup deferred) When 1C's expression mode lands, validate the Vega/VL expression string with the library's own parseExpr (Lyra §3.6) and surface errors inline; autocomplete the dataset's own column names (we have the schema from profile.ts) (Lyra §3.10). Record the parseExpr technique in architecture/08. Spec impact: folded into 1C.

Phase 2 — Interaction substrate (enables Tier C)

Two changes to how the user touches fields. They have standalone value but their main job is to be the substrate Phase 3 (and any future added channels) stands on — sequence them here so Tier C lands cleanly.

2A · Value-or-field channels (the Property model) — capability gain; medium Source: Lyra §3.3 (Property.tsx — one droppable control that is either a literal value or a bound field). Today a channel is field-only. Let a channel also hold a constant value (fixed colour / size) with one consistent control and a chip showing the binding kind. VL encodes exactly this (field vs value vs datum). Generalizes cleanly to any channel we add later. Spec impact: §06 "Encoding channels" — a channel may carry a constant.

2B · Field shelf + in-place type cycling — the largest interaction shift; the Tier-C/facet substrate Source: Lyra §3.4 (drop-zones) + Voyager (field list with type chips) + Lyra §3.5 (FieldType — the type icon is the control, click cycles N→O→Q→T within the valid set). Flip from channel-first ("pick a channel, then its column") to field-first ("here are your columns — drag/click onto channels"), matching how people think. This is the natural way to assign many fields across many channels, so it is the interaction substrate for Tier C and faceting, not a standalone task. Spec impact: §06 layout revision (field shelf alongside the channel rows).

Phase 3 — Tier C (the intent-first front door) — the defining feature of this push

The grand idea, stated once: the builder is the app's rapid, intuitive on-ramp; Monaco is the expert surface; the JSON spec stays the document. Phase 3 is where that becomes true — 3A gives the builder an intent-first opening screen, 3D makes that screen the app's guided creation flow (not a feature hidden in a modal), and 3B/3C seed and re-enter it.

3A · Intent-first front door — the B→C step Source: research §5/§8 Tier C (FT Visual Vocabulary + Datawrapper intent taxonomy). A "what do you want to show?" entry mapping intent × column types → recommended mark + channel layout:

Intent (FT / Datawrapper)	Our expression (within 5 marks / our channels)
Magnitude / Comparison	Bar (x=N, y=Q; horizontal for long labels)
Ranking	Bar, sorted by value
Change over time	Line (x=T, y=Q; color=N for series)
Correlation	Point (x=Q, y=Q); Circle + size=Q for a 3rd measure
Distribution	Bar of binned counts (histogram) — uses Bin
Part-to-whole	Stacked / 100% bar (uses Stack); true pie needs theta → Phase 4
Deviation	diverging signed Bar

Honest coverage gaps stay honest (Spatial / Flow excluded; Part-to-whole partial until theta). The front door is an on-ramp, not a gate — it pre-populates the mark-first builder, which the user can then adjust or ignore. Munzner's typology and Wilke's directory become seatable council sources at this point (research §2). Built on the 2B field shelf. Run the front-door copy + flow through /council. Spec impact: substantial §06 amendment — a new "Intent" front-door subsection; the M4 spec note explicitly flagged this as the deferred tier, so this is the planned amendment, not drift.

3B · Starter examples gallery — cheap; pairs with 3A Source: Lyra §3.7. A small set of curated starter snippets, one per covered FT intent (Magnitude/Bar, Change-over-time/Line, Correlation/Point, Distribution/histogram, Part-to-whole/stacked). Improves first-run, doubles as living documentation of what the app does well. Natural home: the snippet library. Author them in the builder dialect so they hydrate via 3C. Spec impact: §02 (library seed content).

3C · Open in builder (strict spec hydration) — decided 2026-06-11; pairs with 3B Reverse the assembler: a pure parseChartSpec(spec) → BuilderConfig | null so an existing snippet can re-enter the builder populated. Strict policy only — a snippet-list button ("Open in builder") enabled exactly when the spec is losslessly representable in the builder's dialect; anything richer stays Monaco-only. Losslessness is checked not by enumerating unsupported features but by re-assembling the parsed config and deep-comparing against the original (ignoring key order, $schema, and the injected tooltip: true) — exact, ~20 lines, and it stays correct automatically as the dialect grows (every Phase-4 addition widens what hydrates for free). No lossy or residue-preserving mode: the moment the builder can overwrite a richer spec, the GUI starts competing with the JSON as the document (the Lyra round-trip trap — §5).

Value: converts the builder from create-only to create-and-revise for its own output, and makes the 3B starters openable as builder seeds. Known fiddly bits (all mechanical): inverse of escapeVegaField, filter-value un-coercion back to the BuilderFilter string shape, sort: "-y" → {sort: 'descending'}. Round-trip property tests (parse(build(config)) ≈ config) are the natural core coverage. Per-row check in the library is cheap but memoize it. Decisions at build time: save semantics on an opened snippet (update-in-place via a builder "edit" mode vs. always-create-new — today the builder only creates) and the disabled-button affordance (disabled-with-reason vs. hidden — park in ux-second-pass.md if non-obvious). Spec impact: §02 (library row action) + §06 (hydration + the strict gate).

3D · Entry points & discoverability — done 2026-06-12 (council-reviewed; see status log); 3A opens onto it next Today the builder has one entry, two levels deep: Datasets modal → per-dataset "Build Chart" (DatasetsModal.tsx) — the data model's shape ("charts come from datasets"), not the user's intent ("I want to make a chart"). Meanwhile the library's pinned primary action, "Create New Snippet", drops straight into blank Monaco — the expert path is the default, and the audience the builder serves may never find it. Council ruling (recorded here; contract updates at build time):

• NN/g #6 (recognition over recall): the build-a-chart action must be visible where the intent forms — the library's creation surface, not recalled as a dataset-row action.
• NN/g #7 (flexibility/efficiency): the guided path is what novices see; raw JSON is the expert accelerator — visible, one click, never hidden. Builder-forward, Monaco intact.
• Carbon empty-states: one primary action per surface ("pick the most important"; a tertiary button for the secondary CTA); a no-data state explains what the space will hold, offers one primary next step, and never dead-ends.

The decided shape:

1. Fork the library's creation surface — primary Build Chart (opens the builder), ghost/tertiary New JSON snippet (today's instant create, unchanged). Two plain buttons with clear hierarchy — no split/menu-button widget (two static choices don't earn an ARIA menu).
2. Dataset picker inside the builder (header select, default = most recently modified), so the builder opens without a preselected dataset. Build-time decision taken: switching with an untouched opening config re-derives fresh smart defaults; switching a built-on config rebases it (rebaseBuilderConfig — chart-level intent kept, bindings to missing columns shed; a same-schema dataset keeps everything). Useful beyond 3D (switch data without leaving the builder).
3. No-datasets empty state in the builder, per the Carbon no-data pattern: what the builder does + one primary action ("Add a dataset" → Datasets modal). No dead end.
4. Onboarding tile — a data-first path ("import your data → build a chart") beside the existing examples gallery.
5. Keep the dataset-row "Build Chart" — contextual shortcut, pre-picks the dataset; it just stops being the only door.

Composition with 3A (the point of the sequencing): the Build-Chart button lands on the intent front door — "what do you want to show, with which data?" is the builder's opening screen. 3D without 3A opens onto the mark-first builder (fine, interim); 3A without 3D is a front door nobody finds. Discoverability is a prerequisite for 3A's value, so: 3D with or just before 3A. Spec impact: §02 (library creation actions + onboarding) + §06 (dataset picker, empty state).

Builder UX & perf — in-flight fixes (2026-06-10, from dogfooding the Superstore dataset)

Pre-existing builder rough edges surfaced while testing on a 10k-row / ~24-col dataset. Fixed in this batch (not part of 1A–3B, but the same surface):

• Modal is a near-fullscreen work surface — new xlarge shell tier (ModalShell, min(1800px, 96vw) × min(1100px, 92vh)); the Chart Builder no longer wastes screen.
• Panes scroll internally, modal keeps its shape — the .builder grid fills the body (grid-template-rows: minmax(0,1fr)), the config pane and the preview each scroll in their own viewport, so a tall one-mark-per-row chart scrolls inside the preview instead of pushing Create/Cancel below the fold.
• Backdrop click no longer discards in-progress work — dismissOnBackdrop: false on the builder (registry flag); Escape and × still close.
• Data-aware default pre-population — the builder no longer blindly takes the first two columns (which opened Superstore on a 9994-bar degenerate chart). When the dataset is profiled, defaultBuilderConfig picks a "safest bet": a low-cardinality category vs a count of records (tidy bar), else a time series of the first measure, else a scatter — each guaranteed to render. Falls back to positional when there are no stats. Pure + tested. This composes with (isn't replaced by) the future intent-first front door — the builder always needs a sane opening state.
• Render-timing diagnostics — BuilderPreview logs parse · prepare · destroy · embed · paint · total (+ mark, row count) to the console (dev always; prod only when slow). The paint phase (a double-rAF after embed()) captures the real freeze.

Perf finding — confirmed and fixed. Diagnostics on the Superstore dataset: embed 237ms · paint 6458ms for the default one-bar-per-row chart (9994 rows), vs paint 6ms once grouped to a few categories. The freeze was entirely SVG layout/paint (one DOM node per mark), not chart compilation. Fix shipped: the builder preview now renders with canvas (renderSpec(…, { renderer: 'canvas' })); SVG stays the default for the editor's LivePreview and for image export. Contract divergence recorded in architecture/05 §2.

Canvas max-dimension guard (measured, not guessed). Canvas (unlike SVG) has a hard max side (~32k px), so a chart that resolves taller than that fails to allocate (the broken-image icon). The cause is physical render size, not cardinality — a vertical bar with thousands of X bands renders fine (width is container-bounded); only an unbounded band axis (e.g. a horizontal bar's Y) overflows. So renderSpec now runs a headless ('none') layout probe for canvas charts, reads the chart's resolved height, and throws ChartTooLargeError(heightPx, limitPx) when it exceeds MAX_CANVAS_PX ÷ dpr. The builder catches it and shows the real numbers ("would be ~200,000px tall — larger than the browser can draw on a canvas (~16,383px max here); aggregate or filter"). The earlier band-count proxy (previewBandCount/MAX_PREVIEW_BANDS) was removed — readability (cardinality) stays a builderWarnings concern; the render-size limit is now measured at its true cause.

Phase 4 — Beyond "moderately capable" (gated — decide later)

These exceed the stated ceiling. List them so they have a home, but do not commit them in this push — revisit once Phases 1–3 land and we see real usage. Each must clear the guardrail: promote a control only when it is both common AND awkward in JSON.

• More channels — theta (unlocks pie/donut → true part-to-whole), opacity, shape. (theta is the most defensible — it closes a real coverage gap.) Ride on 2A/2B.
• Faceting (Row / Column → small multiples) — research §8 B8. The clean way to compare many categories. Design crux: VL facets default to shared scales (keep that default); expose an "independent axes" toggle (resolve.scale) only as advanced. Verify against the preview's "container" fit modes (per-cell sizing on facets is finicky).
• Light styling / scale controls — colour-scheme picker (categorical / sequential / diverging), measure-axis zero / log toggle, custom axis title, legend title / hide. Implement as auto-derived override panels that start empty (inheriting VL defaults) and emit JSON only when touched; clearing a channel drops its overrides (Lyra §3.9 cleanupUnused — no orphaned scale/axis in the output spec).
• Builder undo/redo / "reset to smart defaults" — Lyra §3.11. Low priority (the modal is short-lived; the smart default already gives a sane start).
• Lookup / join a second dataset — Lyra §3.12; data-model change (multi-dataset snippets). Larger, separate effort.
• Transform-aware data inspector — evolve 1D from raw-source rows to the resolved, post-transform data (filtered rows + calculated columns), and generalize it to a togglable data panel below the main Live Preview (vega-editor's "Data Viewer", a debugging aid for any snippet, not only builder output). Reads runtime rows via view.data(name) through the RenderHandle. Cross-cutting (editor + builder), so it has its own home: docs/data-inspector-exploration.md.

---

4. Recommended build order

Phase 1  1A actionable hints      ✓ done
         1B per-chart export      ✓ done
         1C filter (+ calculate)  ✓ done
         1D data preview          ✓ done
         1E expr-validate         ✓ done (syntax + unknown-field; completion popup deferred)
Phase 2  2A value-or-field channels (Property model)  ✓ done
         2B field shelf + in-place type cycling       ✓ done (field-first + on-chart shelves)
Phase 3  3D entry points & discoverability         ✓ done (2026-06-12)
         3A intent-first front door (Tier C)       ✓ done (2026-06-13); persistent strip
         3B starter examples                       ← next; pairs with 3C
         3C open in builder (strict hydration)     ← added 2026-06-11; pairs with 3B
Marks    +rect (Heatmap)                           ✓ done (2026-06-13)
Phase 4  (gated) theta/facets/styling-overrides/undo/lookup — decide after Phase 3

Also shipped (builder UX/perf, from dogfooding): near-fullscreen modal, internal-scroll
panes, backdrop-dismiss guard, render diagnostics, canvas preview + canvas-size guard,
data-aware default pre-population. See "Builder UX & perf" above.

Rationale for the order: Phase 1 is the cheapest large quality jump and needs no new interaction model, so it ships value while the bigger design settles. Phase 2 is pure substrate — low user-visible payoff alone, but Phase 3 is much cleaner on top of it than bolted onto the channel-first UI. Phase 3 delivers the brief's headline ("recommendations on the fly") — 3D first, because a front door nobody finds delivers nothing: the entry points make the builder the app's guided creation flow, then 3A gives that flow its intent-first opening. Phase 4 is deliberately deferred to protect the "moderately capable" ceiling.

---

5. Constraints & anti-scope (the ceiling)

What "moderately capable" rules out — load-bearing, from lyra-review.md §2.1/§4:

• No everything-inspector. Lyra surfaces ~50 direct controls per primitive because the GUI was its document. We split the work on purpose: a small guarded builder + a first-class Monaco editor for the long tail. Styling/scale/axis breadth for its own sake belongs in Monaco, not the builder.
• The promotion test: a control enters the builder only when it is both common AND awkward in JSON. Otherwise it stays in Monaco.
• JSON stays the source of truth. The builder emits spec; it is never the document (the Lyra one-way-export trap that forbids round-trips).
• No interaction-by-demonstration, no direct-manipulation canvas, no general data-pipeline editor. If we ever add interactivity, expose VL params/selections as a small guarded action — never port Lyra's signal generator.
• Take ideas from the reference clones, read no code into the repo (AGENTS.md: no shared lib; patterns adapted, not imported).

---

6. Cross-cutting notes

• Spec deltas: 1A (minor §06), 1B (§08 + §02/§03), 1C (new §06 transforms subsection), 1D (§06 + §05), 2A/2B (§06 layout), 3A (substantial §06 intent front-door amendment), 3B (§02). Per "spec follows code now": build the decision, then amend the spec to match — don't let code and §06 drift.
• Council: auto-fires on guidance copy and new interactive-widget keyboard/focus work — so 1A (hint affordance + copy) and 3A (front-door flow + copy) both go through /council before committing. It advises; architecture 09/10 decide.
• Verification: pure rules get chart-builder.test.ts cases; every UI/affordance change gets a manual pass against the live builder — a green build proves nothing about what the user sees (AGENTS.md; docs/manual-verification.md). Items 1A and 1C are mostly src/core/, squarely the "core-first, tested hardest" rule.
• Source of truth going forward: this doc. chart-builder-research.md §8 and lyra-review.md §5 remain the research record; their forward sequences are superseded by §4 here.