oleh/astrolabe
1
0
Fork 0
mirror of https://github.com/olehomelchenko/astrolabe.git synced 2026-08-08 02:02:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Docs: move point-in-time records to exploration/, trim build-narration, lift open items to plan

Browse Source
This commit is contained in:
Oleh Omelchenko
2026-06-14 23:58:06 +03:00
parent de9dbf3ec8
commit 66c123e15e
16 changed files with 77 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/exploration/chart-builder-enhancement-scope.md
+660
View File
@@ -0,0 +1,660 @@
# Chart Builder — Enhancement Scope
> **Status:** scope consolidated 2026-06-10. This is the **single forward-looking home**
> for chart-builder enhancement work — it merges the research backlog from
> [`chart-builder-research.md`](./chart-builder-research.md) §8 (the M4 decision and its
> deferred items) with the interaction ideas from [`lyra-review.md`](./lyra-review.md) §5,
> read against the current spec ([`spec/06-chart-builder.md`](../spec/06-chart-builder.md))
> and the shipped code (`src/core/chart-builder.ts`).
>
> **Goal (the brief):** a **rapid, intuitive GUI for building Vega-Lite specs**, with
> **guidance and recommendations on the fly**, **moderately capable** — not a full
> visual-design IDE.
>
> **Decision (2026-06-10):** push the builder from its shipped **Tier B** ("smart +
> guarded") up to **Tier C** ("intent-first aid"). "Moderately capable" is the ceiling:
> we add the controls that are _both common and awkward in JSON_ and stop there; the long
> tail of styling/scale/axis breadth stays in Monaco. The two source docs remain the
> research record (the _why_ and the citations); this doc is the _plan_ (the _what next_
> and the _order_).
---
## Status log
Newest first. The at-a-glance build-order tracker is §4; per-item detail is §3. This log is
the quick "where are we" — read it first.
- **2026-06-13 (guidance: reason over role, not raw type)** — closed a **false-positive class**
in `builderWarnings` (eng-council + council consult). A histogram (bar, binned-Q X, count Y)
tripped "two measures → scatter" because `isMeasureMapping`/`effectiveType` ignored `bin`.
Fix: `isMeasureMapping` is now **bin-aware** (a binned field is a discretized dimension —
mirrors Vega-Lite's own `isDiscrete(fieldDef)`); a new `isReorderableCategory` keeps Sort
from being offered on a histogram's binned axis (the council-flagged regression); the scatter
rule routes through the role predicate + a positive mark list (deletes the `mark !== 'rect'`
bolt-on); `stackMeasureChannel` excludes binned. Pinned with regression tests (no
scatter/Sort on a histogram; real bar still sorts; two raw measures still nudge). Promoted to
**/alignment check #15** ("reason over role, not raw type"). **Not done (deliberately):**
intent-gating the taste warnings — verified no current intent layout trips one, so the gate
would be dead code; recorded as a standing principle in `builderWarnings` instead. The
broader posture (curate good via the front door > enumerate bad via warnings) is the council's
recommended bottom for the combinatorial-warning worry. Verified: typecheck + test (913) +
eslint + build.
- **2026-06-13 (3A + heatmap mark)** — **the intent-first front door shipped (Tier C), and
the mark set gained `rect` (Heatmap).**
- **Heatmap mark** — `rect` added to `MARK_TYPES` (six marks; labelled **Heatmap** in the
picker, which now wraps to two rows in the 320–360px pane). Guidance rewired: both-axes
rule covers heatmaps; a new hint nudges a Colour measure on a two-axis heatmap with a
one-click **Colour by count**; `rect` is exempt from the two-measures→scatter nudge (a
binned 2-D histogram is valid). Never the auto-default mark; names read "Heatmap of …".
Spec §06 mark-type + guidance updated.
- **3A intent front door** — a **persistent strip** at the top of the config pane (the
interaction model chosen with the user over a replace-on-open screen — "shows all
controls + a do-it-for-me", the Tableau _Show Me_ parallel). Core (`chart-builder.ts`):
`CHART_INTENTS` (Compare/Ranking/Change-over-time/Correlation/Distribution/Part-to-whole/
Heatmap), `intentLayout` (intent × column-roles → mark + channels), `intentApplicable`
(Show-Me gating), `applyIntent` (reshape, keep dataset/transforms/title), and
`activeIntent` (structural match → the live chart's intent highlights with **no stored
state**; lights the smart default on open). Store: `setIntent`. UI: an APG **toolbar** of
toggle chips (roving tabindex, arrows move / Enter applies, accent-ring selection,
`aria-disabled`+reason on inapplicable intents).
- **Council** run on the front-door copy/flow → toolbar (not radiogroup — select-on-arrow
would reshape the chart), disabled-focusable-with-reason, intent-framing (Heatmap kept as
the one chart-type label, mirroring the mark). Recorded in `architecture/10` §5; spec §06
gained an **Intent (the front door)** subsection.
- **Verified:** `typecheck` + `test` (909, +17: intent core/store, heatmap guidance/naming)
- `eslint` + `build`. **Owed:** a visual pass on the live strip (chips, disabled states,
keyboard) + a heatmap rendered from real data. **Open (user's call):** whether the
_Heatmap_ chip should read as an intent phrase instead (council's intent-framing point).
- **2026-06-12 (3D)** — **entry points & discoverability shipped. Up next: 3A (the
front door now has somewhere to be found).**
- **Library creation surface forked:** primary **Build Chart** (accent, takes the
slack) + ghost **New JSON snippet** (the old instant create) — two plain buttons,
no menu widget; labels collapse in two stages as the pane narrows (the long ghost
label first). New `chart` icon (rising columns, pane-icon rect style).
- **The builder opens un-targeted:** `init(null)` picks the most recently modified
dataset; a **Dataset picker** heads the config pane (replaces the "Building from"
line). Switching with an untouched config re-derives smart defaults; a built-on
config is **rebased** via the new core `rebaseBuilderConfig` (mark/title/size/
sort/stack/calculates/expression-filters kept; encodings + predicate filters bound
to columns the new dataset lacks shed; same-schema switch keeps everything).
- **No-datasets state** (Carbon no-data): what the builder does + primary "Add a
dataset" → Datasets create form. A defensive "Choose a dataset" chooser covers the
datasets-exist-but-none-loaded case. New `#build` hash serializes the
no-dataset builder (spec §01E table).
- **Onboarding gains the data-first door** ("Build a chart from your data", ghost
beside the primary; applies the workspace split up front since the builder can
create the first snippet). Dataset-row "Build Chart" unchanged (the contextual
shortcut).
- **Contract updated:** spec §02 (creation surface, onboarding), §06 (Opening
rewritten: doors, dataset picker, no-datasets state), §01E (`#build`); council
resolutions recorded in `architecture/10` §5 ("builder-forward, Monaco-intact").
- **Verified:** `typecheck` + `test` (873, +10: core rebase, store init(null)/
switchDataset, url-hash `#build`, modal empty states) + `eslint` + prettier.
**Owed:** a visual pass on the forked creation surface + builder picker/empty
states (user-driven, batched with the 2B visual debt).
- **2026-06-12 (polish batch)** — **the 2B visual-pass findings fixed; the parked council
batch resolved and applied; two capability gaps closed.**
- **Aggregation for any field type.** `validAggregateOps` replaces the quantitative-only
gate: **Count distinct** on anything (a nominal Colour/Y now measures unique values —
emitted as a quantitative `{ aggregate: 'distinct' }`), Min/Max on temporal/ordinal,
arithmetic still quantitative-only. Retyping keeps a still-valid aggregate.
- **Title/subtitle** (pulled forward from the placement discussion): top-level `title`
emission; a user title becomes the snippet name; subtitle gated on a title.
- **Council batch applied:** the type chip and every native `<select>` in the builder are
now **`SelectControl`** — a reusable value-picker disclosure (the SortControl primitive
generalized; replaces native selects app-wide where the control is part of a designed
surface). Field chips open an explicit **channel chooser** (occupied channels say what
they'd replace); an **armed** channel short-circuits it and is now visible (accent ring +
status line + Esc disarms). "or constant" → **"Use a constant"** ghost button. Chart
properties (Title · Subtitle · W · H) moved to a **strip under the preview**; Sort/Stack
stay by the encodings. Resolutions recorded in `architecture/10` §5; `ux-second-pass.md`
cleared (drag stays deferred).
- **Visual-pass fixes:** pill ✕ flush right (pills hug content), Swap X/Y beside the Axes
heading, field shelf scrolls (280px viewport, sticky group heads), per-channel transforms
inline beside the pill, preview honors explicit width/height (fit-mode only while auto).
- **Verified:** `typecheck` + `test` (774, +10) + `eslint` + `build`. **Owed:** a fresh
visual look at the reworked surfaces (user-driven).
- **2026-06-12 (scope)** — **3D added to Phase 3: builder entry points & discoverability**
(council-reviewed: NN/g #6 recognition + #7 expert accelerators, Carbon empty-states).
Today the builder's only door is Datasets → row "Build Chart" while the primary "Create
New Snippet" lands in blank Monaco — the data model's shape, not the user's intent.
Decided: fork the library's creation surface (primary **Build Chart**, ghost **New JSON
snippet**), give the builder an internal dataset picker + a no-datasets empty state, add
an onboarding tile. 3D lands **with or just before 3A** — the Build-Chart button opens
onto the intent front door, making 3A the app's guided creation flow. Detail in §3 · 3D.
- **2026-06-11 (scope)** — **3C added to Phase 3: open-in-builder (strict spec hydration).**
A snippet-list button, enabled exactly when the spec round-trips losslessly through the
builder dialect (checked by re-assemble + deep-compare, not feature enumeration). Strict
only — no lossy/residue modes (§5's round-trip trap stays closed). Detail in §3 · 3C.
- **2026-06-11 (Phase 2)** — **2A + 2B shipped: the field-first interaction substrate.** Up next: **3A intent-first front door**.
- **2A · Value-or-field channels (the Property model).** `ChannelMapping` gained a constant
`value` arm; the assembler emits `{ value }` and the measure/stack/area-split/prune logic
all treat a constant Colour as "no series." New pure helpers (`isValueMapping`,
`channelAcceptsValue`, `defaultChannelValue`, `coerceChannelValue`,
`isColumnAllowedOnChannel`). `type` stays required (a constant carries a preserved-but-
ignored type, so a field↔constant toggle round-trips). Tested.
- **2B · Field-first builder.** The channel-first dropdown rows are replaced by a **field
shelf** (columns as type-glyphed chips, auto Dimensions/Measures split past a column
threshold), **click-to-assign** (armed channel, else first empty that accepts it —
`assignField`/`focusChannel`), Tableau-style **pills** (a type chip that **cycles** the
field type in-place, transforms beneath), and the **on-chart Columns/Rows shelves** above
the preview (X/Y as a property of the chart) with a **reserved faceting placeholder** in
each. Colour/Size live in a left **Marks card**, each switchable to a **constant** (2A).
- **Decisions taken (overridable):** drag deferred (click/keyboard-first, fully tested);
field grouping auto; aggregate stays a control under the pill (no pill context-menu yet).
- **Spec §06** rewritten for the field-first model (Layout, Encoding channels, faceting
placeholder, constants, validation/output).
- **Verified:** `typecheck` + `test` (762, +20 core/store/modal) + `eslint` + `build`.
**Owed:** a manual/visual pass against the live builder; a **`/council`** look at the
type-cycle chip (a cycling button gives no direct type pick — explicit menu vs cycle is a
real a11y trade-off) and the constant affordance.
- **2026-06-11 (later)** — **Owed debts on the Data section closed. Up next: Phase 2 (2A value-or-field channels, then 2B field shelf).**
- **Council pass** on the new error/disclosure copy (the previously-deferred auto-fire
surface). Three a11y conformance gaps against `architecture/10` were fixed: the inline
expression feedback now carries a **status glyph** (round error / triangle warning), not
colour alone (§3, WCAG 1.4.1); the parse error is a **polite `role="status"`**, not a
per-keystroke assertive alert (APG Alert / WCAG 2.2.4); and the message is linked to its
input via **`aria-describedby`** (GOV.UK error-message). Unknown-field copy clarified to
"…— not a column in this dataset." Resolution recorded in `architecture/10` §5.
- **Manual/visual pass** run via a headless-Chrome (Playwright) walk-through of the live
builder — filter shelf (predicate + `is between`), field↔expression toggle, expression
error/unknown-field glyphs (filter _and_ calc inputs), calculated field, data-preview
table with type chips, smart-default chart. All surfaces render as intended.
- **Bug found + fixed (mid-edit preview resilience).** `filterTransformObject` /
`calculateTransformObject` emitted **any non-empty** expression — including a half-typed,
unparseable one — so the preview blanked with a raw render error while the user typed.
They now **drop a syntactically-invalid expression** like an empty/incomplete entry
(guarded by core `validateExpression`), matching `buildTransforms`' own "a config
mid-edit still renders" contract; the inline feedback still flags the typo. Pure core,
tested (+2 cases). Visually confirmed: an invalid filter/calc now keeps the last-good
chart instead of breaking the preview.
- **Verified:** `typecheck` + `test` (741 passing) + `eslint` clean.
- **Still open (copy judgment, user's call):** the preview's catch-all
`"Couldn't render this chart: {raw Vega message}"` puts a diagnostic in the headline
(arch 10 says diagnostics go in a disclosure) — correct in the editor, debatable in the
builder; and the builder's terse `"Dataset «X» not found."` drops the next-step the
contract mandates (near-unreachable in the builder). Both noted, not changed.
- **2026-06-11** — **1C + 1D + 1E shipped (the Data section).**
- A new **Data section** at the top of the builder's left pane — "here are your rows;
shape them, then encode them" — emits the spec's top-level `transform` array.
- **1C · Filters** — a guarded **field + operator + value** predicate shelf. Operators
narrow by field type (`validFilterOps`): a measure/temporal field offers ordering
(`< ≤ > ≥`) + `is between`; a category offers `is` / `is not` / `is one of`. Values
coerce by type (quantitative → number; others → string, so ISO dates sort right).
`notEqual` emits a `{ not: { …equal } }` wrapper. A reversible **expression** power-mode
takes a raw `datum.…` predicate. Incomplete filters are skipped so the preview keeps
rendering. Multiple filters AND together. Pure core (`buildTransforms`,
`validFilterOps`, `filterOpArity`), tested.
- **1C · Calculated fields** — `{ calculate, as }` derived columns. A named field appears
in the channel dropdowns via `effectiveColumns` (defaults Quantitative); emitted
**before** filters (a row-wise calculate is order-independent, so calc-first is
equivalent and lets filters reference derived fields). Removing/renaming a referenced
field clears the dangling channel (`pruneEncodings`, in the store on calc edit/remove).
- **1E · Expression validation** — new pure core `expr-validate.ts` using Vega's own
`parseExpression` (already in the `vega` chunk, so ~zero bundle cost): inline syntax
errors on both expression inputs, plus a soft **unknown-field** warning when a
`datum.<field>` reference doesn't match a column (`referencedFields` walks the AST).
Field discoverability is served by **dataset-derived placeholder examples** (e.g.
`datum.revenue * 2`); a full Monaco-style completion popup is **deferred** (a bare
`<input>` doesn't warrant it — noted, not built).
- **1D · Data preview** — a collapsible, read-only first-N-rows table with a per-column
**type chip** in each header, to sanity-check inferred types before building (reuses
core `tabularRows`). Default collapsed; the scroll region is keyboard-reachable
(`tabIndex=0` + labelled group — avoids the Datasets-manager a11y gap).
- **Spec §06** gained a "Data (filters, calculated fields, preview)" section; the Layout
and Output blocks cross-reference it.
- **Council not yet run** on the new error/disclosure copy (the soft auto-fire surface:
expression-error + unknown-field copy, the preview disclosure). Conventions were matched
to the existing warnings region (arch 10 §5) and `SettingsPopover` disclosure; flag for a
council pass on review if desired.
- **Expression reference:** a contextual link to the Vega expression-language docs is
shown when an expression input is in play (a calculated field, or a filter in
expression mode) — the place the user needs to know the available functions/operators.
- **Verified:** `typecheck` + `test` (full suite green; +103 new core/store cases, +4 modal
smoke tests) + `eslint` + `build` (PWA, 45 precache entries). **Owed:** a manual/visual
pass against the live builder (filter shelf, calc → channel, expr errors, preview
table) — tests don't cover what the surface looks/feels like.
- **Surfaced direction (now parked):** the 1D preview shows **raw source** rows; a
transform-aware **data inspector** (resolved rows, à la vega-editor, in the builder
_and_ below the main Live Preview) is the wanted evolution — documented in
**[`docs/data-inspector-exploration.md`](./data-inspector-exploration.md)**, deferred.
- **2026-06-10** — **Up next: 1C (filter + calculate transforms), paired with 1D (data preview).**
- **1B · Per-chart export** shipped: an **Export** disclosure in the Live Preview header
(distinct from the workspace Export) — **Copy spec** + **Download JSON** (`.vl.json`) of
the shown text, and **Download PNG/SVG** of the live chart. Image output rides a new
`RenderHandle.toImageURL(format, { scale, background })` (PNG via `view.toCanvas` →
`blob:` URL; SVG via `view.toSVG` → `data:` URL), so no component touches the Vega
`view`. Filenames derive from the snippet name — filesystem-safe, script-preserving
(pure `core/chart-export.ts`, tested). Home: the **preview header**, not the plan's
"library row / editor toolbar" — image export needs the live view; the
disclosure-of-controls (not an ARIA menu) mirrors `SettingsPopover`. - **Export options (from first-round feedback):** PNG **Resolution** `1×/2×/3×` is a
multiplier of `devicePixelRatio`, so the default `1×` is Retina-crisp — the soft-1×
export was a dpr bug (raw `toImageURL` scaleFactor ignores dpr). **Background**
`Theme`(default)/`White`/`None` fixes transparent PNGs (the chart config is
transparent so the on-screen pane colour shows; export composites the chosen colour
under the PNG / adds an SVG `<rect>`). **Referenced data** `Inline`(default)/`Keep
refs` (shown only when the spec references saved datasets) inlines dataset values so
the exported spec renders standalone (`inlineReferencedDatasets`, tested). - Spec §08 gained a _Per-chart export_ section (with the options); §04 cross-references
it; `architecture/05` §2 records the handle's dpr-aware scale + background compositing.
- **1A · Actionable hints** shipped: one-click fixes on guidance warnings
(`BuilderWarning.fixes` + `applyWarningFix`), council-reviewed, with focus/announce a11y.
- **Builder UX/perf batch** (from dogfooding the Superstore dataset) shipped: near-fullscreen
`xlarge` modal tier; panes scroll internally (preview in its own viewport); backdrop-dismiss
guard (`dismissOnBackdrop: false`); render-timing diagnostics; **canvas** preview renderer
(SVG stays default elsewhere) + **canvas max-dimension guard** (`ChartTooLargeError` via a
headless probe); **data-aware default pre-population** (`smartDefaultEncodings`).
- Scope consolidated and **Tier-C target** set (this doc created); the research/Lyra forward
sequences are superseded by §4 here.
---
## 1. Where the builder is today (the floor — don't rebuild)
Tier B is shipped and tested (M4 done). The intelligence that decides _which chart and
why_ already exists in `src/core/chart-builder.ts`:
- **Smart default mark** from the (X, Y) field-type shape — not unconditionally Bar
(`defaultMark`).
- **Valid-type-only** field-type menus per column + **Size discipline** (Nominal /
Temporal / negative-extent columns are _blocked_ on Size, not merely warned)
(`validFieldTypes`, `isChannelTypeAllowed`).
- **Non-blocking guidance** — `builderWarnings` (6 rules: line/area needs both axes,
all-categorical, two-measures-want-scatter, area-split-many-series, crowded category
axis, negative-size guard).
- **Per-channel transforms** — Aggregate / Bin / `timeUnit`, chart-level Sort / Stack,
and a field-less "Count of records" measure.
- **Swap X/Y**, debounced live preview, validation gate (≥1 channel mapped).
Everything in §3 below is **backlog** — verified not yet built: `BuilderWarning` has no
`fix` field, there is no per-chart export, no top-level `transform`, no data-table
preview, no channels beyond X/Y/Color/Size, no styling/scale controls.
---
## 2. The target experience (Tier C, concretely)
Reading the brief's four words against the research:
| Brief word | What it means here | The levers (from §3) |
| ----------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Rapid** | shortest path from "a dataset" to "a chart I'd keep" | smart defaults (have), intent front door, starter examples, field shelf |
| **Intuitive** | matches how people think ("I have fields; what shows my point?") | intent front door, field shelf, data preview, value-or-field channels |
| **Guidance & recommendations on the fly** | the app proposes and corrects, not just validates | **actionable hints**, **intent front door** (the defining Tier-C feature) |
| **Moderately capable** | covers the common data-shaping + a few high-value encodings; _not_ every knob | filter / calculate, a _small_ set of promoted controls — and a hard stop short of Lyra's everything-inspector |
The defining shift from B→C is the **intent-first front door**: a _"what do you want to
show?"_ entry (FT / Datawrapper intent categories) that recommends a mark + channel layout
from `intent × column-types`, instead of starting the user at a blank mark picker. It is
the feature the brief most directly asks for, and it is the one piece the M4 research
_deferred_. Choosing Tier C is choosing to pull it forward.
Tier C **extends** §06 — it does not replace the mark-first builder. The front door is an
on-ramp; the user can still ignore it and drive the channels directly, and can always drop
to Monaco. This preserves Astrolabe's core invariant: **the JSON spec is the source of
truth; the builder is a view that emits it** (the Lyra anti-lesson — never let the GUI
become the document).
---
## 3. The consolidated enhancement set
Organized into build phases by dependency and value. Each item carries its **source**,
**value/effort**, **code home**, and **spec impact**. Phases 1–3 are the committed Tier-C
scope; Phase 4 is explicitly _beyond_ "moderately capable" and gated on a later decision.
### Phase 1 — Guidance + I/O (no new interaction model) — _do first_
High value-to-effort, mostly pure-core + thin UI, no architectural change. These make the
_current_ builder dramatically better and de-risk the bigger phases.
**1A · Actionable hints** — _done (2026-06-10)_
Turn advisory warnings into one-click fixes. Source: Lyra §3.1 (`Hints` carries an
`action`). Several existing warnings have an obvious remedy:
- _"draws one mark per row → "_ **[Aggregate as Sum]** (set the measure's `aggregate`)
- _"long labels → "_ **[Swap X/Y]** (the action already exists — just wire it)
- _"two measures usually read as a scatter → "_ **[Switch to Point]**
- _"area split into many series → "_ **[Stack]** or **[Remove colour]**
Implementation: extended `BuilderWarning` with `fixes?: BuilderWarningFix[]` (`{ label, apply }`, pure,
unit-tested in `chart-builder.test.ts`); the modal renders each as a ghost button wired to a
new `applyWarningFix` store action. Wired fixes: **[Aggregate as Sum]** + **[Swap X/Y]**
(one-mark-per-row), **[Swap X/Y]** (high-cardinality axis), **[Switch to Point]** (two
measures), **[Stack]** + **[Remove colour]** (area split — and a stacked area is no longer
flagged, so [Stack] resolves it). Council run (Carbon Actionable notification + APG Alert):
ghost buttons, remedy-in-button, polite "Applied: …" announcement, focus moved off the
removed button — resolution recorded in `architecture/10` §5. §06 "Guidance" amended to
document the one-click fixes.
**1B · Per-chart export** — _done (2026-06-10)_
Today export was workspace-backup only (§08); there was **no way to get one chart out**.
Source: Lyra §3.8. Shipped as an **Export disclosure in the Live Preview header**:
- **Copy spec** (clipboard) + **Download `.vl.json`** of the currently-shown text.
- **Download PNG / SVG** of the live chart via a new `RenderHandle.toImageURL` wrapping
`view.toImageURL` (PNG at 2× → `blob:` URL, revoked after download; SVG → `data:` URL),
so the embedding boundary holds — no component touches the raw view.
- Filenames from the snippet name, filesystem-safe and script-preserving (pure
`core/chart-export.ts`, tested). Standalone HTML left out (the deferred optional).
Home decision: the **preview header**, not the plan's original "library row / editor
toolbar" suggestion — the image formats need the live rendered view, and "export this
chart" reads best beside the chart. The widget is a disclosure-of-action-buttons (not an
ARIA menu), mirroring `SettingsPopover` and sharing its single-open registry. _Spec impact:
new §08 "Per-chart export" section; §04 cross-reference; `architecture/05` §2 handle note._
**1C · Filter (+ Calculate) dataset transforms** — _done (2026-06-11; see status log)_
The transform layer the builder doesn't touch: top-level `transform: []`. Source: Lyra
§3.12. The builder _already tells users to filter_ in three warnings
(`chart-builder.ts:475,476,511`) while offering no way to do it.
- **Filter** _(highest)_ — a guarded **field + operator + value** predicate shelf
(Voyager-style, no expression needed for the common case); power form is a raw `datum.…`
expression validated with Vega's `parseExpr` (see 1E). VL applies top-level transforms
_before_ encoding aggregation, so "filter raw rows, then aggregate" is the natural
default; filtering on an aggregated value (HAVING) is the advanced case — defer.
- **Calculate / derived field** _(second)_ — `transform: [{calculate, as}]`; the new field
then appears in column dropdowns like any other.
- **Lookup / join a second dataset** — larger data-model change (one dataset per snippet
today) → **defer to Phase 4**.
Home: a new **"Data" section in the builder's left pane, above the channels** (`here are
your rows [+ Filter] [+ Calculate] → now encode them`), paired with 1D. _Spec impact: new
§06 "Data / transforms" subsection — this is genuinely new behaviour, write it._
**1D · Data-table preview** — _done (2026-06-11; see status log)_
Neither the builder nor the Datasets manager ever shows the **actual rows**. Source: Lyra
§3.2. A compact, **read-only** first-N-rows grid with a **per-column type chip** in each
header lets users sanity-check inferred types _before_ building — exactly when inference is
most likely to surprise. Type + cardinality + extent already come from `profile.ts`; we
only need the row sample. Homes: a collapsible "Data" strip in the builder's left pane
(under the dataset name) and/or the Datasets manager. Keep it read-only (editing data is
out of scope). _Spec impact: §06 + §05 (Datasets) additions._
**1E · Inline expression validation + field autocomplete** — _done (2026-06-11; completion popup deferred)_
When 1C's expression mode lands, validate the Vega/VL expression string with the library's
own `parseExpr` (Lyra §3.6) and surface errors inline; autocomplete the dataset's own
column names (we have the schema from `profile.ts`) (Lyra §3.10). Record the `parseExpr`
technique in `architecture/08`. _Spec impact: folded into 1C._
### Phase 2 — Interaction substrate (enables Tier C)
Two changes to _how the user touches fields_. They have standalone value but their main job
is to be the substrate Phase 3 (and any future added channels) stands on — sequence them
here so Tier C lands cleanly.
**2A · Value-or-field channels (the Property model)** — _capability gain; medium_
Source: Lyra §3.3 (`Property.tsx` — one droppable control that is _either_ a literal value
_or_ a bound field). Today a channel is field-only. Let a channel also hold a **constant
`value`** (fixed colour / size) with one consistent control and a chip showing the binding
kind. VL encodes exactly this (`field` vs `value` vs `datum`). Generalizes cleanly to any
channel we add later. _Spec impact: §06 "Encoding channels" — a channel may carry a
constant._
**2B · Field shelf + in-place type cycling** — _the largest interaction shift; the Tier-C/facet substrate_
Source: Lyra §3.4 (drop-zones) + Voyager (field list with type chips) + Lyra §3.5
(`FieldType` — the type icon _is_ the control, click cycles N→O→Q→T within the valid set).
Flip from **channel-first** ("pick a channel, then its column") to **field-first** ("here
are your columns — drag/click onto channels"), matching how people think. This is the
natural way to assign many fields across many channels, so it is the **interaction
substrate for Tier C and faceting**, not a standalone task. _Spec impact: §06 layout
revision (field shelf alongside the channel rows)._
### Phase 3 — Tier C (the intent-first front door) — _the defining feature of this push_
The grand idea, stated once: **the builder is the app's rapid, intuitive on-ramp; Monaco
is the expert surface; the JSON spec stays the document.** Phase 3 is where that becomes
true — 3A gives the builder an intent-first opening screen, 3D makes that screen the
app's guided creation flow (not a feature hidden in a modal), and 3B/3C seed and re-enter
it.
**3A · Intent-first front door** — _the B→C step_
Source: research §5/§8 Tier C (FT Visual Vocabulary + Datawrapper intent taxonomy). A
_"what do you want to show?"_ entry mapping **intent × column types → recommended mark +
channel layout**:
| Intent (FT / Datawrapper) | Our expression (within 5 marks / our channels) |
| -------------------------- | ------------------------------------------------------------------- |
| **Magnitude / Comparison** | Bar (x=N, y=Q; horizontal for long labels) |
| **Ranking** | Bar, sorted by value |
| **Change over time** | Line (x=T, y=Q; color=N for series) |
| **Correlation** | Point (x=Q, y=Q); Circle + size=Q for a 3rd measure |
| **Distribution** | Bar of binned counts (histogram) — uses Bin |
| **Part-to-whole** | Stacked / 100% bar (uses Stack); _true pie needs `theta` → Phase 4_ |
| **Deviation** | diverging signed Bar |
Honest coverage gaps stay honest (Spatial / Flow excluded; Part-to-whole partial until
`theta`). The front door is an **on-ramp, not a gate** — it pre-populates the mark-first
builder, which the user can then adjust or ignore. Munzner's typology and Wilke's directory
become seatable council sources at this point (research §2). Built **on the 2B field
shelf**. **Run the front-door copy + flow through `/council`.** _Spec impact: substantial
§06 amendment — a new "Intent" front-door subsection; the M4 spec note explicitly flagged
this as the deferred tier, so this is the planned amendment, not drift._
**3B · Starter examples gallery** — _cheap; pairs with 3A_
Source: Lyra §3.7. A small set of **curated starter snippets**, one per covered FT intent
(Magnitude/Bar, Change-over-time/Line, Correlation/Point, Distribution/histogram,
Part-to-whole/stacked). Improves first-run, doubles as living documentation of what the app
does well. Natural home: the snippet library. **Author them in the builder dialect so they
hydrate via 3C.** _Spec impact: §02 (library seed content)._
**3C · Open in builder (strict spec hydration)** — _decided 2026-06-11; pairs with 3B_
Reverse the assembler: a pure `parseChartSpec(spec) → BuilderConfig | null` so an existing
snippet can re-enter the builder populated. **Strict policy only** — a snippet-list button
("Open in builder") enabled exactly when the spec is **losslessly** representable in the
builder's dialect; anything richer stays Monaco-only. Losslessness is checked not by
enumerating unsupported features but by **re-assembling the parsed config and
deep-comparing against the original** (ignoring key order, `$schema`, and the injected
`tooltip: true`) — exact, ~20 lines, and it stays correct automatically as the dialect
grows (every Phase-4 addition widens what hydrates for free). No lossy or
residue-preserving mode: the moment the builder can overwrite a richer spec, the GUI starts
competing with the JSON as the document (the Lyra round-trip trap — §5).
Value: converts the builder from **create-only to create-and-revise** for its own output,
and makes the 3B starters openable as builder seeds. Known fiddly bits (all mechanical):
inverse of `escapeVegaField`, filter-value un-coercion back to the `BuilderFilter` string
shape, `sort: "-y"` → `{sort: 'descending'}`. Round-trip property tests
(`parse(build(config))` ≈ `config`) are the natural core coverage. Per-row check in the
library is cheap but memoize it. **Decisions at build time:** save semantics on an opened
snippet (update-in-place via a builder "edit" mode vs. always-create-new — today the
builder only creates) and the disabled-button affordance (disabled-with-reason vs. hidden —
park in `ux-second-pass.md` if non-obvious). _Spec impact: §02 (library row action) + §06
(hydration + the strict gate)._
**3D · Entry points & discoverability** — _done 2026-06-12 (council-reviewed; see status log); 3A opens onto it next_
Today the builder has **one entry, two levels deep**: Datasets modal → per-dataset "Build
Chart" (`DatasetsModal.tsx`) — the data model's shape ("charts come from datasets"), not
the user's intent ("I want to make a chart"). Meanwhile the library's pinned primary
action, "Create New Snippet", drops straight into blank Monaco — the **expert** path is
the default, and the audience the builder serves may never find it. Council ruling
(recorded here; contract updates at build time):
- **NN/g #6 (recognition over recall):** the build-a-chart action must be visible where
the intent forms — the library's creation surface, not recalled as a dataset-row action.
- **NN/g #7 (flexibility/efficiency):** the guided path is what novices see; raw JSON is
the expert accelerator — visible, one click, never hidden. Builder-forward, Monaco
intact.
- **Carbon empty-states:** one primary action per surface ("pick the most important"; a
tertiary button for the secondary CTA); a no-data state explains what the space will
hold, offers one primary next step, and never dead-ends.
The decided shape:
1. **Fork the library's creation surface** — primary **Build Chart** (opens the builder),
ghost/tertiary **New JSON snippet** (today's instant create, unchanged). Two plain
buttons with clear hierarchy — no split/menu-button widget (two static choices don't
earn an ARIA menu).
2. **Dataset picker inside the builder** (header select, default = most recently
modified), so the builder opens without a preselected dataset. Build-time decision
taken: switching with an **untouched** opening config re-derives fresh smart defaults;
switching a **built-on** config rebases it (`rebaseBuilderConfig` — chart-level intent
kept, bindings to missing columns shed; a same-schema dataset keeps everything).
Useful beyond 3D (switch data without leaving the builder).
3. **No-datasets empty state** in the builder, per the Carbon no-data pattern: what the
builder does + one primary action ("Add a dataset" → Datasets modal). No dead end.
4. **Onboarding tile** — a data-first path ("import your data → build a chart") beside the
existing examples gallery.
5. **Keep the dataset-row "Build Chart"** — contextual shortcut, pre-picks the dataset; it
just stops being the only door.
**Composition with 3A (the point of the sequencing):** the Build-Chart button lands on the
intent front door — "what do you want to show, with which data?" _is_ the builder's
opening screen. 3D without 3A opens onto the mark-first builder (fine, interim); 3A
without 3D is a front door nobody finds. Discoverability is a prerequisite for 3A's value,
so: **3D with or just before 3A.** _Spec impact: §02 (library creation actions +
onboarding) + §06 (dataset picker, empty state)._
### Builder UX & perf — in-flight fixes (2026-06-10, from dogfooding the Superstore dataset)
Pre-existing builder rough edges surfaced while testing on a 10k-row / ~24-col dataset.
Fixed in this batch (not part of 1A–3B, but the same surface):
- **Modal is a near-fullscreen work surface** — new `xlarge` shell tier (`ModalShell`,
`min(1800px, 96vw) × min(1100px, 92vh)`); the Chart Builder no longer wastes screen.
- **Panes scroll internally, modal keeps its shape** — the `.builder` grid fills the body
(`grid-template-rows: minmax(0,1fr)`), the config pane and the **preview** each scroll in
their own viewport, so a tall one-mark-per-row chart scrolls inside the preview instead of
pushing Create/Cancel below the fold.
- **Backdrop click no longer discards in-progress work** — `dismissOnBackdrop: false` on the
builder (registry flag); Escape and × still close.
- **Data-aware default pre-population** — the builder no longer blindly takes the first two
columns (which opened Superstore on a 9994-bar degenerate chart). When the dataset is
profiled, `defaultBuilderConfig` picks a "safest bet": a low-cardinality category vs a
count of records (tidy bar), else a time series of the first measure, else a scatter — each
guaranteed to render. Falls back to positional when there are no stats. Pure + tested. This
composes with (isn't replaced by) the future intent-first front door — the builder always
needs a sane opening state.
- **Render-timing diagnostics** — `BuilderPreview` logs `parse · prepare · destroy · embed ·
paint · total` (+ mark, row count) to the console (dev always; prod only when slow). The
**paint** phase (a double-rAF after `embed()`) captures the real freeze.
**Perf finding — confirmed and fixed.** Diagnostics on the Superstore dataset:
`embed 237ms · paint 6458ms` for the default one-bar-per-row chart (9994 rows), vs
`paint 6ms` once grouped to a few categories. The freeze was entirely **SVG layout/paint**
(one DOM node per mark), not chart compilation. **Fix shipped:** the builder preview now
renders with **canvas** (`renderSpec(…, { renderer: 'canvas' })`); SVG stays the default for
the editor's LivePreview and for image export. Contract divergence recorded in
`architecture/05` §2.
**Canvas max-dimension guard (measured, not guessed).** Canvas (unlike SVG) has a hard
max side (~32k px), so a chart that resolves taller than that fails to allocate (the
broken-image icon). The cause is **physical render size, not cardinality** — a vertical bar
with thousands of _X_ bands renders fine (width is container-bounded); only an unbounded
band axis (e.g. a horizontal bar's _Y_) overflows. So `renderSpec` now runs a **headless
(`'none'`) layout probe** for canvas charts, reads the chart's resolved **height**, and
throws `ChartTooLargeError(heightPx, limitPx)` when it exceeds `MAX_CANVAS_PX ÷ dpr`. The
builder catches it and shows the real numbers ("would be ~200,000px tall — larger than the
browser can draw on a canvas (~16,383px max here); aggregate or filter"). The earlier
band-count proxy (`previewBandCount`/`MAX_PREVIEW_BANDS`) was removed — readability
(cardinality) stays a `builderWarnings` concern; the render-size limit is now measured at
its true cause.
### Phase 4 — Beyond "moderately capable" (gated — decide later)
These exceed the stated ceiling. List them so they have a home, but **do not commit them in
this push** — revisit once Phases 1–3 land and we see real usage. Each must clear the
guardrail: _promote a control only when it is **both common AND awkward in JSON**._
- **More channels** — `theta` (unlocks pie/donut → _true_ part-to-whole), `opacity`,
`shape`. (`theta` is the most defensible — it closes a real coverage gap.) Ride on 2A/2B.
- **Faceting (Row / Column → small multiples)** — research §8 B8. The clean way to compare
many categories. Design crux: VL facets default to **shared scales** (keep that default);
expose an "independent axes" toggle (`resolve.scale`) only as advanced. **Verify against
the preview's `"container"` fit modes** (per-cell sizing on facets is finicky).
- **Light styling / scale controls** — colour-scheme picker (categorical / sequential /
diverging), measure-axis `zero` / `log` toggle, custom axis title, legend title / hide.
Implement as **auto-derived override panels** that start empty (inheriting VL defaults)
and emit JSON **only when touched**; clearing a channel **drops its overrides** (Lyra
§3.9 `cleanupUnused` — no orphaned `scale`/`axis` in the output spec).
- **Builder undo/redo / "reset to smart defaults"** — Lyra §3.11. Low priority (the modal
is short-lived; the smart default already gives a sane start).
- **Lookup / join a second dataset** — Lyra §3.12; data-model change (multi-dataset
snippets). Larger, separate effort.
- **Transform-aware data inspector** — evolve 1D from raw-source rows to the _resolved,
post-transform_ data (filtered rows + calculated columns), and generalize it to a
togglable data panel below the **main** Live Preview (vega-editor's "Data Viewer", a
debugging aid for any snippet, not only builder output). Reads runtime rows via
`view.data(name)` through the `RenderHandle`. Cross-cutting (editor + builder), so it
has its own home: **[`docs/data-inspector-exploration.md`](./data-inspector-exploration.md)**.
---
## 4. Recommended build order
```
Phase 1 1A actionable hints ✓ done
1B per-chart export ✓ done
1C filter (+ calculate) ✓ done
1D data preview ✓ done
1E expr-validate ✓ done (syntax + unknown-field; completion popup deferred)
Phase 2 2A value-or-field channels (Property model) ✓ done
2B field shelf + in-place type cycling ✓ done (field-first + on-chart shelves)
Phase 3 3D entry points & discoverability ✓ done (2026-06-12)
3A intent-first front door (Tier C) ✓ done (2026-06-13); persistent strip
3B starter examples ← next; pairs with 3C
3C open in builder (strict hydration) ← added 2026-06-11; pairs with 3B
Marks +rect (Heatmap) ✓ done (2026-06-13)
Phase 4 (gated) theta/facets/styling-overrides/undo/lookup — decide after Phase 3
Also shipped (builder UX/perf, from dogfooding): near-fullscreen modal, internal-scroll
panes, backdrop-dismiss guard, render diagnostics, canvas preview + canvas-size guard,
data-aware default pre-population. See "Builder UX & perf" above.
```
Rationale for the order: Phase 1 is the cheapest large quality jump and needs no new
interaction model, so it ships value while the bigger design settles. Phase 2 is pure
substrate — low _user-visible_ payoff alone, but Phase 3 is much cleaner on top of it than
bolted onto the channel-first UI. Phase 3 delivers the brief's headline ("recommendations
on the fly") — 3D first, because a front door nobody finds delivers nothing: the entry
points make the builder the app's guided creation flow, then 3A gives that flow its
intent-first opening. Phase 4 is deliberately deferred to protect the "moderately capable"
ceiling.
---
## 5. Constraints & anti-scope (the ceiling)
What "moderately capable" rules **out** — load-bearing, from `lyra-review.md` §2.1/§4:
- **No everything-inspector.** Lyra surfaces ~50 direct controls per primitive because the
GUI _was_ its document. We split the work on purpose: a small **guarded** builder + a
first-class **Monaco** editor for the long tail. Styling/scale/axis breadth for its own
sake belongs in Monaco, not the builder.
- **The promotion test:** a control enters the builder only when it is **both common AND
awkward in JSON**. Otherwise it stays in Monaco.
- **JSON stays the source of truth.** The builder _emits_ spec; it is never the document
(the Lyra one-way-export trap that forbids round-trips).
- **No interaction-by-demonstration, no direct-manipulation canvas, no general
data-pipeline editor.** If we ever add interactivity, expose VL `params`/selections as a
small guarded action — never port Lyra's signal generator.
- **Take ideas from the reference clones, read no code into the repo** (`AGENTS.md`: no
shared lib; patterns adapted, not imported).
---
## 6. Cross-cutting notes
- **Spec deltas:** 1A (minor §06), 1B (§08 + §02/§03), 1C (new §06 transforms subsection),
1D (§06 + §05), 2A/2B (§06 layout), 3A (substantial §06 intent front-door amendment), 3B
(§02). Per _"spec follows code now"_: build the decision, then amend the spec to match —
don't let code and §06 drift.
- **Council:** auto-fires on guidance copy and new interactive-widget keyboard/focus work —
so 1A (hint affordance + copy) and 3A (front-door flow + copy) both go through `/council`
before committing. It advises; architecture 09/10 decide.
- **Verification:** pure rules get `chart-builder.test.ts` cases; every UI/affordance change
gets a manual pass against the _live_ builder — a green build proves nothing about what
the user sees (`AGENTS.md`; `docs/manual-verification.md`). Items 1A and 1C are mostly
`src/core/`, squarely the "core-first, tested hardest" rule.
- **Source of truth going forward:** this doc. `chart-builder-research.md` §8 and
`lyra-review.md` §5 remain the _research record_; their forward sequences are superseded
by §4 here.