oleh/astrolabe
1
0
Fork 0
mirror of https://github.com/olehomelchenko/astrolabe.git synced 2026-08-08 02:02:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Docs: move point-in-time records to exploration/, trim build-narration, lift open items to plan

Browse Source
This commit is contained in:
Oleh Omelchenko
2026-06-14 23:58:06 +03:00
parent de9dbf3ec8
commit 66c123e15e
16 changed files with 77 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/exploration/README.md
+18
View File
@@ -0,0 +1,18 @@
# docs/exploration
Point-in-time records: research, tool reviews, and scope/planning memos written to think a
decision through. Unlike `docs/spec/` (the contract — the _what_) and `docs/architecture/`
(the maintained _how_), nothing here is kept current — each file reflects what was known
when it was written. Decisions that survived are folded into the spec and architecture;
these remain as the reasoning behind them.
Active follow-up work is tracked in [`../IMPLEMENTATION-PLAN.md`](../IMPLEMENTATION-PLAN.md),
not here. Exact figures in these files (timings, line counts, test counts) are frozen
snapshots, not live numbers.
- `chart-builder-research.md` — cross-source chart-choice research (Draco, Voyager, FT Visual Vocabulary, Datawrapper).
- `lyra-review.md` — review of vega/lyra for Chart Builder interaction ideas.
- `chart-builder-enhancement-scope.md` — consolidated Tier-B → Tier-C forward plan for the Chart Builder.
- `chart-theming-scope.md` — chart theming plan and slice breakdown.
- `monetization-and-sync-exploration.md` — monetization + BYO-cloud-sync direction memo.
- `data-inspector-exploration.md` — transform-aware data-inspector direction.
docs/exploration/chart-builder-enhancement-scope.md
+660
View File
@@ -0,0 +1,660 @@
# Chart Builder — Enhancement Scope
> **Status:** scope consolidated 2026-06-10. This is the **single forward-looking home**
> for chart-builder enhancement work — it merges the research backlog from
> [`chart-builder-research.md`](./chart-builder-research.md) §8 (the M4 decision and its
> deferred items) with the interaction ideas from [`lyra-review.md`](./lyra-review.md) §5,
> read against the current spec ([`spec/06-chart-builder.md`](../spec/06-chart-builder.md))
> and the shipped code (`src/core/chart-builder.ts`).
>
> **Goal (the brief):** a **rapid, intuitive GUI for building Vega-Lite specs**, with
> **guidance and recommendations on the fly**, **moderately capable** — not a full
> visual-design IDE.
>
> **Decision (2026-06-10):** push the builder from its shipped **Tier B** ("smart +
> guarded") up to **Tier C** ("intent-first aid"). "Moderately capable" is the ceiling:
> we add the controls that are _both common and awkward in JSON_ and stop there; the long
> tail of styling/scale/axis breadth stays in Monaco. The two source docs remain the
> research record (the _why_ and the citations); this doc is the _plan_ (the _what next_
> and the _order_).
---
## Status log
Newest first. The at-a-glance build-order tracker is §4; per-item detail is §3. This log is
the quick "where are we" — read it first.
- **2026-06-13 (guidance: reason over role, not raw type)** — closed a **false-positive class**
in `builderWarnings` (eng-council + council consult). A histogram (bar, binned-Q X, count Y)
tripped "two measures → scatter" because `isMeasureMapping`/`effectiveType` ignored `bin`.
Fix: `isMeasureMapping` is now **bin-aware** (a binned field is a discretized dimension —
mirrors Vega-Lite's own `isDiscrete(fieldDef)`); a new `isReorderableCategory` keeps Sort
from being offered on a histogram's binned axis (the council-flagged regression); the scatter
rule routes through the role predicate + a positive mark list (deletes the `mark !== 'rect'`
bolt-on); `stackMeasureChannel` excludes binned. Pinned with regression tests (no
scatter/Sort on a histogram; real bar still sorts; two raw measures still nudge). Promoted to
**/alignment check #15** ("reason over role, not raw type"). **Not done (deliberately):**
intent-gating the taste warnings — verified no current intent layout trips one, so the gate
would be dead code; recorded as a standing principle in `builderWarnings` instead. The
broader posture (curate good via the front door > enumerate bad via warnings) is the council's
recommended bottom for the combinatorial-warning worry. Verified: typecheck + test (913) +
eslint + build.
- **2026-06-13 (3A + heatmap mark)** — **the intent-first front door shipped (Tier C), and
the mark set gained `rect` (Heatmap).**
- **Heatmap mark** — `rect` added to `MARK_TYPES` (six marks; labelled **Heatmap** in the
picker, which now wraps to two rows in the 320–360px pane). Guidance rewired: both-axes
rule covers heatmaps; a new hint nudges a Colour measure on a two-axis heatmap with a
one-click **Colour by count**; `rect` is exempt from the two-measures→scatter nudge (a
binned 2-D histogram is valid). Never the auto-default mark; names read "Heatmap of …".
Spec §06 mark-type + guidance updated.
- **3A intent front door** — a **persistent strip** at the top of the config pane (the
interaction model chosen with the user over a replace-on-open screen — "shows all
controls + a do-it-for-me", the Tableau _Show Me_ parallel). Core (`chart-builder.ts`):
`CHART_INTENTS` (Compare/Ranking/Change-over-time/Correlation/Distribution/Part-to-whole/
Heatmap), `intentLayout` (intent × column-roles → mark + channels), `intentApplicable`
(Show-Me gating), `applyIntent` (reshape, keep dataset/transforms/title), and
`activeIntent` (structural match → the live chart's intent highlights with **no stored
state**; lights the smart default on open). Store: `setIntent`. UI: an APG **toolbar** of
toggle chips (roving tabindex, arrows move / Enter applies, accent-ring selection,
`aria-disabled`+reason on inapplicable intents).
- **Council** run on the front-door copy/flow → toolbar (not radiogroup — select-on-arrow
would reshape the chart), disabled-focusable-with-reason, intent-framing (Heatmap kept as
the one chart-type label, mirroring the mark). Recorded in `architecture/10` §5; spec §06
gained an **Intent (the front door)** subsection.
- **Verified:** `typecheck` + `test` (909, +17: intent core/store, heatmap guidance/naming)
- `eslint` + `build`. **Owed:** a visual pass on the live strip (chips, disabled states,
keyboard) + a heatmap rendered from real data. **Open (user's call):** whether the
_Heatmap_ chip should read as an intent phrase instead (council's intent-framing point).
- **2026-06-12 (3D)** — **entry points & discoverability shipped. Up next: 3A (the
front door now has somewhere to be found).**
- **Library creation surface forked:** primary **Build Chart** (accent, takes the
slack) + ghost **New JSON snippet** (the old instant create) — two plain buttons,
no menu widget; labels collapse in two stages as the pane narrows (the long ghost
label first). New `chart` icon (rising columns, pane-icon rect style).
- **The builder opens un-targeted:** `init(null)` picks the most recently modified
dataset; a **Dataset picker** heads the config pane (replaces the "Building from"
line). Switching with an untouched config re-derives smart defaults; a built-on
config is **rebased** via the new core `rebaseBuilderConfig` (mark/title/size/
sort/stack/calculates/expression-filters kept; encodings + predicate filters bound
to columns the new dataset lacks shed; same-schema switch keeps everything).
- **No-datasets state** (Carbon no-data): what the builder does + primary "Add a
dataset" → Datasets create form. A defensive "Choose a dataset" chooser covers the
datasets-exist-but-none-loaded case. New `#build` hash serializes the
no-dataset builder (spec §01E table).
- **Onboarding gains the data-first door** ("Build a chart from your data", ghost
beside the primary; applies the workspace split up front since the builder can
create the first snippet). Dataset-row "Build Chart" unchanged (the contextual
shortcut).
- **Contract updated:** spec §02 (creation surface, onboarding), §06 (Opening
rewritten: doors, dataset picker, no-datasets state), §01E (`#build`); council
resolutions recorded in `architecture/10` §5 ("builder-forward, Monaco-intact").
- **Verified:** `typecheck` + `test` (873, +10: core rebase, store init(null)/
switchDataset, url-hash `#build`, modal empty states) + `eslint` + prettier.
**Owed:** a visual pass on the forked creation surface + builder picker/empty
states (user-driven, batched with the 2B visual debt).
- **2026-06-12 (polish batch)** — **the 2B visual-pass findings fixed; the parked council
batch resolved and applied; two capability gaps closed.**
- **Aggregation for any field type.** `validAggregateOps` replaces the quantitative-only
gate: **Count distinct** on anything (a nominal Colour/Y now measures unique values —
emitted as a quantitative `{ aggregate: 'distinct' }`), Min/Max on temporal/ordinal,
arithmetic still quantitative-only. Retyping keeps a still-valid aggregate.
- **Title/subtitle** (pulled forward from the placement discussion): top-level `title`
emission; a user title becomes the snippet name; subtitle gated on a title.
- **Council batch applied:** the type chip and every native `<select>` in the builder are
now **`SelectControl`** — a reusable value-picker disclosure (the SortControl primitive
generalized; replaces native selects app-wide where the control is part of a designed
surface). Field chips open an explicit **channel chooser** (occupied channels say what
they'd replace); an **armed** channel short-circuits it and is now visible (accent ring +
status line + Esc disarms). "or constant" → **"Use a constant"** ghost button. Chart
properties (Title · Subtitle · W · H) moved to a **strip under the preview**; Sort/Stack
stay by the encodings. Resolutions recorded in `architecture/10` §5; `ux-second-pass.md`
cleared (drag stays deferred).
- **Visual-pass fixes:** pill ✕ flush right (pills hug content), Swap X/Y beside the Axes
heading, field shelf scrolls (280px viewport, sticky group heads), per-channel transforms
inline beside the pill, preview honors explicit width/height (fit-mode only while auto).
- **Verified:** `typecheck` + `test` (774, +10) + `eslint` + `build`. **Owed:** a fresh
visual look at the reworked surfaces (user-driven).
- **2026-06-12 (scope)** — **3D added to Phase 3: builder entry points & discoverability**
(council-reviewed: NN/g #6 recognition + #7 expert accelerators, Carbon empty-states).
Today the builder's only door is Datasets → row "Build Chart" while the primary "Create
New Snippet" lands in blank Monaco — the data model's shape, not the user's intent.
Decided: fork the library's creation surface (primary **Build Chart**, ghost **New JSON
snippet**), give the builder an internal dataset picker + a no-datasets empty state, add
an onboarding tile. 3D lands **with or just before 3A** — the Build-Chart button opens
onto the intent front door, making 3A the app's guided creation flow. Detail in §3 · 3D.
- **2026-06-11 (scope)** — **3C added to Phase 3: open-in-builder (strict spec hydration).**
A snippet-list button, enabled exactly when the spec round-trips losslessly through the
builder dialect (checked by re-assemble + deep-compare, not feature enumeration). Strict
only — no lossy/residue modes (§5's round-trip trap stays closed). Detail in §3 · 3C.
- **2026-06-11 (Phase 2)** — **2A + 2B shipped: the field-first interaction substrate.** Up next: **3A intent-first front door**.
- **2A · Value-or-field channels (the Property model).** `ChannelMapping` gained a constant
`value` arm; the assembler emits `{ value }` and the measure/stack/area-split/prune logic
all treat a constant Colour as "no series." New pure helpers (`isValueMapping`,
`channelAcceptsValue`, `defaultChannelValue`, `coerceChannelValue`,
`isColumnAllowedOnChannel`). `type` stays required (a constant carries a preserved-but-
ignored type, so a field↔constant toggle round-trips). Tested.
- **2B · Field-first builder.** The channel-first dropdown rows are replaced by a **field
shelf** (columns as type-glyphed chips, auto Dimensions/Measures split past a column
threshold), **click-to-assign** (armed channel, else first empty that accepts it —
`assignField`/`focusChannel`), Tableau-style **pills** (a type chip that **cycles** the
field type in-place, transforms beneath), and the **on-chart Columns/Rows shelves** above
the preview (X/Y as a property of the chart) with a **reserved faceting placeholder** in
each. Colour/Size live in a left **Marks card**, each switchable to a **constant** (2A).
- **Decisions taken (overridable):** drag deferred (click/keyboard-first, fully tested);
field grouping auto; aggregate stays a control under the pill (no pill context-menu yet).
- **Spec §06** rewritten for the field-first model (Layout, Encoding channels, faceting
placeholder, constants, validation/output).
- **Verified:** `typecheck` + `test` (762, +20 core/store/modal) + `eslint` + `build`.
**Owed:** a manual/visual pass against the live builder; a **`/council`** look at the
type-cycle chip (a cycling button gives no direct type pick — explicit menu vs cycle is a
real a11y trade-off) and the constant affordance.
- **2026-06-11 (later)** — **Owed debts on the Data section closed. Up next: Phase 2 (2A value-or-field channels, then 2B field shelf).**
- **Council pass** on the new error/disclosure copy (the previously-deferred auto-fire
surface). Three a11y conformance gaps against `architecture/10` were fixed: the inline
expression feedback now carries a **status glyph** (round error / triangle warning), not
colour alone (§3, WCAG 1.4.1); the parse error is a **polite `role="status"`**, not a
per-keystroke assertive alert (APG Alert / WCAG 2.2.4); and the message is linked to its
input via **`aria-describedby`** (GOV.UK error-message). Unknown-field copy clarified to
"…— not a column in this dataset." Resolution recorded in `architecture/10` §5.
- **Manual/visual pass** run via a headless-Chrome (Playwright) walk-through of the live
builder — filter shelf (predicate + `is between`), field↔expression toggle, expression
error/unknown-field glyphs (filter _and_ calc inputs), calculated field, data-preview
table with type chips, smart-default chart. All surfaces render as intended.
- **Bug found + fixed (mid-edit preview resilience).** `filterTransformObject` /
`calculateTransformObject` emitted **any non-empty** expression — including a half-typed,
unparseable one — so the preview blanked with a raw render error while the user typed.
They now **drop a syntactically-invalid expression** like an empty/incomplete entry
(guarded by core `validateExpression`), matching `buildTransforms`' own "a config
mid-edit still renders" contract; the inline feedback still flags the typo. Pure core,
tested (+2 cases). Visually confirmed: an invalid filter/calc now keeps the last-good
chart instead of breaking the preview.
- **Verified:** `typecheck` + `test` (741 passing) + `eslint` clean.
- **Still open (copy judgment, user's call):** the preview's catch-all
`"Couldn't render this chart: {raw Vega message}"` puts a diagnostic in the headline
(arch 10 says diagnostics go in a disclosure) — correct in the editor, debatable in the
builder; and the builder's terse `"Dataset «X» not found."` drops the next-step the
contract mandates (near-unreachable in the builder). Both noted, not changed.
- **2026-06-11** — **1C + 1D + 1E shipped (the Data section).**
- A new **Data section** at the top of the builder's left pane — "here are your rows;
shape them, then encode them" — emits the spec's top-level `transform` array.
- **1C · Filters** — a guarded **field + operator + value** predicate shelf. Operators
narrow by field type (`validFilterOps`): a measure/temporal field offers ordering
(`< ≤ > ≥`) + `is between`; a category offers `is` / `is not` / `is one of`. Values
coerce by type (quantitative → number; others → string, so ISO dates sort right).
`notEqual` emits a `{ not: { …equal } }` wrapper. A reversible **expression** power-mode
takes a raw `datum.…` predicate. Incomplete filters are skipped so the preview keeps
rendering. Multiple filters AND together. Pure core (`buildTransforms`,
`validFilterOps`, `filterOpArity`), tested.
- **1C · Calculated fields** — `{ calculate, as }` derived columns. A named field appears
in the channel dropdowns via `effectiveColumns` (defaults Quantitative); emitted
**before** filters (a row-wise calculate is order-independent, so calc-first is
equivalent and lets filters reference derived fields). Removing/renaming a referenced
field clears the dangling channel (`pruneEncodings`, in the store on calc edit/remove).
- **1E · Expression validation** — new pure core `expr-validate.ts` using Vega's own
`parseExpression` (already in the `vega` chunk, so ~zero bundle cost): inline syntax
errors on both expression inputs, plus a soft **unknown-field** warning when a
`datum.<field>` reference doesn't match a column (`referencedFields` walks the AST).
Field discoverability is served by **dataset-derived placeholder examples** (e.g.
`datum.revenue * 2`); a full Monaco-style completion popup is **deferred** (a bare
`<input>` doesn't warrant it — noted, not built).
- **1D · Data preview** — a collapsible, read-only first-N-rows table with a per-column
**type chip** in each header, to sanity-check inferred types before building (reuses
core `tabularRows`). Default collapsed; the scroll region is keyboard-reachable
(`tabIndex=0` + labelled group — avoids the Datasets-manager a11y gap).
- **Spec §06** gained a "Data (filters, calculated fields, preview)" section; the Layout
and Output blocks cross-reference it.
- **Council not yet run** on the new error/disclosure copy (the soft auto-fire surface:
expression-error + unknown-field copy, the preview disclosure). Conventions were matched
to the existing warnings region (arch 10 §5) and `SettingsPopover` disclosure; flag for a
council pass on review if desired.
- **Expression reference:** a contextual link to the Vega expression-language docs is
shown when an expression input is in play (a calculated field, or a filter in
expression mode) — the place the user needs to know the available functions/operators.
- **Verified:** `typecheck` + `test` (full suite green; +103 new core/store cases, +4 modal
smoke tests) + `eslint` + `build` (PWA, 45 precache entries). **Owed:** a manual/visual
pass against the live builder (filter shelf, calc → channel, expr errors, preview
table) — tests don't cover what the surface looks/feels like.
- **Surfaced direction (now parked):** the 1D preview shows **raw source** rows; a
transform-aware **data inspector** (resolved rows, à la vega-editor, in the builder
_and_ below the main Live Preview) is the wanted evolution — documented in
**[`docs/data-inspector-exploration.md`](./data-inspector-exploration.md)**, deferred.
- **2026-06-10** — **Up next: 1C (filter + calculate transforms), paired with 1D (data preview).**
- **1B · Per-chart export** shipped: an **Export** disclosure in the Live Preview header
(distinct from the workspace Export) — **Copy spec** + **Download JSON** (`.vl.json`) of
the shown text, and **Download PNG/SVG** of the live chart. Image output rides a new
`RenderHandle.toImageURL(format, { scale, background })` (PNG via `view.toCanvas` →
`blob:` URL; SVG via `view.toSVG` → `data:` URL), so no component touches the Vega
`view`. Filenames derive from the snippet name — filesystem-safe, script-preserving
(pure `core/chart-export.ts`, tested). Home: the **preview header**, not the plan's
"library row / editor toolbar" — image export needs the live view; the
disclosure-of-controls (not an ARIA menu) mirrors `SettingsPopover`. - **Export options (from first-round feedback):** PNG **Resolution** `1×/2×/3×` is a
multiplier of `devicePixelRatio`, so the default `1×` is Retina-crisp — the soft-1×
export was a dpr bug (raw `toImageURL` scaleFactor ignores dpr). **Background**
`Theme`(default)/`White`/`None` fixes transparent PNGs (the chart config is
transparent so the on-screen pane colour shows; export composites the chosen colour
under the PNG / adds an SVG `<rect>`). **Referenced data** `Inline`(default)/`Keep
refs` (shown only when the spec references saved datasets) inlines dataset values so
the exported spec renders standalone (`inlineReferencedDatasets`, tested). - Spec §08 gained a _Per-chart export_ section (with the options); §04 cross-references
it; `architecture/05` §2 records the handle's dpr-aware scale + background compositing.
- **1A · Actionable hints** shipped: one-click fixes on guidance warnings
(`BuilderWarning.fixes` + `applyWarningFix`), council-reviewed, with focus/announce a11y.
- **Builder UX/perf batch** (from dogfooding the Superstore dataset) shipped: near-fullscreen
`xlarge` modal tier; panes scroll internally (preview in its own viewport); backdrop-dismiss
guard (`dismissOnBackdrop: false`); render-timing diagnostics; **canvas** preview renderer
(SVG stays default elsewhere) + **canvas max-dimension guard** (`ChartTooLargeError` via a
headless probe); **data-aware default pre-population** (`smartDefaultEncodings`).
- Scope consolidated and **Tier-C target** set (this doc created); the research/Lyra forward
sequences are superseded by §4 here.
---
## 1. Where the builder is today (the floor — don't rebuild)
Tier B is shipped and tested (M4 done). The intelligence that decides _which chart and
why_ already exists in `src/core/chart-builder.ts`:
- **Smart default mark** from the (X, Y) field-type shape — not unconditionally Bar
(`defaultMark`).
- **Valid-type-only** field-type menus per column + **Size discipline** (Nominal /
Temporal / negative-extent columns are _blocked_ on Size, not merely warned)
(`validFieldTypes`, `isChannelTypeAllowed`).
- **Non-blocking guidance** — `builderWarnings` (6 rules: line/area needs both axes,
all-categorical, two-measures-want-scatter, area-split-many-series, crowded category
axis, negative-size guard).
- **Per-channel transforms** — Aggregate / Bin / `timeUnit`, chart-level Sort / Stack,
and a field-less "Count of records" measure.
- **Swap X/Y**, debounced live preview, validation gate (≥1 channel mapped).
Everything in §3 below is **backlog** — verified not yet built: `BuilderWarning` has no
`fix` field, there is no per-chart export, no top-level `transform`, no data-table
preview, no channels beyond X/Y/Color/Size, no styling/scale controls.
---
## 2. The target experience (Tier C, concretely)
Reading the brief's four words against the research:
| Brief word | What it means here | The levers (from §3) |
| ----------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Rapid** | shortest path from "a dataset" to "a chart I'd keep" | smart defaults (have), intent front door, starter examples, field shelf |
| **Intuitive** | matches how people think ("I have fields; what shows my point?") | intent front door, field shelf, data preview, value-or-field channels |
| **Guidance & recommendations on the fly** | the app proposes and corrects, not just validates | **actionable hints**, **intent front door** (the defining Tier-C feature) |
| **Moderately capable** | covers the common data-shaping + a few high-value encodings; _not_ every knob | filter / calculate, a _small_ set of promoted controls — and a hard stop short of Lyra's everything-inspector |
The defining shift from B→C is the **intent-first front door**: a _"what do you want to
show?"_ entry (FT / Datawrapper intent categories) that recommends a mark + channel layout
from `intent × column-types`, instead of starting the user at a blank mark picker. It is
the feature the brief most directly asks for, and it is the one piece the M4 research
_deferred_. Choosing Tier C is choosing to pull it forward.
Tier C **extends** §06 — it does not replace the mark-first builder. The front door is an
on-ramp; the user can still ignore it and drive the channels directly, and can always drop
to Monaco. This preserves Astrolabe's core invariant: **the JSON spec is the source of
truth; the builder is a view that emits it** (the Lyra anti-lesson — never let the GUI
become the document).
---
## 3. The consolidated enhancement set
Organized into build phases by dependency and value. Each item carries its **source**,
**value/effort**, **code home**, and **spec impact**. Phases 1–3 are the committed Tier-C
scope; Phase 4 is explicitly _beyond_ "moderately capable" and gated on a later decision.
### Phase 1 — Guidance + I/O (no new interaction model) — _do first_
High value-to-effort, mostly pure-core + thin UI, no architectural change. These make the
_current_ builder dramatically better and de-risk the bigger phases.
**1A · Actionable hints** — _done (2026-06-10)_
Turn advisory warnings into one-click fixes. Source: Lyra §3.1 (`Hints` carries an
`action`). Several existing warnings have an obvious remedy:
- _"draws one mark per row → "_ **[Aggregate as Sum]** (set the measure's `aggregate`)
- _"long labels → "_ **[Swap X/Y]** (the action already exists — just wire it)
- _"two measures usually read as a scatter → "_ **[Switch to Point]**
- _"area split into many series → "_ **[Stack]** or **[Remove colour]**
Implementation: extended `BuilderWarning` with `fixes?: BuilderWarningFix[]` (`{ label, apply }`, pure,
unit-tested in `chart-builder.test.ts`); the modal renders each as a ghost button wired to a
new `applyWarningFix` store action. Wired fixes: **[Aggregate as Sum]** + **[Swap X/Y]**
(one-mark-per-row), **[Swap X/Y]** (high-cardinality axis), **[Switch to Point]** (two
measures), **[Stack]** + **[Remove colour]** (area split — and a stacked area is no longer
flagged, so [Stack] resolves it). Council run (Carbon Actionable notification + APG Alert):
ghost buttons, remedy-in-button, polite "Applied: …" announcement, focus moved off the
removed button — resolution recorded in `architecture/10` §5. §06 "Guidance" amended to
document the one-click fixes.
**1B · Per-chart export** — _done (2026-06-10)_
Today export was workspace-backup only (§08); there was **no way to get one chart out**.
Source: Lyra §3.8. Shipped as an **Export disclosure in the Live Preview header**:
- **Copy spec** (clipboard) + **Download `.vl.json`** of the currently-shown text.
- **Download PNG / SVG** of the live chart via a new `RenderHandle.toImageURL` wrapping
`view.toImageURL` (PNG at 2× → `blob:` URL, revoked after download; SVG → `data:` URL),
so the embedding boundary holds — no component touches the raw view.
- Filenames from the snippet name, filesystem-safe and script-preserving (pure
`core/chart-export.ts`, tested). Standalone HTML left out (the deferred optional).
Home decision: the **preview header**, not the plan's original "library row / editor
toolbar" suggestion — the image formats need the live rendered view, and "export this
chart" reads best beside the chart. The widget is a disclosure-of-action-buttons (not an
ARIA menu), mirroring `SettingsPopover` and sharing its single-open registry. _Spec impact:
new §08 "Per-chart export" section; §04 cross-reference; `architecture/05` §2 handle note._
**1C · Filter (+ Calculate) dataset transforms** — _done (2026-06-11; see status log)_
The transform layer the builder doesn't touch: top-level `transform: []`. Source: Lyra
§3.12. The builder _already tells users to filter_ in three warnings
(`chart-builder.ts:475,476,511`) while offering no way to do it.
- **Filter** _(highest)_ — a guarded **field + operator + value** predicate shelf
(Voyager-style, no expression needed for the common case); power form is a raw `datum.…`
expression validated with Vega's `parseExpr` (see 1E). VL applies top-level transforms
_before_ encoding aggregation, so "filter raw rows, then aggregate" is the natural
default; filtering on an aggregated value (HAVING) is the advanced case — defer.
- **Calculate / derived field** _(second)_ — `transform: [{calculate, as}]`; the new field
then appears in column dropdowns like any other.
- **Lookup / join a second dataset** — larger data-model change (one dataset per snippet
today) → **defer to Phase 4**.
Home: a new **"Data" section in the builder's left pane, above the channels** (`here are
your rows [+ Filter] [+ Calculate] → now encode them`), paired with 1D. _Spec impact: new
§06 "Data / transforms" subsection — this is genuinely new behaviour, write it._
**1D · Data-table preview** — _done (2026-06-11; see status log)_
Neither the builder nor the Datasets manager ever shows the **actual rows**. Source: Lyra
§3.2. A compact, **read-only** first-N-rows grid with a **per-column type chip** in each
header lets users sanity-check inferred types _before_ building — exactly when inference is
most likely to surprise. Type + cardinality + extent already come from `profile.ts`; we
only need the row sample. Homes: a collapsible "Data" strip in the builder's left pane
(under the dataset name) and/or the Datasets manager. Keep it read-only (editing data is
out of scope). _Spec impact: §06 + §05 (Datasets) additions._
**1E · Inline expression validation + field autocomplete** — _done (2026-06-11; completion popup deferred)_
When 1C's expression mode lands, validate the Vega/VL expression string with the library's
own `parseExpr` (Lyra §3.6) and surface errors inline; autocomplete the dataset's own
column names (we have the schema from `profile.ts`) (Lyra §3.10). Record the `parseExpr`
technique in `architecture/08`. _Spec impact: folded into 1C._
### Phase 2 — Interaction substrate (enables Tier C)
Two changes to _how the user touches fields_. They have standalone value but their main job
is to be the substrate Phase 3 (and any future added channels) stands on — sequence them
here so Tier C lands cleanly.
**2A · Value-or-field channels (the Property model)** — _capability gain; medium_
Source: Lyra §3.3 (`Property.tsx` — one droppable control that is _either_ a literal value
_or_ a bound field). Today a channel is field-only. Let a channel also hold a **constant
`value`** (fixed colour / size) with one consistent control and a chip showing the binding
kind. VL encodes exactly this (`field` vs `value` vs `datum`). Generalizes cleanly to any
channel we add later. _Spec impact: §06 "Encoding channels" — a channel may carry a
constant._
**2B · Field shelf + in-place type cycling** — _the largest interaction shift; the Tier-C/facet substrate_
Source: Lyra §3.4 (drop-zones) + Voyager (field list with type chips) + Lyra §3.5
(`FieldType` — the type icon _is_ the control, click cycles N→O→Q→T within the valid set).
Flip from **channel-first** ("pick a channel, then its column") to **field-first** ("here
are your columns — drag/click onto channels"), matching how people think. This is the
natural way to assign many fields across many channels, so it is the **interaction
substrate for Tier C and faceting**, not a standalone task. _Spec impact: §06 layout
revision (field shelf alongside the channel rows)._
### Phase 3 — Tier C (the intent-first front door) — _the defining feature of this push_
The grand idea, stated once: **the builder is the app's rapid, intuitive on-ramp; Monaco
is the expert surface; the JSON spec stays the document.** Phase 3 is where that becomes
true — 3A gives the builder an intent-first opening screen, 3D makes that screen the
app's guided creation flow (not a feature hidden in a modal), and 3B/3C seed and re-enter
it.
**3A · Intent-first front door** — _the B→C step_
Source: research §5/§8 Tier C (FT Visual Vocabulary + Datawrapper intent taxonomy). A
_"what do you want to show?"_ entry mapping **intent × column types → recommended mark +
channel layout**:
| Intent (FT / Datawrapper) | Our expression (within 5 marks / our channels) |
| -------------------------- | ------------------------------------------------------------------- |
| **Magnitude / Comparison** | Bar (x=N, y=Q; horizontal for long labels) |
| **Ranking** | Bar, sorted by value |
| **Change over time** | Line (x=T, y=Q; color=N for series) |
| **Correlation** | Point (x=Q, y=Q); Circle + size=Q for a 3rd measure |
| **Distribution** | Bar of binned counts (histogram) — uses Bin |
| **Part-to-whole** | Stacked / 100% bar (uses Stack); _true pie needs `theta` → Phase 4_ |
| **Deviation** | diverging signed Bar |
Honest coverage gaps stay honest (Spatial / Flow excluded; Part-to-whole partial until
`theta`). The front door is an **on-ramp, not a gate** — it pre-populates the mark-first
builder, which the user can then adjust or ignore. Munzner's typology and Wilke's directory
become seatable council sources at this point (research §2). Built **on the 2B field
shelf**. **Run the front-door copy + flow through `/council`.** _Spec impact: substantial
§06 amendment — a new "Intent" front-door subsection; the M4 spec note explicitly flagged
this as the deferred tier, so this is the planned amendment, not drift._
**3B · Starter examples gallery** — _cheap; pairs with 3A_
Source: Lyra §3.7. A small set of **curated starter snippets**, one per covered FT intent
(Magnitude/Bar, Change-over-time/Line, Correlation/Point, Distribution/histogram,
Part-to-whole/stacked). Improves first-run, doubles as living documentation of what the app
does well. Natural home: the snippet library. **Author them in the builder dialect so they
hydrate via 3C.** _Spec impact: §02 (library seed content)._
**3C · Open in builder (strict spec hydration)** — _decided 2026-06-11; pairs with 3B_
Reverse the assembler: a pure `parseChartSpec(spec) → BuilderConfig | null` so an existing
snippet can re-enter the builder populated. **Strict policy only** — a snippet-list button
("Open in builder") enabled exactly when the spec is **losslessly** representable in the
builder's dialect; anything richer stays Monaco-only. Losslessness is checked not by
enumerating unsupported features but by **re-assembling the parsed config and
deep-comparing against the original** (ignoring key order, `$schema`, and the injected
`tooltip: true`) — exact, ~20 lines, and it stays correct automatically as the dialect
grows (every Phase-4 addition widens what hydrates for free). No lossy or
residue-preserving mode: the moment the builder can overwrite a richer spec, the GUI starts
competing with the JSON as the document (the Lyra round-trip trap — §5).
Value: converts the builder from **create-only to create-and-revise** for its own output,
and makes the 3B starters openable as builder seeds. Known fiddly bits (all mechanical):
inverse of `escapeVegaField`, filter-value un-coercion back to the `BuilderFilter` string
shape, `sort: "-y"` → `{sort: 'descending'}`. Round-trip property tests
(`parse(build(config))` ≈ `config`) are the natural core coverage. Per-row check in the
library is cheap but memoize it. **Decisions at build time:** save semantics on an opened
snippet (update-in-place via a builder "edit" mode vs. always-create-new — today the
builder only creates) and the disabled-button affordance (disabled-with-reason vs. hidden —
park in `ux-second-pass.md` if non-obvious). _Spec impact: §02 (library row action) + §06
(hydration + the strict gate)._
**3D · Entry points & discoverability** — _done 2026-06-12 (council-reviewed; see status log); 3A opens onto it next_
Today the builder has **one entry, two levels deep**: Datasets modal → per-dataset "Build
Chart" (`DatasetsModal.tsx`) — the data model's shape ("charts come from datasets"), not
the user's intent ("I want to make a chart"). Meanwhile the library's pinned primary
action, "Create New Snippet", drops straight into blank Monaco — the **expert** path is
the default, and the audience the builder serves may never find it. Council ruling
(recorded here; contract updates at build time):
- **NN/g #6 (recognition over recall):** the build-a-chart action must be visible where
the intent forms — the library's creation surface, not recalled as a dataset-row action.
- **NN/g #7 (flexibility/efficiency):** the guided path is what novices see; raw JSON is
the expert accelerator — visible, one click, never hidden. Builder-forward, Monaco
intact.
- **Carbon empty-states:** one primary action per surface ("pick the most important"; a
tertiary button for the secondary CTA); a no-data state explains what the space will
hold, offers one primary next step, and never dead-ends.
The decided shape:
1. **Fork the library's creation surface** — primary **Build Chart** (opens the builder),
ghost/tertiary **New JSON snippet** (today's instant create, unchanged). Two plain
buttons with clear hierarchy — no split/menu-button widget (two static choices don't
earn an ARIA menu).
2. **Dataset picker inside the builder** (header select, default = most recently
modified), so the builder opens without a preselected dataset. Build-time decision
taken: switching with an **untouched** opening config re-derives fresh smart defaults;
switching a **built-on** config rebases it (`rebaseBuilderConfig` — chart-level intent
kept, bindings to missing columns shed; a same-schema dataset keeps everything).
Useful beyond 3D (switch data without leaving the builder).
3. **No-datasets empty state** in the builder, per the Carbon no-data pattern: what the
builder does + one primary action ("Add a dataset" → Datasets modal). No dead end.
4. **Onboarding tile** — a data-first path ("import your data → build a chart") beside the
existing examples gallery.
5. **Keep the dataset-row "Build Chart"** — contextual shortcut, pre-picks the dataset; it
just stops being the only door.
**Composition with 3A (the point of the sequencing):** the Build-Chart button lands on the
intent front door — "what do you want to show, with which data?" _is_ the builder's
opening screen. 3D without 3A opens onto the mark-first builder (fine, interim); 3A
without 3D is a front door nobody finds. Discoverability is a prerequisite for 3A's value,
so: **3D with or just before 3A.** _Spec impact: §02 (library creation actions +
onboarding) + §06 (dataset picker, empty state)._
### Builder UX & perf — in-flight fixes (2026-06-10, from dogfooding the Superstore dataset)
Pre-existing builder rough edges surfaced while testing on a 10k-row / ~24-col dataset.
Fixed in this batch (not part of 1A–3B, but the same surface):
- **Modal is a near-fullscreen work surface** — new `xlarge` shell tier (`ModalShell`,
`min(1800px, 96vw) × min(1100px, 92vh)`); the Chart Builder no longer wastes screen.
- **Panes scroll internally, modal keeps its shape** — the `.builder` grid fills the body
(`grid-template-rows: minmax(0,1fr)`), the config pane and the **preview** each scroll in
their own viewport, so a tall one-mark-per-row chart scrolls inside the preview instead of
pushing Create/Cancel below the fold.
- **Backdrop click no longer discards in-progress work** — `dismissOnBackdrop: false` on the
builder (registry flag); Escape and × still close.
- **Data-aware default pre-population** — the builder no longer blindly takes the first two
columns (which opened Superstore on a 9994-bar degenerate chart). When the dataset is
profiled, `defaultBuilderConfig` picks a "safest bet": a low-cardinality category vs a
count of records (tidy bar), else a time series of the first measure, else a scatter — each
guaranteed to render. Falls back to positional when there are no stats. Pure + tested. This
composes with (isn't replaced by) the future intent-first front door — the builder always
needs a sane opening state.
- **Render-timing diagnostics** — `BuilderPreview` logs `parse · prepare · destroy · embed ·
paint · total` (+ mark, row count) to the console (dev always; prod only when slow). The
**paint** phase (a double-rAF after `embed()`) captures the real freeze.
**Perf finding — confirmed and fixed.** Diagnostics on the Superstore dataset:
`embed 237ms · paint 6458ms` for the default one-bar-per-row chart (9994 rows), vs
`paint 6ms` once grouped to a few categories. The freeze was entirely **SVG layout/paint**
(one DOM node per mark), not chart compilation. **Fix shipped:** the builder preview now
renders with **canvas** (`renderSpec(…, { renderer: 'canvas' })`); SVG stays the default for
the editor's LivePreview and for image export. Contract divergence recorded in
`architecture/05` §2.
**Canvas max-dimension guard (measured, not guessed).** Canvas (unlike SVG) has a hard
max side (~32k px), so a chart that resolves taller than that fails to allocate (the
broken-image icon). The cause is **physical render size, not cardinality** — a vertical bar
with thousands of _X_ bands renders fine (width is container-bounded); only an unbounded
band axis (e.g. a horizontal bar's _Y_) overflows. So `renderSpec` now runs a **headless
(`'none'`) layout probe** for canvas charts, reads the chart's resolved **height**, and
throws `ChartTooLargeError(heightPx, limitPx)` when it exceeds `MAX_CANVAS_PX ÷ dpr`. The
builder catches it and shows the real numbers ("would be ~200,000px tall — larger than the
browser can draw on a canvas (~16,383px max here); aggregate or filter"). The earlier
band-count proxy (`previewBandCount`/`MAX_PREVIEW_BANDS`) was removed — readability
(cardinality) stays a `builderWarnings` concern; the render-size limit is now measured at
its true cause.
### Phase 4 — Beyond "moderately capable" (gated — decide later)
These exceed the stated ceiling. List them so they have a home, but **do not commit them in
this push** — revisit once Phases 1–3 land and we see real usage. Each must clear the
guardrail: _promote a control only when it is **both common AND awkward in JSON**._
- **More channels** — `theta` (unlocks pie/donut → _true_ part-to-whole), `opacity`,
`shape`. (`theta` is the most defensible — it closes a real coverage gap.) Ride on 2A/2B.
- **Faceting (Row / Column → small multiples)** — research §8 B8. The clean way to compare
many categories. Design crux: VL facets default to **shared scales** (keep that default);
expose an "independent axes" toggle (`resolve.scale`) only as advanced. **Verify against
the preview's `"container"` fit modes** (per-cell sizing on facets is finicky).
- **Light styling / scale controls** — colour-scheme picker (categorical / sequential /
diverging), measure-axis `zero` / `log` toggle, custom axis title, legend title / hide.
Implement as **auto-derived override panels** that start empty (inheriting VL defaults)
and emit JSON **only when touched**; clearing a channel **drops its overrides** (Lyra
§3.9 `cleanupUnused` — no orphaned `scale`/`axis` in the output spec).
- **Builder undo/redo / "reset to smart defaults"** — Lyra §3.11. Low priority (the modal
is short-lived; the smart default already gives a sane start).
- **Lookup / join a second dataset** — Lyra §3.12; data-model change (multi-dataset
snippets). Larger, separate effort.
- **Transform-aware data inspector** — evolve 1D from raw-source rows to the _resolved,
post-transform_ data (filtered rows + calculated columns), and generalize it to a
togglable data panel below the **main** Live Preview (vega-editor's "Data Viewer", a
debugging aid for any snippet, not only builder output). Reads runtime rows via
`view.data(name)` through the `RenderHandle`. Cross-cutting (editor + builder), so it
has its own home: **[`docs/data-inspector-exploration.md`](./data-inspector-exploration.md)**.
---
## 4. Recommended build order
```
Phase 1 1A actionable hints ✓ done
1B per-chart export ✓ done
1C filter (+ calculate) ✓ done
1D data preview ✓ done
1E expr-validate ✓ done (syntax + unknown-field; completion popup deferred)
Phase 2 2A value-or-field channels (Property model) ✓ done
2B field shelf + in-place type cycling ✓ done (field-first + on-chart shelves)
Phase 3 3D entry points & discoverability ✓ done (2026-06-12)
3A intent-first front door (Tier C) ✓ done (2026-06-13); persistent strip
3B starter examples ← next; pairs with 3C
3C open in builder (strict hydration) ← added 2026-06-11; pairs with 3B
Marks +rect (Heatmap) ✓ done (2026-06-13)
Phase 4 (gated) theta/facets/styling-overrides/undo/lookup — decide after Phase 3
Also shipped (builder UX/perf, from dogfooding): near-fullscreen modal, internal-scroll
panes, backdrop-dismiss guard, render diagnostics, canvas preview + canvas-size guard,
data-aware default pre-population. See "Builder UX & perf" above.
```
Rationale for the order: Phase 1 is the cheapest large quality jump and needs no new
interaction model, so it ships value while the bigger design settles. Phase 2 is pure
substrate — low _user-visible_ payoff alone, but Phase 3 is much cleaner on top of it than
bolted onto the channel-first UI. Phase 3 delivers the brief's headline ("recommendations
on the fly") — 3D first, because a front door nobody finds delivers nothing: the entry
points make the builder the app's guided creation flow, then 3A gives that flow its
intent-first opening. Phase 4 is deliberately deferred to protect the "moderately capable"
ceiling.
---
## 5. Constraints & anti-scope (the ceiling)
What "moderately capable" rules **out** — load-bearing, from `lyra-review.md` §2.1/§4:
- **No everything-inspector.** Lyra surfaces ~50 direct controls per primitive because the
GUI _was_ its document. We split the work on purpose: a small **guarded** builder + a
first-class **Monaco** editor for the long tail. Styling/scale/axis breadth for its own
sake belongs in Monaco, not the builder.
- **The promotion test:** a control enters the builder only when it is **both common AND
awkward in JSON**. Otherwise it stays in Monaco.
- **JSON stays the source of truth.** The builder _emits_ spec; it is never the document
(the Lyra one-way-export trap that forbids round-trips).
- **No interaction-by-demonstration, no direct-manipulation canvas, no general
data-pipeline editor.** If we ever add interactivity, expose VL `params`/selections as a
small guarded action — never port Lyra's signal generator.
- **Take ideas from the reference clones, read no code into the repo** (`AGENTS.md`: no
shared lib; patterns adapted, not imported).
---
## 6. Cross-cutting notes
- **Spec deltas:** 1A (minor §06), 1B (§08 + §02/§03), 1C (new §06 transforms subsection),
1D (§06 + §05), 2A/2B (§06 layout), 3A (substantial §06 intent front-door amendment), 3B
(§02). Per _"spec follows code now"_: build the decision, then amend the spec to match —
don't let code and §06 drift.
- **Council:** auto-fires on guidance copy and new interactive-widget keyboard/focus work —
so 1A (hint affordance + copy) and 3A (front-door flow + copy) both go through `/council`
before committing. It advises; architecture 09/10 decide.
- **Verification:** pure rules get `chart-builder.test.ts` cases; every UI/affordance change
gets a manual pass against the _live_ builder — a green build proves nothing about what
the user sees (`AGENTS.md`; `docs/manual-verification.md`). Items 1A and 1C are mostly
`src/core/`, squarely the "core-first, tested hardest" rule.
- **Source of truth going forward:** this doc. `chart-builder-research.md` §8 and
`lyra-review.md` §5 remain the _research record_; their forward sequences are superseded
by §4 here.
docs/exploration/chart-builder-research.md
+278
View File
@@ -0,0 +1,278 @@
# Chart Builder — Design Research (M4)
> **Status:** research complete; informs the M4 build (spec §06).
> **Decision:** build **Tier B — "smart + guarded"** (mark-first, still §06-shaped).
> **Why this doc exists:** the Chart Builder is the point where Astrolabe stops being
> a pass-through JSON editor and starts making chart-shaped suggestions/defaults.
> "Which chart, and why" becomes a decision the app owns, so we researched it
> deliberately before building. This is the record of what we studied and what we
> took from each source — the citations behind every default and guardrail in
> `src/core/chart-builder.ts`.
---
## 1. Scope of the builder (the constraint everything maps into)
Spec §06: compose a Vega-Lite chart from a dataset with **one mark** ∈
{Bar, Line, Point, Area, Circle}, mapping columns to **four channels** (X, Y, Color,
Size), each carrying a **field type** ∈ {Quantitative, Nominal, Ordinal, Temporal},
plus optional pixel width/height → a complete spec saved as a snippet that
references the dataset by name. Column types are inferred upstream as
`number | string | date | boolean` (`src/core/type-inference.ts`).
No transforms (no binning, aggregation, stacking, regression), no second axis, no
geo. That narrow surface is the lens through which every source below was read:
"what does this canon tell us to do **within Bar/Line/Point/Area/Circle and
X/Y/Color/Size?**"
## 2. The sources
Two kinds: **formal CS** (how recommendation engines actually rank charts) and
**chart-choice canon** (how practitioners pick). They were chosen for being
**cloneable/grep-able offline** (the council's working model) and authoritative for
"which chart," which our other seats (Carbon/GOV.UK/APG/NN/g) don't cover.
| Source | What it is | Local path |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| **Draco** (uwdata) | Visualization design knowledge as ASP constraints — the formal "what makes a good chart," with hard (validity) + soft (preference) rules and weights, some learned from human perception experiments (Kim 2018, Saket 2018). | `reference/draco` |
| **Voyager** (vega) | UW IDL's recommendation/exploration tool on CompassQL — the _interaction_ model (field shelves, auto-add, type chips) and effectiveness-ranked encoding suggestions. | `reference/voyager` |
| **FT Visual Vocabulary** (Financial Times) | A poster/taxonomy mapping _what you want to show_ (9 data-relationship categories) → chart types. **Seated** in the council. | `reference/chart-doctor/visual-vocabulary/` |
| **Datawrapper** | Practitioner chart-choice in plain language; intent-first ("the chart's main statement becomes a compass"). **Seated** (distilled). | `reference/principles/datawrapper.md` |
**Theoretical basis, not seated (deliberately):** **Munzner**, _Visualization Analysis
and Design_ (marks & channels; the channel-effectiveness rankings — magnitude:
position → length → angle → area …; identity: spatial region → hue → shape;
expressiveness & effectiveness principles) and **Wilke**, _Fundamentals of Data
Visualization_ (`clauswilke/dataviz`; example directory by intent + "ugly/bad/wrong"
pedagogy). They are the _why_ beneath Draco and Voyager — Draco's soft weights are an
operationalization of exactly these Mackinlay/APT/Munzner effectiveness rankings — but
they restate the same rules the seated sources already give us, so seating them would
add overlap, not coverage. Cited here as grounding; revisit if we ever build the
intent-first "Tier C" front door, where Munzner's typology and Wilke's directory
would earn their place.
## 3. What we take from each source
### From Draco — validity guardrails + a preference ranking (the rigorous core)
Draco models a chart as ASP facts and rejects/ranks them with **hard** (∞ cost) and
**soft** (weighted) constraints (`asp/optimize.lp`). We can't ship an ASP solver in a
browser, but the rules are a lookup table. The portable subset:
- **Hard validity (block in the UI):** `reference/draco/asp/hard.lp`
- Quantitative on a string/boolean column — illegal (`:6`). Temporal only on a
datetime column (`:7`).
- **Size encoding a Nominal field — illegal** ("size implies order; nominal is
misleading", `:53`). Size cannot encode **negative** values (`:56`). Size only on
point/text marks (`:110`).
- Bar/Area must include a **zero baseline** on the measure axis (`:103-104`).
- Bar needs a categorical axis — both x and y continuous on a bar is malformed
(`:97`); Line/Area need **both** x and y, and not both discrete (`:91,:94`).
- Same field on x and y — illegal (`:122`). >20 categorical colors — illegal (`:172`).
- **Soft preference (the weights, `asp/weights.lp` + `asp/soft.lp`):**
- Channel-by-type appropriateness (lower = better): continuous data is free on x/y,
costs to put on color (10) or size (1); nominal cheapest on y then x then color;
ordered data expensive on size. → **fill X/Y before Color/Size.**
- Mark by data shape: continuous×continuous → **point** (line/area heavily
penalized); continuous×discrete aggregated → **bar**; discrete×discrete → point/rect.
- Prefer time on x (`temporal_y`, `:147`); never type a number as nominal
(`number_nominal`, weight 10); the loudest nudge is an all-discrete chart with no
measure (`only_discrete`, weight 30).
The hand-tuned `weights.lp` is the portable "common-sense" set; the learned
`weights_learned.lp` corroborates direction, not magnitude.
### From Voyager — the interaction model + the valid-type table
- **`getValidTypes` (`src/components/data-pane/field-list.tsx:140-155`) — adopted
almost verbatim:** number→{quantitative, nominal}, integer→{quantitative, nominal},
datetime→{temporal}, string→{nominal}, boolean→{nominal}. The type toggle shows only
when ≥2 valid types exist. (We extend slightly — see §4 — to also offer Ordinal,
which Voyager deliberately omits, `encoding.ts:131-134`.)
- **Auto-add / "auto" mark (`models/shelf/index.ts:72-81`):** Voyager lets a field be
added with `channel:'?'` and asks CompassQL to place it by `effectiveness`. The small
builder analogue is a **non-empty smart default** (`defaultBuilderConfig`) so the
preview is never blank.
- **Type chips + swap:** per-field type indicator with a click-to-change popover, and a
cheap x↔y swap (Voyager's `SPEC_FIELD_MOVE` is remove-both + re-add).
- **Out of scope (Voyager scope creep we reject):** wildcard shelves, the full Related
Views gallery, faceting (row/column), and embedding CompassQL/`compassql@0.20.2`
itself. We hand-roll a small decision table in `src/core/` instead of pulling the
engine.
### From FT Visual Vocabulary — the intent→chart taxonomy (and our coverage gaps)
`reference/chart-doctor/visual-vocabulary/README.md` (taxonomy is prose). Nine
categories; mapped to **our five marks**:
| FT category | What it shows | Our expression |
| -------------------- | --------------------------- | ----------------------------------------------------------------------------- |
| **Magnitude** | size comparisons | **Bar** (x=N, y=Q; horizontal x=Q, y=N for long labels) — primary |
| **Ranking** | position in an ordered list | **Bar, sorted** by value (the sort _is_ the feature) |
| **Change over Time** | trends | **Line** (x=T, y=Q; color=N for series); Bar/Area alternatives, single series |
| **Correlation** | relationship of 2+ measures | **Point** (x=Q, y=Q); **Circle/bubble** + size=Q for a third measure |
| **Deviation** | +/− from a reference | **Bar** with signed Q (diverging bar only) |
| **Distribution** | spread/frequency | weak: raw **Point** strip, or **Bar** of pre-binned counts (no bin transform) |
| **Part-to-whole** | component shares | **none well** — redirect to Magnitude/Bar; we can't show true proportions |
| **Spatial** | geography | **none** — exclude |
| **Flow** | movement between states | **none** — exclude |
**Coverage:** strong on Magnitude, Ranking, Change-over-Time, Correlation; partial on
Deviation/Distribution; none on Part-to-whole/Spatial/Flow. Honest gaps, not silent
degradation.
### From Datawrapper — plain-language rules + intent labels
`reference/principles/datawrapper.md`. Corroborates the same default-mark-by-intent
table (comparison→Bar, time→Line, correlation→Point/bubble) and supplies friendlier
intent words (Developments over time / Shares / Comparison / Correlation). Bindable
rules: bar is the safe default; bar over column on small screens; line for continuous
time, columns for a few points; circles are hard to compare precisely; size encodes a
quantity; area = single total (warn on multi-series).
## 4. The convergent rules — what all four agree on (high-confidence)
These are not a judgment call; the formal engines and the practitioner canon land on
the same place. They are the spec for `src/core/chart-builder.ts`:
1. **Column type → valid field types** (Voyager `getValidTypes`; Draco `hard.lp:6-7`):
`number`→{Quantitative (default), Ordinal, Nominal}; `date`→{Temporal only};
`string`→{Nominal (default), Ordinal}; `boolean`→{Nominal}. Never offer Q for
string/boolean, never Temporal for a non-date. (We add Ordinal where it's a defensible
user assertion of order; Voyager omits it for UX simplicity — our deliberate superset.)
2. **Default mark from the (X, Y) shape** (Draco mark-by-shape; Voyager effectiveness;
FT; Datawrapper): temporal × quantitative → **Line**; quantitative × quantitative →
**Point**; (nominal/ordinal) × quantitative → **Bar**; both-discrete → **Point**
(Bar/Line/Area are invalid with no continuous axis); single axis or unknown → Bar.
3. **Channel priority + Size discipline** (Draco `hard.lp:53,56,110` + non-positional
pref): fill X/Y before Color/Size; Color before Size. **Size is only valid for
Quantitative/Ordinal positive measures on Point/Circle marks** — disabled for Nominal,
Temporal, and negative data (not merely discouraged).
4. **Bar/Area zero-baseline; Line exempt** (Draco `hard.lp:103-104`; FT; ONS/Vox sources
FT links). We expose no axis-truncation control, so Vega-Lite's own defaults already
give zero-baseline bars and free-baseline lines — the rule is satisfied by _not adding_
an override, nothing to emit.
5. **Chart-choice polish** (FT; Datawrapper): sort bars when ranking; horizontal bar for
long category labels; Size encodes a quantity, Color a category; Area is for a single
series (warn against color-splitting into many).
## 5. The decision: Tier B — "smart + guarded"
Three tiers were on the table. **Tier B** was chosen (2026-06-05).
- **Tier A — spec-literal:** Bar default, four channel dropdowns, type override, smart
pre-population. Matches §06 verbatim but uses almost none of the research; stays a
"dumb" composer.
- **Tier B — smart + guarded (chosen):** Tier A **+** default _mark_ from the (X, Y) type
shape (not always Bar) **+** valid-type-only menus **+** inline non-blocking warnings
from the Draco rules **+** swap-X/Y **+** Size disabled for Nominal/Temporal/negative.
Still mark-first and §06-shaped, but genuinely intelligent. Requires a small §06
amendment (documented in the spec).
- **Tier C — intent-first aid:** Tier B **+** a "what do you want to show?" front door
(FT/Datawrapper intents → recommended mark + channel layout from intent × column
types). Highest "which chart & why" value; biggest UI; clearly extends §06. Deferred —
if revisited, this is where Munzner's typology and Wilke's directory would be seated.
## 6. How it maps to implementation
The convergent rules become pure functions in `src/core/chart-builder.ts`
(tested in `chart-builder.test.ts`), consumed by the builder store/modal:
- `validFieldTypes(columnType)` → the type menu (rule 1); `defaultFieldType` = its head.
- `defaultMark(xType, yType)` → smart default mark (rule 2); used by
`defaultBuilderConfig`.
- `isChannelTypeAllowed(channel, type)` → Size discipline gate (rule 3).
- `builderWarnings(config)` → inline non-blocking hints (rules 3–5: line/area need both
axes, area + many series, two measures better as a scatter, etc.).
- `buildChartSpec` / `buildSnippetSpecText` → assemble the final spec; zero-baseline is
Vega-Lite-default (rule 4), so nothing is emitted for it.
## 7. Anti-recommendations (what a naive builder would happily produce, and we don't)
The highest-value guardrails — encodings a naive UI emits that the canon rejects:
- A categorical column on **Size** (Draco hard `:53`) — blocked, not warned.
- A **truncated-axis bar** — prevented by never exposing an axis override (Draco `:103`).
- A high-cardinality category on **Color** → unreadable legend (soft w=10; >20 hard).
- A **Line between two raw measures** instead of a scatter (Draco soft w=20) — warned.
- An **all-categorical chart with no measure** (Draco soft w=30, the loudest) — warned.
- A **number typed Nominal** (Draco soft w=10) — discouraged via default = Quantitative.
## 8. Future enhancements (backlog)
The Tier-B build is the floor, not the ceiling. The enhancements below were surfaced by
the research. Status as of 2026-06-06.
> **Forward plan moved (2026-06-10):** the _prioritized, sequenced_ enhancement plan now
> lives in [`chart-builder-enhancement-scope.md`](./chart-builder-enhancement-scope.md),
> which merges this backlog with the [`lyra-review.md`](./lyra-review.md) §5 ideas and sets
> the **Tier-C** target. This section remains the research _record_ (the citations behind
> each item); consult the scope doc for _what to build next and in what order_.
**A · Cheap wins inside the current 5-mark / 4-channel scope**
- **A1 · Sort-on-ranking** _(done)_ — chart-level Sort control (Asc/Desc/None) sorts the
categorical axis by the measure (FT: "bars display ranks much more easily when sorted").
Appears only for a category-vs-measure pair.
- **A2 · Bar orientation** _(partly done)_ — the **Swap X/Y** control is the manual path to
a horizontal bar, and the crowded-axis hint (A3, below) now auto-suggests it for the
un-aggregated case. A general "long labels → go horizontal" suggestion on _any_ vertical
bar is still deferred (needs a label-length / cardinality signal); a blanket warning was
rejected — it would fire on every ordinary vertical bar.
- **A3 · Crowded-axis & high-cardinality warnings** _(done)_ —
- _Done:_ the **un-aggregated crowded axis** — a bar/line/area with a category axis and a
**raw** measure draws one mark (and one label) per row, so over `CROWDED_CATEGORY_ROWS`
(30) rows it warns and points to aggregating, or a horizontal bar. Row-count-based:
detects exactly the mark-count == row-count case (URL/non-tabular → `rowCount` null → skipped).
- _Done (via the profiling extension below):_ an **aggregated** axis that still has many
distinct **categories** (`CROWDED_CATEGORY_DISTINCT` = 30) and an unreadable **Color
legend** (`CROWDED_LEGEND_DISTINCT` = 12, discrete colour only) now warn from per-column
cardinality. (Number-typed-Nominal remains a possible future nudge; not yet flagged.)
- **A4 · Data-aware Size guard** _(done)_ — Size mapped to a field whose profiled numeric
**extent** goes negative warns (Draco `hard.lp:56`; size implies positive magnitude),
on top of the type-level Size discipline.
> **Done — profiling extension (the A3 / A4 enabler).** `profile.ts` now derives, in the
> **same sample pass** that feeds `inferColumnType`, a per-column **capped distinct count**
> (`DISTINCT_CAP` = 50, with a `distinctCapped` overflow flag) and a **numeric extent**
> (min/max, numeric columns only), surfaced on `DatasetProfile.columnStats` and the `Dataset`
> record. `builderWarnings(config, rowCount, columns)` consumes them for the legend/axis
> crowding hints and the negative-value Size guard. **Caveats handled:** URL / non-tabular
> data has no rows → `columnStats` is `[]` and the dependent hints skip; datasets stored
> before the field default to `[]` via `dataset-migrations` / import normalization (no forced
> re-profile on read — they pick up stats on next save). Thresholds live in `chart-builder.ts`
> constants alongside `CROWDED_CATEGORY_ROWS`.
**B · Transform-enabled coverage (new core capability + §06 extension)** _(done)_
- **B5 · Aggregation** _(done)_ — per-channel `sum` / `mean` / `median` / `min` / `max`, plus
a field-less "Count of records" measure (Voyager's `count(*)`). The priority item.
- **B6 · Binning** _(done)_ — `bin` on a quantitative field → true histograms (closes the
Distribution gap); mutually exclusive with aggregate on the same field.
- **B7 · Stacking** _(done)_ — `stack` (`zero` / `normalize`) for bar/area + a Color series →
part-to-whole (closes that gap; enables 100%-stacked).
- **Temporal granularity** _(done)_ — Vega-Lite `timeUnit` (Year / Quarter / Month / Week /
Day / Hour, plus combined units) on a Temporal field; defaults to None (raw).
- **B8 · Faceting (Row / Column → small multiples)** _(next increment, after A+B + UI land)_
— two more channels that multiply the chart into a trellis, the clean way to compare many
categories (Voyager has it; FT/Datawrapper recommend small multiples; we currently can't
express them). Still mark-first, so a Tier-B extension. **Axis alignment** is the design
crux: Vega-Lite facets default to **shared scales** (aligned axes) — keep that as the
default; expose an "independent axes" toggle (`resolve.scale`) only as an advanced option.
**Verify** faceting against the preview's `"container"` fit modes before trusting it
(per-cell sizing on facets is finicky). Sequenced as additive after the current build.
**C · Intent-first front door (Tier C)** _(deferred)_ — see §5. "What do you want to
show?" → recommend mark + channels from the FT/Datawrapper taxonomy × column types; where
Munzner + Wilke would be seated.
**D · Plumbing** — **D9** URL hash routing for the open builder (owned by **M6**, spec
§01E); **D10** a GOV.UK/NN-g copy pass over the guidance-hint wording (the M4 council
seating's residual one-off debt).
---
_Citations are to files under `/Users/oleh/code/reference/`. The seated chart-choice
canon (FT clone + Datawrapper distill) lives in the council roster
(`.claude/skills/council/SKILL.md`); Draco/Voyager are reference clones, not council
seats — they're engineering sources, not user-facing design authorities._
docs/exploration/chart-theming-scope.md
+340
View File
@@ -0,0 +1,340 @@
# Chart Theming — Enhancement Scope
> **Status:** scope consolidated 2026-06-12. Single forward-looking home for chart-theme
> work: separating the opinionated house style from the legibility minimum, a preview
> theme selector, config merge/extract, custom named themes, and fonts (shipped roster +
> user-loaded). Read against `src/core/vega-themes.ts`,
> `src/app/services/chart-renderer.ts`, and `docs/architecture/05` §3.
>
> **Goal (the brief):** "here's how you can easily transform your Vega-Lite charts to not
> look like stock Vega-Lite charts" — make the house style one option among several,
> let a user apply custom branding (colors **and fonts**) quickly, and keep every byte
> self-hosted and offline-capable.
---
## 1. Where we stand (the audit)
Everything opinionated lives in **one file** — `src/core/vega-themes.ts` — injected as
vega-embed's `config` option at embed time (`chart-renderer.ts`). It is never baked into
the stored spec; pasting a snippet into the Vega editor renders stock. No CSS reaches
into the chart DOM. Exports (PNG/SVG) render through the same view, so they **carry the
theme**.
Merge precedence (verified in `vega-lite/src/compile/compile.ts`):
`mergeConfig(opt.config, spec.config)` — **the spec's own `config` wins** over our
injected theme, property by property. A snippet can already opt out of any of it.
Exact diff vs. stock Vega-Lite (defaults read from `vega-parser/src/config.js`):
| Property | Astrolabe (light / dark) | Stock Vega-Lite |
| -------------------------- | --------------------------- | -------------------- |
| `background` | transparent | `white` |
| `font` | IBM Plex Sans stack | `sans-serif` |
| `title` | 16px / 600 / app text color | 13px / bold / black |
| `axis.domainColor` | `#c6c6c6` / `#525252` | `#888` |
| `axis.gridColor` | `#e0e0e0` / `#393939` | `#ddd` |
| `axis.gridDash` | `[2,2]` | solid |
| `axis.labelColor` | `#525252` / `#a8a8a8` | black |
| `axis.titleColor` | `#161616` / `#f4f4f4` | black |
| `axis.label/titleFontSize` | 11 / 12 | 10 / 11 |
| `axis.titleFontWeight` | 600 | bold (700) |
| `range.category` | Carbon data-viz 14-color | tableau10 (10-color) |
| `view.stroke` | transparent | `#ddd` plot border |
Untouched: everything else — notably the **default mark color stays Vega blue
`#4c78a8`**; the Carbon palette only kicks in once a color encoding exists.
The config splits into two layers with different standing:
- **Base (legibility/integration)** — required for charts to be readable on our panes at
all, dark mode especially: `background: transparent` + the guide _colors_ (stock black
text on a dark pane is illegible). Structurally the same job `[data-theme]` does for
the rest of the app.
- **Expressive (house style)** — genuinely opinionated: Plex, the Carbon categorical
palette, dotted grid, bumped guide sizes/weights, 16px title, no plot border. Strip it
and charts still work in both UI themes; they just look like Vega-Lite.
## 2. What vega-editor does (and what we take)
Read from the local clone (`reference/vega-editor`, `components/config-editor/`):
- **Theme dropdown = the `vega-themes` npm package** (~14 preset configs: excel,
ggplot2, fivethirtyeight, latimes, powerbi, googlecharts, urbaninstitute, dark, four
Carbon themes) + a `custom` sentinel. Already in our tree — vega-embed depends on it.
- **Picking a theme is a one-shot copy** of the preset JSON into a config editor pane;
any hand-edit flips back to `custom`. No live binding.
- **The config pane** feeds `opt.config` at compile — the slot we already use. Two
Monaco context-menu commands bridge pane ↔ spec: **Merge Config Into Spec** (pane →
`spec.config`, spec's existing keys win, pane empties) and **Extract Config From
Spec** (the inverse).
- **No custom-theme saving.** One global localStorage state blob; `custom` is "whatever
is in the pane". Nothing to borrow for named themes — that part is our own design.
The structural mismatch: vega-editor is a scratchpad for one transient document;
Astrolabe is a library. **Decided 2026-06-12:** theme choice is **not per-snippet** —
`spec.config` _is_ the per-snippet mechanism, and merge/extract makes it ergonomic. The
app-level selector is a global preference.
## 3. Fonts (researched 2026-06-12)
**The hard constraint:** `vega-scenegraph/src/util/text.js` measures every label via
canvas `measureText` **regardless of renderer**. A font that finishes loading after
embed leaves the whole layout measured with fallback metrics. Any custom-font path must
`await document.fonts.load('<weight> 11px "Family"')` per used face **before**
`renderSpec`. Once loaded, SVG view, canvas view, and PNG export all work for free.
**Known limitation:** SVG _export_ carries only the family name — a viewer without the
font sees fallback (industry standard; data-URI `@font-face` embedding is a heavy
maybe-later).
**Shipped roster (self-hosted, no CDN — same `@fontsource` mechanism as Plex).**
Measured latin woff2 sizes (jsdelivr, 2026-06-12): regular text faces run **13–25KB per
weight**; handwriting (Caveat) ~50KB. A ~9-family roster at ~2 weights ≈ **400–450KB
latin**. All-subsets multiplier ≈ 3–4× (Inter: 87KB all-subsets vs 23KB latin, one
weight). Current dist is 7.8MB with 412KB of Plex — the roster roughly doubles font
payload; acceptable.
Candidate roster (final pick deserves a visual specimen pass, not a chat decision):
| Role | Faces (weights) |
| --------------------- | ----------------------------------------------------------------------------------------------------- |
| Already shipped, free | IBM Plex Sans, IBM Plex Mono |
| Dataviz sans | Inter (400/600), Roboto Condensed (400/600), Libre Franklin (400/600) |
| Brand-coherent | IBM Plex Serif (400/600), IBM Plex Sans Condensed (400/600) |
| Editorial serif | Source Serif 4 _or_ Spectral (400/600) |
| Exotic / display | Space Grotesk (400/600), Playfair Display (400/700), Caveat (400/600, "sketch"), Space Mono (400/700) |
**Subset/precache strategy:** chart fonts are decoration with automatic per-glyph
fallback (`unicode-range`), not app capability — non-latin data labels falling back to
the system font is degraded styling, not a broken app (contrast the Plex Cyrillic
lesson, which was UI capability). Plan: **ship all subsets in dist** (~1.2–1.5MB dist
growth), **precache latin only** (~+400KB), runtime-cache the remaining subsets
same-origin (CacheFirst) so a used subset persists offline after first render.
**User-loaded fonts (the branding case — primary).** Real brand fonts are licensed and
usually _not_ on Google Fonts. Path: upload woff2/ttf → bytes in IndexedDB (the
datasets persistence pattern) → `new FontFace(family, bytes)` + `document.fonts.add()`
at startup and before render. Fully local, offline-native, no privacy question.
**Google Fonts CDN tier — deferred, opt-in only.** Verified: keyless catalog at
`fonts.google.com/metadata/fonts` (1,936 families; a names-only list is ~30KB raw, so
the _picker_ can ship static and offline), CSS2 endpoint live, Workbox CacheFirst on
`fonts.gstatic.com` makes a chosen font offline after first use. Tension: `base.css`
says fonts are "never a CDN", and font requests expose the user's IP to Google. If this
ships, it is an explicit per-font user action, never automatic.
## 4. Build order
1. **Layer split** ✅ (refactor, no visible change) — `vega-themes.ts` is base +
expressive per UI theme, merged into the existing exports via `mergeChartLayers`.
2. **Preview theme selector** ✅ — global pref `ui.chartTheme`: **Astrolabe** (follows
UI theme, default) · **Stock Vega-Lite** (empty config) · all 14 vega-themes presets ·
(later) custom themes. Governs LivePreview **and export** (same view). Resolved
design points: the control is a `SelectControl` in the **preview header** (a select
nested inside PreviewSettings would close its own parent — SelectControl and
SettingsPopover share the one-open-popover registry); Onboarding/Chart-Builder
previews stay house-styled; preset/stock backgrounds render verbatim (a white chart
card on the dark pane is an honest destination preview).
3. **Merge/extract config** ✅ — `core/spec-config.ts` (`mergeConfigIntoSpec`,
`extractConfigFromSpec`), surfaced as the editor toolbar's **Config** menu
(SelectControl action picker — council: NN/g #6, Carbon overflow; arch 10 §5 records
the rule) with Monaco context-menu/palette as accelerators on the same functions
(spec §03G): bake the active theme into `spec.config` (existing keys win,
render-identical), or lift `spec.config` out to the clipboard (copy before remove —
a failed copy aborts).
4. **Custom named themes** ✅ (2026-06-12) — IndexedDB entity
`{ id, name, config }` (`core/custom-theme.ts`, themes store @ DB v2) + the **Theme
Builder** modal: theme list, JSON config editor, a font control that populates one
family across every font slot (`applyFontToConfig`), and a live multi-chart gallery
(`core/theme-preview-specs.ts`) so one edit is previewed across titles, axes,
legends, headers, and the major marks. Created by duplicating the currently-selected
theme (house/preset/custom) or via the editor's **Extract Config to New Theme**
action (spec §03G); appears in the selector as `custom:<id>` (the "Edit themes…"
action row sits right after the customs, before the preset roster); deleting the
active one falls back to Astrolabe. Custom themes travel in the §08 workspace
export/import envelope (additive `themes` array, name auto-suffix on clash, ids
reassigned by the store, rolled back with datasets on a failed import).
5. **Shipped font roster** ✅ (2026-06-14) — 11 self-hosted families via @fontsource
(`styles/chart-fonts.css`, full subsets bundled) extending `THEME_FONT_OPTIONS` to 17
entries; `collectFontFamilies` (core) + a `document.fonts.load` gate at the top of
`renderSpec` (before the layout/probe pass, which measures text regardless of
renderer); Workbox precaches the `latin` subset of the roster (~520KB) plus every
subset of the UI Plex Sans/Mono, and runtime-caches the rest (latin-ext + non-latin)
CacheFirst so a script works offline after first use. Roster picked from a visual
specimen. Not done here: theme↔font pairing metadata (a suggestion nicety, deferred).
6. **User font upload** — FontFace-from-IndexedDB tier; theme entity's `fonts` field
carries `{ family, source: 'file' }`.
7. **Deferred** — Google Fonts opt-in tier; SVG export font embedding; built-in
expressive preset gallery ("Editorial", "Terminal", "Sketch") showcasing the roster.
**Rejected:** per-snippet theme field (2026-06-12 — `spec.config` + merge/extract covers
it without a second mechanism).
## 5. Structured controls (slice 4b)
The builder today is a raw JSON textarea + one font dropdown + the live gallery. Slice 4b
adds a strip of structured controls above the editor — accelerators that write into the
JSON, never replacing it. The JSON stays the source of truth and the full-power escape
hatch; controls cover the common ~80% (color, type, spacing, grid), not all 72 config
properties (that is the trap vega-editor deliberately avoids by staying JSON).
**Hard constraint — the builder must preserve unknown keys.** Verified by compiling: the
vega-themes presets carry Vega-_layer_ keys (`symbol`, `shape`, `path`, `group`) that are
not in the Vega-Lite `Config` schema, and Vega-Lite forwards the whole config to Vega
unchanged — they take effect. So a structured control must **merge into** the existing
config (immutable path-set that spreads siblings), never rebuild it from a closed
schema-typed model, or it silently drops those keys on a round-trip. Same shape as
`applyFontToConfig`, which walks and rewrites rather than reconstructing.
**Resolved design points:**
- **Surfacing — inline tab strip** (not popovers, not sub-modals). Tabs (Color / Type /
Layout / Axes & grid / Legend) sit between the toolbar and the JSON+gallery, all in the
one xlarge modal. No nested overlays/focus traps, no contention with the one-open-popover
registry, and panels + JSON + gallery stay visible together.
- **Color model — scheme picker that materializes to swatches.** A `range` family takes
either an explicit color array or a named scheme written as Vega's range-scheme object
`{ scheme: name }`. (A bare scheme-name _string_ passes vega-lite compile but Vega
rejects it at render — "Unrecognized scale range value" — so the controls write the
object form, read either, and `normalizeRangeSchemes` heals the bare form at render.)
Pick a named scheme for the quick path; "materialize" expands it to an editable swatch
array for brand tuning. Catalog ships 15 categorical + 24 sequential + 10 diverging
schemes; categorical schemes resolve to arrays, continuous ones to interpolators sampled
into stops for the gradient preview and the materialize action.
- Structured controls are gated on valid JSON (same as the font control): a parse error
disables them and the textarea is the fix.
- Controls write **minimal** config — clearing a value deletes the key rather than writing
a default, so a theme stays a diff against stock, not a full dump.
**Panels:** Color (`range.category` swatches/scheme, `mark.color`, `range.heatmap`/`ramp`/
`diverging`) · Type (base `font`, title + axis title/label size+weight) · Layout (`background`
incl. transparent, `padding`, `view.stroke`/`fill`/cornerRadius) · Axes & grid (grid on/off +
color + dash, domain, label color/angle, title color — base `axis` only; the 25 variants stay
JSON) · Legend (orient, title/label color+size, symbol size). Legend _type_ (size) lives in
the Legend panel rather than Type, so every legend property a brand tunes sits together.
**Build order:** (a) core foundation — scheme catalog + immutable config path get/set +
`schemeColors` materialize, with tests; (b) Color panel (highest payoff); (c) Type, Layout,
Axes, Legend panels; (d) wire the tab strip into the modal.
**Not in slice 4b:** the house style's own gaps — no `mark.color` (single-series charts stay
Vega-blue), unset legend/header/padding — are left for a separate house-style redo, not
papered over here. Minor cleanup noted: `theme-preview-specs.ts` declares `$schema` v5 while
the app standardizes on v6.
## 6. Status log
- **2026-06-14 (Color panel bugfix)** — **scheme picks rendered blank.** A named scheme
was written into `config.range.*` as a bare string, which vega-lite compiles but Vega
rejects at render ("Unrecognized scale range value") — silently caught by the gallery's
per-card try/catch, so the categorical/sequential/diverging charts blanked the moment a
scheme was picked. Predates this session's panels/fonts (shipped with the Color panel).
Fix: the controls write Vega's range-scheme object `{ scheme: name }` and read either
form; `normalizeRangeSchemes` (core) heals a bare-form config at the render-resolution
points (`chartConfigForSelection` for the live preview/export, and the builder gallery),
so themes saved/imported with the old form self-heal. The gallery's catch now surfaces the
error message in the card (fail-loud, arch 02) so a render failure on valid JSON isn't
invisible again. Regression cover: a real
vega-lite→vega compile/parse/run asserting `{ scheme }` renders and the bare string
throws, plus `normalizeRangeSchemes` unit tests. Verified: typecheck, lint, tests (983).
- **2026-06-14 (slice 5)** — **shipped font roster.** 11 self-hosted families
(`styles/chart-fonts.css`, imported in main.tsx, separate from the UI Plex in base.css):
Inter · Libre Franklin · Roboto Condensed · IBM Plex Sans Condensed · IBM Plex Serif ·
Source Serif 4 · Spectral · Space Grotesk · Playfair Display · Caveat · Space Mono, at
400 + 600 (Space Mono 400 + 700). `THEME_FONT_OPTIONS` grew to 17 (roster grouped by
role, then the system stacks); each roster stack carries a category fallback. The render
path now gates on fonts: `collectFontFamilies` (core, the read-counterpart of
`applyFontToConfig`; skips `data`/`datasets`) gathers the families a spec+config use and
`renderSpec` awaits `document.fonts.load` for them before the first layout pass — Vega
measures text via canvas `measureText` regardless of renderer, so a face loading after
embed would lay out with fallback metrics. Best-effort + 3s-capped so a slow first fetch
never freezes the preview. Precache strategy (vite.config Workbox): the `latin` subset
of every family (~520KB for the roster) + all Plex Sans/Mono subsets (UI capability) are
precached; latin-ext and non-latin scripts are runtime-cached CacheFirst (`*-latin-[0-9]*`
excludes latin-ext; the Plex Sans brace-list avoids matching the condensed roster font).
Verified: typecheck, lint, full tests (976), production build + precache-manifest
inspection. Note: @fontsource ships legacy `.woff` beside `.woff2`; modern browsers use
woff2, so the `.woff` sit unused in dist (pre-existing for Plex — neither precached nor
runtime-cached).
- **2026-06-14 (slice 4b complete)** — **Layout / Axes & grid / Legend panels + Type
size/weight.** The remaining structured-control panels, built on a small shared
primitives module `ThemeFields.tsx` (`ControlSection`, `ColorRow`, `NumberRow`,
`SelectRow`) so the panels read declaratively and match the Color panel's look. Each
control writes one config path through the same inline `mutateDraftConfig` +
`setConfigValue` the Color panel uses, with the minimal-diff delete (clearing a value
removes the key, pruning emptied objects). Leaf coercion (`asString`/`asNumber`/
`asBoolean`) moved into core `theme-controls.ts` beside the path get/set, tested there.
Panels: **Layout** (background and `view` fill/border as tri-state default·transparent/
none·custom, corner radius, scalar padding with a JSON hint when it's a per-side object);
**Axes & grid** (grid visibility/color/dash-preset, domain/label/title color, label
angle — base `axis` only); **Legend** (orient, title/label color+size, symbol size);
**Type** rounded out with title and axis title/label size+weight (font family relocated
into the extracted `TypeControls`). Resolved while building: each generic row label
("Size", "Color", "Weight") repeats across sections, so `ControlSection` is a
`role="group"` labelled by its heading and rows take an accessible-name override — the
visible label stays short, the control's announced name is qualified ("Title size"). The
font-roster decision (slice 5) was teed up with a throwaway visual specimen. Verified:
typecheck, lint, full tests (969). Remaining in 4b: swatch reorder (Color panel).
- **2026-06-14 (slice 4b, first increment)** — **structured-control foundation + Color
panel.** Core `theme-controls.ts`: immutable config path get/set (preserves siblings —
the Vega-layer-key guarantee — and prunes on delete) + the named-scheme catalog (15
categorical / 24 sequential / 10 diverging) + `schemeColors` resolution (categorical
arrays passthrough, continuous interpolators sampled to hex), all tested. `vega-scale`
added as a declared dep (focused sub-package, like `vega-expression`) with a typings
shim in `vite-env.d.ts` (its package.json `exports` omits `types`). Store gains the
generic `mutateDraftConfig(fn)` write path; `applyDraftFont` refactored onto it. Modal
gains an APG tab strip — **Color** (categorical scheme/swatches + materialize, default
`mark.color`, sequential/diverging gradient pickers) and **Type** (the relocated font
control). Tabpanel gated on valid JSON. From first-use feedback, same day: the modal
body is now **controls + JSON on the left, gallery as a full-height right rail** (the
previews were starved before); `SelectControl` gained an optional per-option `preview`
so the scheme dropdowns show swatch strips (categorical) / gradient bars (continuous);
every swatch is a reusable `SwatchRow` (color picker + copyable/editable hex field); and
sequential/diverging gained **Materialize → editable stops**, so custom gradient colors
are possible, not just named schemes. Second feedback pass: the raw JSON is now a
**collapsed disclosure** at the bottom of the controls column (it was eating half the
first screen), forced open only on a parse error; the structured controls fill the
column. `SelectControl` options gained a `labelStyle`, so the **font dropdown renders
each name in its own family** (the type analogue of the color swatches) and its trigger
shows the current font in-face. Verified: typecheck, lint, full tests (950).
Remaining: Layout / Axes & grid / Legend panels; swatch reorder.
- **2026-06-12 (slice 4 close-out)** — **custom themes in the §08 envelope.** The
workspace export now writes a `themes` array (additive — no format bump; importers
treat it as optional, so pre-theme envelopes stay valid). Import normalizes each
record (`normalizeCustomTheme`), auto-suffixes name clashes via the generalized
`dedupeIncomingNames` (the dataset dedupe, now shared), reassigns ids through
`CustomThemeStore.addThemes` (selection untouched), and rolls themes back together
with datasets when the atomic snippet write fails. Toast counts gain a theme
clause. Spec §08 updated ("Dataset conflicts" → "Name conflicts"). Slice 4 is now
fully done; next is slice 5 (shipped font roster).
- **2026-06-12 (slice 4)** — **custom named themes + Theme Builder shipped.**
`CustomTheme` entity through the full stack (core → theme-store @ DB v2 →
CustomThemeStore → theme-persistence → startup hydrate); selection model extended to
`custom:<id>` with missing-record fallback to the house style; Theme Builder modal
(xlarge, list + name + config JSON + font-apply control + 7-card live gallery, canvas
renderer, per-card chain-lock); picker gains custom entries + an "Edit themes…"
action row. The font control ships with render-safe faces only (Plex + web-safe
stacks) — the roster slice (5) extends `THEME_FONT_OPTIONS` and adds the
`document.fonts.load` gate. Spec updated (§01C, §04 Chart theme + Theme Builder,
§09C/E/G) + architecture 05 §3. Same-day follow-ups from first use: `openDB` now
verifies the store layout and self-heals an interrupted upgrade (arch 02 §2.1,
`db.test.ts` on fake-indexeddb); "Edit themes…" moved before the preset roster
(discoverability); **Extract Config to New Theme** added as the third Config-menu
action (spec §03G) — config block → saved theme, selected, removed from the spec.
Not yet done: themes in the §08 export/import envelope.
- **2026-06-12 (slices 2–3)** — **theme selector + merge/extract shipped.**
`ChartThemeId`/`chartConfigForSelection` in core; `ui.chartTheme` persisted via the
`previewFitMode` orchestration pattern; `SelectControl` picker in the preview header;
LivePreview renders (and therefore exports) with the selection; `core/spec-config.ts`
merge/extract behind two Monaco editor actions. Spec updated (§03G, §04 Chart theme,
§07, §09C) + architecture 05 §3 rewritten to the layered/selectable model. Verified:
typecheck, eslint, full tests (802), build. Custom named themes (slice 4) and fonts
(5–6) remain.
- **2026-06-12** — scope written; audit, vega-editor read, font research done (numbers
above). Slice 1 (layer split) implemented.
docs/exploration/data-inspector-exploration.md
+64
View File
@@ -0,0 +1,64 @@
# Data Inspector — exploration (future direction)
> **Status:** parked idea (2026-06-11). A deliberate divergence from the Phase-1
> chart-builder scope, recorded here so it isn't lost. **Not built.** The shipped
> seed is the builder's 1D data preview (raw source rows); this is its transform-aware
> evolution, generalized to the main editor.
## What
A vega-editor-style **data inspector**: show the data tables the chart actually uses at
runtime — _after_ the spec's transforms run — not just the raw source rows. Two homes:
1. **Chart Builder** — make the existing data preview (1D) transform-aware: show the
filtered rows and any calculated columns, so the preview reflects the `transform`
block the builder emits. (Today 1D shows the **raw dataset rows**, pre-transform — an
"inspect the source" view. This adds the "inspect the result" view alongside it.)
2. **Main window** — a **togglable data panel below the Live Preview** in the editor,
for any snippet (not only builder-made ones): inspect each named dataset / the
post-transform rows while authoring a spec by hand.
## Why
vega-editor surfaces every internal data table below the chart (its "Data Viewer"), and
it is the primary way to debug "why is my chart empty / wrong" — you look at what the
transforms produced. Astrolabe has no equivalent today: the only data views are of the
**raw source** (the Datasets manager, and builder 1D). Seeing the **resolved** data
closes that debugging gap and is useful well beyond the builder.
## Technique
Vega exposes runtime datasets via `view.data(name)` — the rows of any named dataset in
the compiled dataflow, including the root data _after_ transforms. The renderer already
owns the only `view` reference behind `RenderHandle` (`src/app/services/chart-renderer.ts`);
add a `RenderHandle.listData()` / `data(name)` accessor that wraps `view.data(...)`,
mirroring how `toImageURL` wraps `view.toImageURL` — so no component touches the raw
view and the embedding boundary holds (architecture/05 §1–§2).
Vega-Lite compiles user transforms into named datasets (e.g. `source_0`, `data_0`);
mapping those to friendly labels and choosing which to expose is the main design work.
## Scope notes / open questions
- **Read-only**, like 1D — it inspects, never mutates (the "moderately capable" ceiling:
a debugging aid, not a data editor).
- **Big datasets** — cap the rows shown (reuse the 50-row preview cap). The view holds
every row; the table must not.
- **Which tables to list** — just the post-transform root/result, or every internal
dataset (the vega-editor dropdown)? Start with the result; expose the rest behind a
selector if it earns its place.
- **Main-window placement** — a third stacked region inside the preview pane (toggle +
persisted visibility, like the existing pane toggles) vs. a tab over the chart. Wants a
small interaction pass (architecture/10) before building.
- **Refresh timing** — the data is only available once a view has rendered; the panel
reads from the live `RenderHandle` and updates when a new render resolves (it must not
read a finalized view — same generation discipline as the preview).
## Relationship to shipped work
- Seed: builder **1D** data preview — see
[`chart-builder-enhancement-scope.md`](./chart-builder-enhancement-scope.md) (§3 1D).
- Renderer/preview home: [`architecture/05`](../architecture/05-rendering-theming-preview.md)
(the `RenderHandle` boundary the `view.data` accessor extends).
- Behaviour, once built, is specified in `docs/spec/04` (Live Preview) and `docs/spec/06`
(the builder preview), per "spec follows code".
docs/exploration/lyra-review.md
+391
View File
@@ -0,0 +1,391 @@
# Lyra — Repository Review & Improvement Ideas
> **Status:** review complete (2026-06-10). Source: `vega/lyra` cloned to
> `reference/lyra` (branch `lyra2019`, last commit `fb284bf`, 2021-05-14 — unmaintained).
> **Purpose:** mine Lyra, UW IDL's _direct-manipulation_ Vega design environment, for
> interaction patterns that could improve Astrolabe's chart-building experience. This is a
> companion to [`chart-builder-research.md`](./chart-builder-research.md) (which already
> seats Voyager, also from UW IDL); Lyra is the _no-code authoring_ sibling we hadn't read.
> **Bottom line:** take Lyra's **interaction primitives and data-context UI**, not its
> architecture. Astrolabe's chart-choice intelligence (Tier B) is already _ahead_ of Lyra;
> the gaps Lyra exposes are about **how the user touches fields and sees their data**, plus
> **making our guidance actionable**.
---
## 1. What Lyra is, and the one fact that decides what transfers
Lyra lets you build a custom visualization **without writing code** — drag data fields onto
graphical mark properties, position marks with connectors, resize with handles, and define
interactions by demonstration. It targets **raw Vega** (not Vega-Lite), is a React + \*\*Redux
- Immutable.js** app on a **d3 v3\*\* runtime, and is a 2019-era research prototype that was
never finished ("does not contain all functionality" — README) and has been dormant since
2021.
The single most important architectural fact:
> **Lyra's source of truth is a _decomposed GUI model_; the Vega spec is a one-way _export_.**
Lyra keeps the design as a Redux store of typed primitives — `Mark`, `Scale`, `Guide`
(axis/legend), `Pipeline`/`Dataset`, `Signal`, `Interaction`, `Widget`
(`src/js/store/factory/*`) — and `ctrl/export.ts` (`exporter()`, 546 lines) serializes that
store **into** a Vega spec on demand. There is no inverse: Lyra **cannot import and edit an
arbitrary Vega spec**. The GUI _is_ the document.
**Astrolabe is the exact inverse.** Our source of truth is the **JSON spec** (a snippet);
the builder is a _view that emits_ JSON (`buildChartSpec`/`buildSnippetSpecText` in
`src/core/chart-builder.ts`), and the user can always drop to Monaco and hand-edit. For a
**snippet manager** this is the right call — Lyra's model would forbid the round-trip our
whole product depends on. So we take **none of Lyra's architecture** and **all of the
transferable lessons are at the interaction layer**, where the GUI-vs-JSON question doesn't
matter.
This also reframes the comparison: Lyra is not a "better chart builder" to catch up to. On
**chart-choice intelligence** Astrolabe is already further along — Lyra has no
recommendation logic at all (that was Voyager's job, which we already mined). Lyra's value to
us is the parts of the _authoring experience_ we haven't built: field-first interaction, live
data context, and one-click assistance.
---
## 2. Where Astrolabe already leads Lyra (so we don't chase the wrong things)
| Dimension | Lyra | Astrolabe (today) |
| ------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Chart-choice guidance | none — manual mark + manual encoding | Tier B: smart default mark, valid-type locking, Size discipline, non-blocking warnings (`builderWarnings`) |
| Transforms in the builder | data-pipeline only (filter/formula/aggregate/lookup), not per-encoding | per-channel Aggregate / Bin / `timeUnit`, chart-level Sort / Stack |
| Target language | raw Vega (verbose, low-level) | Vega-Lite (the right altitude for "pick mark + channels") |
| Import / round-trip | impossible (GUI is the doc) | native — JSON is the doc, builder emits it |
| Stack | d3 v3 / Redux / Immutable / class components (stale) | React + Zustand + CSS Modules (current) |
The takeaways below deliberately **avoid** re-building anything in the right column.
### 2.1 — Counterpoint: Lyra exposes a far larger _control surface_
On chart-choice _intelligence_ we lead, but on **raw number of direct controls** Lyra is well
ahead. Selecting a primitive (mark / scale / axis / legend) opens an inspector of literal
knobs — and the inventory dwarfs our builder's:
| Primitive (`components/inspectors/*`) | Lyra's direct controls |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Mark — **Point** (`Symbol.tsx`) | x, y, **shape**, size, fill **color**, fill **opacity**, stroke color, stroke width |
| Mark — **Bar/Rect** (`Rect.tsx`) | X extent (start/center/end _spatial preset_), Y extent, fill color/opacity, stroke color/width |
| Mark — **Line** (`Line.tsx`) | x, y, stroke color/width, **interpolate (curve)**, tension |
| Mark — **Area** (`Area.tsx`) | baseline extent, fill color/opacity, stroke, interpolate, tension, orient |
| Mark — **Text** (`Text.tsx`) | text (expr/template), **font face/size/weight/style**, color, opacity, x/y, dx/dy offset, **align**, baseline, **rotation** |
| **Scale** (`Scale.tsx`) | **type** (linear/log/time/ordinal/band/point), **zero/nice/clamp/reverse/round**, **domain** (auto fields or manual min/max / value list), **range / colour scheme** (tableau, category10/20, sequential, diverging), padding(Inner/Outer) |
| **Axis** (`Axis.tsx`) | orient, axis-line colour/width, **title text + font/size/colour/offset**, **label size + angle + colour**, **grid on/off** + colour/opacity/width, **tick count/size/colour** |
| **Legend** (`Legend.tsx`) | orient, border, title text/size/colour, label font, symbol shape/size/fill/opacity, gradient |
And the structural multiplier: **every mark property is _droppable_** — each is _either_ a
literal value _or_ a bound field. Lyra therefore has **no fixed channel set**; you can bind
data to strokeWidth, opacity, shape, fontSize, dx… Our builder caps at **X / Y / Color /
Size** and exposes **zero** styling, scale, axis, or legend controls.
**Why this is not simply "Lyra is more capable":**
1. **It's a _builder-surface_ gap, not a capability gap.** Everything above, Vega-Lite
expresses too (mark config, `scale`/`axis`/`legend`, channels like `opacity`/`shape`/
`theta`/`row`/`column`). In Astrolabe you reach it by **hand-editing the JSON in Monaco**.
Lyra had **no editor and no round-trip** — the GUI was the _only_ way to touch the spec, so
it was forced to surface every knob. We split the work on purpose: a small **guarded**
encoding builder + a first-class JSON editor for the long tail.
2. **Lyra's breadth carries zero intelligence and high cost** — no defaults, no valid-type
locking, no guardrails, on raw low-level Vega, in an unfinished inspector.
3. So the question is not "match Lyra's control count" but **"which slices of that surface are
common _and_ awkward enough in JSON to deserve promotion into the guarded builder?"**
**Slices worth promoting (ranked):**
- **Constant-value styling** via the value-or-field channel (§3.3): fixed fill colour /
opacity / point shape / line curve without touching JSON — the most common "tweak the look."
- **Colour-scheme picker** on the Color channel (categorical / sequential / diverging) + a
measure-axis **`zero` / `log`** toggle. Palette changes are common and fiddly in JSON.
- **A few more channels** — `opacity`, `shape`, `tooltip` (already auto-enabled), **`theta`**
(unlocks pie/donut → a true part-to-whole), **`row`/`column`** facets (backlog **B8**). These
_are_ the field-shelf model (§3.4): adding channels and adding the shelf are one project.
- **Light axis/legend text** — custom **axis title**, **legend title / hide**. Common, low-risk.
**Guardrail:** promote a control only when it is _both_ common _and_ awkward in JSON.
Otherwise the clean guarded builder drifts toward Lyra's fiddly everything-inspector — the
very complexity that helped leave Lyra unfinished. Styling/scale/axis breadth for its own sake
belongs in Monaco, not the builder.
---
## 3. High-value ideas to adopt (ranked, in-scope, mapped to our code)
### 3.1 — Make `builderWarnings` **actionable** (Lyra's `Hints`) — _highest value, smallest lift_
Lyra's hint system (`components/hints/Hints.tsx`) is a contextual nudge that carries an
**action**: `{title, text, action, action_text}` — a one-click button that _applies the fix_
and clears the hint. Our guidance is strictly advisory text today: `BuilderWarning`
(`chart-builder.ts:333`) renders as a passive hint with no affordance.
Several of our existing warnings have an obvious one-click remedy and should become
**suggested actions** (Carbon "inline notification with action" / NN-g "make the system do the
work"):
- _"…draws one mark per row; aggregate the measure"_ → **[Aggregate as Sum]** sets the
measure channel's `aggregate`.
- _"…long labels; flip to a horizontal bar"_ → **[Swap X/Y]** (we already have the action —
just wire it to the hint).
- _"Two measures usually read better as a scatter"_ → **[Switch to Point]**.
- _"Area split into many colour series…"_ → **[Stack]** or **[Remove colour]**.
This keeps the Tier-B "guarded, non-blocking" philosophy — the action is an _offer_, never a
forced change — while turning a wall of advice into a guided improvement loop. Concretely:
extend `BuilderWarning` with an optional `fix?: { label: string; apply: (c) => BuilderConfig }`
(pure, lives in `chart-builder.ts`, fully unit-testable), and render the button in
`ChartBuilderModal`. **Run the copy + affordance through `/council`** (it auto-fires on
guidance copy and new interactive affordances).
### 3.2 — A live **data-table preview** in the builder (Lyra's `DataTable`) — _closes a real gap_
Confirmed gap: neither `ChartBuilderModal` nor `DatasetsModal` ever shows the **actual data
rows**. The builder picks columns from a dropdown and renders a _chart_ preview, but the user
never sees the values they're encoding. Lyra always shows the data — `DataTable.tsx` is a
paged, scrollable grid with a **per-field type icon** in each column header and hover
inspection.
A compact, read-only **data preview** (first N rows, per-column type chip in the header)
would let the user sanity-check "is this column really a date / really numeric" _before_
building — exactly when type inference is most likely to surprise them. Good homes:
- a collapsible "Data" strip in the builder's left pane (under the dataset name), and/or
- the Datasets manager, where a dataset's shape is otherwise invisible.
We already compute per-column type + cardinality + extent in `profile.ts` (the A3/A4
profiling extension), so the header chips are free; we only need the row sample. Keep it
**read-only** — editing data is out of scope (and is where Lyra's pipeline complexity lives).
### 3.3 — The unified **Property** primitive: one control = value · field · scale (Lyra's signature UI)
Lyra's best idea is `components/inspectors/Property.tsx` — a **single droppable control** that
is, depending on binding state, _either_:
- a **literal value** editor (number / color / range / select / text — `FormInputProperty`), **or**
- a **field chip** (drag a column onto it; chip shows source-vs-derived, click to unbind), **or**
- a **scale chip** (the scale the field flows through; click to unbind).
Every mark inspector is then just a **declarative list** of these — e.g. `Symbol.tsx`
declares Position/Geometry/Fill/Stroke groups as ~10 `<Property>` lines. One primitive, one
drop-to-bind gesture, one unbind gesture, reused for _everything_ including the transform
expression fields (`Filter.tsx`, `Formula.tsx` reuse the same `Property` in autocomplete
mode).
For Astrolabe this is the conceptual model for a **richer channel row**: a channel is "a field
**or** a constant **or** (future) a datum," and Vega-Lite encodes exactly that distinction
(`field` vs `value` vs `datum`). Today our channel row is field-only (column dropdown +
type segmented control). Adopting the Property model would let a channel also hold a **constant
value** (e.g. a fixed color/size) with one consistent control and a chip showing the binding
kind. This is a medium lift and a genuine capability gain, and it generalizes cleanly if we
ever add channels beyond X/Y/Color/Size.
### 3.4 — Field-first interaction: a **field shelf** you drag onto channels (Lyra drop-zones + Voyager)
Both UW tools converge on **fields as the primary objects**: Lyra drags fields onto mark
properties; Voyager has a field list with type chips you add to channel shelves. Astrolabe is
**channel-first** (pick a channel, then choose its column). A **field shelf** — the dataset's
columns listed with type chips, dragged or clicked onto channels — inverts that to match how
people actually think ("I have these fields; where do they go?").
This is the largest interaction shift here and overlaps our **deferred Tier-C / faceting**
work (`chart-builder-research.md` §8 B8/C). Recommend treating it as the **interaction
substrate for Tier C**, not a standalone task: when we build the intent-first front door and
add Row/Column facet channels, a field shelf is the natural way to assign many fields across
many channels. Note now; sequence with Tier C.
### 3.5 — One-click **type cycling** on the field chip (Lyra's `FieldType`)
Lyra's `FieldType.tsx` is a tiny, nice touch: the field's **type icon is the control** —
click it and it cycles N→O→Q→T (within the field's valid set). We already show a type
indicator next to each column option and a full `N|O|Q|T` segmented control per channel. The
cheap win is making the **per-option type chip itself interactive** (and, in a future field
shelf, the chip on each field), so type is a property of the _field_ the user can toggle in
place — fewer controls, the same guard (only valid types, via our `validFieldTypes`).
### 3.6 — Inline **expression validation** via Vega's own parser (Lyra's transform technique)
Lyra validates a filter/calculate expression with `parseExpr` from `vega-parser` **before
committing** it to the model (`Filter.tsx`, `Formula.tsx`), catching malformed expressions at
the keystroke. We don't expose raw expressions in the builder, so this isn't an immediate
builder feature — but the **technique** (validate Vega/VL expression strings with the
library's parser and surface the error inline) is directly reusable for our Monaco editor's
diagnostics and for any future calculate/filter affordance. Worth recording in
`architecture/08` (vega-editor techniques) as a known approach; Lyra even left the
"_indicate error in parsing_" TODO unfinished, so we'd be completing the idea, not copying it.
### 3.7 — **Starter examples** gallery (Lyra's `ExampleGallery`) — _cheap, optional_
Lyra's toolbar has an `ExampleGallery` modal that `hydrate()`s a curated example into the
editor — a "start from something" affordance. Astrolabe's snippet library is the natural home
for a small set of **curated starter snippets** (one per FT intent category we cover —
Magnitude/Bar, Change-over-Time/Line, Correlation/Point…). Low effort, improves first-run and
doubles as living documentation of what the app does well. Lower priority than 3.1–3.2.
### 3.8 — **Per-chart export: image + standalone spec** (Lyra's `Export` menu) — _biggest miss; cheap_
The one squarely-useful idea I almost overlooked. Lyra's toolbar `Export` (`components/toolbar/Export.tsx`)
offers **PNG**, **SVG**, **JSON spec**, and a **standalone HTML** scaffold — using
`view.toImageURL(type)` on the live Vega view.
Astrolabe's export today is **workspace-JSON backup only** (`docs/spec/08`) — there is **no way
to get a single chart out** as an image, as its own spec file, or onto the clipboard. Yet our
renderer already holds exactly the Vega `view` Lyra exports from (`services/chart-renderer.ts`,
`renderer: 'svg'`), so this is a handful of lines:
- **Copy spec** (clipboard) and **Download `.vl.json`** for the active snippet — the most-asked
"get my chart out" actions, both trivial given the snippet _is_ the spec.
- **Download PNG / SVG** of the rendered chart via `view.toImageURL('png'|'svg')` — drop a chart
into a doc/slide without a screenshot.
- (Optional) **standalone HTML** — a self-contained vega-embed page; ties to the
share/"private-move" direction in `monetization-and-sync-exploration.md`.
Home: a snippet-level "Export / Share" affordance (library row action or editor toolbar),
distinct from the workspace Export. High value-to-effort; should jump near the top of the queue.
### 3.9 — Scale / axis / legend as **auto-derived _override_ panels** (Lyra's `bindChannel` pipeline)
If we ever promote scale/axis/legend controls (§2.1), Lyra shows the right _editing model_.
Binding a field in Lyra runs a pipeline (`actions/bindChannel/`): `parseScales` + `parseGuides`
**auto-materialize** the scale and axis/legend, `aggregateDependencies` re-points them when the
data is aggregated, and **`cleanupUnused`** garbage-collects scales/datasets no longer
referenced. The user edits a scale/axis only to _override_ an auto-derived default.
Vega-Lite already auto-derives scales/axes/legends from encodings, so we get the inference for
free — the borrow is purely UX: surface scale/axis/legend as **optional override panels that
start empty (inheriting VL defaults) and only emit JSON when touched**, and **clearing a channel
drops its overrides** (the `cleanupUnused` lesson — no orphaned `scale`/`axis` config left behind).
This is the discipline that keeps §2.1's promoted controls from bloating the output spec.
### 3.10 — Expression **autocomplete with field/signal chips** (Lyra's `AutoComplete`)
`components/inspectors/AutoComplete.tsx` is a `contenteditable` expression field that
autocompletes **dataset field names** (and signals), rendering each `datum.field` reference as a
styled, non-editable **chip**. For any future filter/calculate affordance — and for Monaco
completions in expression positions — autocompleting the dataset's own column names (we already
have the schema from `profile.ts`) is the high-value half; the chip rendering is polish. Niche
until we expose expressions, but cheap to remember.
### 3.11 — **Builder-state undo/redo** (Lyra's `redux-undo`) — _low priority_
Lyra makes the whole design undoable (`redux-undo`, global `vis.present`/`past`). Our builder
modal has **no undo** (Monaco covers the _editor_, not builder config). A session-scoped
undo/redo — or even a lighter **"reset to smart defaults"** — would soften experimentation in
the builder. Low priority: the builder is a short-lived modal and the smart default already
gives a sane starting point. Noted for completeness, not urgency.
### 3.12 — **Dataset-level transforms: Filter / Calculate** (Lyra's data pipeline) — _the cleanest gap_
The transform dimension where Astrolabe and Lyra differ most. The key fact is that **Vega-Lite
has two transform layers**:
1. **Inline encoding transforms** — `aggregate` / `bin` / `timeUnit` / `sort` / `stack` written
straight onto a channel. **This is the layer Astrolabe's builder uses**, and uses _well_ —
per-channel and guarded, arguably cleaner than Lyra (whose Aggregate is a separate pipeline step).
2. **Top-level `transform: []`** — dataset-level operations applied in sequence before encoding:
**`filter`**, **`calculate`**, **`lookup`**, window, fold, pivot, regression… **Astrolabe's
builder exposes _none_ of these.** Lyra's data pipeline is built almost entirely _around_ them
(`components/pipelines/transforms/`: Filter, Formula, Lookup, Sort — each an ordered card in a
`TransformList` on `datasets.<id>.transform`).
The operations in layer 2 have **no encoding-level shorthand** — there is no way to express them
except by hand-editing the JSON. The two worth borrowing:
- **Filter _(highest value)_** — row filtering (`transform: [{filter: …}]`). The single most
common data-shaping need, and the builder is _already telling users to do it_ — three existing
`builderWarnings` literally say "filter to fewer categories / to non-negative values"
(`chart-builder.ts:475,476,511`) while offering no way to. Guarded form: a **field + operator +
value** predicate shelf (Voyager-style), no expression required for the common case; power form:
a raw `datum.…` expression validated with Lyra's `parseExpr` technique (§3.6) + field
autocomplete (§3.10). VL applies top-level transforms _before_ encoding aggregation, so "filter
raw rows, then the chart aggregates" is the natural — and usually wanted — default; filtering on
an _aggregated_ value (HAVING) is the advanced case, defer it.
- **Calculate / derived field** — `transform: [{calculate: "datum.a / datum.b", as: "ratio"}]`,
exactly Lyra's `Formula`. The new field then appears in the column dropdowns like any other.
High value, second to Filter.
- **Lookup (join a second dataset)** — Lyra's `Lookup` joins one pipeline's fields into another by
key. Astrolabe references **one dataset per snippet**, so this is a larger data-model change —
note and **defer**.
**Where it lives:** Lyra keeps these in a separate _dataset_ pane, distinct from the encoding
inspector — the right instinct. In Astrolabe they belong in a **"Data" section in the builder's
left pane, above the channels**, paired with the §3.2 data-table preview: _here are your rows
[+ Filter] [+ Calculate] → now encode them_, with the preview table updating live as filters
apply. This composes cleanly and keeps the channel rows about encoding, not data prep.
**Scope fit:** a field+operator+value filter shelf is squarely Tier-B "smart + guarded," closes
the loop on warnings the builder already emits, and is arguably **higher value than faceting (B8)**
— it should enter the §8 backlog as a new first-class item (it is absent today).
---
## 4. Anti-recommendations — what to deliberately **not** take from Lyra
These are as load-bearing as the adopt list; Lyra is a research prototype and several of its
defining features are traps for us.
- **The decomposed GUI-as-document architecture** (`store/factory/*` + one-way `export.ts`).
It forbids the spec→builder round-trip our product is built on. Our JSON-source-of-truth +
emit model is correct; keep it.
- **Interaction-by-demonstration** (`ctrl/demonstrations.ts` — **1454 lines** that synthesize
raw Vega signals/selections from brush/click/hover, plus `signals.ts`/`listeners.ts`).
Impressive research, wrong layer for us: **Vega-Lite's `params`/selections** are the right
abstraction. _If_ we ever add interactivity, expose VL selections as a small guarded builder
action — never port Lyra's signal generator.
- **The direct-manipulation canvas** (handles, connectors, `manipulators.ts`,
`transforms/manipulators/*`). Enormous surface tied to the Vega-runtime model, and it
competes with — rather than complements — our JSON-first-with-guardrails value proposition.
- **The stack**: d3 v3, Redux, Immutable.js, `react-modal`, class components, `datalib`. We're
React + Zustand + CSS Modules with our own modal registry (`architecture/03`). **Take ideas,
read no code into the repo.** (Per `AGENTS.md`: no shared lib, patterns adapted not imported.)
- **The full data-pipeline editor** (lookup/aggregate/formula transform _chains_ as a
first-class pipeline UI). Our per-channel transforms cover the in-scope need; a general
pipeline editor is a different, much larger product.
---
## 5. Recommended sequence (mapped to the existing backlog)
> **Consolidated (2026-06-10):** this sequence is now merged with
> `chart-builder-research.md` §8 into the forward plan at
> [`chart-builder-enhancement-scope.md`](./chart-builder-enhancement-scope.md) (Tier-C
> target). The list below is the original Lyra-side reasoning; the scope doc §4 is the
> authoritative build order.
Slot these into `chart-builder-research.md` §8 rather than inventing a new track:
1. **Actionable hints** (§3.1) — extend `BuilderWarning` with an optional pure `fix`; wire the
four obvious ones; `/council` the copy. _Small, high-value, pure-core + thin UI._ **Do first.**
2. **Per-chart export** (§3.8) — Copy spec / Download `.vl.json` / PNG / SVG for a snippet, from
the `view` the renderer already holds. _Highest value-to-effort; near the top._
3. **Filter (+ Calculate) dataset transforms** (§3.12) — a guarded field/operator/value filter
shelf (then a derived-field calculate) in a "Data" section above the channels. _Closes the gap
the builder's own warnings point at; not in the backlog today; arguably > faceting._
4. **Data preview** (§3.2) — read-only row sample + header type chips in the builder (and
Datasets manager); reuse `profile.ts` stats. _Closes a confirmed gap; mostly UI; pairs with #3._
5. **Value-or-field channels via the Property model** (§3.3) — let a channel hold a constant
`value`, not just a `field`; one consistent control. _Capability gain; medium._ Pairs with the
§2.1 control-promotion shortlist (colour scheme, `zero`/`log`, axis/legend titles).
6. **Inline expression validation + field autocomplete** (§3.6, §3.10) — record the `parseExpr`
technique in `architecture/08`; apply when the Filter/Calculate expression mode (#3) lands. _Build with the feature._
7. **Field shelf + in-place type chip** (§3.4, §3.5) — design as the **Tier-C / faceting +
added-channels substrate** (§2.1: shape/opacity/theta/facets), not standalone. _Largest; with Tier C._
8. **Starter examples** (§3.7) — curated seed snippets, one per covered FT intent. _Cheap, optional._
9. **Scale/axis override panels + builder undo** (§3.9, §3.11) — only alongside control promotion;
override-panels-that-emit-only-when-touched, with orphan cleanup. _Deferred polish._
Verification for each lands the usual way: pure rules get `chart-builder.test.ts` cases;
UI/affordance changes get a manual pass against the live builder (a green build proves nothing
about what the user sees — `AGENTS.md`). Items 1 and 3 are mostly `src/core/` and squarely fit
our "core-first, tested hardest" rule.
---
_Citations are to `reference/lyra/src/js/…` (cloned, branch `lyra2019`). Lyra and Voyager are
both UW IDL; Voyager is already seated in `chart-builder-research.md`, so this review focuses
on the no-code-authoring ideas Voyager doesn't cover. Lyra is an engineering/interaction
source, not a council seat._
docs/exploration/monetization-and-sync-exploration.md
+186
View File
@@ -0,0 +1,186 @@
# Monetization & Sync — Exploration
> **Status:** Exploration, not a commitment. Captured 2026-06-10 from a strategy
> conversation. Nothing here is scheduled or in the plan; this is a memo to return to
> later so the _reasoning_ — not just the conclusion — survives.
>
> **Question:** If Astrolabe were ever monetized — e.g. a login that stores encrypted
> snippets in a database — what would that look like, technically and as a product?
>
> **Short answer:** The naive version (mandatory account, cloud-stored library) betrays
> the project's stated identity and isn't worth building. The version that survives
> scrutiny is the _opposite_ of a database: **private "bring-your-own-cloud" sync** that
> moves the user's own library between their own machines through storage they already
> control, with no server, no account, and no data custody. It's the smallest price and the
> smallest severity — and it barely dents the SOUL. Charging for it, if at all, is a
> one-time fee or donations, not a subscription.
---
## 1. The tension this collides with
A login + encrypted-DB system isn't a neutral feature add. [`SOUL.md`](../../SOUL.md) makes
"no account" a **named value**, and lists the opposite under _What We're Not_:
- Value #5, _Own Your Data_: "Your library is a file you control, **not a row in someone's
database**."
- _Not a collaboration platform_: "No multi-user, **no sync**, no comments."
- _Not a server app_: "**No account system.** No backend, no rendering service."
So there are really two different proposals hiding in the question:
1. **Mandatory account, cloud-stored library** — monetizes by making the cloud the _home_
of the data. This turns "a file you control" into "a row in our database," the exact
thing the product is defined against. **Rejected** regardless of revenue.
2. **Optional, opt-in sync layered on a local-first app that still works fully offline with
no account** — monetizes _without_ touching the soul. This is the only branch worth
exploring, and it's well-trodden (Obsidian: free local app, paid Sync/Publish; the free
local app _is_ the marketing for the paid layer).
Everything below lives in branch 2. If branch 1 is ever wanted, the honest move is to
change `SOUL.md` **first**, deliberately — not to let the product drift into it.
---
## 2. The options, and why most were set aside
| Option | What it is | Verdict |
| ------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------- |
| Mandatory account + cloud DB | Cloud is the canonical home of the library | **Rejected** — betrays the SOUL outright |
| Hosted E2EE sync (subscription) | We run a zero-knowledge server holding ciphertext blobs | **Too heavy** — see §4; disproportionate at our scale |
| Publish / embed | A snippet (or library) opens at a shareable URL | **Deferred** — bigger soul-violation + contested market; see §5 |
| **Private-move BYO-cloud sync** | Library rides the user's _own_ cloud between their _own_ machines | **Chosen direction** — smallest price, smallest severity; see §6 |
---
## 3. Build vs. "buy" vs. BYO — the real axis
"Buy the sync engine" was a confusing phrase: it never meant _pay money_. The recommended
engines (Evolu, Jazz) are themselves open source. The axis that matters is three rungs:
1. **Build it yourself** — write the crypto, the sync protocol, and the server. A sync
engine is doing six jobs, and each is a place to silently lose or leak data: change
tracking, push/pull, conflict resolution, encryption + key recovery, auth, and the
server. **Building the crypto/sync core yourself is one of the most bug-prone things in
software** — failures are catastrophic _and_ silent ("a library quietly corrupts and the
user notices when it's gone"). Wrong rung for a first foray.
2. **Open-source engine, you host it** — the hard logic is the library's (Evolu = local-
first + E2EE + SQLite, the closest fit; Jazz = adds auth/permissions; PouchDB+CouchDB =
battle-tested replication but _not_ E2EE by default). You run a server (Cloudflare
Workers keeps fixed cost near zero).
3. **Open-source engine, someone else hosts it** — you pay them to run the server; you
still write the client. The _only_ rung that costs money, and all it buys is "not
running a server."
So open source absolutely does it. Money only ever buys away the operational burden — and
the chosen direction (§6) removes the server entirely, so it doesn't even arise.
---
## 4. Why charging money is the heavy part (not the code)
The engine is the easy 20%. Taking payment changes the _category_ of the project, inheriting
obligations that have nothing to do with code:
- **Data custody, forever.** You hold the canonical copy. A server loss or a sync-corruption
bug is _their work gone, and they paid you to keep it safe._ E2EE makes this **worse**:
you can't read the ciphertext, so you can't inspect or repair damage either.
- **The "can never walk away" tax.** A free local app is static files — stop touching it and
it keeps working. A paid server-backed service breaks for every paying customer the day
you stop paying the hosting bill. You've converted "ship it and move on" into "responsible
indefinitely."
- **Uptime, support, and the forgotten-passphrase trap.** With E2EE, "recover my data" has
the answer "I can't," and users will be angry.
- **Billing + tax + legal.** International VAT/sales-tax (offload to a merchant-of-record
like Paddle/Lemon Squeezy), plus privacy policy, ToS, account deletion.
**The proportionality insight — and it cuts against scale.** Almost every obligation above
is _fixed, not per-user_. So the economics are **worst** at small scale: maximum fixed
responsibility, minimum revenue to justify it. 50 users × $5/mo = $250/mo against a forever
guarantee of their data's survival, support, on-call, and never abandoning it. For a small,
passionate audience and a builder who isn't chasing profit, full paid-custody sync is
plausibly a _bad trade even when well-intentioned._ The danger was never "money grab" — it's
that the responsibility dwarfs both the money and the community it serves.
---
## 5. The Vega-editor insight (and how it defuses "publish")
The Vega editor's **share** feature is _itself_ bring-your-own-cloud: "share" doesn't write
to a Vega server — it writes to _your_ GitHub Gist and hands you a URL that renders that Gist
**client-side** in anyone's browser. The editor is a pure client; GitHub is the storage.
Two consequences:
1. "Like the Vega editor, but for a whole library" and "BYO-cloud" are the **same idea** —
and the tool Astrolabe descends from already proves it works with zero server and zero
custody.
2. It defuses the earlier worry that _publish_ violates the SOUL's "no rendering service."
It only does if _we_ run the renderer. If a snippet lives in the user's own storage and
renders **client-side in the viewer's browser**, there's no backend we operate and the
data still belongs to the user. The SOUL survives.
Publish was still **deferred** (not killed): bigger soul departure than private sync, and it
walks into Datawrapper/Flourish's contested, free-tier territory. Private-move is the cleaner,
less-contested wedge.
---
## 6. The decision: private-move BYO-cloud sync
The chosen direction. The point is **me getting my own library onto my other machine** —
not showing charts to others. Storage is private; there is no "viewer," just me.
Why it's the smallest price: it's barely a new system. Private-move sync is essentially
**"continuous export/import to a file the user holds, plus a merge rule."** The export
format already exists, so this adds a live file handle and a conflict rule on top of
something shipped — no infrastructure, no account, no encryption-key-recovery burden. The
user's own cloud supplies the auth and the durability for free.
Three decisions define the whole feature:
1. **Transport.** The user points Astrolabe at a file (via the File System Access API) that
lives inside their Dropbox/iCloud/Drive folder. Astrolabe reads/writes the library doc
there; _their_ cloud moves it between machines. We never see the bytes.
**Caveat:** the File System Access API is Chromium-only — on Firefox/Safari, fall back to
the manual export/import already shipped. So "automatic" is a Chromium upgrade over a
baseline that works everywhere.
2. **Conflict rule.** Two machines edit offline, both write the file. Per-snippet
last-write-wins with a version stamp, surfacing a visible "conflicted copy" duplicate
when stamps clash (Dropbox's own behavior — and exactly SOUL value #6, _predictable, not
clever_). This is the only genuinely new logic, and it belongs in `core/` where it's
testable.
3. **Format.** Essentially the existing export doc, perhaps with per-snippet version stamps
added. An extension of a schema, not a new one.
**Monetization, if any:** a **one-time purchase** (fits "not a money grab," no recurring-
billing or expired-subscription support load, matches an audience that distrusts
subscriptions), or **donations / GitHub Sponsors** for pure sustainability that gates
nothing. Not a subscription.
---
## 7. SOUL impact — minimal
This barely touches the SOUL, which is the point. _Local-by-default_, _own-your-data_,
_no-account_, _no-server_ all stay literally true — "your library is a file you control"
becomes **more** true, since it's now a real file in the user's own cloud. The only line
needing softening is "no sync," and single-user move-my-own-library isn't the multi-user
**collaboration** that line was written to exclude. The deliberate SOUL amendment shrinks to
roughly one sentence.
---
## 8. If/when this is revisited — next steps
- **Validate the need first, for free.** Export/import is already a manual sync ("export,
drop in Dropbox, import on the other machine"). If people won't do _that_, they won't pay
for the automatic version — and you've learned it without building anything.
- **Map the three touch-points** before committing: the `app/infrastructure/` file-handle
adapter, the `core/` merge rule (with tests), and the one-sentence `SOUL.md` carve-out.
- **Decide the conflict UX deliberately** (likely a `/council` question): how the
"conflicted copy" surfaces to the user.
- **Never DIY the crypto** if this ever grows toward hosted/E2EE — reach for Evolu/Jazz.
- **Keep the core free, local, and offline forever.** It's both the soul and the growth
engine; the moment a power user hits a paywall on the thing the app _is_, the word-of-mouth
that local-first products live on is lost.