diff --git a/docs/lyra-review.md b/docs/lyra-review.md new file mode 100644 index 0000000..70d6d32 --- /dev/null +++ b/docs/lyra-review.md @@ -0,0 +1,385 @@ +# Lyra — Repository Review & Improvement Ideas + +> **Status:** review complete (2026-06-10). Source: `vega/lyra` cloned to +> `reference/lyra` (branch `lyra2019`, last commit `fb284bf`, 2021-05-14 — unmaintained). +> **Purpose:** mine Lyra, UW IDL's _direct-manipulation_ Vega design environment, for +> interaction patterns that could improve Astrolabe's chart-building experience. This is a +> companion to [`chart-builder-research.md`](./chart-builder-research.md) (which already +> seats Voyager, also from UW IDL); Lyra is the _no-code authoring_ sibling we hadn't read. +> **Bottom line:** take Lyra's **interaction primitives and data-context UI**, not its +> architecture. Astrolabe's chart-choice intelligence (Tier B) is already _ahead_ of Lyra; +> the gaps Lyra exposes are about **how the user touches fields and sees their data**, plus +> **making our guidance actionable**. + +--- + +## 1. What Lyra is, and the one fact that decides what transfers + +Lyra lets you build a custom visualization **without writing code** — drag data fields onto +graphical mark properties, position marks with connectors, resize with handles, and define +interactions by demonstration. It targets **raw Vega** (not Vega-Lite), is a React + \*\*Redux + +- Immutable.js** app on a **d3 v3\*\* runtime, and is a 2019-era research prototype that was + never finished ("does not contain all functionality" — README) and has been dormant since + +2021. + +The single most important architectural fact: + +> **Lyra's source of truth is a _decomposed GUI model_; the Vega spec is a one-way _export_.** + +Lyra keeps the design as a Redux store of typed primitives — `Mark`, `Scale`, `Guide` +(axis/legend), `Pipeline`/`Dataset`, `Signal`, `Interaction`, `Widget` +(`src/js/store/factory/*`) — and `ctrl/export.ts` (`exporter()`, 546 lines) serializes that +store **into** a Vega spec on demand. There is no inverse: Lyra **cannot import and edit an +arbitrary Vega spec**. The GUI _is_ the document. + +**Astrolabe is the exact inverse.** Our source of truth is the **JSON spec** (a snippet); +the builder is a _view that emits_ JSON (`buildChartSpec`/`buildSnippetSpecText` in +`src/core/chart-builder.ts`), and the user can always drop to Monaco and hand-edit. For a +**snippet manager** this is the right call — Lyra's model would forbid the round-trip our +whole product depends on. So we take **none of Lyra's architecture** and **all of the +transferable lessons are at the interaction layer**, where the GUI-vs-JSON question doesn't +matter. + +This also reframes the comparison: Lyra is not a "better chart builder" to catch up to. On +**chart-choice intelligence** Astrolabe is already further along — Lyra has no +recommendation logic at all (that was Voyager's job, which we already mined). Lyra's value to +us is the parts of the _authoring experience_ we haven't built: field-first interaction, live +data context, and one-click assistance. + +--- + +## 2. Where Astrolabe already leads Lyra (so we don't chase the wrong things) + +| Dimension | Lyra | Astrolabe (today) | +| ------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| Chart-choice guidance | none — manual mark + manual encoding | Tier B: smart default mark, valid-type locking, Size discipline, non-blocking warnings (`builderWarnings`) | +| Transforms in the builder | data-pipeline only (filter/formula/aggregate/lookup), not per-encoding | per-channel Aggregate / Bin / `timeUnit`, chart-level Sort / Stack | +| Target language | raw Vega (verbose, low-level) | Vega-Lite (the right altitude for "pick mark + channels") | +| Import / round-trip | impossible (GUI is the doc) | native — JSON is the doc, builder emits it | +| Stack | d3 v3 / Redux / Immutable / class components (stale) | React + Zustand + CSS Modules (current) | + +The takeaways below deliberately **avoid** re-building anything in the right column. + +### 2.1 — Counterpoint: Lyra exposes a far larger _control surface_ + +On chart-choice _intelligence_ we lead, but on **raw number of direct controls** Lyra is well +ahead. Selecting a primitive (mark / scale / axis / legend) opens an inspector of literal +knobs — and the inventory dwarfs our builder's: + +| Primitive (`components/inspectors/*`) | Lyra's direct controls | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Mark — **Point** (`Symbol.tsx`) | x, y, **shape**, size, fill **color**, fill **opacity**, stroke color, stroke width | +| Mark — **Bar/Rect** (`Rect.tsx`) | X extent (start/center/end _spatial preset_), Y extent, fill color/opacity, stroke color/width | +| Mark — **Line** (`Line.tsx`) | x, y, stroke color/width, **interpolate (curve)**, tension | +| Mark — **Area** (`Area.tsx`) | baseline extent, fill color/opacity, stroke, interpolate, tension, orient | +| Mark — **Text** (`Text.tsx`) | text (expr/template), **font face/size/weight/style**, color, opacity, x/y, dx/dy offset, **align**, baseline, **rotation** | +| **Scale** (`Scale.tsx`) | **type** (linear/log/time/ordinal/band/point), **zero/nice/clamp/reverse/round**, **domain** (auto fields or manual min/max / value list), **range / colour scheme** (tableau, category10/20, sequential, diverging), padding(Inner/Outer) | +| **Axis** (`Axis.tsx`) | orient, axis-line colour/width, **title text + font/size/colour/offset**, **label size + angle + colour**, **grid on/off** + colour/opacity/width, **tick count/size/colour** | +| **Legend** (`Legend.tsx`) | orient, border, title text/size/colour, label font, symbol shape/size/fill/opacity, gradient | + +And the structural multiplier: **every mark property is _droppable_** — each is _either_ a +literal value _or_ a bound field. Lyra therefore has **no fixed channel set**; you can bind +data to strokeWidth, opacity, shape, fontSize, dx… Our builder caps at **X / Y / Color / +Size** and exposes **zero** styling, scale, axis, or legend controls. + +**Why this is not simply "Lyra is more capable":** + +1. **It's a _builder-surface_ gap, not a capability gap.** Everything above, Vega-Lite + expresses too (mark config, `scale`/`axis`/`legend`, channels like `opacity`/`shape`/ + `theta`/`row`/`column`). In Astrolabe you reach it by **hand-editing the JSON in Monaco**. + Lyra had **no editor and no round-trip** — the GUI was the _only_ way to touch the spec, so + it was forced to surface every knob. We split the work on purpose: a small **guarded** + encoding builder + a first-class JSON editor for the long tail. +2. **Lyra's breadth carries zero intelligence and high cost** — no defaults, no valid-type + locking, no guardrails, on raw low-level Vega, in an unfinished inspector. +3. So the question is not "match Lyra's control count" but **"which slices of that surface are + common _and_ awkward enough in JSON to deserve promotion into the guarded builder?"** + +**Slices worth promoting (ranked):** + +- **Constant-value styling** via the value-or-field channel (§3.3): fixed fill colour / + opacity / point shape / line curve without touching JSON — the most common "tweak the look." +- **Colour-scheme picker** on the Color channel (categorical / sequential / diverging) + a + measure-axis **`zero` / `log`** toggle. Palette changes are common and fiddly in JSON. +- **A few more channels** — `opacity`, `shape`, `tooltip` (already auto-enabled), **`theta`** + (unlocks pie/donut → a true part-to-whole), **`row`/`column`** facets (backlog **B8**). These + _are_ the field-shelf model (§3.4): adding channels and adding the shelf are one project. +- **Light axis/legend text** — custom **axis title**, **legend title / hide**. Common, low-risk. + +**Guardrail:** promote a control only when it is _both_ common _and_ awkward in JSON. +Otherwise the clean guarded builder drifts toward Lyra's fiddly everything-inspector — the +very complexity that helped leave Lyra unfinished. Styling/scale/axis breadth for its own sake +belongs in Monaco, not the builder. + +--- + +## 3. High-value ideas to adopt (ranked, in-scope, mapped to our code) + +### 3.1 — Make `builderWarnings` **actionable** (Lyra's `Hints`) — _highest value, smallest lift_ + +Lyra's hint system (`components/hints/Hints.tsx`) is a contextual nudge that carries an +**action**: `{title, text, action, action_text}` — a one-click button that _applies the fix_ +and clears the hint. Our guidance is strictly advisory text today: `BuilderWarning` +(`chart-builder.ts:333`) renders as a passive hint with no affordance. + +Several of our existing warnings have an obvious one-click remedy and should become +**suggested actions** (Carbon "inline notification with action" / NN-g "make the system do the +work"): + +- _"…draws one mark per row; aggregate the measure"_ → **[Aggregate as Sum]** sets the + measure channel's `aggregate`. +- _"…long labels; flip to a horizontal bar"_ → **[Swap X/Y]** (we already have the action — + just wire it to the hint). +- _"Two measures usually read better as a scatter"_ → **[Switch to Point]**. +- _"Area split into many colour series…"_ → **[Stack]** or **[Remove colour]**. + +This keeps the Tier-B "guarded, non-blocking" philosophy — the action is an _offer_, never a +forced change — while turning a wall of advice into a guided improvement loop. Concretely: +extend `BuilderWarning` with an optional `fix?: { label: string; apply: (c) => BuilderConfig }` +(pure, lives in `chart-builder.ts`, fully unit-testable), and render the button in +`ChartBuilderModal`. **Run the copy + affordance through `/council`** (it auto-fires on +guidance copy and new interactive affordances). + +### 3.2 — A live **data-table preview** in the builder (Lyra's `DataTable`) — _closes a real gap_ + +Confirmed gap: neither `ChartBuilderModal` nor `DatasetsModal` ever shows the **actual data +rows**. The builder picks columns from a dropdown and renders a _chart_ preview, but the user +never sees the values they're encoding. Lyra always shows the data — `DataTable.tsx` is a +paged, scrollable grid with a **per-field type icon** in each column header and hover +inspection. + +A compact, read-only **data preview** (first N rows, per-column type chip in the header) +would let the user sanity-check "is this column really a date / really numeric" _before_ +building — exactly when type inference is most likely to surprise them. Good homes: + +- a collapsible "Data" strip in the builder's left pane (under the dataset name), and/or +- the Datasets manager, where a dataset's shape is otherwise invisible. + +We already compute per-column type + cardinality + extent in `profile.ts` (the A3/A4 +profiling extension), so the header chips are free; we only need the row sample. Keep it +**read-only** — editing data is out of scope (and is where Lyra's pipeline complexity lives). + +### 3.3 — The unified **Property** primitive: one control = value · field · scale (Lyra's signature UI) + +Lyra's best idea is `components/inspectors/Property.tsx` — a **single droppable control** that +is, depending on binding state, _either_: + +- a **literal value** editor (number / color / range / select / text — `FormInputProperty`), **or** +- a **field chip** (drag a column onto it; chip shows source-vs-derived, click to unbind), **or** +- a **scale chip** (the scale the field flows through; click to unbind). + +Every mark inspector is then just a **declarative list** of these — e.g. `Symbol.tsx` +declares Position/Geometry/Fill/Stroke groups as ~10 `` lines. One primitive, one +drop-to-bind gesture, one unbind gesture, reused for _everything_ including the transform +expression fields (`Filter.tsx`, `Formula.tsx` reuse the same `Property` in autocomplete +mode). + +For Astrolabe this is the conceptual model for a **richer channel row**: a channel is "a field +**or** a constant **or** (future) a datum," and Vega-Lite encodes exactly that distinction +(`field` vs `value` vs `datum`). Today our channel row is field-only (column dropdown + +type segmented control). Adopting the Property model would let a channel also hold a **constant +value** (e.g. a fixed color/size) with one consistent control and a chip showing the binding +kind. This is a medium lift and a genuine capability gain, and it generalizes cleanly if we +ever add channels beyond X/Y/Color/Size. + +### 3.4 — Field-first interaction: a **field shelf** you drag onto channels (Lyra drop-zones + Voyager) + +Both UW tools converge on **fields as the primary objects**: Lyra drags fields onto mark +properties; Voyager has a field list with type chips you add to channel shelves. Astrolabe is +**channel-first** (pick a channel, then choose its column). A **field shelf** — the dataset's +columns listed with type chips, dragged or clicked onto channels — inverts that to match how +people actually think ("I have these fields; where do they go?"). + +This is the largest interaction shift here and overlaps our **deferred Tier-C / faceting** +work (`chart-builder-research.md` §8 B8/C). Recommend treating it as the **interaction +substrate for Tier C**, not a standalone task: when we build the intent-first front door and +add Row/Column facet channels, a field shelf is the natural way to assign many fields across +many channels. Note now; sequence with Tier C. + +### 3.5 — One-click **type cycling** on the field chip (Lyra's `FieldType`) + +Lyra's `FieldType.tsx` is a tiny, nice touch: the field's **type icon is the control** — +click it and it cycles N→O→Q→T (within the field's valid set). We already show a type +indicator next to each column option and a full `N|O|Q|T` segmented control per channel. The +cheap win is making the **per-option type chip itself interactive** (and, in a future field +shelf, the chip on each field), so type is a property of the _field_ the user can toggle in +place — fewer controls, the same guard (only valid types, via our `validFieldTypes`). + +### 3.6 — Inline **expression validation** via Vega's own parser (Lyra's transform technique) + +Lyra validates a filter/calculate expression with `parseExpr` from `vega-parser` **before +committing** it to the model (`Filter.tsx`, `Formula.tsx`), catching malformed expressions at +the keystroke. We don't expose raw expressions in the builder, so this isn't an immediate +builder feature — but the **technique** (validate Vega/VL expression strings with the +library's parser and surface the error inline) is directly reusable for our Monaco editor's +diagnostics and for any future calculate/filter affordance. Worth recording in +`architecture/08` (vega-editor techniques) as a known approach; Lyra even left the +"_indicate error in parsing_" TODO unfinished, so we'd be completing the idea, not copying it. + +### 3.7 — **Starter examples** gallery (Lyra's `ExampleGallery`) — _cheap, optional_ + +Lyra's toolbar has an `ExampleGallery` modal that `hydrate()`s a curated example into the +editor — a "start from something" affordance. Astrolabe's snippet library is the natural home +for a small set of **curated starter snippets** (one per FT intent category we cover — +Magnitude/Bar, Change-over-Time/Line, Correlation/Point…). Low effort, improves first-run and +doubles as living documentation of what the app does well. Lower priority than 3.1–3.2. + +### 3.8 — **Per-chart export: image + standalone spec** (Lyra's `Export` menu) — _biggest miss; cheap_ + +The one squarely-useful idea I almost overlooked. Lyra's toolbar `Export` (`components/toolbar/Export.tsx`) +offers **PNG**, **SVG**, **JSON spec**, and a **standalone HTML** scaffold — using +`view.toImageURL(type)` on the live Vega view. + +Astrolabe's export today is **workspace-JSON backup only** (`docs/spec/08`) — there is **no way +to get a single chart out** as an image, as its own spec file, or onto the clipboard. Yet our +renderer already holds exactly the Vega `view` Lyra exports from (`services/chart-renderer.ts`, +`renderer: 'svg'`), so this is a handful of lines: + +- **Copy spec** (clipboard) and **Download `.vl.json`** for the active snippet — the most-asked + "get my chart out" actions, both trivial given the snippet _is_ the spec. +- **Download PNG / SVG** of the rendered chart via `view.toImageURL('png'|'svg')` — drop a chart + into a doc/slide without a screenshot. +- (Optional) **standalone HTML** — a self-contained vega-embed page; ties to the + share/"private-move" direction in `monetization-and-sync-exploration.md`. + +Home: a snippet-level "Export / Share" affordance (library row action or editor toolbar), +distinct from the workspace Export. High value-to-effort; should jump near the top of the queue. + +### 3.9 — Scale / axis / legend as **auto-derived _override_ panels** (Lyra's `bindChannel` pipeline) + +If we ever promote scale/axis/legend controls (§2.1), Lyra shows the right _editing model_. +Binding a field in Lyra runs a pipeline (`actions/bindChannel/`): `parseScales` + `parseGuides` +**auto-materialize** the scale and axis/legend, `aggregateDependencies` re-points them when the +data is aggregated, and **`cleanupUnused`** garbage-collects scales/datasets no longer +referenced. The user edits a scale/axis only to _override_ an auto-derived default. + +Vega-Lite already auto-derives scales/axes/legends from encodings, so we get the inference for +free — the borrow is purely UX: surface scale/axis/legend as **optional override panels that +start empty (inheriting VL defaults) and only emit JSON when touched**, and **clearing a channel +drops its overrides** (the `cleanupUnused` lesson — no orphaned `scale`/`axis` config left behind). +This is the discipline that keeps §2.1's promoted controls from bloating the output spec. + +### 3.10 — Expression **autocomplete with field/signal chips** (Lyra's `AutoComplete`) + +`components/inspectors/AutoComplete.tsx` is a `contenteditable` expression field that +autocompletes **dataset field names** (and signals), rendering each `datum.field` reference as a +styled, non-editable **chip**. For any future filter/calculate affordance — and for Monaco +completions in expression positions — autocompleting the dataset's own column names (we already +have the schema from `profile.ts`) is the high-value half; the chip rendering is polish. Niche +until we expose expressions, but cheap to remember. + +### 3.11 — **Builder-state undo/redo** (Lyra's `redux-undo`) — _low priority_ + +Lyra makes the whole design undoable (`redux-undo`, global `vis.present`/`past`). Our builder +modal has **no undo** (Monaco covers the _editor_, not builder config). A session-scoped +undo/redo — or even a lighter **"reset to smart defaults"** — would soften experimentation in +the builder. Low priority: the builder is a short-lived modal and the smart default already +gives a sane starting point. Noted for completeness, not urgency. + +### 3.12 — **Dataset-level transforms: Filter / Calculate** (Lyra's data pipeline) — _the cleanest gap_ + +The transform dimension where Astrolabe and Lyra differ most. The key fact is that **Vega-Lite +has two transform layers**: + +1. **Inline encoding transforms** — `aggregate` / `bin` / `timeUnit` / `sort` / `stack` written + straight onto a channel. **This is the layer Astrolabe's builder uses**, and uses _well_ — + per-channel and guarded, arguably cleaner than Lyra (whose Aggregate is a separate pipeline step). +2. **Top-level `transform: []`** — dataset-level operations applied in sequence before encoding: + **`filter`**, **`calculate`**, **`lookup`**, window, fold, pivot, regression… **Astrolabe's + builder exposes _none_ of these.** Lyra's data pipeline is built almost entirely _around_ them + (`components/pipelines/transforms/`: Filter, Formula, Lookup, Sort — each an ordered card in a + `TransformList` on `datasets..transform`). + +The operations in layer 2 have **no encoding-level shorthand** — there is no way to express them +except by hand-editing the JSON. The two worth borrowing: + +- **Filter _(highest value)_** — row filtering (`transform: [{filter: …}]`). The single most + common data-shaping need, and the builder is _already telling users to do it_ — three existing + `builderWarnings` literally say "filter to fewer categories / to non-negative values" + (`chart-builder.ts:475,476,511`) while offering no way to. Guarded form: a **field + operator + + value** predicate shelf (Voyager-style), no expression required for the common case; power form: + a raw `datum.…` expression validated with Lyra's `parseExpr` technique (§3.6) + field + autocomplete (§3.10). VL applies top-level transforms _before_ encoding aggregation, so "filter + raw rows, then the chart aggregates" is the natural — and usually wanted — default; filtering on + an _aggregated_ value (HAVING) is the advanced case, defer it. +- **Calculate / derived field** — `transform: [{calculate: "datum.a / datum.b", as: "ratio"}]`, + exactly Lyra's `Formula`. The new field then appears in the column dropdowns like any other. + High value, second to Filter. +- **Lookup (join a second dataset)** — Lyra's `Lookup` joins one pipeline's fields into another by + key. Astrolabe references **one dataset per snippet**, so this is a larger data-model change — + note and **defer**. + +**Where it lives:** Lyra keeps these in a separate _dataset_ pane, distinct from the encoding +inspector — the right instinct. In Astrolabe they belong in a **"Data" section in the builder's +left pane, above the channels**, paired with the §3.2 data-table preview: _here are your rows +[+ Filter] [+ Calculate] → now encode them_, with the preview table updating live as filters +apply. This composes cleanly and keeps the channel rows about encoding, not data prep. + +**Scope fit:** a field+operator+value filter shelf is squarely Tier-B "smart + guarded," closes +the loop on warnings the builder already emits, and is arguably **higher value than faceting (B8)** +— it should enter the §8 backlog as a new first-class item (it is absent today). + +--- + +## 4. Anti-recommendations — what to deliberately **not** take from Lyra + +These are as load-bearing as the adopt list; Lyra is a research prototype and several of its +defining features are traps for us. + +- **The decomposed GUI-as-document architecture** (`store/factory/*` + one-way `export.ts`). + It forbids the spec→builder round-trip our product is built on. Our JSON-source-of-truth + + emit model is correct; keep it. +- **Interaction-by-demonstration** (`ctrl/demonstrations.ts` — **1454 lines** that synthesize + raw Vega signals/selections from brush/click/hover, plus `signals.ts`/`listeners.ts`). + Impressive research, wrong layer for us: **Vega-Lite's `params`/selections** are the right + abstraction. _If_ we ever add interactivity, expose VL selections as a small guarded builder + action — never port Lyra's signal generator. +- **The direct-manipulation canvas** (handles, connectors, `manipulators.ts`, + `transforms/manipulators/*`). Enormous surface tied to the Vega-runtime model, and it + competes with — rather than complements — our JSON-first-with-guardrails value proposition. +- **The stack**: d3 v3, Redux, Immutable.js, `react-modal`, class components, `datalib`. We're + React + Zustand + CSS Modules with our own modal registry (`architecture/03`). **Take ideas, + read no code into the repo.** (Per `AGENTS.md`: no shared lib, patterns adapted not imported.) +- **The full data-pipeline editor** (lookup/aggregate/formula transform _chains_ as a + first-class pipeline UI). Our per-channel transforms cover the in-scope need; a general + pipeline editor is a different, much larger product. + +--- + +## 5. Recommended sequence (mapped to the existing backlog) + +Slot these into `chart-builder-research.md` §8 rather than inventing a new track: + +1. **Actionable hints** (§3.1) — extend `BuilderWarning` with an optional pure `fix`; wire the + four obvious ones; `/council` the copy. _Small, high-value, pure-core + thin UI._ **Do first.** +2. **Per-chart export** (§3.8) — Copy spec / Download `.vl.json` / PNG / SVG for a snippet, from + the `view` the renderer already holds. _Highest value-to-effort; near the top._ +3. **Filter (+ Calculate) dataset transforms** (§3.12) — a guarded field/operator/value filter + shelf (then a derived-field calculate) in a "Data" section above the channels. _Closes the gap + the builder's own warnings point at; not in the backlog today; arguably > faceting._ +4. **Data preview** (§3.2) — read-only row sample + header type chips in the builder (and + Datasets manager); reuse `profile.ts` stats. _Closes a confirmed gap; mostly UI; pairs with #3._ +5. **Value-or-field channels via the Property model** (§3.3) — let a channel hold a constant + `value`, not just a `field`; one consistent control. _Capability gain; medium._ Pairs with the + §2.1 control-promotion shortlist (colour scheme, `zero`/`log`, axis/legend titles). +6. **Inline expression validation + field autocomplete** (§3.6, §3.10) — record the `parseExpr` + technique in `architecture/08`; apply when the Filter/Calculate expression mode (#3) lands. _Build with the feature._ +7. **Field shelf + in-place type chip** (§3.4, §3.5) — design as the **Tier-C / faceting + + added-channels substrate** (§2.1: shape/opacity/theta/facets), not standalone. _Largest; with Tier C._ +8. **Starter examples** (§3.7) — curated seed snippets, one per covered FT intent. _Cheap, optional._ +9. **Scale/axis override panels + builder undo** (§3.9, §3.11) — only alongside control promotion; + override-panels-that-emit-only-when-touched, with orphan cleanup. _Deferred polish._ + +Verification for each lands the usual way: pure rules get `chart-builder.test.ts` cases; +UI/affordance changes get a manual pass against the live builder (a green build proves nothing +about what the user sees — `AGENTS.md`). Items 1 and 3 are mostly `src/core/` and squarely fit +our "core-first, tested hardest" rule. + +--- + +_Citations are to `reference/lyra/src/js/…` (cloned, branch `lyra2019`). Lyra and Voyager are +both UW IDL; Voyager is already seated in `chart-builder-research.md`, so this review focuses +on the no-code-authoring ideas Voyager doesn't cover. Lyra is an engineering/interaction +source, not a council seat._