Chart builder: filter/calculate transforms, data preview, and inline expression validation

docs/architecture/05-rendering-theming-preview.md

@@ -40,6 +40,13 @@ above it is data; everything below it is a Vega `View` we own and must tear down
   `subscribe` listener), not as reactive state itself.
 - **Don't** scatter `vegaEmbed(...)` calls across components.
 
+> **Forward note — data inspector.** A planned vega-editor-style inspector (show the
+> _resolved, post-transform_ data tables, in the builder and as a togglable panel below
+> the main Live Preview) reads runtime rows via `view.data(name)`. It rides this same
+> boundary: a `RenderHandle.data()` accessor wraps the view (like `toImageURL`), so no
+> component touches the raw view. Parked, not built — see
+> [`docs/data-inspector-exploration.md`](../data-inspector-exploration.md).
+
 ---
 
 ## 2. vega-embed Integration

docs/chart-builder-enhancement-scope.md

@@ -25,6 +25,51 @@
 Newest first. The at-a-glance build-order tracker is §4; per-item detail is §3. This log is
 the quick "where are we" — read it first.
 
+- **2026-06-11** — **1C + 1D + 1E shipped (the Data section). Up next: Phase 2 (2A value-or-field channels, then 2B field shelf).**
+  - A new **Data section** at the top of the builder's left pane — "here are your rows;
+    shape them, then encode them" — emits the spec's top-level `transform` array.
+  - **1C · Filters** — a guarded **field + operator + value** predicate shelf. Operators
+    narrow by field type (`validFilterOps`): a measure/temporal field offers ordering
+    (`< ≤ > ≥`) + `is between`; a category offers `is` / `is not` / `is one of`. Values
+    coerce by type (quantitative → number; others → string, so ISO dates sort right).
+    `notEqual` emits a `{ not: { …equal } }` wrapper. A reversible **expression** power-mode
+    takes a raw `datum.…` predicate. Incomplete filters are skipped so the preview keeps
+    rendering. Multiple filters AND together. Pure core (`buildTransforms`,
+    `validFilterOps`, `filterOpArity`), tested.
+  - **1C · Calculated fields** — `{ calculate, as }` derived columns. A named field appears
+    in the channel dropdowns via `effectiveColumns` (defaults Quantitative); emitted
+    **before** filters (a row-wise calculate is order-independent, so calc-first is
+    equivalent and lets filters reference derived fields). Removing/renaming a referenced
+    field clears the dangling channel (`pruneEncodings`, in the store on calc edit/remove).
+  - **1E · Expression validation** — new pure core `expr-validate.ts` using Vega's own
+    `parseExpression` (already in the `vega` chunk, so ~zero bundle cost): inline syntax
+    errors on both expression inputs, plus a soft **unknown-field** warning when a
+    `datum.<field>` reference doesn't match a column (`referencedFields` walks the AST).
+    Field discoverability is served by **dataset-derived placeholder examples** (e.g.
+    `datum.revenue * 2`); a full Monaco-style completion popup is **deferred** (a bare
+    `<input>` doesn't warrant it — noted, not built).
+  - **1D · Data preview** — a collapsible, read-only first-N-rows table with a per-column
+    **type chip** in each header, to sanity-check inferred types before building (reuses
+    core `tabularRows`). Default collapsed; the scroll region is keyboard-reachable
+    (`tabIndex=0` + labelled group — avoids the Datasets-manager a11y gap).
+  - **Spec §06** gained a "Data (filters, calculated fields, preview)" section; the Layout
+    and Output blocks cross-reference it.
+  - **Council not yet run** on the new error/disclosure copy (the soft auto-fire surface:
+    expression-error + unknown-field copy, the preview disclosure). Conventions were matched
+    to the existing warnings region (arch 10 §5) and `SettingsPopover` disclosure; flag for a
+    council pass on review if desired.
+  - **Expression reference:** a contextual link to the Vega expression-language docs is
+    shown when an expression input is in play (a calculated field, or a filter in
+    expression mode) — the place the user needs to know the available functions/operators.
+  - **Verified:** `typecheck` + `test` (full suite green; +103 new core/store cases, +4 modal
+    smoke tests) + `eslint` + `build` (PWA, 45 precache entries). **Owed:** a manual/visual
+    pass against the live builder (filter shelf, calc → channel, expr errors, preview
+    table) — tests don't cover what the surface looks/feels like.
+  - **Surfaced direction (now parked):** the 1D preview shows **raw source** rows; a
+    transform-aware **data inspector** (resolved rows, à la vega-editor, in the builder
+    _and_ below the main Live Preview) is the wanted evolution — documented in
+    **[`docs/data-inspector-exploration.md`](./data-inspector-exploration.md)**, deferred.
+
 - **2026-06-10** — **Up next: 1C (filter + calculate transforms), paired with 1D (data preview).**
   - **1B · Per-chart export** shipped: an **Export** disclosure in the Live Preview header
     (distinct from the workspace Export) — **Copy spec** + **Download JSON** (`.vl.json`) of
@@ -34,17 +79,15 @@ the quick "where are we" — read it first.
     `view`. Filenames derive from the snippet name — filesystem-safe, script-preserving
     (pure `core/chart-export.ts`, tested). Home: the **preview header**, not the plan's
     "library row / editor toolbar" — image export needs the live view; the
-    disclosure-of-controls (not an ARIA menu) mirrors `SettingsPopover`.
-    - **Export options (from first-round feedback):** PNG **Resolution** `1×/2×/3×` is a
-      multiplier of `devicePixelRatio`, so the default `1×` is Retina-crisp — the soft-1×
-      export was a dpr bug (raw `toImageURL` scaleFactor ignores dpr). **Background**
-      `Theme`(default)/`White`/`None` fixes transparent PNGs (the chart config is
-      transparent so the on-screen pane colour shows; export composites the chosen colour
-      under the PNG / adds an SVG `<rect>`). **Referenced data** `Inline`(default)/`Keep
+    disclosure-of-controls (not an ARIA menu) mirrors `SettingsPopover`. - **Export options (from first-round feedback):** PNG **Resolution** `1×/2×/3×` is a
+    multiplier of `devicePixelRatio`, so the default `1×` is Retina-crisp — the soft-1×
+    export was a dpr bug (raw `toImageURL` scaleFactor ignores dpr). **Background**
+    `Theme`(default)/`White`/`None` fixes transparent PNGs (the chart config is
+    transparent so the on-screen pane colour shows; export composites the chosen colour
+    under the PNG / adds an SVG `<rect>`). **Referenced data** `Inline`(default)/`Keep
 refs` (shown only when the spec references saved datasets) inlines dataset values so
-      the exported spec renders standalone (`inlineReferencedDatasets`, tested).
-    - Spec §08 gained a _Per-chart export_ section (with the options); §04 cross-references
-      it; `architecture/05` §2 records the handle's dpr-aware scale + background compositing.
+    the exported spec renders standalone (`inlineReferencedDatasets`, tested). - Spec §08 gained a _Per-chart export_ section (with the options); §04 cross-references
+    it; `architecture/05` §2 records the handle's dpr-aware scale + background compositing.
   - **1A · Actionable hints** shipped: one-click fixes on guidance warnings
     (`BuilderWarning.fixes` + `applyWarningFix`), council-reviewed, with focus/announce a11y.
   - **Builder UX/perf batch** (from dogfooding the Superstore dataset) shipped: near-fullscreen
@@ -152,7 +195,7 @@ chart" reads best beside the chart. The widget is a disclosure-of-action-buttons
 ARIA menu), mirroring `SettingsPopover` and sharing its single-open registry. _Spec impact:
 new §08 "Per-chart export" section; §04 cross-reference; `architecture/05` §2 handle note._
 
-**1C · Filter (+ Calculate) dataset transforms** — _closes the loop the builder's own warnings open_
+**1C · Filter (+ Calculate) dataset transforms** — _done (2026-06-11; see status log)_
 The transform layer the builder doesn't touch: top-level `transform: []`. Source: Lyra
 §3.12. The builder _already tells users to filter_ in three warnings
 (`chart-builder.ts:475,476,511`) while offering no way to do it.
@@ -171,7 +214,7 @@ Home: a new **"Data" section in the builder's left pane, above the channels** (`
 your rows [+ Filter] [+ Calculate] → now encode them`), paired with 1D. _Spec impact: new
 §06 "Data / transforms" subsection — this is genuinely new behaviour, write it._
 
-**1D · Data-table preview** — _closes a confirmed gap; pairs with 1C_
+**1D · Data-table preview** — _done (2026-06-11; see status log)_
 Neither the builder nor the Datasets manager ever shows the **actual rows**. Source: Lyra
 §3.2. A compact, **read-only** first-N-rows grid with a **per-column type chip** in each
 header lets users sanity-check inferred types _before_ building — exactly when inference is
@@ -180,7 +223,7 @@ only need the row sample. Homes: a collapsible "Data" strip in the builder's lef
 (under the dataset name) and/or the Datasets manager. Keep it read-only (editing data is
 out of scope). _Spec impact: §06 + §05 (Datasets) additions._
 
-**1E · Inline expression validation + field autocomplete** — _build with 1C, not standalone_
+**1E · Inline expression validation + field autocomplete** — _done (2026-06-11; completion popup deferred)_
 When 1C's expression mode lands, validate the Vega/VL expression string with the library's
 own `parseExpr` (Lyra §3.6) and surface errors inline; autocomplete the dataset's own
 column names (we have the schema from `profile.ts`) (Lyra §3.10). Record the `parseExpr`
@@ -306,6 +349,12 @@ guardrail: _promote a control only when it is **both common AND awkward in JSON*
   is short-lived; the smart default already gives a sane start).
 - **Lookup / join a second dataset** — Lyra §3.12; data-model change (multi-dataset
   snippets). Larger, separate effort.
+- **Transform-aware data inspector** — evolve 1D from raw-source rows to the _resolved,
+  post-transform_ data (filtered rows + calculated columns), and generalize it to a
+  togglable data panel below the **main** Live Preview (vega-editor's "Data Viewer", a
+  debugging aid for any snippet, not only builder output). Reads runtime rows via
+  `view.data(name)` through the `RenderHandle`. Cross-cutting (editor + builder), so it
+  has its own home: **[`docs/data-inspector-exploration.md`](./data-inspector-exploration.md)**.
 
 ---
 
@@ -314,10 +363,10 @@ guardrail: _promote a control only when it is **both common AND awkward in JSON*
 ```
 Phase 1  1A actionable hints      ✓ done
          1B per-chart export      ✓ done
-         1C filter (+ calculate)  ← next: closes the loop on warnings the builder already emits
-         1D data preview          ← pairs with 1C
-         1E expr-validate + autocomplete (with 1C)
-Phase 2  2A value-or-field channels (Property model)
+         1C filter (+ calculate)  ✓ done
+         1D data preview          ✓ done
+         1E expr-validate         ✓ done (syntax + unknown-field; completion popup deferred)
+Phase 2  2A value-or-field channels (Property model)  ← next
          2B field shelf + in-place type cycling   ← Tier-C substrate
 Phase 3  3A intent-first front door (Tier C)       ← built on 2B; the defining feature
          3B starter examples

docs/data-inspector-exploration.md

@@ -0,0 +1,64 @@
+# Data Inspector — exploration (future direction)
+
+> **Status:** parked idea (2026-06-11). A deliberate divergence from the Phase-1
+> chart-builder scope, recorded here so it isn't lost. **Not built.** The shipped
+> seed is the builder's 1D data preview (raw source rows); this is its transform-aware
+> evolution, generalized to the main editor.
+
+## What
+
+A vega-editor-style **data inspector**: show the data tables the chart actually uses at
+runtime — _after_ the spec's transforms run — not just the raw source rows. Two homes:
+
+1. **Chart Builder** — make the existing data preview (1D) transform-aware: show the
+   filtered rows and any calculated columns, so the preview reflects the `transform`
+   block the builder emits. (Today 1D shows the **raw dataset rows**, pre-transform — an
+   "inspect the source" view. This adds the "inspect the result" view alongside it.)
+2. **Main window** — a **togglable data panel below the Live Preview** in the editor,
+   for any snippet (not only builder-made ones): inspect each named dataset / the
+   post-transform rows while authoring a spec by hand.
+
+## Why
+
+vega-editor surfaces every internal data table below the chart (its "Data Viewer"), and
+it is the primary way to debug "why is my chart empty / wrong" — you look at what the
+transforms produced. Astrolabe has no equivalent today: the only data views are of the
+**raw source** (the Datasets manager, and builder 1D). Seeing the **resolved** data
+closes that debugging gap and is useful well beyond the builder.
+
+## Technique
+
+Vega exposes runtime datasets via `view.data(name)` — the rows of any named dataset in
+the compiled dataflow, including the root data _after_ transforms. The renderer already
+owns the only `view` reference behind `RenderHandle` (`src/app/services/chart-renderer.ts`);
+add a `RenderHandle.listData()` / `data(name)` accessor that wraps `view.data(...)`,
+mirroring how `toImageURL` wraps `view.toImageURL` — so no component touches the raw
+view and the embedding boundary holds (architecture/05 §1–§2).
+
+Vega-Lite compiles user transforms into named datasets (e.g. `source_0`, `data_0`);
+mapping those to friendly labels and choosing which to expose is the main design work.
+
+## Scope notes / open questions
+
+- **Read-only**, like 1D — it inspects, never mutates (the "moderately capable" ceiling:
+  a debugging aid, not a data editor).
+- **Big datasets** — cap the rows shown (reuse the 50-row preview cap). The view holds
+  every row; the table must not.
+- **Which tables to list** — just the post-transform root/result, or every internal
+  dataset (the vega-editor dropdown)? Start with the result; expose the rest behind a
+  selector if it earns its place.
+- **Main-window placement** — a third stacked region inside the preview pane (toggle +
+  persisted visibility, like the existing pane toggles) vs. a tab over the chart. Wants a
+  small interaction pass (architecture/10) before building.
+- **Refresh timing** — the data is only available once a view has rendered; the panel
+  reads from the live `RenderHandle` and updates when a new render resolves (it must not
+  read a finalized view — same generation discipline as the preview).
+
+## Relationship to shipped work
+
+- Seed: builder **1D** data preview — see
+  [`chart-builder-enhancement-scope.md`](./chart-builder-enhancement-scope.md) (§3 1D).
+- Renderer/preview home: [`architecture/05`](./architecture/05-rendering-theming-preview.md)
+  (the `RenderHandle` boundary the `view.data` accessor extends).
+- Behaviour, once built, is specified in `docs/spec/04` (Live Preview) and `docs/spec/06`
+  (the builder preview), per "spec follows code".

docs/spec/06-chart-builder.md

@@ -14,9 +14,40 @@ The Chart Builder is a visual, no-JSON way to compose a Vega-Lite chart from a s
 
 A two-pane modal:
 
-- **Left — configuration:** dataset name, mark type selector, one row per encoding channel, optional width/height inputs, and a "Create Snippet" action.
+- **Left — configuration:** dataset name, a **Data** section (row filters, calculated fields, and a collapsible row preview — see _Data_ below), mark type selector, one row per encoding channel, optional width/height inputs, and a "Create Snippet" action.
 - **Right — live preview:** a rendered chart that updates as the configuration changes, with a placeholder/error area.
 
+## Data (preview, filters, calculated fields)
+
+A **Data** section sits at the top of the configuration pane — "here are your rows; shape them, then encode them". It previews the source rows first, then offers controls to shape them before encoding. Everything here is optional; a chart can be built with none of it. The shaping controls emit the spec's top-level `transform` array (see _Output_).
+
+The section is ordered **input → shaping** so the distinction reads at a glance: the row preview (the **input** data) comes first, the filters and calculated fields (which shape what the chart actually draws) come below it.
+
+### Data preview
+
+- The first item in the section: a collapsible, **read-only** sample of the dataset's first rows (capped), with a per-column **type chip** in each header. It lets the user sanity-check inferred types against the actual values _before_ building — exactly when type inference is most likely to surprise. Editing the data is out of scope. A non-tabular payload (a single JSON object, TopoJSON, an unfetched URL) has no rows to show.
+- The preview shows the dataset's **raw source rows** — _before_ the filters and calculated fields below are applied; it does not show derived columns. Its position above those controls makes that explicit: it is the input, not the result. (The transformed result is what the right-hand chart renders. A transform-aware "data inspector" that shows the resolved rows is a planned future direction — see [`docs/data-inspector-exploration.md`](../data-inspector-exploration.md).)
+
+### Filters
+
+- A list of row filters, each added via **Add filter** and removable. All filters combine (logical **AND**) and apply to the raw rows **before** any encoding aggregation, so "filter rows, then aggregate" is the natural reading.
+- A filter is, by default, a **guarded predicate** — a **field**, an **operator**, and a **value** — needing no expression for the common case:
+  - The **field** dropdown offers the dataset's columns plus any calculated fields.
+  - The **operators** offered depend on the field's type: a measure or temporal field offers `is` / `is not` / `<` / `≤` / `>` / `≥` / `is between` (two bounds); a category offers `is` / `is not` / `is one of` (a comma-separated membership list). A quantitative value compares as a number; other types compare as text (ISO dates sort correctly as text).
+- A filter can be switched to an **expression** power-mode — a raw Vega predicate expression (e.g. `datum.value > 0`) — for what the guarded shelf can't say. The toggle is reversible.
+- An **incomplete** filter (no value yet, a blank range bound, an empty expression) is ignored, so the live preview keeps rendering while the user types.
+
+### Calculated fields
+
+- A list of derived fields, each added via **Add field** and removable: a **name** and a **Vega expression** that produces a new column (e.g. `profit` = `datum.revenue - datum.cost`).
+- A named calculated field appears in the encoding **channel dropdowns** like any real column (it defaults to **Quantitative**, the common arithmetic case, and its type can be overridden on the channel within the valid set). Calculated fields are applied **before** filters, so a filter may reference a derived field.
+- Removing or renaming a calculated field that a channel referenced **clears that channel** (the produced spec never encodes a field that no longer exists).
+
+### Expression validation
+
+- Both expression inputs (a filter's expression mode, a calculated field) are validated with **Vega's own expression parser** — the same one the chart uses — so a syntax error is reported **inline** the moment it appears, matching exactly what the chart would accept.
+- A `datum.<field>` reference that does not match a known column raises a soft **"unknown field"** warning (a typo guard) without blocking — the value is genuinely valid Vega, it just won't resolve.
+
 ## Inputs and Controls
 
 ### Mark type
@@ -93,7 +124,7 @@ A clean configuration shows no hints.
 
 Selecting "Create Snippet" produces the final artifact:
 
-- Builds a complete Vega-Lite spec containing: the schema reference, a named data reference to the dataset, the chosen mark (with tooltips enabled), the mapped encodings (each with its field and field type, plus any aggregate / bin / `timeUnit` transform), chart-level sort and stacking where set, and any explicit width/height.
+- Builds a complete Vega-Lite spec containing: the schema reference, a named data reference to the dataset, any top-level `transform` (calculated fields first, then row filters — see _Data_), the chosen mark (with tooltips enabled), the mapped encodings (each with its field and field type, plus any aggregate / bin / `timeUnit` transform), chart-level sort and stacking where set, and any explicit width/height.
 - Channels left unmapped are omitted; if no encodings exist the spec omits the encoding block entirely (prevented by validation here).
 - Creates a new snippet from that spec with an auto-generated descriptive name, adds it to the snippet library, and records that it was built from the dataset.
 - Links the snippet to the dataset by recording the dataset reference, so the bidirectional snippet↔dataset relationship is established (see _Datasets_).