mirror of
https://github.com/olehomelchenko/astrolabe.git
synced 2026-08-08 02:02:33 +00:00
Core: classify spec data blocks by Vega-Lite fidelity (spec-data) for dataset refs
This commit is contained in:
@@ -74,6 +74,10 @@ This is the at-a-glance list; keep it in sync with them.
|
||||
|
||||
**Next (flagged for build):**
|
||||
|
||||
- **Multi-view data model** ([`multi-view-data-model-scope.md`](exploration/multi-view-data-model-scope.md)) —
|
||||
durable composition support across the data-facing features. M1 (Vega-Lite-fidelity
|
||||
reference classifier, `core/spec-data`) is done; M2–M4 extend the editor data context,
|
||||
Extract, and the data inspector to be view-scoped.
|
||||
- **Chart Builder · 3B starter examples** ([`chart-builder-enhancement-scope.md`](exploration/chart-builder-enhancement-scope.md) §3) —
|
||||
a small set of curated starters, one per covered FT intent. Reshaped by 3C: a
|
||||
builder-openable starter must reference a dataset, so it ships paired sample datasets (or is
|
||||
|
||||
@@ -135,70 +135,67 @@ without waiting for a publish. Recomputation runs only on a _valid_ spec —
|
||||
auto-save is debounced and parse-gated (spec §03B) — so a transiently-invalid
|
||||
draft never disturbs the links.
|
||||
|
||||
### 3.1 Extracting referenced names from a spec (pure — `src/core/spec-refs.ts`)
|
||||
### 3.1 What counts as a reference, and extracting them (pure — `src/core/spec-data.ts`, `spec-refs.ts`)
|
||||
|
||||
A Vega-Lite spec can reference named data in several places: the top-level
|
||||
`data`, per-layer `data`, `data` inside `spec`/`facet`/`hconcat`/`vconcat`, and a
|
||||
lookup transform's `from.data`. A spec may also define its OWN inline datasets via
|
||||
a top-level `datasets` map — those are self-defined, not library references.
|
||||
Rather than enumerate Vega-Lite's grammar, we walk the spec recursively and
|
||||
collect every `{ data: { name } }` — but **prune two keys**: never recurse into a
|
||||
`data` object's payload (its `values`/rows) or the top-level `datasets` map,
|
||||
because those hold user data, not nested specs. Without the prune, a data _row_
|
||||
carrying a field literally named `data: { name: "x" }` is misread as a reference.
|
||||
This is pure, deterministic, and the most heavily unit-tested function here.
|
||||
A library reference is exactly Vega-Lite **named data**: a `data` block with a
|
||||
string `name` and no `values`, `url`, or generator key (`sequence`/`sphere`/
|
||||
`graticule`), whose name the spec does not define for itself via a top-level
|
||||
`datasets` map. A `name` riding on inline `values` or a `url` is Vega-Lite's
|
||||
runtime-rebind label — not a dependency — and self-defined `datasets` names
|
||||
resolve natively; both are left untouched. This mirrors Vega-Lite's own
|
||||
`isNamedData`, so Astrolabe **extends** Vega-Lite rather than diverging: every
|
||||
native data form keeps working, and only true references are tracked and resolved.
|
||||
|
||||
That classification lives in **`core/spec-data`** (`classifyData`,
|
||||
`libraryRefName`) — the single predicate that reference extraction, rename
|
||||
(`spec-refs`), and render-time resolution (`rendering`) all route through, so they
|
||||
cannot disagree on what is a dependency.
|
||||
|
||||
```ts
|
||||
// src/core/spec-refs.ts
|
||||
// src/core/spec-data.ts — the shared classifier (mirrors vega-lite/src/data.ts)
|
||||
|
||||
type Json = unknown;
|
||||
/** The library name a `data` block references, or null for native VL data / self-defined names. */
|
||||
export function libraryRefName(data: unknown, selfDefined: ReadonlySet<string>): string | null {
|
||||
if (classifyData(data) !== 'named') return null; // url | values | generator → not a reference
|
||||
const { name } = data as { name: string };
|
||||
return selfDefined.has(name) ? null : name;
|
||||
}
|
||||
```
|
||||
|
||||
References appear in several places — top-level `data`, per-layer `data`, `data`
|
||||
inside `spec`/`facet`/`hconcat`/`vconcat`, and a lookup transform's `from.data`.
|
||||
Rather than enumerate the grammar, each pass walks the spec recursively but
|
||||
**prunes two keys**: a `data` object's payload (its `values`/rows) and the
|
||||
top-level `datasets` map hold user data, not nested specs. Without the prune, a
|
||||
data _row_ carrying a field literally named `data: { name: "x" }` is misread as a
|
||||
reference.
|
||||
|
||||
```ts
|
||||
// src/core/spec-refs.ts — the recursive walk; classification routes through spec-data
|
||||
|
||||
/** Collects every library dataset name referenced by `{ data: { name } }`, excluding self-defined ones. */
|
||||
export function extractDatasetRefs(spec: Json): string[] {
|
||||
const root = typeof spec === 'string' ? safeParse(spec) : spec;
|
||||
const selfDefined = selfDefinedNames(root); // names from the spec's own top-level `datasets`
|
||||
const root = typeof spec === 'string' ? safeParse(spec) : spec; // unparseable → no refs
|
||||
const selfDefined = selfDefinedNames(root);
|
||||
const names = new Set<string>();
|
||||
|
||||
const walk = (node: Json): void => {
|
||||
if (Array.isArray(node)) {
|
||||
for (const item of node) walk(item);
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(node)) return void node.forEach(walk);
|
||||
if (node && typeof node === 'object') {
|
||||
const obj = node as Record<string, Json>;
|
||||
const data = obj.data as Record<string, Json> | undefined;
|
||||
if (data && typeof data === 'object' && typeof data.name === 'string') {
|
||||
if (!selfDefined.has(data.name)) names.add(data.name);
|
||||
}
|
||||
// Prune: a `data` payload and the `datasets` map hold user data, not refs.
|
||||
const refName = libraryRefName(obj.data, selfDefined);
|
||||
if (refName !== null) names.add(refName);
|
||||
for (const key of Object.keys(obj)) {
|
||||
if (key === 'data' || key === 'datasets') continue;
|
||||
if (key === 'data' || key === 'datasets') continue; // prune user-data payloads
|
||||
walk(obj[key]);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
walk(root);
|
||||
return [...names];
|
||||
}
|
||||
|
||||
function safeParse(s: string): Json {
|
||||
try {
|
||||
return JSON.parse(s);
|
||||
} catch {
|
||||
return null; // an unparseable draft simply has no resolvable refs
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```ts
|
||||
// src/core/spec-refs.ts — thin wrapper, run on every draft change and on publish
|
||||
|
||||
/** The list stored on snippet.datasetRefs. Sorted + de-duped for stable diffs. */
|
||||
export function recomputeDatasetRefs(spec: Json): string[] {
|
||||
return extractDatasetRefs(spec).sort();
|
||||
}
|
||||
```
|
||||
`recomputeDatasetRefs` is the thin sorted/de-duped wrapper stored on
|
||||
`snippet.datasetRefs`, run on every draft change and on publish.
|
||||
|
||||
> A `spec` may be an object or a string (see the Data Model). Normalize once,
|
||||
> at the boundary, so the recursive walk never has to care.
|
||||
@@ -207,17 +204,19 @@ export function recomputeDatasetRefs(spec: Json): string[] {
|
||||
|
||||
- Treat `extractDatasetRefs` as the single source of truth for "what does this
|
||||
spec reference". The reverse-lookup and rename paths both depend on it
|
||||
agreeing with what the renderer actually resolves.
|
||||
agreeing with what the renderer actually resolves — which holds because all of
|
||||
them classify through the same `core/spec-data` predicate.
|
||||
- Recompute and store `datasetRefs` on **every draft change and on publish** —
|
||||
but only through the parse-gated, debounced auto-save (`commitDraft`) and the
|
||||
programmatic extract/revert rewrites, never on raw keystrokes. That keeps the
|
||||
links in step with the edited draft while never recomputing from a
|
||||
transiently-invalid spec.
|
||||
- Prune the **same two keys** (`data`, `datasets`) in all three ref walks —
|
||||
extraction here, `renameDatasetInSpec`, and the renderer's `resolveDatasetRefs`
|
||||
(`src/core/rendering.ts`). They must agree on what counts as a reference; if one
|
||||
descends into data payloads and another doesn't, extraction and rendering
|
||||
disagree and a row field named `data` either gets counted, rewritten, or throws
|
||||
- Keep the three ref walks in lockstep — extraction here, `renameDatasetInSpec`,
|
||||
and the renderer's `resolveDatasetRefs` (`src/core/rendering.ts`). They agree
|
||||
because they share both halves: the `libraryRefName` classifier (what is a
|
||||
reference) and the prune of the **same two keys** (`data`, `datasets`). If one
|
||||
classified differently, or descended into data payloads while another didn't, a
|
||||
row field named `data` would get counted, rewritten, or throw
|
||||
`DatasetNotFoundError`.
|
||||
|
||||
**Don't**
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# Multi-view data model — scope & plan
|
||||
|
||||
Astrolabe authors arbitrary Vega-Lite, including **composed** specs (`layer`,
|
||||
`hconcat`/`vconcat`/`concat`, `facet`, `repeat`). Most of the spec-structure
|
||||
machinery already handles composition; the data-facing features carried
|
||||
single-view assumptions. This memo records the assessment, the data-model
|
||||
contract that anchors the work, and the milestone plan to make multi-view support
|
||||
durable. The guiding constraint: **extend Vega-Lite, never break it** — every
|
||||
native data form must keep working.
|
||||
|
||||
## The data-model contract
|
||||
|
||||
A dataset reference is exactly Vega-Lite **named data**: a `data` block with a
|
||||
string `name` and no `values`/`url`/generator key, whose name the spec does not
|
||||
self-define via top-level `datasets`. This mirrors Vega-Lite's `isNamedData`
|
||||
(`reference/vega-lite/src/data.ts`). The classification is owned by
|
||||
`core/spec-data` (`classifyData`, `libraryRefName`); reference extraction
|
||||
(`spec-refs`), rename (`spec-refs`), and render-time resolution (`rendering`) all
|
||||
route through it. See `docs/architecture/07` §3.1.
|
||||
|
||||
Library references resolve to inline data before embedding
|
||||
(`core/rendering` → `prepareSpecForRender`); a self-defined `datasets` name is
|
||||
left for Vega-Lite to resolve natively.
|
||||
|
||||
## Assessment: already multi-view vs. single-view assumptions
|
||||
|
||||
**Already composition-aware** (recurse all view operators): ref extraction/rename
|
||||
(`spec-refs`), reference resolution + fit-mode (`rendering`), structural wrap/
|
||||
unwrap/add-view (`spec-transforms`, `spec-cursor`, `spec-insert`), derived-field
|
||||
collection (`spec-fields`), config baking (`spec-config`), standalone export
|
||||
(`chart-export`, reuses `prepareSpecForRender`).
|
||||
|
||||
**Single-view assumptions** (the work):
|
||||
|
||||
- **Editor data context** (`app/services/active-dataset`) resolves _one_ dataset
|
||||
for the whole draft (first ref, or first inline data), with no notion of which
|
||||
view the cursor sits in. Completion/hover/inlay (`spec-dataset-hints`) and the
|
||||
facet/repeat field defaults (`spec-transform-actions`) therefore offer the wrong
|
||||
view's columns in a composition whose views bind different datasets.
|
||||
- **Extract inline data** (`app/stores/ExtractStore`) lifts only the top-level
|
||||
`data` block.
|
||||
- **Data inspector** (`core/result-data`, `DataInspector`) surfaces one input +
|
||||
one resolved table; a composition produces several `source_<n>`/`data_<n>`.
|
||||
- **`deriveSnippetName`** (`core/snippet`) reads top-level `mark`/`encoding` only,
|
||||
so a composed spec falls back to the default name (graceful, not a bug).
|
||||
- **Chart builder** is single-view by design; its strict round-trip hydration
|
||||
returns `null` for composed specs, so they stay Monaco-only (correct).
|
||||
|
||||
## Vega-Lite fidelity clashes
|
||||
|
||||
1. **Named inline/url data misread as a reference** — classifying on "has a
|
||||
string `name`" alone caught named-inline (`{ name, values }`) and named-url,
|
||||
breaking valid specs (spurious `DatasetNotFoundError`, or clobbered inline
|
||||
values). _Resolved_ by the `core/spec-data` classifier (M1).
|
||||
2. **Shadowing** — a library dataset whose name equals a self-defined `datasets`
|
||||
key is silently ignored (self-defined wins). Documented precedence; candidate
|
||||
for a user-facing note, no code change required.
|
||||
3. **Case-rule split** — library matching is case-insensitive (`naming.ts`);
|
||||
self-defined exclusion and Vega-Lite's own named-data lookup are case-sensitive.
|
||||
These are distinct namespaces, so the split is defensible; minor.
|
||||
4. **Runtime-injected named data** — Vega-Lite allows binding `{ name }` at
|
||||
runtime; Astrolabe always pre-resolves, so an imported spec relying on runtime
|
||||
injection won't render. Out of scope.
|
||||
|
||||
## Milestone plan
|
||||
|
||||
- **M1 — data-model foundation** ✅ — `core/spec-data` classifier mirroring
|
||||
`isNamedData`; `spec-refs` + `rendering` routed through it. Closes clash 1.
|
||||
- **M2 — view-scoped editor context** — pure `dataContextAtPath(spec, path)`:
|
||||
climb the cursor's JSON path to the nearest enclosing `data` (honoring
|
||||
Vega-Lite's parent→child data inheritance), classify it, and collect
|
||||
ancestor-chain derived fields. Rework `active-dataset` to be cursor-scoped and
|
||||
thread the offset through the three Monaco providers and the facet/repeat
|
||||
defaults. Resolve columns for every form: library ref, inline `values`,
|
||||
named-inline, self-defined `datasets`, url (no static rows), generator (none).
|
||||
- **M3 — view-scoped extract** — seed Extract from the focused view's inline data
|
||||
(reusing the cursor-scope machinery) and rewrite that view's `data`.
|
||||
- **M4 — multi-view inspection** — group the dataflow's datasets into per-view
|
||||
input/resolved pairs (`result-data`) and add a view selector to `DataInspector`
|
||||
(new interactive widget → `/council`).
|
||||
|
||||
Delivery is incremental, one milestone per commit, verified against real behavior.
|
||||
The consolidated data-model contract write-up into `docs/architecture` (05/08)
|
||||
lands once the shape is final.
|
||||
Reference in New Issue
Block a user