astrolabe/docs/architecture/07-naming-and-relationships.md

# Naming & Relationships

How Astrolabe keeps entity **names unique** within a collection, and how it
tracks the **bidirectional links** between snippets and datasets so they stay
consistent as entities are created, imported, and renamed.

Two concerns live here, and they reinforce each other:

1. **Name uniqueness** — every dataset has a unique name. Names are the primary
   key users see and the key snippets reference, so duplicates would be
   ambiguous. We reject duplicate names on create/rename, and auto-suffix
   collisions during bulk import.
2. **Relationship tracking** — a snippet references datasets _by name_ through
   its `datasetRefs: string[]` field. This is a bidirectional, name-based link:
   from a snippet you read its refs; from a dataset you scan snippets to find
   who uses it. Renaming a dataset must propagate to every snippet that points
   at it, in both the spec and the `datasetRefs` array, or the links rot.

The hard, testable logic is **pure** and lives in `src/core/`. The parts that
read and mutate stores live in `src/app/services/`.

---

## 1. Why names, not IDs, are the link

Datasets carry a numeric `id`, but snippets reference them **by name** because
that is what Vega-Lite uses: a spec resolves data through a named-data
reference, `{ "data": { "name": "MyDataset" } }`. The name _is_ the contract
between a spec and the dataset library. Storing a numeric id in the spec would
mean the spec is no longer a standalone, paste-anywhere Vega-Lite document.

The consequence: names must be unique (two datasets named `Sales` would make
`{ "data": { "name": "Sales" } }` ambiguous), and renaming a dataset is a
**graph operation**, not a single field write — every reference to the old name
must move with it.

---

## 2. Name uniqueness (pure — `src/core/naming.ts`)

### 2.1 Uniqueness check

Comparisons are **case-insensitive** (`Sales` and `sales` collide), so a single
display name maps to a single dataset regardless of how a user types a
reference. The check takes an optional `excludeId` so a rename can ignore the
record being renamed (renaming `Sales` to `Sales` is not a collision with
itself).

```ts
// src/core/naming.ts

/** Case-insensitive set of names already in use, minus an optional excluded id. */
export function isNameTaken(
  desired: string,
  datasets: ReadonlyArray<{ id: number; name: string }>,
  excludeId?: number,
): boolean {
  const lower = desired.trim().toLowerCase();
  return datasets.some((d) => d.id !== excludeId && d.name.toLowerCase() === lower);
}
```

### 2.2 Making a unique name

When a desired name is taken — during import, "extract inline data", or
"build chart" — we do **not** overwrite the existing dataset. We derive the
next free name by appending a numeric suffix: `Name` → `Name 2` → `Name 3`.
The function takes the set of existing names so it has no store dependency and
is trivially unit-testable.

```ts
// src/core/naming.ts

/**
 * Returns `desired` if free, else the first available `${desired} ${n}` (n >= 2).
 * `existingNames` is the set of names already in the collection.
 * Comparison is case-insensitive; the returned name preserves `desired`'s casing.
 */
export function makeUniqueName(desired: string, existingNames: Iterable<string>): string {
  const taken = new Set<string>();
  for (const n of existingNames) taken.add(n.toLowerCase());

  const base = desired.trim();
  if (!taken.has(base.toLowerCase())) return base;

  let n = 2;
  while (taken.has(`${base} ${n}`.toLowerCase())) n++;
  return `${base} ${n}`;
}
```

> If a base name already ends in a number (`Q1 2024`), the suffix still appends
> (`Q1 2024 2`). That is intentional: we never parse meaning out of the name,
> we only guarantee a free slot. Keep this dumb and predictable.

**Do**

- Use `isNameTaken` to reject duplicate create/rename in the UI before saving,
  and surface an error toast.
- Use `makeUniqueName` for every non-interactive path (import, extract, build)
  where blocking the user would be worse than a silent, reported rename.
- Pass `excludeId` on rename so an unchanged or case-only edit is allowed.

**Don't**

- Don't compare names case-sensitively anywhere — pick `toLowerCase()` once and
  use it consistently.
- Don't let `makeUniqueName` mutate a store or read store state; it takes plain
  data and returns a string.

---

## 3. The bidirectional snippet ↔ dataset link

```
            datasetRefs: ["Sales", "Regions"]          (forward, on the snippet)
   Snippet ───────────────────────────────────────────────────►  Dataset "Sales"
        ▲                                                                 │
        └──────────── scan all snippets for "Sales" in datasetRefs ◄──────┘
                                  (reverse, derived)
```

- **Forward** (snippet → datasets): read `snippet.datasetRefs`. Cheap, stored.
- **Reverse** (dataset → snippets): there is no stored back-pointer. We compute
  it by scanning snippets. Keeping it _derived_ means it can never disagree with
  the forward links — there is one source of truth.

`datasetRefs` is **derived from the spec**, not hand-maintained. It mirrors the
dataset names referenced by the **draft** spec — the version being edited — and
is recomputed on every change to the draft (auto-save, the Extract-to-Dataset
rewrite, revert) and on publish. Tracking the draft (not only the last publish)
keeps a snippet's linked-datasets display and the reverse lookup in step with
what the editor shows — so a hand-typed reference or an Extract links its dataset
without waiting for a publish. Recomputation runs only on a _valid_ spec —
auto-save is debounced and parse-gated (spec §03B) — so a transiently-invalid
draft never disturbs the links.

### 3.1 What counts as a reference, and extracting them (pure — `src/core/spec-data.ts`, `spec-refs.ts`)

A library reference is exactly Vega-Lite **named data**: a `data` block with a
string `name` and no `values`, `url`, or generator key (`sequence`/`sphere`/
`graticule`), whose name the spec does not define for itself via a top-level
`datasets` map. A `name` riding on inline `values` or a `url` is Vega-Lite's
runtime-rebind label — not a dependency — and self-defined `datasets` names
resolve natively; both are left untouched. This mirrors Vega-Lite's own
`isNamedData`, so Astrolabe **extends** Vega-Lite rather than diverging: every
native data form keeps working, and only true references are tracked and resolved.

That classification lives in **`core/spec-data`** (`classifyData`,
`libraryRefName`) — the single predicate that reference extraction, rename
(`spec-refs`), and render-time resolution (`rendering`) all route through, so they
cannot disagree on what is a dependency.

```ts
// src/core/spec-data.ts — the shared classifier (mirrors vega-lite/src/data.ts)

/** The library name a `data` block references, or null for native VL data / self-defined names. */
export function libraryRefName(data: unknown, selfDefined: ReadonlySet<string>): string | null {
  if (classifyData(data) !== 'named') return null; // url | values | generator → not a reference
  const { name } = data as { name: string };
  return selfDefined.has(name) ? null : name;
}
```

References appear in several places — top-level `data`, per-layer `data`, `data`
inside `spec`/`facet`/`hconcat`/`vconcat`, and a lookup transform's `from.data`.
Rather than enumerate the grammar, the walk recurses the spec but **prunes two
keys**: a `data` object's payload (its `values`/rows) and the top-level `datasets`
map hold user data, not nested specs. Without the prune, a data _row_ carrying a
field literally named `data: { name: "x" }` is misread as a reference.

That walk is **one shared pair** in `core/spec-data`, not re-implemented per pass:
`forEachDataBinding(spec, visit)` (read-only, `visit` returns `true` to stop early)
and `mapDataBindings(spec, mapData)` (returns a copy). Both compute the spec's
self-defined names once and hand them to the callback, so a caller writes only its
own rule — _what is a reference_ (`libraryRefName`) or _what to rewrite_ — never the
walk. `extractDatasetRefs` collects through `forEachDataBinding`; `renameDatasetInSpec`
and `promoteSelfDefinedDataset` (§3.2) rewrite through `mapDataBindings`.

`recomputeDatasetRefs` is the thin sorted/de-duped wrapper stored on
`snippet.datasetRefs`, run on every draft change and on publish.

> A `spec` may be an object or a string (see the Data Model). Normalize once,
> at the boundary, so the recursive walk never has to care.

**Do**

- Treat `extractDatasetRefs` as the single source of truth for "what does this
  spec reference". The reverse-lookup and rename paths both depend on it
  agreeing with what the renderer actually resolves — which holds because all of
  them classify through the same `core/spec-data` predicate.
- Recompute and store `datasetRefs` on **every draft change and on publish** —
  but only through the parse-gated, debounced auto-save (`commitDraft`) and the
  programmatic extract/revert rewrites, never on raw keystrokes. That keeps the
  links in step with the edited draft while never recomputing from a
  transiently-invalid spec.
- Route every binding pass through the shared `forEachDataBinding` /
  `mapDataBindings`. They share both halves that must agree — the `libraryRefName`
  classifier (what is a reference) and the prune of the **same two keys** (`data`,
  `datasets`) — so extraction, rename, and the reverse extraction cannot drift. If
  one classified differently, or descended into data payloads while another didn't,
  a row field named `data` would get counted, rewritten, or throw
  `DatasetNotFoundError`. (The renderer's `resolveDatasetRefs` mutates in place and
  can throw mid-walk, so it keeps its own copy of the walk — the one exception, held
  in step by the same prune-two-keys rule.)

**Don't**

- Don't let two code paths each have their own idea of "referenced names".
  Renamer, ref-recomputer, and renderer must use the same walk shape.
- Don't add a fresh prune-walk for a new binding pass — reuse the shared pair. A
  hand-written copy is one classifier tweak away from disagreeing with the others.
- Don't "enumerate the grammar" (scope the walk to a fixed list of container
  keys) to fix the payload-descent problem — pruning the two data-bearing keys
  stays correct as Vega-Lite's composition grammar grows; an allow-list rots.

### 3.2 Extracting embedded data into a dataset — the reverse (`spec-inline-data.ts`, `spec-refs.ts`)

Extract-to-Dataset is the inverse of a reference: it lifts a view's **embedded**
data into a stored dataset and rewrites the spec to reference it by name. It is a
cursor-scoped editor action — `services/extract-action` resolves the focused view's
binding (`dataBindingAtPath`), seeds the modal, then opens it; the gate
`specHasExtractableData` hides the toolbar action when no view carries liftable
data. Two embedded shapes lift, both routing through the same `spec-data` classifier
so they never disagree with reference detection:

- **Inline `values`** — `inlineValuesOf` captures the payload verbatim (a CSV/TSV
  string is kept as-is); confirm rewrites that view's `data` block, at its anchor
  path, to `{ name }`.
- **A self-defined `datasets` entry** the view references — `selfDefinedPayloadOf`
  reads the named rows and `promoteSelfDefinedDataset` drops the `datasets` entry
  (and the map when it empties). Keeping the name needs no reference rewrite — it
  un-shadows onto the new library dataset; renaming rewrites every matching
  reference. This is the reverse direction of the self-defined-vs-library
  precedence in §3.1.

A `lookup` transform's inline `from.data` lifts like any view binding —
`dataBindingAtPath` finds it. A `url`, a generator, or an existing library
reference carries nothing to lift.

---

## 4. Reverse lookup: who uses this dataset?

The Dataset Manager shows a **usage badge** and a **Linked Snippets** list; the
Snippet Library shows a snippet's linked datasets. Both come from one scan — no
stored back-pointer to drift.

The scan itself is **pure** and lives in core, taking the snippets as a parameter
so the _same_ implementation serves two callers: the reactive UI (which passes its
live `snippets` selection straight in) and the non-reactive service wrapper (which
passes a store snapshot for programmatic callers). This is what makes "who
references this" have exactly one implementation (§6).

```ts
// src/core/relationships.ts — pure scan (unit-tested)

/** Snippets whose datasetRefs include `name` (case-insensitive). */
export function snippetsReferencingDataset<T extends { datasetRefs: readonly string[] }>(
  snippets: readonly T[],
  name: string,
): T[] {
  const lower = name.toLowerCase();
  return snippets.filter((s) => s.datasetRefs.some((ref) => ref.toLowerCase() === lower));
}

/** Usage counts for the whole library in one pass, keyed by lower-cased name. */
export function datasetUsageCounts(
  snippets: readonly { datasetRefs: readonly string[] }[],
): Map<string, number>;
```

There is no app-layer wrapper module around the scan. The reactive UI — a
component subscribed to `snippets` — calls the core helper directly, so the
badge and Linked Snippets list update the moment any snippet's draft changes
its refs (auto-save) or it is published. A non-reactive caller binds the same
helper to a snapshot inline at its call site
(`snippetsReferencingDataset(useSnippetStore.getState().snippets, name)`) —
one matching implementation, no `getState()` wrapper that would go stale in
reactive code.

**Do**

- Keep reverse lookup a pure scan, parameterized on the snippets so both the
  reactive and snapshot callers share it. It is O(snippets) but the collections
  are small (library budget ~5 MB); clarity beats an index.
- Call the core helper directly from a store-subscribed component for reactivity;
  bind it to a `getState()` snapshot inline for non-reactive code.

**Don't**

- Don't add a `referencedBy` array to datasets. A stored reverse pointer is a
  second source of truth that _will_ fall out of sync with `datasetRefs`.

---

## 5. Import: auto-suffix collisions, then report

On import we never overwrite an existing record. A record whose name collides
is renamed to a unique name via `makeUniqueName`, and **every rename is
collected and reported to the user** (toast / summary) so the change is never
silent. Crucially, names are reserved _as we go_ — within a single import, two
incoming `Sales` datasets become `Sales 2` and `Sales 3`, not two `Sales 2`.
The helper is generic over `{ name: string }` because two record kinds key on a
unique name: datasets and custom chart themes (only dataset renames need
propagation — nothing references a theme by name).

```ts
// src/core/import-normalize.ts

import { makeUniqueName } from './naming';

export interface NameRename {
  from: string;
  to: string;
}

/**
 * Returns incoming records with collision-free names, plus the renames applied.
 * `existing` are names already in the collection; `incoming` are records to add.
 */
export function dedupeIncomingNames<T extends { name: string }>(
  existing: ReadonlyArray<string>,
  incoming: ReadonlyArray<T>,
): { records: T[]; renames: NameRename[] } {
  const reserved = new Set(existing.map((n) => n.toLowerCase()));
  const renames: NameRename[] = [];

  const records = incoming.map((r) => {
    const unique = makeUniqueName(r.name, reserved);
    reserved.add(unique.toLowerCase()); // reserve so later imports don't collide
    if (unique !== r.name) renames.push({ from: r.name, to: unique });
    return unique === r.name ? r : { ...r, name: unique };
  });

  return { records, renames };
}
```

> If imported snippets reference the renamed dataset, their `datasetRefs` and
> specs must be rewritten to the new name too. The import flow does this purely,
> before anything is committed: `applyDatasetRenamesToSnippets`
> (`core/import-normalize`) rewrites the incoming snippet set per applied rename
> using the same `renameDatasetInSpec` machinery as §6.

### 5.1 Where the import/export flow lives, and its rules

**Flow:** header (Import/Export buttons in `App.tsx`) → `services/transfer.ts`
(the only store-touching layer) → pure core (`core/import-normalize.ts` shape
detection + normalization + the dedupe/rename/id-reassign helpers; `core/export-envelope.ts`)

- browser IO (`infrastructure/file-transfer.ts`). The pure helpers are unit-tested
  hardest; `transfer.ts` only orchestrates (read stores → call core → commit →
  notify). The behavior is recorded in spec §08.

Three rules a future change must keep:

- **Datasets commit before snippets** (`DatasetStore.addDatasets` then
  `SnippetStore.addSnippets`) so a snippet's by-name reference resolves against the
  just-added (possibly suffixed) dataset.
- **Imported datasets get fresh monotonic numeric ids** (`addDatasets`), not their
  envelope ids. Safe — and necessary — because datasets are linked **by name, not
  id** (§1): id reuse would collide in IndexedDB, but renaming the _id_ breaks
  nothing. (This is why the old `Date.now()`-collision TODO on `add` doesn't bite import.)
- **Rename propagation reads the spec, not only `datasetRefs`.**
  `applyDatasetRenamesToSnippets` finds the referenced name via
  `extractDatasetRefs(spec)` ∪ `datasetRefs`, so an imported snippet whose
  `datasetRefs` is absent/stale (a hand-crafted or foreign file) still gets its
  spec rewritten — the renderer resolves by spec, so a missed rename would break it.

**Do**

- Reserve each chosen name immediately so collisions _within_ one import are
  also resolved.
- Return the rename list and show it; a silent rename looks like data loss.

**Don't**

- Don't overwrite or merge a same-named existing dataset on import. Suffix and
  keep both — the user decides what to delete.

---

## 6. Rename propagation: keep the link consistent

Renaming a dataset is the operation that ties §2–§5 together. A rename must, in
one atomic step:

1. Update the dataset's own `name`.
2. For **every snippet referencing the old name**: rewrite the named-data
   references inside its spec (`{ "data": { "name": "old" } }` →
   `{ "data": { "name": "new" } }`) — in **both** `spec` and `draftSpec`.
3. Recompute that snippet's `datasetRefs` from the rewritten **draft** spec (the
   tracked surface), so the forward link mirrors reality and the reverse scan
   stays correct.

The spec rewrite is pure; the orchestration reads and writes stores.

```ts
// src/core/spec-refs.ts — pure rewrite

/** Returns a copy of `spec` with every data.name === oldName replaced by newName. */
export function renameDatasetInSpec(spec: Json, oldName: string, newName: string): Json {
  const obj = typeof spec === 'string' ? safeParse(spec) : spec;
  const selfDefined = selfDefinedNames(obj); // never rename a spec's own inline dataset name

  const rewrite = (node: Json): Json => {
    if (Array.isArray(node)) return node.map(rewrite);
    if (node && typeof node === 'object') {
      const out: Record<string, Json> = {};
      for (const [k, v] of Object.entries(node as Record<string, Json>)) {
        // A `data` object is a reference site: rename a matching name, but never
        // recurse into its payload. `datasets` (self-defined inline data) is left
        // whole. Same prune as extractDatasetRefs — see §3.1.
        if (k === 'data' && v && typeof v === 'object' && !Array.isArray(v)) {
          const dv = v as Record<string, Json>;
          out[k] = dv.name === oldName && !selfDefined.has(oldName) ? { ...dv, name: newName } : dv;
        } else if (k === 'data' || k === 'datasets') {
          out[k] = v;
        } else {
          out[k] = rewrite(v);
        }
      }
      return out;
    }
    return node;
  };

  const rewritten = rewrite(obj);
  // Preserve the original spec's stored shape (string vs object).
  return typeof spec === 'string' ? JSON.stringify(rewritten, null, 2) : rewritten;
}
```

The one rename implementation in the live app is the store action
**`SnippetStore.renameDatasetRefs(oldName, newName, now)`** — the action that
owns the snippet collection. It matches by spec+draft _content_ (not
`datasetRefs`, which mirrors only the draft), rewrites `spec`/`draftSpec` via
`renameDatasetInSpec`, recomputes `datasetRefs` from the rewritten draft, and
returns the number of snippets changed. Keeping the loop in the store means the
reactive editor buffer is refreshed in the same atomic update when the active
snippet's draft is rewritten. `DatasetStore.update` calls it whenever a save
changes a dataset's name, so a rename propagates everywhere as part of the one
user action — there is no separate coordinator module to call.

> **Collision on rename.** The UI rename form rejects a name already in use via
> `isNameTaken(newName, datasets, dataset.id)`. Programmatic renames (e.g. an
> import flow) instead resolve with `makeUniqueName` first. The propagation
> action itself does not invent a name — it assumes `newName` is the agreed
> target.

**Do**

- Rewrite `spec` **and** `draftSpec`. A user mid-edit must not see their draft
  silently break because the dataset was renamed underneath them.
- Recompute `datasetRefs` from the rewritten **draft** spec rather than
  string-replacing the array — the draft is the source of truth, the array is
  its mirror.
- Find affected snippets by scanning each one's **spec and draft content** for
  the old name (`extractDatasetRefs`), not by its `datasetRefs` array. Because
  `datasetRefs` mirrors the draft, a name referenced only by the still-published
  spec (the user removed it from the draft but hasn't published) is absent from
  the array; matching on content rewrites it anyway, so the published spec can't
  rot to a renamed-away dataset. Scan the content (don't reserialize) so a
  non-referencing snippet's text stays byte-for-byte intact. The import-side
  `applyDatasetRenamesToSnippets` already follows this spec-content rule (§5.1).

**Don't**

- Don't update `datasetRefs` without also rewriting the spec — the rendered
  named-data reference would still point at the old, now-missing name.
- Don't rename the dataset and skip propagation "for now". A half-applied rename
  is the exact inconsistency this whole document exists to prevent.

---

## 7. Snippet name provenance — the naming hierarchy

Snippet names (unlike dataset names) need no uniqueness; what they need is a rule for
**who may rewrite them**. Each snippet carries `nameSource` (spec §09A): `'user'` names
are frozen — set by an explicit rename (`SnippetStore.renameSnippet`) and never touched
by the app again; `'auto'` names are app-picked and keep tracking the spec. On publish,
an auto-named snippet is re-named from the now-published content in priority order: the
spec's `title` (string, line array, or `{ text }` forms), else a mark + encodings
description, else the existing name stands. The derivation dialect is deliberately the
same one `generateChartName` uses for builder output, so manually authored and
builder-built snippets read alike in the library.

Flow: `core/snippet.ts` (`deriveSnippetName`, `isAutoNamed`, `isDefaultSnippetName`) →
`SnippetStore.publish` (the only rewrite site) / `renameSnippet` (the freeze site) →
`SnippetLibrary`'s metadata panel (which must adopt a publish rename — arch 01 §2,
editing buffers). Records predating `nameSource` have no provenance; `isAutoNamed`
treats them as user-named unless the name is **provably** app-picked — the timestamp
default shape, or identical to what `deriveSnippetName` returns for the record's own
published spec. The conservative default is deliberate: rewriting a chosen name is worse
than failing to track an auto one.

---

## 8. Where things live

| Concern                                                                  | Location                         | Pure?               | Tested      |
| ------------------------------------------------------------------------ | -------------------------------- | ------------------- | ----------- |
| `makeUniqueName`, `isNameTaken`                                          | `src/core/naming.ts`             | yes                 | unit        |
| `extractDatasetRefs`, `recomputeDatasetRefs`                             | `src/core/spec-refs.ts`          | yes                 | unit        |
| `renameDatasetInSpec`                                                    | `src/core/spec-refs.ts`          | yes                 | unit        |
| `snippetsReferencingDataset`, `datasetUsageCounts` (reverse-lookup scan) | `src/core/relationships.ts`      | yes                 | unit        |
| `renameDatasetRefs` → updated count (rename propagation)                 | `src/app/stores/SnippetStore.ts` | no (mutates stores) | integration |
| `dedupeIncomingNames` (datasets + custom themes)                         | `src/core/import-normalize.ts`   | yes                 | unit        |
| `deriveSnippetName`, `isAutoNamed` (snippet name provenance)             | `src/core/snippet.ts`            | yes                 | unit        |

The dividing line: anything that takes plain data and returns plain data is
**core** and unit-tested in isolation; anything that reaches into a Zustand store
is a **store action or app service**. The rule of thumb — _the **draft** spec is the source of
truth, `datasetRefs` mirrors it, the reverse lookup is derived_ — is what keeps
the bidirectional link from ever needing manual repair.