astrolabe/docs/architecture/07-naming-and-relationships.md

# Naming & Relationships

How Astrolabe keeps entity **names unique** within a collection, and how it
tracks the **bidirectional links** between snippets and datasets so they stay
consistent as entities are created, imported, and renamed.

Two concerns live here, and they reinforce each other:

1. **Name uniqueness** — every dataset has a unique name. Names are the primary
   key users see and the key snippets reference, so duplicates would be
   ambiguous. We reject duplicate names on create/rename, and auto-suffix
   collisions during bulk import.
2. **Relationship tracking** — a snippet references datasets _by name_ through
   its `datasetRefs: string[]` field. This is a bidirectional, name-based link:
   from a snippet you read its refs; from a dataset you scan snippets to find
   who uses it. Renaming a dataset must propagate to every snippet that points
   at it, in both the spec and the `datasetRefs` array, or the links rot.

The hard, testable logic is **pure** and lives in `src/core/`. The parts that
read and mutate stores live in `src/app/services/`.

---

## 1. Why names, not IDs, are the link

Datasets carry a numeric `id`, but snippets reference them **by name** because
that is what Vega-Lite uses: a spec resolves data through a named-data
reference, `{ "data": { "name": "MyDataset" } }`. The name _is_ the contract
between a spec and the dataset library. Storing a numeric id in the spec would
mean the spec is no longer a standalone, paste-anywhere Vega-Lite document.

The consequence: names must be unique (two datasets named `Sales` would make
`{ "data": { "name": "Sales" } }` ambiguous), and renaming a dataset is a
**graph operation**, not a single field write — every reference to the old name
must move with it.

---

## 2. Name uniqueness (pure — `src/core/naming.ts`)

### 2.1 Uniqueness check

Comparisons are **case-insensitive** (`Sales` and `sales` collide), so a single
display name maps to a single dataset regardless of how a user types a
reference. The check takes an optional `excludeId` so a rename can ignore the
record being renamed (renaming `Sales` to `Sales` is not a collision with
itself).

```ts
// src/core/naming.ts

/** Case-insensitive set of names already in use, minus an optional excluded id. */
export function isNameTaken(
  desired: string,
  datasets: ReadonlyArray<{ id: number; name: string }>,
  excludeId?: number,
): boolean {
  const lower = desired.trim().toLowerCase();
  return datasets.some((d) => d.id !== excludeId && d.name.toLowerCase() === lower);
}
```

### 2.2 Making a unique name

When a desired name is taken — during import, "extract inline data", or
"build chart" — we do **not** overwrite the existing dataset. We derive the
next free name by appending a numeric suffix: `Name` → `Name 2` → `Name 3`.
The function takes the set of existing names so it has no store dependency and
is trivially unit-testable.

```ts
// src/core/naming.ts

/**
 * Returns `desired` if free, else the first available `${desired} ${n}` (n >= 2).
 * `existingNames` is the set of names already in the collection.
 * Comparison is case-insensitive; the returned name preserves `desired`'s casing.
 */
export function makeUniqueName(desired: string, existingNames: Iterable<string>): string {
  const taken = new Set<string>();
  for (const n of existingNames) taken.add(n.toLowerCase());

  const base = desired.trim();
  if (!taken.has(base.toLowerCase())) return base;

  let n = 2;
  while (taken.has(`${base} ${n}`.toLowerCase())) n++;
  return `${base} ${n}`;
}
```

> If a base name already ends in a number (`Q1 2024`), the suffix still appends
> (`Q1 2024 2`). That is intentional: we never parse meaning out of the name,
> we only guarantee a free slot. Keep this dumb and predictable.

**Do**

- Use `isNameTaken` to reject duplicate create/rename in the UI before saving,
  and surface an error toast.
- Use `makeUniqueName` for every non-interactive path (import, extract, build)
  where blocking the user would be worse than a silent, reported rename.
- Pass `excludeId` on rename so an unchanged or case-only edit is allowed.

**Don't**

- Don't compare names case-sensitively anywhere — pick `toLowerCase()` once and
  use it consistently.
- Don't let `makeUniqueName` mutate a store or read store state; it takes plain
  data and returns a string.

---

## 3. The bidirectional snippet ↔ dataset link

```
            datasetRefs: ["Sales", "Regions"]          (forward, on the snippet)
   Snippet ───────────────────────────────────────────────────►  Dataset "Sales"
        ▲                                                                 │
        └──────────── scan all snippets for "Sales" in datasetRefs ◄──────┘
                                  (reverse, derived)
```

- **Forward** (snippet → datasets): read `snippet.datasetRefs`. Cheap, stored.
- **Reverse** (dataset → snippets): there is no stored back-pointer. We compute
  it by scanning snippets. Keeping it _derived_ means it can never disagree with
  the forward links — there is one source of truth.

`datasetRefs` is **derived from the spec**, not hand-maintained. It mirrors the
dataset names referenced by the **draft** spec — the version being edited — and
is recomputed on every change to the draft (auto-save, the Extract-to-Dataset
rewrite, revert) and on publish. Tracking the draft (not only the last publish)
keeps a snippet's linked-datasets display and the reverse lookup in step with
what the editor shows — so a hand-typed reference or an Extract links its dataset
without waiting for a publish. Recomputation runs only on a _valid_ spec —
auto-save is debounced and parse-gated (spec §03B) — so a transiently-invalid
draft never disturbs the links.

### 3.1 Extracting referenced names from a spec (pure — `src/core/spec-refs.ts`)

A Vega-Lite spec can reference named data in several places: the top-level
`data`, per-layer `data`, `data` inside `spec`/`facet`/`hconcat`/`vconcat`, and a
lookup transform's `from.data`. A spec may also define its OWN inline datasets via
a top-level `datasets` map — those are self-defined, not library references.
Rather than enumerate Vega-Lite's grammar, we walk the spec recursively and
collect every `{ data: { name } }` — but **prune two keys**: never recurse into a
`data` object's payload (its `values`/rows) or the top-level `datasets` map,
because those hold user data, not nested specs. Without the prune, a data _row_
carrying a field literally named `data: { name: "x" }` is misread as a reference.
This is pure, deterministic, and the most heavily unit-tested function here.

```ts
// src/core/spec-refs.ts

type Json = unknown;

/** Collects every library dataset name referenced by `{ data: { name } }`, excluding self-defined ones. */
export function extractDatasetRefs(spec: Json): string[] {
  const root = typeof spec === 'string' ? safeParse(spec) : spec;
  const selfDefined = selfDefinedNames(root); // names from the spec's own top-level `datasets`
  const names = new Set<string>();

  const walk = (node: Json): void => {
    if (Array.isArray(node)) {
      for (const item of node) walk(item);
      return;
    }
    if (node && typeof node === 'object') {
      const obj = node as Record<string, Json>;
      const data = obj.data as Record<string, Json> | undefined;
      if (data && typeof data === 'object' && typeof data.name === 'string') {
        if (!selfDefined.has(data.name)) names.add(data.name);
      }
      // Prune: a `data` payload and the `datasets` map hold user data, not refs.
      for (const key of Object.keys(obj)) {
        if (key === 'data' || key === 'datasets') continue;
        walk(obj[key]);
      }
    }
  };

  walk(root);
  return [...names];
}

function safeParse(s: string): Json {
  try {
    return JSON.parse(s);
  } catch {
    return null; // an unparseable draft simply has no resolvable refs
  }
}
```

```ts
// src/core/spec-refs.ts — thin wrapper, run on every draft change and on publish

/** The list stored on snippet.datasetRefs. Sorted + de-duped for stable diffs. */
export function recomputeDatasetRefs(spec: Json): string[] {
  return extractDatasetRefs(spec).sort();
}
```

> A `spec` may be an object or a string (see the Data Model). Normalize once,
> at the boundary, so the recursive walk never has to care.

**Do**

- Treat `extractDatasetRefs` as the single source of truth for "what does this
  spec reference". The reverse-lookup and rename paths both depend on it
  agreeing with what the renderer actually resolves.
- Recompute and store `datasetRefs` on **every draft change and on publish** —
  but only through the parse-gated, debounced auto-save (`commitDraft`) and the
  programmatic extract/revert rewrites, never on raw keystrokes. That keeps the
  links in step with the edited draft while never recomputing from a
  transiently-invalid spec.
- Prune the **same two keys** (`data`, `datasets`) in all three ref walks —
  extraction here, `renameDatasetInSpec`, and the renderer's `resolveDatasetRefs`
  (`src/core/rendering.ts`). They must agree on what counts as a reference; if one
  descends into data payloads and another doesn't, extraction and rendering
  disagree and a row field named `data` either gets counted, rewritten, or throws
  `DatasetNotFoundError`.

**Don't**

- Don't let two code paths each have their own idea of "referenced names".
  Renamer, ref-recomputer, and renderer must use the same walk shape.
- Don't "enumerate the grammar" (scope the walk to a fixed list of container
  keys) to fix the payload-descent problem — pruning the two data-bearing keys
  stays correct as Vega-Lite's composition grammar grows; an allow-list rots.

---

## 4. Reverse lookup: who uses this dataset?

The Dataset Manager shows a **usage badge** and a **Linked Snippets** list; the
Snippet Library shows a snippet's linked datasets. Both come from one scan — no
stored back-pointer to drift.

The scan itself is **pure** and lives in core, taking the snippets as a parameter
so the _same_ implementation serves two callers: the reactive UI (which passes its
live `snippets` selection straight in) and the non-reactive service wrapper (which
passes a store snapshot for programmatic callers). This is what makes "who
references this" have exactly one implementation (§6).

```ts
// src/core/relationships.ts — pure scan (unit-tested)

/** Snippets whose datasetRefs include `name` (case-insensitive). */
export function snippetsReferencingDataset<T extends { datasetRefs: readonly string[] }>(
  snippets: readonly T[],
  name: string,
): T[] {
  const lower = name.toLowerCase();
  return snippets.filter((s) => s.datasetRefs.some((ref) => ref.toLowerCase() === lower));
}

/** Usage counts for the whole library in one pass, keyed by lower-cased name. */
export function datasetUsageCounts(
  snippets: readonly { datasetRefs: readonly string[] }[],
): Map<string, number>;
```

There is no app-layer wrapper module around the scan. The reactive UI — a
component subscribed to `snippets` — calls the core helper directly, so the
badge and Linked Snippets list update the moment any snippet's draft changes
its refs (auto-save) or it is published. A non-reactive caller binds the same
helper to a snapshot inline at its call site
(`snippetsReferencingDataset(useSnippetStore.getState().snippets, name)`) —
one matching implementation, no `getState()` wrapper that would go stale in
reactive code.

**Do**

- Keep reverse lookup a pure scan, parameterized on the snippets so both the
  reactive and snapshot callers share it. It is O(snippets) but the collections
  are small (library budget ~5 MB); clarity beats an index.
- Call the core helper directly from a store-subscribed component for reactivity;
  bind it to a `getState()` snapshot inline for non-reactive code.

**Don't**

- Don't add a `referencedBy` array to datasets. A stored reverse pointer is a
  second source of truth that _will_ fall out of sync with `datasetRefs`.

---

## 5. Import: auto-suffix collisions, then report

On import we never overwrite an existing dataset. A dataset whose name collides
is renamed to a unique name via `makeUniqueName`, and **every rename is
collected and reported to the user** (toast / summary) so the change is never
silent. Crucially, names are reserved _as we go_ — within a single import, two
incoming `Sales` datasets become `Sales 2` and `Sales 3`, not two `Sales 2`.

```ts
// src/core/import-normalize.ts

import { makeUniqueName } from './naming';
import type { Dataset } from './dataset';

export interface DatasetRename {
  from: string;
  to: string;
}

/**
 * Returns incoming datasets with collision-free names, plus the renames applied.
 * `existing` are names already in the library; `incoming` are datasets to add.
 */
export function dedupeIncomingDatasetNames(
  existing: ReadonlyArray<string>,
  incoming: ReadonlyArray<Dataset>,
): { datasets: Dataset[]; renames: DatasetRename[] } {
  const reserved = new Set(existing.map((n) => n.toLowerCase()));
  const renames: DatasetRename[] = [];

  const datasets = incoming.map((d) => {
    const unique = makeUniqueName(d.name, reserved);
    reserved.add(unique.toLowerCase()); // reserve so later imports don't collide
    if (unique !== d.name) renames.push({ from: d.name, to: unique });
    return unique === d.name ? d : { ...d, name: unique };
  });

  return { datasets, renames };
}
```

> If imported snippets reference the renamed dataset, their `datasetRefs` and
> specs must be rewritten to the new name too. The import flow does this purely,
> before anything is committed: `applyDatasetRenamesToSnippets`
> (`core/import-normalize`) rewrites the incoming snippet set per applied rename
> using the same `renameDatasetInSpec` machinery as §6.

### 5.1 Where the import/export flow lives, and its rules

**Flow:** header (Import/Export buttons in `App.tsx`) → `services/transfer.ts`
(the only store-touching layer) → pure core (`core/import-normalize.ts` shape
detection + normalization + the dedupe/rename/id-reassign helpers; `core/export-envelope.ts`)

- browser IO (`infrastructure/file-transfer.ts`). The pure helpers are unit-tested
  hardest; `transfer.ts` only orchestrates (read stores → call core → commit →
  notify). The behavioral contract is spec §08.

Three rules a future change must keep:

- **Datasets commit before snippets** (`DatasetStore.addDatasets` then
  `SnippetStore.addSnippets`) so a snippet's by-name reference resolves against the
  just-added (possibly suffixed) dataset.
- **Imported datasets get fresh monotonic numeric ids** (`addDatasets`), not their
  envelope ids. Safe — and necessary — because datasets are linked **by name, not
  id** (§1): id reuse would collide in IndexedDB, but renaming the _id_ breaks
  nothing. (This is why the old `Date.now()`-collision TODO on `add` doesn't bite import.)
- **Rename propagation reads the spec, not only `datasetRefs`.**
  `applyDatasetRenamesToSnippets` finds the referenced name via
  `extractDatasetRefs(spec)` ∪ `datasetRefs`, so an imported snippet whose
  `datasetRefs` is absent/stale (a hand-crafted or foreign file) still gets its
  spec rewritten — the renderer resolves by spec, so a missed rename would break it.

**Do**

- Reserve each chosen name immediately so collisions _within_ one import are
  also resolved.
- Return the rename list and show it; a silent rename looks like data loss.

**Don't**

- Don't overwrite or merge a same-named existing dataset on import. Suffix and
  keep both — the user decides what to delete.

---

## 6. Rename propagation: keep the link consistent

Renaming a dataset is the operation that ties §2–§5 together. A rename must, in
one atomic step:

1. Update the dataset's own `name`.
2. For **every snippet referencing the old name**: rewrite the named-data
   references inside its spec (`{ "data": { "name": "old" } }` →
   `{ "data": { "name": "new" } }`) — in **both** `spec` and `draftSpec`.
3. Recompute that snippet's `datasetRefs` from the rewritten **draft** spec (the
   tracked surface), so the forward link mirrors reality and the reverse scan
   stays correct.

The spec rewrite is pure; the orchestration reads and writes stores.

```ts
// src/core/spec-refs.ts — pure rewrite

/** Returns a copy of `spec` with every data.name === oldName replaced by newName. */
export function renameDatasetInSpec(spec: Json, oldName: string, newName: string): Json {
  const obj = typeof spec === 'string' ? safeParse(spec) : spec;
  const selfDefined = selfDefinedNames(obj); // never rename a spec's own inline dataset name

  const rewrite = (node: Json): Json => {
    if (Array.isArray(node)) return node.map(rewrite);
    if (node && typeof node === 'object') {
      const out: Record<string, Json> = {};
      for (const [k, v] of Object.entries(node as Record<string, Json>)) {
        // A `data` object is a reference site: rename a matching name, but never
        // recurse into its payload. `datasets` (self-defined inline data) is left
        // whole. Same prune as extractDatasetRefs — see §3.1.
        if (k === 'data' && v && typeof v === 'object' && !Array.isArray(v)) {
          const dv = v as Record<string, Json>;
          out[k] = dv.name === oldName && !selfDefined.has(oldName) ? { ...dv, name: newName } : dv;
        } else if (k === 'data' || k === 'datasets') {
          out[k] = v;
        } else {
          out[k] = rewrite(v);
        }
      }
      return out;
    }
    return node;
  };

  const rewritten = rewrite(obj);
  // Preserve the original spec's stored shape (string vs object).
  return typeof spec === 'string' ? JSON.stringify(rewritten, null, 2) : rewritten;
}
```

The one rename implementation in the live app is the store action
**`SnippetStore.renameDatasetRefs(oldName, newName, now)`** — the action that
owns the snippet collection. It matches by spec+draft _content_ (not
`datasetRefs`, which mirrors only the draft), rewrites `spec`/`draftSpec` via
`renameDatasetInSpec`, recomputes `datasetRefs` from the rewritten draft, and
returns the number of snippets changed. Keeping the loop in the store means the
reactive editor buffer is refreshed in the same atomic update when the active
snippet's draft is rewritten. `DatasetStore.update` calls it whenever a save
changes a dataset's name, so a rename propagates everywhere as part of the one
user action — there is no separate coordinator module to call.

> **Collision on rename.** The UI rename form rejects a name already in use via
> `isNameTaken(newName, datasets, dataset.id)`. Programmatic renames (e.g. an
> import flow) instead resolve with `makeUniqueName` first. The propagation
> action itself does not invent a name — it assumes `newName` is the agreed
> target.

**Do**

- Rewrite `spec` **and** `draftSpec`. A user mid-edit must not see their draft
  silently break because the dataset was renamed underneath them.
- Recompute `datasetRefs` from the rewritten **draft** spec rather than
  string-replacing the array — the draft is the source of truth, the array is
  its mirror.
- Find affected snippets by scanning each one's **spec and draft content** for
  the old name (`extractDatasetRefs`), not by its `datasetRefs` array. Because
  `datasetRefs` mirrors the draft, a name referenced only by the still-published
  spec (the user removed it from the draft but hasn't published) is absent from
  the array; matching on content rewrites it anyway, so the published spec can't
  rot to a renamed-away dataset. Scan the content (don't reserialize) so a
  non-referencing snippet's text stays byte-for-byte intact. The import-side
  `applyDatasetRenamesToSnippets` already follows this spec-content rule (§5.1).

**Don't**

- Don't update `datasetRefs` without also rewriting the spec — the rendered
  named-data reference would still point at the old, now-missing name.
- Don't rename the dataset and skip propagation "for now". A half-applied rename
  is the exact inconsistency this whole document exists to prevent.

---

## 7. Where things live

| Concern                                                                  | Location                         | Pure?               | Tested      |
| ------------------------------------------------------------------------ | -------------------------------- | ------------------- | ----------- |
| `makeUniqueName`, `isNameTaken`                                          | `src/core/naming.ts`             | yes                 | unit        |
| `extractDatasetRefs`, `recomputeDatasetRefs`                             | `src/core/spec-refs.ts`          | yes                 | unit        |
| `renameDatasetInSpec`                                                    | `src/core/spec-refs.ts`          | yes                 | unit        |
| `snippetsReferencingDataset`, `datasetUsageCounts` (reverse-lookup scan) | `src/core/relationships.ts`      | yes                 | unit        |
| `renameDatasetRefs` → updated count (rename propagation)                 | `src/app/stores/SnippetStore.ts` | no (mutates stores) | integration |
| `dedupeIncomingDatasetNames`                                             | `src/core/import-normalize.ts`   | yes                 | unit        |

The dividing line: anything that takes plain data and returns plain data is
**core** and unit-tested in isolation; anything that reaches into a Zustand store
is a **store action or app service**. The rule of thumb — _the **draft** spec is the source of
truth, `datasetRefs` mirrors it, the reverse lookup is derived_ — is what keeps
the bidirectional link from ever needing manual repair.