astrolabe/docs/architecture/07-naming-and-relationships.md

# Naming & Relationships

How Astrolabe keeps entity **names unique** within a collection, and how it
tracks the **bidirectional links** between snippets and datasets so they stay
consistent as entities are created, imported, and renamed.

Two concerns live here, and they reinforce each other:

1. **Name uniqueness** — every dataset has a unique name. Names are the primary
   key users see and the key snippets reference, so duplicates would be
   ambiguous. We reject duplicate names on create/rename, and auto-suffix
   collisions during bulk import.
2. **Relationship tracking** — a snippet references datasets _by name_ through
   its `datasetRefs: string[]` field. This is a bidirectional, name-based link:
   from a snippet you read its refs; from a dataset you scan snippets to find
   who uses it. Renaming a dataset must propagate to every snippet that points
   at it, in both the spec and the `datasetRefs` array, or the links rot.

The hard, testable logic is **pure** and lives in `src/core/`. The parts that
read and mutate stores live in `src/app/services/`.

---

## 1. Why names, not IDs, are the link

Datasets carry a numeric `id`, but snippets reference them **by name** because
that is what Vega-Lite uses: a spec resolves data through a named-data
reference, `{ "data": { "name": "MyDataset" } }`. The name _is_ the contract
between a spec and the dataset library. Storing a numeric id in the spec would
mean the spec is no longer a standalone, paste-anywhere Vega-Lite document.

The consequence: names must be unique (two datasets named `Sales` would make
`{ "data": { "name": "Sales" } }` ambiguous), and renaming a dataset is a
**graph operation**, not a single field write — every reference to the old name
must move with it.

---

## 2. Name uniqueness (pure — `src/core/naming.ts`)

### 2.1 Uniqueness check

Comparisons are **case-insensitive** (`Sales` and `sales` collide), so a single
display name maps to a single dataset regardless of how a user types a
reference. The check takes an optional `excludeId` so a rename can ignore the
record being renamed (renaming `Sales` to `Sales` is not a collision with
itself).

```ts
// src/core/naming.ts

/** Case-insensitive set of names already in use, minus an optional excluded id. */
export function isNameTaken(
  desired: string,
  datasets: ReadonlyArray<{ id: number; name: string }>,
  excludeId?: number,
): boolean {
  const lower = desired.trim().toLowerCase();
  return datasets.some((d) => d.id !== excludeId && d.name.toLowerCase() === lower);
}
```

### 2.2 Making a unique name

When a desired name is taken — during import, "extract inline data", or
"build chart" — we do **not** overwrite the existing dataset. We derive the
next free name by appending a numeric suffix: `Name` → `Name 2` → `Name 3`.
The function takes the set of existing names so it has no store dependency and
is trivially unit-testable.

```ts
// src/core/naming.ts

/**
 * Returns `desired` if free, else the first available `${desired} ${n}` (n >= 2).
 * `existingNames` is the set of names already in the collection.
 * Comparison is case-insensitive; the returned name preserves `desired`'s casing.
 */
export function makeUniqueName(desired: string, existingNames: Iterable<string>): string {
  const taken = new Set<string>();
  for (const n of existingNames) taken.add(n.toLowerCase());

  const base = desired.trim();
  if (!taken.has(base.toLowerCase())) return base;

  let n = 2;
  while (taken.has(`${base} ${n}`.toLowerCase())) n++;
  return `${base} ${n}`;
}
```

> If a base name already ends in a number (`Q1 2024`), the suffix still appends
> (`Q1 2024 2`). That is intentional: we never parse meaning out of the name,
> we only guarantee a free slot. Keep this dumb and predictable.

**Do**

- Use `isNameTaken` to reject duplicate create/rename in the UI before saving,
  and surface an error toast.
- Use `makeUniqueName` for every non-interactive path (import, extract, build)
  where blocking the user would be worse than a silent, reported rename.
- Pass `excludeId` on rename so an unchanged or case-only edit is allowed.

**Don't**

- Don't compare names case-sensitively anywhere — pick `toLowerCase()` once and
  use it consistently.
- Don't let `makeUniqueName` mutate a store or read store state; it takes plain
  data and returns a string.

---

## 3. The bidirectional snippet ↔ dataset link

```
            datasetRefs: ["Sales", "Regions"]          (forward, on the snippet)
   Snippet ───────────────────────────────────────────────────►  Dataset "Sales"
        ▲                                                                 │
        └──────────── scan all snippets for "Sales" in datasetRefs ◄──────┘
                                  (reverse, derived)
```

- **Forward** (snippet → datasets): read `snippet.datasetRefs`. Cheap, stored.
- **Reverse** (dataset → snippets): there is no stored back-pointer. We compute
  it by scanning snippets. Keeping it _derived_ means it can never disagree with
  the forward links — there is one source of truth.

`datasetRefs` is **derived from the spec**, not hand-maintained. It is
recomputed whenever a snippet is published (its draft spec is promoted), so it
always mirrors the dataset names actually referenced in the published spec.

### 3.1 Extracting referenced names from a spec (pure — `src/core/spec-refs.ts`)

A Vega-Lite spec can reference named data in several places: the top-level
`data`, per-layer `data`, `data` inside `spec`/`facet`/`hconcat`/`vconcat`, and a
lookup transform's `from.data`. A spec may also define its OWN inline datasets via
a top-level `datasets` map — those are self-defined, not library references.
Rather than enumerate Vega-Lite's grammar, we walk the spec recursively and
collect every `{ data: { name } }` — but **prune two keys**: never recurse into a
`data` object's payload (its `values`/rows) or the top-level `datasets` map,
because those hold user data, not nested specs. Without the prune, a data _row_
carrying a field literally named `data: { name: "x" }` is misread as a reference.
This is pure, deterministic, and the most heavily unit-tested function here.

```ts
// src/core/spec-refs.ts

type Json = unknown;

/** Collects every library dataset name referenced by `{ data: { name } }`, excluding self-defined ones. */
export function extractDatasetRefs(spec: Json): string[] {
  const root = typeof spec === 'string' ? safeParse(spec) : spec;
  const selfDefined = selfDefinedNames(root); // names from the spec's own top-level `datasets`
  const names = new Set<string>();

  const walk = (node: Json): void => {
    if (Array.isArray(node)) {
      for (const item of node) walk(item);
      return;
    }
    if (node && typeof node === 'object') {
      const obj = node as Record<string, Json>;
      const data = obj.data as Record<string, Json> | undefined;
      if (data && typeof data === 'object' && typeof data.name === 'string') {
        if (!selfDefined.has(data.name)) names.add(data.name);
      }
      // Prune: a `data` payload and the `datasets` map hold user data, not refs.
      for (const key of Object.keys(obj)) {
        if (key === 'data' || key === 'datasets') continue;
        walk(obj[key]);
      }
    }
  };

  walk(root);
  return [...names];
}

function safeParse(s: string): Json {
  try {
    return JSON.parse(s);
  } catch {
    return null; // an unparseable draft simply has no resolvable refs
  }
}
```

```ts
// src/core/spec-refs.ts — thin wrapper used at publish time

/** The list stored on snippet.datasetRefs. Sorted + de-duped for stable diffs. */
export function recomputeDatasetRefs(spec: Json): string[] {
  return extractDatasetRefs(spec).sort();
}
```

> A `spec` may be an object or a string (see the Data Model). Normalize once,
> at the boundary, so the recursive walk never has to care.

**Do**

- Treat `extractDatasetRefs` as the single source of truth for "what does this
  spec reference". The reverse-lookup and rename paths both depend on it
  agreeing with what the renderer actually resolves.
- Recompute and store `datasetRefs` on **publish**, not on every keystroke —
  the draft can be transiently invalid, and only the published spec is shared.
- Prune the **same two keys** (`data`, `datasets`) in all three ref walks —
  extraction here, `renameDatasetInSpec`, and the renderer's `resolveDatasetRefs`
  (`src/core/rendering.ts`). They must agree on what counts as a reference; if one
  descends into data payloads and another doesn't, extraction and rendering
  disagree and a row field named `data` either gets counted, rewritten, or throws
  `DatasetNotFoundError`.

**Don't**

- Don't let two code paths each have their own idea of "referenced names".
  Renamer, ref-recomputer, and renderer must use the same walk shape.
- Don't "enumerate the grammar" (scope the walk to a fixed list of container
  keys) to fix the payload-descent problem — pruning the two data-bearing keys
  stays correct as Vega-Lite's composition grammar grows; an allow-list rots.

---

## 4. Reverse lookup: who uses this dataset?

The Dataset Manager shows a **usage badge** and a **Linked Snippets** list; the
Snippet Library shows a snippet's linked datasets. Both come from one selector
scan — no stored back-pointer to drift.

```ts
// src/app/services/RelationshipService.ts

import { useSnippetStore } from '../stores/SnippetStore';
import type { Snippet } from '../../core/types';

/** Snippets whose datasetRefs include `name` (case-insensitive). */
export function findSnippetsReferencingDataset(name: string): Snippet[] {
  const lower = name.toLowerCase();
  return useSnippetStore
    .getState()
    .snippets.filter((s) => s.datasetRefs.some((ref) => ref.toLowerCase() === lower));
}

/** Count for the usage badge. */
export function datasetUsageCount(name: string): number {
  return findSnippetsReferencingDataset(name).length;
}
```

Because this reads `useSnippetStore.getState().snippets`, exposing it as a
selector for the UI makes the badge and Linked Snippets list reactive for free —
they update the moment any snippet is published with changed refs.

**Do**

- Keep reverse lookup a pure scan over the store. It is O(snippets) but the
  collections are small (library budget ~5 MB); clarity beats an index.
- Expose it as a selector where the UI needs reactivity.

**Don't**

- Don't add a `referencedBy` array to datasets. A stored reverse pointer is a
  second source of truth that _will_ fall out of sync with `datasetRefs`.

---

## 5. Import: auto-suffix collisions, then report

On import we never overwrite an existing dataset. A dataset whose name collides
is renamed to a unique name via `makeUniqueName`, and **every rename is
collected and reported to the user** (toast / summary) so the change is never
silent. Crucially, names are reserved _as we go_ — within a single import, two
incoming `Sales` datasets become `Sales 2` and `Sales 3`, not two `Sales 2`.

```ts
// src/app/services/ImportService.ts

import { makeUniqueName } from '../../core/naming';
import type { Dataset } from '../../core/types';

export interface DatasetRename {
  from: string;
  to: string;
}

/**
 * Returns incoming datasets with collision-free names, plus the renames applied.
 * `existing` are names already in the library; `incoming` are datasets to add.
 */
export function dedupeIncomingDatasetNames(
  existing: ReadonlyArray<string>,
  incoming: ReadonlyArray<Dataset>,
): { datasets: Dataset[]; renames: DatasetRename[] } {
  const reserved = new Set(existing.map((n) => n.toLowerCase()));
  const renames: DatasetRename[] = [];

  const datasets = incoming.map((d) => {
    const unique = makeUniqueName(d.name, reserved);
    reserved.add(unique.toLowerCase()); // reserve so later imports don't collide
    if (unique !== d.name) renames.push({ from: d.name, to: unique });
    return unique === d.name ? d : { ...d, name: unique };
  });

  return { datasets, renames };
}
```

> If imported snippets reference the renamed dataset, their `datasetRefs` and
> specs must be rewritten to the new name too — reuse the rename machinery in
> §6 over the imported snippet set, or run `renameDatasetEverywhere` per applied
> rename after the import is committed.

### 5.1 Where the import/export flow lives, and its rules

**Flow:** header (Import/Export buttons in `App.tsx`) → `services/transfer.ts`
(the only store-touching layer) → pure core (`core/import-normalize.ts` shape
detection + normalization + the dedupe/rename/id-reassign helpers; `core/export-envelope.ts`)

- browser IO (`infrastructure/file-transfer.ts`). The pure helpers are unit-tested
  hardest; `transfer.ts` only orchestrates (read stores → call core → commit →
  notify). The behavioral contract is spec §08.

Three rules a future change must keep:

- **Datasets commit before snippets** (`DatasetStore.addDatasets` then
  `SnippetStore.addSnippets`) so a snippet's by-name reference resolves against the
  just-added (possibly suffixed) dataset.
- **Imported datasets get fresh monotonic numeric ids** (`addDatasets`), not their
  envelope ids. Safe — and necessary — because datasets are linked **by name, not
  id** (§1): id reuse would collide in IndexedDB, but renaming the _id_ breaks
  nothing. (This is why the old `Date.now()`-collision TODO on `add` doesn't bite import.)
- **Rename propagation reads the spec, not only `datasetRefs`.**
  `applyDatasetRenamesToSnippets` finds the referenced name via
  `extractDatasetRefs(spec)` ∪ `datasetRefs`, so an imported snippet whose
  `datasetRefs` is absent/stale (a hand-crafted or foreign file) still gets its
  spec rewritten — the renderer resolves by spec, so a missed rename would break it.

**Do**

- Reserve each chosen name immediately so collisions _within_ one import are
  also resolved.
- Return the rename list and show it; a silent rename looks like data loss.

**Don't**

- Don't overwrite or merge a same-named existing dataset on import. Suffix and
  keep both — the user decides what to delete.

---

## 6. Rename propagation: keep the link consistent

Renaming a dataset is the operation that ties §2–§5 together. A rename must, in
one atomic step:

1. Update the dataset's own `name`.
2. For **every snippet referencing the old name**: rewrite the named-data
   references inside its spec (`{ "data": { "name": "old" } }` →
   `{ "data": { "name": "new" } }`) — in **both** `spec` and `draftSpec`.
3. Recompute that snippet's `datasetRefs` from the rewritten spec, so the
   forward link mirrors reality and the reverse scan stays correct.

The spec rewrite is pure; the orchestration reads and writes stores.

```ts
// src/core/spec-refs.ts — pure rewrite

/** Returns a copy of `spec` with every data.name === oldName replaced by newName. */
export function renameDatasetInSpec(spec: Json, oldName: string, newName: string): Json {
  const obj = typeof spec === 'string' ? safeParse(spec) : spec;
  const selfDefined = selfDefinedNames(obj); // never rename a spec's own inline dataset name

  const rewrite = (node: Json): Json => {
    if (Array.isArray(node)) return node.map(rewrite);
    if (node && typeof node === 'object') {
      const out: Record<string, Json> = {};
      for (const [k, v] of Object.entries(node as Record<string, Json>)) {
        // A `data` object is a reference site: rename a matching name, but never
        // recurse into its payload. `datasets` (self-defined inline data) is left
        // whole. Same prune as extractDatasetRefs — see §3.1.
        if (k === 'data' && v && typeof v === 'object' && !Array.isArray(v)) {
          const dv = v as Record<string, Json>;
          out[k] = dv.name === oldName && !selfDefined.has(oldName) ? { ...dv, name: newName } : dv;
        } else if (k === 'data' || k === 'datasets') {
          out[k] = v;
        } else {
          out[k] = rewrite(v);
        }
      }
      return out;
    }
    return node;
  };

  const rewritten = rewrite(obj);
  // Preserve the original spec's stored shape (string vs object).
  return typeof spec === 'string' ? JSON.stringify(rewritten, null, 2) : rewritten;
}
```

```ts
// src/app/services/RelationshipService.ts — store coordination

import { isNameTaken, makeUniqueName } from '../../core/naming';
import { renameDatasetInSpec, recomputeDatasetRefs } from '../../core/spec-refs';
import { useDatasetStore } from '../stores/DatasetStore';
import { useSnippetStore } from '../stores/SnippetStore';

/**
 * Renames a dataset and propagates the rename to every referencing snippet
 * (spec, draftSpec, and datasetRefs). Returns the snippets that changed.
 * Caller is responsible for collision policy on `newName` (reject vs suffix).
 */
export function renameDatasetEverywhere(oldName: string, newName: string): { updated: number } {
  if (oldName === newName) return { updated: 0 };

  // 1. Rename the dataset record itself.
  const dataset = useDatasetStore.getState().datasets.find((d) => d.name === oldName);
  if (!dataset) return { updated: 0 };
  useDatasetStore.getState().update(dataset.id, { name: newName });

  // 2 + 3. Rewrite every referencing snippet's specs and refs.
  let updated = 0;
  for (const snippet of useSnippetStore.getState().snippets) {
    if (!snippet.datasetRefs.some((r) => r.toLowerCase() === oldName.toLowerCase())) continue;

    const spec = renameDatasetInSpec(snippet.spec, oldName, newName);
    const draftSpec = renameDatasetInSpec(snippet.draftSpec, oldName, newName);
    useSnippetStore.getState().update(snippet.id, {
      spec,
      draftSpec,
      datasetRefs: recomputeDatasetRefs(spec),
    });
    updated++;
  }

  return { updated };
}
```

> **Collision on rename.** The UI rename form rejects a name already in use via
> `isNameTaken(newName, datasets, dataset.id)`. Programmatic renames (e.g. an
> import flow) instead resolve with `makeUniqueName` before calling
> `renameDatasetEverywhere`. The propagation function itself does not invent a
> name — it assumes `newName` is the agreed target.

**Do**

- Rewrite `spec` **and** `draftSpec`. A user mid-edit must not see their draft
  silently break because the dataset was renamed underneath them.
- Recompute `datasetRefs` from the rewritten spec rather than string-replacing
  the array — the spec is the source of truth, the array is its mirror.
- Use the §4 reverse lookup to find affected snippets, so "who references this"
  has exactly one implementation.

**Don't**

- Don't update `datasetRefs` without also rewriting the spec — the rendered
  named-data reference would still point at the old, now-missing name.
- Don't rename the dataset and skip propagation "for now". A half-applied rename
  is the exact inconsistency this whole document exists to prevent.

---

## 7. Where things live

| Concern                                       | Location                                  | Pure?                          | Tested           |
| --------------------------------------------- | ----------------------------------------- | ------------------------------ | ---------------- |
| `makeUniqueName`, `isNameTaken`               | `src/core/naming.ts`                      | yes                            | unit             |
| `extractDatasetRefs`, `recomputeDatasetRefs`  | `src/core/spec-refs.ts`                   | yes                            | unit             |
| `renameDatasetInSpec`                         | `src/core/spec-refs.ts`                   | yes                            | unit             |
| `findSnippetsReferencingDataset`, usage count | `src/app/services/RelationshipService.ts` | no (reads store)               | integration      |
| `renameDatasetEverywhere`                     | `src/app/services/RelationshipService.ts` | no (mutates stores)            | integration      |
| `dedupeIncomingDatasetNames`                  | `src/app/services/ImportService.ts`       | nearly (uses `makeUniqueName`) | unit/integration |

The dividing line: anything that takes plain data and returns plain data is
**core** and unit-tested in isolation; anything that reaches into a Zustand store
is an **app service**. The rename rule of thumb — _the spec is the source of
truth, `datasetRefs` mirrors it, the reverse lookup is derived_ — is what keeps
the bidirectional link from ever needing manual repair.