astrolabe/docs/spec/05-datasets.md

# 05 · Datasets

The **Dataset Manager** is a modal for creating and managing named, reusable datasets that snippets can reference by name. It is the home of the dataset library: a place separate from snippets where data lives once and is shared across many visualizations.

## Purpose & Model

Datasets are named blobs of data stored in the user's local library, independent of any single snippet. A snippet references a dataset by name rather than embedding the data inline, so the same data can power many snippets and be edited in one place.

- Datasets persist locally across sessions in a high-capacity local store, far larger than the budget available to snippets — large datasets belong here, not inline in specs.
- A snippet references a dataset using a Vega-Lite named-data reference, e.g. `{ "data": { "name": "MyDataset" } }`. When the _Live Preview_ renders a spec, it resolves any such named reference against the dataset library (see _Live Preview_).
- See _Data Model_ for the stored shape of a dataset.

## Opening & Navigation

- Opened from a header control or via the keyboard shortcut Cmd/Ctrl+K.
- The current view and the selected dataset are reflected in the URL, so a selected dataset produces a shareable/back-navigable location (see _Application Shell & Navigation_).
- Closing the modal clears the current selection and any open create form.

## Layout

A two-pane modal:

- **List pane** (left): a "New Dataset" action plus the list of all datasets, sorted most-recently-modified first.
- **Detail pane** (right): shows the selected dataset's details, the create form when creating, or an empty prompt ("Select a dataset or create a new one") when nothing is selected.

### List item

Each list item shows:

- The dataset **name**.
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). A URL dataset that has not yet been fetched shows "not fetched" in place of the figures it does not have.
- A **usage badge** when one or more snippets reference the dataset, indicating how many.

Clicking an item selects it and shows its detail. Per-item actions (delete, plus copy-reference and build-chart) live in the detail pane for the selected dataset.

## Source Types

A dataset has one of two source types, chosen when creating it:

- **Inline** — the data itself is pasted in and stored directly in the library.
- **URL** — the dataset is fetched once from a remote http/https address when it is created, and the fetched data is **snapshotted** into the library, along with the address (kept so the snapshot can be re-fetched). A URL dataset then works offline and renders from its local copy; it is never re-fetched at render time, only on demand (see _Refresh_, below).

Either way the library holds the full data and profiles it. A URL dataset additionally records the address it was fetched from and when it was last fetched. Because the fetch happens in the browser, an address the browser cannot reach — offline, or one that blocks cross-origin requests — cannot be snapshotted; see _Actions → New / Create New_ for how that failure is handled.

## Supported Formats

Four data formats are supported, named in the UI and stored on the dataset:

- **JSON** — an array of objects (most common, profilable) or a single object.
- **CSV** — comma-separated with a header row.
- **TSV** — tab-separated with a header row.
- **TopoJSON** — topology/map data (a JSON object whose type marks it as a topology).

### Auto-detection

When the user pastes inline data, the app auto-detects the format and reports a **confidence** level (high / medium / low):

- Valid JSON parses to JSON, or to TopoJSON when it is a topology object — high confidence.
- Otherwise, multi-line text with a header row is detected as TSV (when tab-separated) or CSV (when comma-separated) — medium confidence.
- Unrecognized input yields no format (low confidence); saving is blocked with a message asking the user to check the input.

The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the create form shows a format hint inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`); the dataset's actual format is determined from the **fetched content** when it is created, falling back to the extension when the content is ambiguous.

## Profiling

For tabular data (JSON array-of-objects, CSV, TSV) — whether pasted inline or fetched from a URL — the app computes and stores a profile:

- **Row count** and **column count**.
- The list of **column names**.
- An **inferred type per column**: number, text/string, date, or boolean. Type inference looks at the column's values: all-numeric becomes number, all `true`/`false` becomes boolean, otherwise string; empty cells are ignored.
- **Size** in bytes of the stored data.

A **truncated data preview** of the raw data is also retained for display. A fetched URL dataset is profiled exactly like inline data; only non-tabular data and a URL dataset that has **not yet been fetched** are unprofiled (counts show "N/A").

## Detail Panel

The detail pane for a selected dataset shows:

- **Name**.
- **Comment** (optional free-text notes), when present.
- **Source** (URL datasets only): the address the snapshot was fetched from, and when it was last fetched (or "Not fetched yet"), alongside the **Refresh** action.
- **Overview**: statistics (rows, columns, size), the **column list** with each column's name and inferred type shown with a simple type indicator, and created/modified timestamps.
- **Preview**: a sample of the data. Tabular datasets (CSV, TSV, or a JSON array-of-objects — inline or fetched) render as a **table** of the first rows under the profiled column names, with a note when more rows exist than are shown. Non-tabular payloads (a single JSON object, TopoJSON) render as pretty-printed JSON, and a URL dataset that has not yet been fetched shows a short placeholder.
- **Linked Snippets**: the list of snippets that reference this dataset by name. This is the dataset side of bidirectional dataset↔snippet linking (see _Snippet Library_).

## Actions

A destructive or off-screen outcome raises a confirming toast; an action whose result is immediately visible is confirmed by that change. Any action may raise an error toast on failure (see _Application Shell & Navigation_ → Toasts).

- **Copy Reference** — copies the by-name reference object to the clipboard, ready to paste into a spec:
  `{ "data": { "name": "MyDataset" } }`
  The clipboard write is invisible, so it is confirmed _inline on the control_ ("Copied"), announced politely to assistive technology — not a toast.
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. For a **URL** dataset, saving fetches and snapshots the address; the Save control shows a busy state while fetching. If the fetch fails — offline, blocked by the host (cross-origin), not found, empty, timed out, or larger than the fetch size limit — the form shows a readable, cause-specific error and offers a one-click **"Paste data inline instead"** that switches the form to an inline paste (keeping the name and comment), rather than saving a broken record. The size-limit case is the exception: because pasting an oversized file inline would not help, its message points at using a smaller or pre-aggregated source rather than the inline fallback. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
- **Refresh** (URL datasets) — re-fetches the dataset's address and re-snapshots it, re-profiling the data and advancing the last-fetched time. The updated figures are the visible confirmation, so success raises no toast; a failed refresh raises an error toast. Refresh is also how a URL dataset that has not yet been fetched (e.g. one migrated from an older version) acquires its snapshot.
- **Edit** — rename, edit the comment, change the source data, and (for URL datasets) change the address. Updating inline data re-profiles it; changing a URL dataset's address re-fetches and re-snapshots it; editing only a URL dataset's name or comment does **not** re-fetch. The modified timestamp advances.
- **Delete** — asks for confirmation ("Delete \"Name\"? This cannot be undone."), then removes the dataset and clears the selection.

## Build Chart From Dataset

From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name. This relies on the dataset's profiled columns, so a URL dataset must have been fetched first — an unfetched URL reference has no schema to map (use _Refresh_ to fetch it).

## Extract Inline Data → Dataset

The reverse flow starts in the editor: a user can extract inline `data.values` out of a spec into a new named dataset (see _Spec Editor & Draft/Published Workflow_). The result appears here as a new dataset, and the originating snippet's spec is rewritten to reference it by name.

## Naming & Uniqueness

- Dataset names must be **unique**. Attempting to create a dataset with a name already in use is rejected with an error toast.
- During bulk operations such as import, conflicting names are automatically suffixed to remain unique rather than overwriting existing datasets (see _Import & Export_).
- Renaming a dataset that is referenced by snippets keeps references consistent by updating the matching named-data references in affected specs.