astrolabe/docs/spec/05-datasets.md

# 05 · Datasets

The **Dataset Manager** is a modal for creating and managing named, reusable datasets that snippets can reference by name. It is the home of the dataset library: a place separate from snippets where data lives once and is shared across many visualizations.

## Purpose & Model

Datasets are named blobs of data stored in the user's local library, independent of any single snippet. A snippet references a dataset by name rather than embedding the data inline, so the same data can power many snippets and be edited in one place.

- Datasets persist locally across sessions in a high-capacity local store, far larger than the budget available to snippets — large datasets belong here, not inline in specs.
- A snippet references a dataset using a Vega-Lite named-data reference, e.g. `{ "data": { "name": "MyDataset" } }`. When the _Live Preview_ renders a spec, it resolves any such named reference against the dataset library (see _Live Preview_).
- See _Data Model_ for the stored shape of a dataset.

## Opening & Navigation

- Opened from a header control or via the keyboard shortcut Cmd/Ctrl+K.
- The current view and the selected dataset are reflected in the URL, so a selected dataset produces a shareable/back-navigable location (see _Application Shell & Navigation_).
- Closing the modal clears the current selection and any open create form.

## Layout

A two-pane modal:

- **List pane** (left): a "New Dataset" action plus the list of all datasets, sorted most-recently-modified first.
- **Detail pane** (right): shows the selected dataset's details, the create form when creating, or an empty prompt ("Select a dataset or create a new one") when nothing is selected.

### List item

Each list item shows:

- The dataset **name**.
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). For URL datasets where counts are not yet known, only the source and format label are shown.
- A **usage badge** when one or more snippets reference the dataset, indicating how many.

Clicking an item selects it and shows its detail. Per-item actions (delete, plus copy-reference and build-chart) live in the detail pane for the selected dataset.

## Source Types

A dataset has one of two source types, chosen when creating it:

- **Inline** — the data itself is pasted in and stored directly in the library.
- **URL** — the dataset stores a remote URL (http/https). The data is not copied locally; it is fetched on demand when a referencing spec is rendered (see _Live Preview_).

For inline datasets the library holds the full data and can profile it. For URL datasets the library holds only the link, so row/column/size figures are typically not computed up front and show as "N/A".

## Supported Formats

Four data formats are supported, named in the UI and stored on the dataset:

- **JSON** — an array of objects (most common, profilable) or a single object.
- **CSV** — comma-separated with a header row.
- **TSV** — tab-separated with a header row.
- **TopoJSON** — topology/map data (a JSON object whose type marks it as a topology).

### Auto-detection

When the user pastes inline data, the app auto-detects the format and reports a **confidence** level (high / medium / low):

- Valid JSON parses to JSON, or to TopoJSON when it is a topology object — high confidence.
- Otherwise, multi-line text with a header row is detected as TSV (when tab-separated) or CSV (when comma-separated) — medium confidence.
- Unrecognized input yields no format (low confidence); saving is blocked with a message asking the user to check the input.

The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the format is inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`) and shown as a hint.

## Profiling

For tabular inline data (JSON array-of-objects, CSV, TSV) the app computes and stores a profile:

- **Row count** and **column count**.
- The list of **column names**.
- An **inferred type per column**: number, text/string, date, or boolean. Type inference looks at the column's values: all-numeric becomes number, all `true`/`false` becomes boolean, otherwise string; empty cells are ignored.
- **Size** in bytes of the stored data.

A **truncated data preview** of the raw data is also retained for display. URL datasets and non-tabular data are not profiled (counts show "N/A").

## Detail Panel

The detail pane for a selected dataset shows:

- **Name**.
- **Comment** (optional free-text notes), when present.
- **Overview**: statistics (rows, columns, size), the **column list** with each column's name and inferred type shown with a simple type indicator, and created/modified timestamps.
- **Preview**: a truncated rendering of the data (raw text for CSV/TSV/URL, pretty-printed for JSON/TopoJSON).
- **Linked Snippets**: the list of snippets that reference this dataset by name. This is the dataset side of bidirectional dataset↔snippet linking (see _Snippet Library_).

## Actions

Each action raises a confirming toast (or an error toast on failure).

- **Copy Reference** — copies the by-name reference object to the clipboard, ready to paste into a spec:
  `{ "data": { "name": "MyDataset" } }`
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. On success the new dataset is selected.
- **Edit** — rename, edit the comment, and update the data (re-paste inline data or refresh the URL). Updating inline data re-profiles it; the modified timestamp advances.
- **Delete** — asks for confirmation ("Delete \"Name\"? This cannot be undone."), then removes the dataset and clears the selection.

## Build Chart From Dataset

From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name.

## Extract Inline Data → Dataset

The reverse flow starts in the editor: a user can extract inline `data.values` out of a spec into a new named dataset (see _Spec Editor & Draft/Published Workflow_). The result appears here as a new dataset, and the originating snippet's spec is rewritten to reference it by name.

## Naming & Uniqueness

- Dataset names must be **unique**. Attempting to create a dataset with a name already in use is rejected with an error toast.
- During bulk operations such as import, conflicting names are automatically suffixed to remain unique rather than overwriting existing datasets (see _Import & Export_).
- Renaming a dataset that is referenced by snippets keeps references consistent by updating the matching named-data references in affected specs.