Snapshot URL datasets locally on add; preview tabular data as a table

This commit is contained in:
2026-06-10 10:24:42 +03:00
parent eb5e7ac53a
commit 2410c6e965
23 changed files with 1239 additions and 148 deletions
+12 -10
View File
@@ -28,7 +28,7 @@ A two-pane modal:
Each list item shows:
- The dataset **name**.
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). For URL datasets where counts are not yet known, only the source and format label are shown.
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). A URL dataset that has not yet been fetched shows "not fetched" in place of the figures it does not have.
- A **usage badge** when one or more snippets reference the dataset, indicating how many.
Clicking an item selects it and shows its detail. Per-item actions (delete, plus copy-reference and build-chart) live in the detail pane for the selected dataset.
@@ -38,9 +38,9 @@ Clicking an item selects it and shows its detail. Per-item actions (delete, plus
A dataset has one of two source types, chosen when creating it:
- **Inline** — the data itself is pasted in and stored directly in the library.
- **URL** — the dataset stores a remote URL (http/https). The data is not copied locally; it is fetched on demand when a referencing spec is rendered (see _Live Preview_).
- **URL** — the dataset is fetched once from a remote http/https address when it is created, and the fetched data is **snapshotted** into the library, along with the address (kept so the snapshot can be re-fetched). A URL dataset then works offline and renders from its local copy; it is never re-fetched at render time, only on demand (see _Refresh_, below).
For inline datasets the library holds the full data and can profile it. For URL datasets the library holds only the link, so row/column/size figures are typically not computed up front and show as "N/A".
Either way the library holds the full data and profiles it. A URL dataset additionally records the address it was fetched from and when it was last fetched. Because the fetch happens in the browser, an address the browser cannot reach — offline, or one that blocks cross-origin requests — cannot be snapshotted; see _Actions → New / Create New_ for how that failure is handled.
## Supported Formats
@@ -59,18 +59,18 @@ When the user pastes inline data, the app auto-detects the format and reports a
- Otherwise, multi-line text with a header row is detected as TSV (when tab-separated) or CSV (when comma-separated) — medium confidence.
- Unrecognized input yields no format (low confidence); saving is blocked with a message asking the user to check the input.
The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the format is inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`) and shown as a hint.
The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the create form shows a format hint inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`); the dataset's actual format is determined from the **fetched content** when it is created, falling back to the extension when the content is ambiguous.
## Profiling
For tabular inline data (JSON array-of-objects, CSV, TSV) the app computes and stores a profile:
For tabular data (JSON array-of-objects, CSV, TSV) — whether pasted inline or fetched from a URL — the app computes and stores a profile:
- **Row count** and **column count**.
- The list of **column names**.
- An **inferred type per column**: number, text/string, date, or boolean. Type inference looks at the column's values: all-numeric becomes number, all `true`/`false` becomes boolean, otherwise string; empty cells are ignored.
- **Size** in bytes of the stored data.
A **truncated data preview** of the raw data is also retained for display. URL datasets and non-tabular data are not profiled (counts show "N/A").
A **truncated data preview** of the raw data is also retained for display. A fetched URL dataset is profiled exactly like inline data; only non-tabular data and a URL dataset that has **not yet been fetched** are unprofiled (counts show "N/A").
## Detail Panel
@@ -78,8 +78,9 @@ The detail pane for a selected dataset shows:
- **Name**.
- **Comment** (optional free-text notes), when present.
- **Source** (URL datasets only): the address the snapshot was fetched from, and when it was last fetched (or "Not fetched yet"), alongside the **Refresh** action.
- **Overview**: statistics (rows, columns, size), the **column list** with each column's name and inferred type shown with a simple type indicator, and created/modified timestamps.
- **Preview**: a truncated rendering of the data (raw text for CSV/TSV/URL, pretty-printed for JSON/TopoJSON).
- **Preview**: a sample of the data. Tabular datasets (CSV, TSV, or a JSON array-of-objects — inline or fetched) render as a **table** of the first rows under the profiled column names, with a note when more rows exist than are shown. Non-tabular payloads (a single JSON object, TopoJSON) render as pretty-printed JSON, and a URL dataset that has not yet been fetched shows a short placeholder.
- **Linked Snippets**: the list of snippets that reference this dataset by name. This is the dataset side of bidirectional dataset↔snippet linking (see _Snippet Library_).
## Actions
@@ -89,13 +90,14 @@ A destructive or off-screen outcome raises a confirming toast; an action whose r
- **Copy Reference** — copies the by-name reference object to the clipboard, ready to paste into a spec:
`{ "data": { "name": "MyDataset" } }`
The clipboard write is invisible, so it is confirmed _inline on the control_ ("Copied"), announced politely to assistive technology — not a toast.
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
- **Edit** — rename, edit the comment, and update the data (re-paste inline data or refresh the URL). Updating inline data re-profiles it; the modified timestamp advances.
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. For a **URL** dataset, saving fetches and snapshots the address; the Save control shows a busy state while fetching. If the fetch fails — offline, blocked by the host (cross-origin), not found, empty, or timed out — the form shows a readable, cause-specific error and offers a one-click **"Paste data inline instead"** that switches the form to an inline paste (keeping the name and comment), rather than saving a broken record. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
- **Refresh** (URL datasets) — re-fetches the dataset's address and re-snapshots it, re-profiling the data and advancing the last-fetched time. The updated figures are the visible confirmation, so success raises no toast; a failed refresh raises an error toast. Refresh is also how a URL dataset that has not yet been fetched (e.g. one migrated from an older version) acquires its snapshot.
- **Edit** — rename, edit the comment, change the source data, and (for URL datasets) change the address. Updating inline data re-profiles it; changing a URL dataset's address re-fetches and re-snapshots it; editing only a URL dataset's name or comment does **not** re-fetch. The modified timestamp advances.
- **Delete** — asks for confirmation ("Delete \"Name\"? This cannot be undone."), then removes the dataset and clears the selection.
## Build Chart From Dataset
From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name.
From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name. This relies on the dataset's profiled columns, so a URL dataset must have been fetched first — an unfetched URL reference has no schema to map (use _Refresh_ to fetch it).
## Extract Inline Data → Dataset