mirror of
https://github.com/olehomelchenko/astrolabe.git
synced 2026-08-08 02:02:33 +00:00
Snapshot URL datasets locally on add; preview tabular data as a table
This commit is contained in:
@@ -16,8 +16,8 @@ When a spec uses inline data, the preview renders it directly. When a spec inste
|
||||
|
||||
- A spec may point at a dataset from the library by name instead of embedding the data inline.
|
||||
- Before rendering, the preview substitutes the referenced dataset's stored contents into the spec.
|
||||
- URL-sourced datasets are fetched as needed at render time.
|
||||
- If a referenced dataset cannot be found or fetched, the preview shows a readable error (see _Error Display_) rather than a broken chart.
|
||||
- URL-sourced datasets render from their **local snapshot** (fetched when the dataset was created or last refreshed; see _Datasets_), so the preview does not fetch at render time and works offline. A URL dataset that has never been fetched has no snapshot and instead renders against its **live URL** as a fallback, until it is refreshed.
|
||||
- If a referenced dataset cannot be found, the preview shows a readable error (see _Error Display_) rather than a broken chart.
|
||||
|
||||
## Fit / Sizing Modes
|
||||
|
||||
@@ -42,12 +42,13 @@ Before the chart is drawn, the spec shown in the editor is transformed into the
|
||||
|
||||
**1. Dataset reference resolution.** Any named-data reference (`data` with a `name`) is replaced in-place with the referenced dataset's actual contents, shaped by the dataset's source and format (see _Datasets_):
|
||||
|
||||
| Dataset source / format | The reference's `data` becomes |
|
||||
| ----------------------- | ---------------------------------------------------------------- |
|
||||
| URL (any format) | a URL reference to the dataset's address, tagged with its format |
|
||||
| Inline JSON | the parsed values, inlined |
|
||||
| Inline CSV / TSV | the raw text, inlined, tagged with its format (CSV or TSV) |
|
||||
| Inline TopoJSON | the value, inlined, tagged as TopoJSON |
|
||||
| Dataset source / format | The reference's `data` becomes |
|
||||
| ------------------------- | -------------------------------------------------------------------------------- |
|
||||
| URL, fetched (any format) | its snapshot, inlined and tagged exactly like the inline rows below |
|
||||
| URL, not yet fetched | a live URL reference to the dataset's address, tagged with its format (fallback) |
|
||||
| Inline JSON | the parsed values, inlined |
|
||||
| Inline CSV / TSV | the raw text, inlined, tagged with its format (CSV or TSV) |
|
||||
| Inline TopoJSON | the value, inlined, tagged as TopoJSON |
|
||||
|
||||
- Resolution recurses into nested sub-specs (layered and concatenated specs, and a parent spec's child `spec`), so references anywhere in the spec are resolved.
|
||||
- If a referenced dataset does not exist, rendering fails with a "dataset not found" error (see _Error Display_).
|
||||
|
||||
+12
-10
@@ -28,7 +28,7 @@ A two-pane modal:
|
||||
Each list item shows:
|
||||
|
||||
- The dataset **name**.
|
||||
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). For URL datasets where counts are not yet known, only the source and format label are shown.
|
||||
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). A URL dataset that has not yet been fetched shows "not fetched" in place of the figures it does not have.
|
||||
- A **usage badge** when one or more snippets reference the dataset, indicating how many.
|
||||
|
||||
Clicking an item selects it and shows its detail. Per-item actions (delete, plus copy-reference and build-chart) live in the detail pane for the selected dataset.
|
||||
@@ -38,9 +38,9 @@ Clicking an item selects it and shows its detail. Per-item actions (delete, plus
|
||||
A dataset has one of two source types, chosen when creating it:
|
||||
|
||||
- **Inline** — the data itself is pasted in and stored directly in the library.
|
||||
- **URL** — the dataset stores a remote URL (http/https). The data is not copied locally; it is fetched on demand when a referencing spec is rendered (see _Live Preview_).
|
||||
- **URL** — the dataset is fetched once from a remote http/https address when it is created, and the fetched data is **snapshotted** into the library, along with the address (kept so the snapshot can be re-fetched). A URL dataset then works offline and renders from its local copy; it is never re-fetched at render time, only on demand (see _Refresh_, below).
|
||||
|
||||
For inline datasets the library holds the full data and can profile it. For URL datasets the library holds only the link, so row/column/size figures are typically not computed up front and show as "N/A".
|
||||
Either way the library holds the full data and profiles it. A URL dataset additionally records the address it was fetched from and when it was last fetched. Because the fetch happens in the browser, an address the browser cannot reach — offline, or one that blocks cross-origin requests — cannot be snapshotted; see _Actions → New / Create New_ for how that failure is handled.
|
||||
|
||||
## Supported Formats
|
||||
|
||||
@@ -59,18 +59,18 @@ When the user pastes inline data, the app auto-detects the format and reports a
|
||||
- Otherwise, multi-line text with a header row is detected as TSV (when tab-separated) or CSV (when comma-separated) — medium confidence.
|
||||
- Unrecognized input yields no format (low confidence); saving is blocked with a message asking the user to check the input.
|
||||
|
||||
The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the format is inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`) and shown as a hint.
|
||||
The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the create form shows a format hint inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`); the dataset's actual format is determined from the **fetched content** when it is created, falling back to the extension when the content is ambiguous.
|
||||
|
||||
## Profiling
|
||||
|
||||
For tabular inline data (JSON array-of-objects, CSV, TSV) the app computes and stores a profile:
|
||||
For tabular data (JSON array-of-objects, CSV, TSV) — whether pasted inline or fetched from a URL — the app computes and stores a profile:
|
||||
|
||||
- **Row count** and **column count**.
|
||||
- The list of **column names**.
|
||||
- An **inferred type per column**: number, text/string, date, or boolean. Type inference looks at the column's values: all-numeric becomes number, all `true`/`false` becomes boolean, otherwise string; empty cells are ignored.
|
||||
- **Size** in bytes of the stored data.
|
||||
|
||||
A **truncated data preview** of the raw data is also retained for display. URL datasets and non-tabular data are not profiled (counts show "N/A").
|
||||
A **truncated data preview** of the raw data is also retained for display. A fetched URL dataset is profiled exactly like inline data; only non-tabular data and a URL dataset that has **not yet been fetched** are unprofiled (counts show "N/A").
|
||||
|
||||
## Detail Panel
|
||||
|
||||
@@ -78,8 +78,9 @@ The detail pane for a selected dataset shows:
|
||||
|
||||
- **Name**.
|
||||
- **Comment** (optional free-text notes), when present.
|
||||
- **Source** (URL datasets only): the address the snapshot was fetched from, and when it was last fetched (or "Not fetched yet"), alongside the **Refresh** action.
|
||||
- **Overview**: statistics (rows, columns, size), the **column list** with each column's name and inferred type shown with a simple type indicator, and created/modified timestamps.
|
||||
- **Preview**: a truncated rendering of the data (raw text for CSV/TSV/URL, pretty-printed for JSON/TopoJSON).
|
||||
- **Preview**: a sample of the data. Tabular datasets (CSV, TSV, or a JSON array-of-objects — inline or fetched) render as a **table** of the first rows under the profiled column names, with a note when more rows exist than are shown. Non-tabular payloads (a single JSON object, TopoJSON) render as pretty-printed JSON, and a URL dataset that has not yet been fetched shows a short placeholder.
|
||||
- **Linked Snippets**: the list of snippets that reference this dataset by name. This is the dataset side of bidirectional dataset↔snippet linking (see _Snippet Library_).
|
||||
|
||||
## Actions
|
||||
@@ -89,13 +90,14 @@ A destructive or off-screen outcome raises a confirming toast; an action whose r
|
||||
- **Copy Reference** — copies the by-name reference object to the clipboard, ready to paste into a spec:
|
||||
`{ "data": { "name": "MyDataset" } }`
|
||||
The clipboard write is invisible, so it is confirmed _inline on the control_ ("Copied"), announced politely to assistive technology — not a toast.
|
||||
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
|
||||
- **Edit** — rename, edit the comment, and update the data (re-paste inline data or refresh the URL). Updating inline data re-profiles it; the modified timestamp advances.
|
||||
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. For a **URL** dataset, saving fetches and snapshots the address; the Save control shows a busy state while fetching. If the fetch fails — offline, blocked by the host (cross-origin), not found, empty, or timed out — the form shows a readable, cause-specific error and offers a one-click **"Paste data inline instead"** that switches the form to an inline paste (keeping the name and comment), rather than saving a broken record. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
|
||||
- **Refresh** (URL datasets) — re-fetches the dataset's address and re-snapshots it, re-profiling the data and advancing the last-fetched time. The updated figures are the visible confirmation, so success raises no toast; a failed refresh raises an error toast. Refresh is also how a URL dataset that has not yet been fetched (e.g. one migrated from an older version) acquires its snapshot.
|
||||
- **Edit** — rename, edit the comment, change the source data, and (for URL datasets) change the address. Updating inline data re-profiles it; changing a URL dataset's address re-fetches and re-snapshots it; editing only a URL dataset's name or comment does **not** re-fetch. The modified timestamp advances.
|
||||
- **Delete** — asks for confirmation ("Delete \"Name\"? This cannot be undone."), then removes the dataset and clears the selection.
|
||||
|
||||
## Build Chart From Dataset
|
||||
|
||||
From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name.
|
||||
From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name. This relies on the dataset's profiled columns, so a URL dataset must have been fetched first — an unfetched URL reference has no schema to map (use _Refresh_ to fetch it).
|
||||
|
||||
## Extract Inline Data → Dataset
|
||||
|
||||
|
||||
+21
-17
@@ -34,29 +34,33 @@ A snippet carries two specs at once. `draftSpec` is the editable working copy; `
|
||||
|
||||
A **Dataset** is a named, reusable data source that snippets can reference by name instead of inlining data. Datasets are managed in the _Datasets_ manager and support multiple formats and two source kinds.
|
||||
|
||||
| Field | Type | Meaning |
|
||||
| ------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `id` | number | Unique numeric identifier. |
|
||||
| `version` | number | Schema version of this record, used for read-time migration (see _Schema versioning_ below). |
|
||||
| `name` | string | Unique, human-readable name; the key snippets reference via `datasetRefs`. |
|
||||
| `data` | JSON value | The payload. For `source = url`: the URL string. For `source = inline`: the raw CSV/TSV text, or the parsed JSON/TopoJSON value. |
|
||||
| `format` | string | One of `json`, `csv`, `tsv`, `topojson`. |
|
||||
| `source` | string | One of `inline` (data embedded in the record) or `url` (data fetched from a remote address). |
|
||||
| `comment` | string | Free-form user note about the dataset. |
|
||||
| `rowCount` | number or null | Number of data rows, or null when unknown/not applicable. |
|
||||
| `columnCount` | number or null | Number of columns, or null when unknown/not applicable. |
|
||||
| `columns` | string[] | Column names, in order. |
|
||||
| `columnTypes` | array of `{ name, type }` | Per-column inferred type. `name` is the column; `type` is one of `number`, `string`, `date`, `boolean`. |
|
||||
| `size` | number | Approximate payload size in bytes. |
|
||||
| `created` | ISO-timestamp string | When the dataset was first added. |
|
||||
| `modified` | ISO-timestamp string | When the dataset was last changed. |
|
||||
| Field | Type | Meaning |
|
||||
| ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `id` | number | Unique numeric identifier. |
|
||||
| `version` | number | Schema version of this record, used for read-time migration (see _Schema versioning_ below). |
|
||||
| `name` | string | Unique, human-readable name; the key snippets reference via `datasetRefs`. |
|
||||
| `data` | JSON value | The payload, shaped by format: raw CSV/TSV text, or the parsed JSON/TopoJSON value. For `source = url` this is the **fetched snapshot**, or `null` before the first successful fetch. |
|
||||
| `format` | string | One of `json`, `csv`, `tsv`, `topojson`. |
|
||||
| `source` | string | One of `inline` (data pasted into the record) or `url` (data fetched once from a remote address and snapshotted into the record). |
|
||||
| `url` | string (url only) | The remote address a `url` dataset was fetched from, retained so it can be re-fetched ("Refresh"). Absent for inline datasets. |
|
||||
| `fetchedAt` | ISO-timestamp or null | For `url` datasets: when the snapshot was last fetched, or `null` if never fetched. Absent for inline datasets. |
|
||||
| `comment` | string | Free-form user note about the dataset. |
|
||||
| `rowCount` | number or null | Number of data rows, or null when unknown/not applicable. |
|
||||
| `columnCount` | number or null | Number of columns, or null when unknown/not applicable. |
|
||||
| `columns` | string[] | Column names, in order. |
|
||||
| `columnTypes` | array of `{ name, type }` | Per-column inferred type. `name` is the column; `type` is one of `number`, `string`, `date`, `boolean`. |
|
||||
| `size` | number | Approximate payload size in bytes. |
|
||||
| `created` | ISO-timestamp string | When the dataset was first added. |
|
||||
| `modified` | ISO-timestamp string | When the dataset was last changed. |
|
||||
|
||||
The `rowCount`, `columnCount`, `columns`, `columnTypes`, and `size` fields are derived summaries computed when data is added or updated; they support previews and type display without re-parsing the full payload.
|
||||
The `rowCount`, `columnCount`, `columns`, `columnTypes`, and `size` fields are derived summaries computed when data is added or updated — including when a `url` dataset is fetched or refreshed; a fetched URL snapshot profiles exactly like inline data. They support previews and type display without re-parsing the full payload.
|
||||
|
||||
### Schema versioning
|
||||
|
||||
Both **Snippet** and **Dataset** records carry a numeric `version` recording the shape of that individual record. When a record is read from storage it is migrated up to the current shape before the app uses it; new writes always store the current version. A record written before versioning existed (no `version` field) is treated as version `1`. This is distinct from the storage container's own layout version, and from the _Import & Export_ envelope `version` (which describes the file format, not a record). Records exported via _Import & Export_ include their `version`.
|
||||
|
||||
The current **Dataset** version is `2`. The v1→v2 migration reflects the URL-snapshot model: a v1 `url` dataset stored its address in `data`, so migration moves that address into the new `url` field and clears `data` to `null` — the record becomes an _unfetched reference_ that renders against its live URL until the user refreshes it, at which point the fetched snapshot is stored and profiled.
|
||||
|
||||
## C. UserSettings
|
||||
|
||||
**UserSettings** holds persisted user preferences as a single structured record. The semantics and UX of each option are covered in _Settings_; the shape below is the storage contract.
|
||||
|
||||
Reference in New Issue
Block a user