oleh/astrolabe
1
0
Fork 0
mirror of https://github.com/olehomelchenko/astrolabe.git synced 2026-08-08 02:02:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Snapshot URL datasets locally on add; preview tabular data as a table

Browse Source
This commit is contained in:
Oleh Omelchenko
2026-06-10 10:24:42 +03:00
parent eb5e7ac53a
commit 2410c6e965
23 changed files with 1239 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/IMPLEMENTATION-PLAN.md
+2 -1
View File
@@ -268,7 +268,8 @@ the reference.
- Snippet `datasetRefs` maintained on publish; library shows dataset icon +
Linked Datasets; dataset detail shows Linked Snippets (bidirectional name link, §09F).
- **Extract-to-Dataset** flow from the editor (§03F).
- URL-sourced datasets fetched at render time.
- URL datasets fetched once on add and **snapshotted** locally (profiled like inline;
render from the snapshot, refreshable on demand — see §05 / §04 rendering contract).
**Tests**
docs/architecture/02-persistence.md
+17 -6
View File
@@ -6,19 +6,20 @@ How Astrolabe stores data in the browser, and the rules that keep that storage t
## 1. The Infrastructure-Adapter Principle
**Rule: nothing outside `src/app/infrastructure/` ever touches `indexedDB`, `localStorage`, `window`, or `location` directly.** Every browser-storage interaction goes through a typed adapter module that exposes plain async functions returning domain objects.
**Rule: nothing outside `src/app/infrastructure/` ever touches `indexedDB`, `localStorage`, `window`, `location`, or `fetch` directly.** Every browser interaction (storage _and_ network) goes through a typed adapter module that exposes plain async functions returning domain objects.
```
src/
├── core/ # portable engine — NO browser APIs, NO React
├── app/
│ ├── stores/ # Zustand stores; calls infrastructure, never IDB
│ ├── services/ # business logic; calls infrastructure, never IDB
│ └── infrastructure/ # the ONLY place that imports indexedDB/localStorage
│ ├── stores/ # Zustand stores; calls infrastructure, never IDB/fetch
│ ├── services/ # business logic; calls infrastructure, never IDB/fetch
│ └── infrastructure/ # the ONLY place that imports indexedDB/localStorage/fetch
│ ├── snippet-store.ts # IndexedDB: snippets (metadata + drafts)
│ ├── dataset-store.ts # IndexedDB: datasets (heavy payloads)
│ ├── settings-store.ts # localStorage: UserSettings
│ └── ux-prefs.ts # localStorage: app/UI prefs (sort, panel layout)
│ ├── ux-prefs.ts # localStorage: app/UI prefs (sort, panel layout)
│ └── remote-data.ts # network: fetch a URL dataset's body (the ONLY fetch)
```
### Why this boundary exists
@@ -29,7 +30,15 @@ src/
- **Failure containment.** Quota errors, corrupt JSON, and missing keys are handled at the boundary and converted into typed results (or sane fallbacks), so the rest of the app never sees a raw `DOMException`.
> **Do:** `import { saveSnippet } from '@/app/infrastructure/snippet-store'`
> **Don't:** `indexedDB.open(...)` or `localStorage.getItem(...)` anywhere in a component, store, or service.
> **Don't:** `indexedDB.open(...)`, `localStorage.getItem(...)`, or `fetch(...)` anywhere in a component, store, or service.
### Background vs. interactive adapters
Most adapters are driven by **background subscribers** (arch 01 §5, _Effects_): a store changes, a startup subscriber writes it through to IndexedDB — the store never calls the adapter itself. The **network** adapter is the exception. Fetching a URL dataset is a user-initiated action with its own pending/error UI, so the **component** calls `remote-data.ts` directly (components may call adapters — cf. `navigator.clipboard`) and hands the fetched body to **pure** store actions (`DatasetStore.commitUrlSnapshot` / `refreshDataset`). The store never fetches, so it stays browser-free and unit-testable on already-fetched text.
> **Rule:** keep `fetch` behind `remote-data.ts`; orchestrate the URL-dataset fetch in the _component_ (busy state + the "paste data inline instead" recovery), not the store. Store commit actions only ever receive already-fetched text.
**URL-dataset snapshot lifecycle** (the files a change to it touches): add / Refresh → component fetches (`infrastructure/remote-data.ts`) → `core/dataset.snapshotFromText` shapes the body by sniffed format → `DatasetStore.commitUrlSnapshot` / `refreshDataset` snapshots + profiles it _exactly like inline data_ → render resolves it cached-first in `core/rendering.resolvedData` (a live-URL fallback applies only while a URL dataset is still unfetched). See spec §05 for the behavior.
---
@@ -221,6 +230,8 @@ export async function saveSnippet(s: Snippet): Promise<void> {
> **Do:** default `version` to the earliest shape (`1`) when the field is absent.
> **Don't:** branch on the presence of individual fields scattered through the app to detect "old data." Centralize that knowledge in the migration function.
> **Mirror shape changes in the import normalizer.** Imported records are built from a file, not read from IndexedDB, so they **never pass through `migrate<Entity>`** — `core/import-normalize.ts` upgrades them independently. A migration that changes a field's _shape_ must be applied in both places or import produces a malformed record. (E.g. the dataset v1→v2 URL-snapshot reshaping — address moves from `data` into `url`, `data` cleared — lives in **both** `migrateDataset` and `normalizeDataset`.)
---
## 5. localStorage Preferences (Settings & App/UI Prefs)
docs/architecture/06-type-inference.md
+7 -3
View File
@@ -183,7 +183,10 @@ how the UI shows **"N/A"** — see §3.2.
### 3.1 What gets profiled
Profiling applies only to **tabular inline data**:
Profiling applies to any **tabular payload**, whether pasted inline or fetched
from a URL (the snapshot model stores a URL dataset's data locally, so it profiles
through the same path as inline data — `snapshotFromText` shapes the fetched body,
then `computeDatasetProfile` runs):
- **JSON** that is an array of objects.
- **CSV** (comma-separated, header row).
@@ -191,8 +194,9 @@ Profiling applies only to **tabular inline data**:
Everything else is **not profiled**:
- **URL datasets** — the library holds only the link, not the data, so there is
nothing to scan. Counts are `null` / N/A.
- **Unfetched URL datasets** — a URL reference with no snapshot yet (e.g. one
migrated from an older record), so there is nothing to scan. Counts are `null` /
N/A until it is refreshed.
- **Non-tabular data** — a single JSON object, TopoJSON, or anything we can't
read as rows-of-columns. Counts are `null` / N/A.
docs/manual-verification.md
+17
View File
@@ -44,6 +44,23 @@
- [ ] Toasts stack, auto-dismiss, and fade as specified; the live-preview busy
indicator appears for slow (>~1s) renders and clears after.
## URL datasets (remote data snapshot)
> Needs a real network, real CORS, and real offline — tests mock the fetch.
- [ ] Add a dataset by URL from a CORS-friendly host (e.g. a GitHub raw `.csv` or a
vega-datasets URL) — Save shows "Fetching…", then the dataset appears profiled
(rows, columns, size) with a "Fetched <time>" line and the source address.
- [ ] Go offline (DevTools → Network → Offline) and reload — a snippet that references
that URL dataset still renders from the local snapshot (no fetch at render time).
- [ ] Add a URL that blocks cross-origin requests (or while offline) — the form shows a
cause-specific error and a "Paste data inline instead" button that switches to an
inline paste keeping the name/comment; no broken record is saved.
- [ ] Refresh a URL dataset whose source changed — rows/size and the "Fetched" time
update; refreshing while offline raises an error toast and leaves the snapshot intact.
- [ ] Build Chart from a fetched URL dataset works (columns are known); from an
unfetched URL reference the builder has no schema until Refresh.
## Visual sweep of the M6 surfaces
- [ ] Library search / sort / empty states, the storage monitor, About & Donate
docs/spec/04-live-preview.md
+9 -8
View File
@@ -16,8 +16,8 @@ When a spec uses inline data, the preview renders it directly. When a spec inste
- A spec may point at a dataset from the library by name instead of embedding the data inline.
- Before rendering, the preview substitutes the referenced dataset's stored contents into the spec.
- URL-sourced datasets are fetched as needed at render time.
- If a referenced dataset cannot be found or fetched, the preview shows a readable error (see _Error Display_) rather than a broken chart.
- URL-sourced datasets render from their **local snapshot** (fetched when the dataset was created or last refreshed; see _Datasets_), so the preview does not fetch at render time and works offline. A URL dataset that has never been fetched has no snapshot and instead renders against its **live URL** as a fallback, until it is refreshed.
- If a referenced dataset cannot be found, the preview shows a readable error (see _Error Display_) rather than a broken chart.
## Fit / Sizing Modes
@@ -42,12 +42,13 @@ Before the chart is drawn, the spec shown in the editor is transformed into the
**1. Dataset reference resolution.** Any named-data reference (`data` with a `name`) is replaced in-place with the referenced dataset's actual contents, shaped by the dataset's source and format (see _Datasets_):
| Dataset source / format | The reference's `data` becomes |
| ----------------------- | ---------------------------------------------------------------- |
| URL (any format) | a URL reference to the dataset's address, tagged with its format |
| Inline JSON | the parsed values, inlined |
| Inline CSV / TSV | the raw text, inlined, tagged with its format (CSV or TSV) |
| Inline TopoJSON | the value, inlined, tagged as TopoJSON |
| Dataset source / format | The reference's `data` becomes |
| ------------------------- | -------------------------------------------------------------------------------- |
| URL, fetched (any format) | its snapshot, inlined and tagged exactly like the inline rows below |
| URL, not yet fetched | a live URL reference to the dataset's address, tagged with its format (fallback) |
| Inline JSON | the parsed values, inlined |
| Inline CSV / TSV | the raw text, inlined, tagged with its format (CSV or TSV) |
| Inline TopoJSON | the value, inlined, tagged as TopoJSON |
- Resolution recurses into nested sub-specs (layered and concatenated specs, and a parent spec's child `spec`), so references anywhere in the spec are resolved.
- If a referenced dataset does not exist, rendering fails with a "dataset not found" error (see _Error Display_).
docs/spec/05-datasets.md
+12 -10
View File
@@ -28,7 +28,7 @@ A two-pane modal:
Each list item shows:
- The dataset **name**.
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). For URL datasets where counts are not yet known, only the source and format label are shown.
- A **meta line** combining: source ("URL" prefix for URL datasets), row count when known, the **format label** (JSON / CSV / TSV / TOPOJSON), and **size** (human-readable, e.g. B / KB / MB). A URL dataset that has not yet been fetched shows "not fetched" in place of the figures it does not have.
- A **usage badge** when one or more snippets reference the dataset, indicating how many.
Clicking an item selects it and shows its detail. Per-item actions (delete, plus copy-reference and build-chart) live in the detail pane for the selected dataset.
@@ -38,9 +38,9 @@ Clicking an item selects it and shows its detail. Per-item actions (delete, plus
A dataset has one of two source types, chosen when creating it:
- **Inline** — the data itself is pasted in and stored directly in the library.
- **URL** — the dataset stores a remote URL (http/https). The data is not copied locally; it is fetched on demand when a referencing spec is rendered (see _Live Preview_).
- **URL** — the dataset is fetched once from a remote http/https address when it is created, and the fetched data is **snapshotted** into the library, along with the address (kept so the snapshot can be re-fetched). A URL dataset then works offline and renders from its local copy; it is never re-fetched at render time, only on demand (see _Refresh_, below).
For inline datasets the library holds the full data and can profile it. For URL datasets the library holds only the link, so row/column/size figures are typically not computed up front and show as "N/A".
Either way the library holds the full data and profiles it. A URL dataset additionally records the address it was fetched from and when it was last fetched. Because the fetch happens in the browser, an address the browser cannot reach — offline, or one that blocks cross-origin requests — cannot be snapshotted; see _Actions → New / Create New_ for how that failure is handled.
## Supported Formats
@@ -59,18 +59,18 @@ When the user pastes inline data, the app auto-detects the format and reports a
- Otherwise, multi-line text with a header row is detected as TSV (when tab-separated) or CSV (when comma-separated) — medium confidence.
- Unrecognized input yields no format (low confidence); saving is blocked with a message asking the user to check the input.
The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the format is inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`) and shown as a hint.
The detected format and source are shown as badges in the create form so the user can confirm or override the source (Inline/URL) before saving. For URL datasets the create form shows a format hint inferred from the URL's file extension (`.csv`, `.tsv`, `.json`, `.topojson`); the dataset's actual format is determined from the **fetched content** when it is created, falling back to the extension when the content is ambiguous.
## Profiling
For tabular inline data (JSON array-of-objects, CSV, TSV) the app computes and stores a profile:
For tabular data (JSON array-of-objects, CSV, TSV) — whether pasted inline or fetched from a URL — the app computes and stores a profile:
- **Row count** and **column count**.
- The list of **column names**.
- An **inferred type per column**: number, text/string, date, or boolean. Type inference looks at the column's values: all-numeric becomes number, all `true`/`false` becomes boolean, otherwise string; empty cells are ignored.
- **Size** in bytes of the stored data.
A **truncated data preview** of the raw data is also retained for display. URL datasets and non-tabular data are not profiled (counts show "N/A").
A **truncated data preview** of the raw data is also retained for display. A fetched URL dataset is profiled exactly like inline data; only non-tabular data and a URL dataset that has **not yet been fetched** are unprofiled (counts show "N/A").
## Detail Panel
@@ -78,8 +78,9 @@ The detail pane for a selected dataset shows:
- **Name**.
- **Comment** (optional free-text notes), when present.
- **Source** (URL datasets only): the address the snapshot was fetched from, and when it was last fetched (or "Not fetched yet"), alongside the **Refresh** action.
- **Overview**: statistics (rows, columns, size), the **column list** with each column's name and inferred type shown with a simple type indicator, and created/modified timestamps.
- **Preview**: a truncated rendering of the data (raw text for CSV/TSV/URL, pretty-printed for JSON/TopoJSON).
- **Preview**: a sample of the data. Tabular datasets (CSV, TSV, or a JSON array-of-objects — inline or fetched) render as a **table** of the first rows under the profiled column names, with a note when more rows exist than are shown. Non-tabular payloads (a single JSON object, TopoJSON) render as pretty-printed JSON, and a URL dataset that has not yet been fetched shows a short placeholder.
- **Linked Snippets**: the list of snippets that reference this dataset by name. This is the dataset side of bidirectional dataset↔snippet linking (see _Snippet Library_).
## Actions
@@ -89,13 +90,14 @@ A destructive or off-screen outcome raises a confirming toast; an action whose r
- **Copy Reference** — copies the by-name reference object to the clipboard, ready to paste into a spec:
`{ "data": { "name": "MyDataset" } }`
The clipboard write is invisible, so it is confirmed _inline on the control_ ("Copied"), announced politely to assistive technology — not a toast.
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
- **Edit** — rename, edit the comment, and update the data (re-paste inline data or refresh the URL). Updating inline data re-profiles it; the modified timestamp advances.
- **New / Create New** — opens the create form in the detail pane with fields: **name** (required, unique), **source** toggle (Inline / URL), the **data** (a paste area for inline, a URL field for URL source), and an optional **comment**. Save is disabled until a name and valid data/URL are present. For a **URL** dataset, saving fetches and snapshots the address; the Save control shows a busy state while fetching. If the fetch fails — offline, blocked by the host (cross-origin), not found, empty, or timed out — the form shows a readable, cause-specific error and offers a one-click **"Paste data inline instead"** that switches the form to an inline paste (keeping the name and comment), rather than saving a broken record. On success the new dataset is shown selected in the detail pane — that visible result is the confirmation, so no toast is raised (a dataset created _off-screen_ via Extract does toast; see _Spec Editor_).
- **Refresh** (URL datasets) — re-fetches the dataset's address and re-snapshots it, re-profiling the data and advancing the last-fetched time. The updated figures are the visible confirmation, so success raises no toast; a failed refresh raises an error toast. Refresh is also how a URL dataset that has not yet been fetched (e.g. one migrated from an older version) acquires its snapshot.
- **Edit** — rename, edit the comment, change the source data, and (for URL datasets) change the address. Updating inline data re-profiles it; changing a URL dataset's address re-fetches and re-snapshots it; editing only a URL dataset's name or comment does **not** re-fetch. The modified timestamp advances.
- **Delete** — asks for confirmation ("Delete \"Name\"? This cannot be undone."), then removes the dataset and clears the selection.
## Build Chart From Dataset
From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name.
From a selected dataset the user can launch the visual _Chart Builder_ (see _Chart Builder_) pre-targeted at that dataset, producing a new snippet whose spec references the dataset by name. This relies on the dataset's profiled columns, so a URL dataset must have been fetched first — an unfetched URL reference has no schema to map (use _Refresh_ to fetch it).
## Extract Inline Data → Dataset
docs/spec/09-data-model.md
+21 -17
View File
@@ -34,29 +34,33 @@ A snippet carries two specs at once. `draftSpec` is the editable working copy; `
A **Dataset** is a named, reusable data source that snippets can reference by name instead of inlining data. Datasets are managed in the _Datasets_ manager and support multiple formats and two source kinds.
| Field | Type | Meaning |
| ------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `id` | number | Unique numeric identifier. |
| `version` | number | Schema version of this record, used for read-time migration (see _Schema versioning_ below). |
| `name` | string | Unique, human-readable name; the key snippets reference via `datasetRefs`. |
| `data` | JSON value | The payload. For `source = url`: the URL string. For `source = inline`: the raw CSV/TSV text, or the parsed JSON/TopoJSON value. |
| `format` | string | One of `json`, `csv`, `tsv`, `topojson`. |
| `source` | string | One of `inline` (data embedded in the record) or `url` (data fetched from a remote address). |
| `comment` | string | Free-form user note about the dataset. |
| `rowCount` | number or null | Number of data rows, or null when unknown/not applicable. |
| `columnCount` | number or null | Number of columns, or null when unknown/not applicable. |
| `columns` | string[] | Column names, in order. |
| `columnTypes` | array of `{ name, type }` | Per-column inferred type. `name` is the column; `type` is one of `number`, `string`, `date`, `boolean`. |
| `size` | number | Approximate payload size in bytes. |
| `created` | ISO-timestamp string | When the dataset was first added. |
| `modified` | ISO-timestamp string | When the dataset was last changed. |
| Field | Type | Meaning |
| ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | number | Unique numeric identifier. |
| `version` | number | Schema version of this record, used for read-time migration (see _Schema versioning_ below). |
| `name` | string | Unique, human-readable name; the key snippets reference via `datasetRefs`. |
| `data` | JSON value | The payload, shaped by format: raw CSV/TSV text, or the parsed JSON/TopoJSON value. For `source = url` this is the **fetched snapshot**, or `null` before the first successful fetch. |
| `format` | string | One of `json`, `csv`, `tsv`, `topojson`. |
| `source` | string | One of `inline` (data pasted into the record) or `url` (data fetched once from a remote address and snapshotted into the record). |
| `url` | string (url only) | The remote address a `url` dataset was fetched from, retained so it can be re-fetched ("Refresh"). Absent for inline datasets. |
| `fetchedAt` | ISO-timestamp or null | For `url` datasets: when the snapshot was last fetched, or `null` if never fetched. Absent for inline datasets. |
| `comment` | string | Free-form user note about the dataset. |
| `rowCount` | number or null | Number of data rows, or null when unknown/not applicable. |
| `columnCount` | number or null | Number of columns, or null when unknown/not applicable. |
| `columns` | string[] | Column names, in order. |
| `columnTypes` | array of `{ name, type }` | Per-column inferred type. `name` is the column; `type` is one of `number`, `string`, `date`, `boolean`. |
| `size` | number | Approximate payload size in bytes. |
| `created` | ISO-timestamp string | When the dataset was first added. |
| `modified` | ISO-timestamp string | When the dataset was last changed. |
The `rowCount`, `columnCount`, `columns`, `columnTypes`, and `size` fields are derived summaries computed when data is added or updated; they support previews and type display without re-parsing the full payload.
The `rowCount`, `columnCount`, `columns`, `columnTypes`, and `size` fields are derived summaries computed when data is added or updated — including when a `url` dataset is fetched or refreshed; a fetched URL snapshot profiles exactly like inline data. They support previews and type display without re-parsing the full payload.
### Schema versioning
Both **Snippet** and **Dataset** records carry a numeric `version` recording the shape of that individual record. When a record is read from storage it is migrated up to the current shape before the app uses it; new writes always store the current version. A record written before versioning existed (no `version` field) is treated as version `1`. This is distinct from the storage container's own layout version, and from the _Import & Export_ envelope `version` (which describes the file format, not a record). Records exported via _Import & Export_ include their `version`.
The current **Dataset** version is `2`. The v1→v2 migration reflects the URL-snapshot model: a v1 `url` dataset stored its address in `data`, so migration moves that address into the new `url` field and clears `data` to `null` — the record becomes an _unfetched reference_ that renders against its live URL until the user refreshes it, at which point the fetched snapshot is stored and profiled.
## C. UserSettings
**UserSettings** holds persisted user preferences as a single structured record. The semantics and UX of each option are covered in _Settings_; the shape below is the storage contract.