From 2410c6e965a0bf6cc6743edaf3327581a27b6d35 Mon Sep 17 00:00:00 2001 From: Oleh Omelchenko Date: Wed, 10 Jun 2026 10:24:42 +0300 Subject: [PATCH] Snapshot URL datasets locally on add; preview tabular data as a table --- docs/IMPLEMENTATION-PLAN.md | 3 +- docs/architecture/02-persistence.md | 23 +- docs/architecture/06-type-inference.md | 10 +- docs/manual-verification.md | 17 ++ docs/spec/04-live-preview.md | 17 +- docs/spec/05-datasets.md | 22 +- docs/spec/09-data-model.md | 38 ++-- src/app/components/DatasetsModal.module.css | 73 ++++++ src/app/components/DatasetsModal.tsx | 209 ++++++++++++++++-- .../infrastructure/dataset-migrations.test.ts | 73 ++++++ src/app/infrastructure/dataset-migrations.ts | 23 +- src/app/infrastructure/remote-data.test.ts | 94 ++++++++ src/app/infrastructure/remote-data.ts | 99 +++++++++ src/app/services/remote-data-errors.test.ts | 38 ++++ src/app/services/remote-data-errors.ts | 51 +++++ src/app/stores/DatasetStore.test.ts | 73 +++++- src/app/stores/DatasetStore.ts | 160 ++++++++++++-- src/core/dataset.test.ts | 143 ++++++++++-- src/core/dataset.ts | 131 ++++++++--- src/core/import-normalize.test.ts | 36 +++ src/core/import-normalize.ts | 21 +- src/core/rendering.test.ts | 20 +- src/core/rendering.ts | 13 +- 23 files changed, 1239 insertions(+), 148 deletions(-) create mode 100644 src/app/infrastructure/dataset-migrations.test.ts create mode 100644 src/app/infrastructure/remote-data.test.ts create mode 100644 src/app/infrastructure/remote-data.ts create mode 100644 src/app/services/remote-data-errors.test.ts create mode 100644 src/app/services/remote-data-errors.ts diff --git a/docs/IMPLEMENTATION-PLAN.md b/docs/IMPLEMENTATION-PLAN.md index 91d03ec..702a8ab 100644 --- a/docs/IMPLEMENTATION-PLAN.md +++ b/docs/IMPLEMENTATION-PLAN.md @@ -268,7 +268,8 @@ the reference. - Snippet `datasetRefs` maintained on publish; library shows dataset icon + Linked Datasets; dataset detail shows Linked Snippets (bidirectional name link, §09F). - **Extract-to-Dataset** flow from the editor (§03F). -- URL-sourced datasets fetched at render time. +- URL datasets fetched once on add and **snapshotted** locally (profiled like inline; + render from the snapshot, refreshable on demand — see §05 / §04 rendering contract). **Tests** diff --git a/docs/architecture/02-persistence.md b/docs/architecture/02-persistence.md index eb40489..aa9fdf8 100644 --- a/docs/architecture/02-persistence.md +++ b/docs/architecture/02-persistence.md @@ -6,19 +6,20 @@ How Astrolabe stores data in the browser, and the rules that keep that storage t ## 1. The Infrastructure-Adapter Principle -**Rule: nothing outside `src/app/infrastructure/` ever touches `indexedDB`, `localStorage`, `window`, or `location` directly.** Every browser-storage interaction goes through a typed adapter module that exposes plain async functions returning domain objects. +**Rule: nothing outside `src/app/infrastructure/` ever touches `indexedDB`, `localStorage`, `window`, `location`, or `fetch` directly.** Every browser interaction (storage _and_ network) goes through a typed adapter module that exposes plain async functions returning domain objects. ``` src/ ├── core/ # portable engine — NO browser APIs, NO React ├── app/ -│ ├── stores/ # Zustand stores; calls infrastructure, never IDB -│ ├── services/ # business logic; calls infrastructure, never IDB -│ └── infrastructure/ # the ONLY place that imports indexedDB/localStorage +│ ├── stores/ # Zustand stores; calls infrastructure, never IDB/fetch +│ ├── services/ # business logic; calls infrastructure, never IDB/fetch +│ └── infrastructure/ # the ONLY place that imports indexedDB/localStorage/fetch │ ├── snippet-store.ts # IndexedDB: snippets (metadata + drafts) │ ├── dataset-store.ts # IndexedDB: datasets (heavy payloads) │ ├── settings-store.ts # localStorage: UserSettings -│ └── ux-prefs.ts # localStorage: app/UI prefs (sort, panel layout) +│ ├── ux-prefs.ts # localStorage: app/UI prefs (sort, panel layout) +│ └── remote-data.ts # network: fetch a URL dataset's body (the ONLY fetch) ``` ### Why this boundary exists @@ -29,7 +30,15 @@ src/ - **Failure containment.** Quota errors, corrupt JSON, and missing keys are handled at the boundary and converted into typed results (or sane fallbacks), so the rest of the app never sees a raw `DOMException`. > **Do:** `import { saveSnippet } from '@/app/infrastructure/snippet-store'` -> **Don't:** `indexedDB.open(...)` or `localStorage.getItem(...)` anywhere in a component, store, or service. +> **Don't:** `indexedDB.open(...)`, `localStorage.getItem(...)`, or `fetch(...)` anywhere in a component, store, or service. + +### Background vs. interactive adapters + +Most adapters are driven by **background subscribers** (arch 01 §5, _Effects_): a store changes, a startup subscriber writes it through to IndexedDB — the store never calls the adapter itself. The **network** adapter is the exception. Fetching a URL dataset is a user-initiated action with its own pending/error UI, so the **component** calls `remote-data.ts` directly (components may call adapters — cf. `navigator.clipboard`) and hands the fetched body to **pure** store actions (`DatasetStore.commitUrlSnapshot` / `refreshDataset`). The store never fetches, so it stays browser-free and unit-testable on already-fetched text. + +> **Rule:** keep `fetch` behind `remote-data.ts`; orchestrate the URL-dataset fetch in the _component_ (busy state + the "paste data inline instead" recovery), not the store. Store commit actions only ever receive already-fetched text. + +**URL-dataset snapshot lifecycle** (the files a change to it touches): add / Refresh → component fetches (`infrastructure/remote-data.ts`) → `core/dataset.snapshotFromText` shapes the body by sniffed format → `DatasetStore.commitUrlSnapshot` / `refreshDataset` snapshots + profiles it _exactly like inline data_ → render resolves it cached-first in `core/rendering.resolvedData` (a live-URL fallback applies only while a URL dataset is still unfetched). See spec §05 for the behavior. --- @@ -221,6 +230,8 @@ export async function saveSnippet(s: Snippet): Promise { > **Do:** default `version` to the earliest shape (`1`) when the field is absent. > **Don't:** branch on the presence of individual fields scattered through the app to detect "old data." Centralize that knowledge in the migration function. +> **Mirror shape changes in the import normalizer.** Imported records are built from a file, not read from IndexedDB, so they **never pass through `migrate`** — `core/import-normalize.ts` upgrades them independently. A migration that changes a field's _shape_ must be applied in both places or import produces a malformed record. (E.g. the dataset v1→v2 URL-snapshot reshaping — address moves from `data` into `url`, `data` cleared — lives in **both** `migrateDataset` and `normalizeDataset`.) + --- ## 5. localStorage Preferences (Settings & App/UI Prefs) diff --git a/docs/architecture/06-type-inference.md b/docs/architecture/06-type-inference.md index 914b2b5..43b2495 100644 --- a/docs/architecture/06-type-inference.md +++ b/docs/architecture/06-type-inference.md @@ -183,7 +183,10 @@ how the UI shows **"N/A"** — see §3.2. ### 3.1 What gets profiled -Profiling applies only to **tabular inline data**: +Profiling applies to any **tabular payload**, whether pasted inline or fetched +from a URL (the snapshot model stores a URL dataset's data locally, so it profiles +through the same path as inline data — `snapshotFromText` shapes the fetched body, +then `computeDatasetProfile` runs): - **JSON** that is an array of objects. - **CSV** (comma-separated, header row). @@ -191,8 +194,9 @@ Profiling applies only to **tabular inline data**: Everything else is **not profiled**: -- **URL datasets** — the library holds only the link, not the data, so there is - nothing to scan. Counts are `null` / N/A. +- **Unfetched URL datasets** — a URL reference with no snapshot yet (e.g. one + migrated from an older record), so there is nothing to scan. Counts are `null` / + N/A until it is refreshed. - **Non-tabular data** — a single JSON object, TopoJSON, or anything we can't read as rows-of-columns. Counts are `null` / N/A. diff --git a/docs/manual-verification.md b/docs/manual-verification.md index 33fe86c..660fb7d 100644 --- a/docs/manual-verification.md +++ b/docs/manual-verification.md @@ -44,6 +44,23 @@ - [ ] Toasts stack, auto-dismiss, and fade as specified; the live-preview busy indicator appears for slow (>~1s) renders and clears after. +## URL datasets (remote data snapshot) + +> Needs a real network, real CORS, and real offline — tests mock the fetch. + +- [ ] Add a dataset by URL from a CORS-friendly host (e.g. a GitHub raw `.csv` or a + vega-datasets URL) — Save shows "Fetching…", then the dataset appears profiled + (rows, columns, size) with a "Fetched