oleh/astrolabe
1
0
Fork 0
mirror of https://github.com/olehomelchenko/astrolabe.git synced 2026-08-08 02:02:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Snapshot URL datasets locally on add; preview tabular data as a table

Browse Source
This commit is contained in:
Oleh Omelchenko
2026-06-10 10:24:42 +03:00
parent eb5e7ac53a
commit 2410c6e965
23 changed files with 1239 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/spec/09-data-model.md
+21 -17
View File
@@ -34,29 +34,33 @@ A snippet carries two specs at once. `draftSpec` is the editable working copy; `
A **Dataset** is a named, reusable data source that snippets can reference by name instead of inlining data. Datasets are managed in the _Datasets_ manager and support multiple formats and two source kinds.
| Field | Type | Meaning |
| ------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `id` | number | Unique numeric identifier. |
| `version` | number | Schema version of this record, used for read-time migration (see _Schema versioning_ below). |
| `name` | string | Unique, human-readable name; the key snippets reference via `datasetRefs`. |
| `data` | JSON value | The payload. For `source = url`: the URL string. For `source = inline`: the raw CSV/TSV text, or the parsed JSON/TopoJSON value. |
| `format` | string | One of `json`, `csv`, `tsv`, `topojson`. |
| `source` | string | One of `inline` (data embedded in the record) or `url` (data fetched from a remote address). |
| `comment` | string | Free-form user note about the dataset. |
| `rowCount` | number or null | Number of data rows, or null when unknown/not applicable. |
| `columnCount` | number or null | Number of columns, or null when unknown/not applicable. |
| `columns` | string[] | Column names, in order. |
| `columnTypes` | array of `{ name, type }` | Per-column inferred type. `name` is the column; `type` is one of `number`, `string`, `date`, `boolean`. |
| `size` | number | Approximate payload size in bytes. |
| `created` | ISO-timestamp string | When the dataset was first added. |
| `modified` | ISO-timestamp string | When the dataset was last changed. |
| Field | Type | Meaning |
| ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | number | Unique numeric identifier. |
| `version` | number | Schema version of this record, used for read-time migration (see _Schema versioning_ below). |
| `name` | string | Unique, human-readable name; the key snippets reference via `datasetRefs`. |
| `data` | JSON value | The payload, shaped by format: raw CSV/TSV text, or the parsed JSON/TopoJSON value. For `source = url` this is the **fetched snapshot**, or `null` before the first successful fetch. |
| `format` | string | One of `json`, `csv`, `tsv`, `topojson`. |
| `source` | string | One of `inline` (data pasted into the record) or `url` (data fetched once from a remote address and snapshotted into the record). |
| `url` | string (url only) | The remote address a `url` dataset was fetched from, retained so it can be re-fetched ("Refresh"). Absent for inline datasets. |
| `fetchedAt` | ISO-timestamp or null | For `url` datasets: when the snapshot was last fetched, or `null` if never fetched. Absent for inline datasets. |
| `comment` | string | Free-form user note about the dataset. |
| `rowCount` | number or null | Number of data rows, or null when unknown/not applicable. |
| `columnCount` | number or null | Number of columns, or null when unknown/not applicable. |
| `columns` | string[] | Column names, in order. |
| `columnTypes` | array of `{ name, type }` | Per-column inferred type. `name` is the column; `type` is one of `number`, `string`, `date`, `boolean`. |
| `size` | number | Approximate payload size in bytes. |
| `created` | ISO-timestamp string | When the dataset was first added. |
| `modified` | ISO-timestamp string | When the dataset was last changed. |
The `rowCount`, `columnCount`, `columns`, `columnTypes`, and `size` fields are derived summaries computed when data is added or updated; they support previews and type display without re-parsing the full payload.
The `rowCount`, `columnCount`, `columns`, `columnTypes`, and `size` fields are derived summaries computed when data is added or updated — including when a `url` dataset is fetched or refreshed; a fetched URL snapshot profiles exactly like inline data. They support previews and type display without re-parsing the full payload.
### Schema versioning
Both **Snippet** and **Dataset** records carry a numeric `version` recording the shape of that individual record. When a record is read from storage it is migrated up to the current shape before the app uses it; new writes always store the current version. A record written before versioning existed (no `version` field) is treated as version `1`. This is distinct from the storage container's own layout version, and from the _Import & Export_ envelope `version` (which describes the file format, not a record). Records exported via _Import & Export_ include their `version`.
The current **Dataset** version is `2`. The v1→v2 migration reflects the URL-snapshot model: a v1 `url` dataset stored its address in `data`, so migration moves that address into the new `url` field and clears `data` to `null` — the record becomes an _unfetched reference_ that renders against its live URL until the user refreshes it, at which point the fetched snapshot is stored and profiled.
## C. UserSettings
**UserSettings** holds persisted user preferences as a single structured record. The semantics and UX of each option are covered in _Settings_; the shape below is the storage contract.