Initial scaffold: spec, architecture playbook, and M0 skeleton

docs/WHY-A-SEPARATE-REBUILD.md

@@ -0,0 +1,204 @@
+# Astrolabe → Syto Integration Analysis
+
+> **Question:** Can Astrolabe (a browser-based Vega-Lite snippet manager) be integrated into
+> Syto's functionality? This document compares the Astrolabe specification (`docs/spec/`)
+> against Syto in its current state, and recommends an integration path.
+>
+> **Short answer:** Not as a wholesale port, and not as the "snippet manager" it is today —
+> that framing collides with Syto's stated non-goals. But the *valuable parts* of Astrolabe
+> (the Chart Builder, the generic spec→render pipeline, the schema-assisted JSON editor) map
+> cleanly onto a Syto-native **"chart a model"** feature, and most of the supporting tech already
+> exists in the codebase. The recommendation is **harvest, don't port** — and the framing decision
+> needs a `SOUL.md` ruling first.
+
+---
+
+## 1. Executive Summary
+
+| | |
+|---|---|
+| **Conceptual fit** | Partial. Astrolabe and Syto are both local-first, browser-only, Vega-Lite-using, three-pane-ish workspaces. But Astrolabe's *primary entity* (a saved chart spec) is a thing Syto deliberately does not have. |
+| **Strategic fit** | **Conflicted.** `SOUL.md` explicitly lists "Not a BI/visualization platform — charts are for exploration during wrangling, not final output" as a non-goal, and "Do One Thing Well." A *snippet library* is chart-authoring-as-product. This is the central tension and must be resolved before any code. |
+| **Technical fit** | **Good for the rendering/editing layer, poor for the data-model and shell layers.** Syto already ships Vega-Lite, vega-embed, CodeMirror 6, IndexedDB persistence, a settings system, URL-hash routing, and a far stronger type/schema engine than Astrolabe's profiler. The friction is in the *parallel systems* a verbatim port would introduce. |
+| **Recommended path** | **Option B (harvest into a native "Visualize" feature).** Reuse Astrolabe's Chart Builder and rendering contract; bind them to Syto **Models** instead of a new "dataset" entity; drop the snippet-as-primary-entity, the draft/published workflow, the separate dataset library, and the separate import/export envelope. |
+
+---
+
+## 2. The Two Products Side by Side
+
+| Dimension | **Astrolabe** | **Syto** |
+|---|---|---|
+| Core artifact | A **snippet** = a saved Vega-Lite spec + metadata | A **workflow** = a declarative transform pipeline over a Source |
+| Primary verb | *Author & organize charts* | *Clean & reshape tabular data* |
+| Data unit | **Dataset** (named blob: JSON/CSV/TSV/TopoJSON, inline or URL) | **Source** (immutable imported table) → **Model** (derived table) |
+| Persistence | Snippets (~5 MB tier) + Datasets (high-capacity tier), both local | Sources + Models in IndexedDB (lazy row data), prefs in localStorage |
+| Editor | JSON editor w/ Vega-Lite schema autocomplete + live validation | CodeMirror 6 — used for transform JSON + the expression language |
+| Rendering | Renders *arbitrary user specs* via reference-resolution + fit-mode transforms | Renders *programmatically generated* EDA specs (`charts.ts`) |
+| Shell | 3 panes: library · editor · preview, + modals | Ribbon + sidebar + data table + slide-panel/modal dialogs |
+| Routing | URL hash: `#snippet-<id>`, `#datasets/...` | URL hash: active source/model/dialog |
+| Export | One JSON envelope of all snippets + datasets | Workflow v2 JSON (transforms, topo-sorted) |
+| Stack stance | Implementation-agnostic spec | Preact + Signals + Arquero + CSS Modules, fixed |
+
+**The key observation:** Astrolabe's "dataset" is conceptually Syto's "Source," and the thing you
+most want to chart in Syto — a cleaned, transformed **Model** — has *no equivalent in Astrolabe at
+all*. Astrolabe charts static blobs; Syto produces living, recomputed tables. A naive port would
+bolt a second, weaker data-library (Astrolabe datasets) next to Syto's existing one (Sources/Models),
+which directly violates SOUL's **"No Parallel Systems"** value.
+
+---
+
+## 3. The Strategic Tension (resolve this first)
+
+This is not a technical blocker; it is a product-identity decision, and per project convention
+(`SOUL.md` is the arbiter for contract/design decisions) it must be settled before implementation.
+
+**What `SOUL.md` currently says:**
+
+- *"Do One Thing Well… It's not trying to become a spreadsheet, a statistical package, a visualization tool, or a database. The EDA features… exist to help users understand their data before transforming it — not to replace dedicated analysis tools."*
+- *"Not a BI/visualization platform: Charts are for exploration during wrangling, not final output. Dashboards and reporting are a separate concern."*
+
+A **snippet manager** — a personal, searchable, import/exportable *library of saved charts* — is
+squarely "charts as final output" and "a visualization tool." Porting Astrolabe as-is would
+contradict two written non-goals.
+
+**However**, there is a reading that is fully *aligned* with the rest of SOUL:
+
+- *"Beginner-Friendly, Not Beginner-Limited"* and *"Progressive Complexity"* — today a user can clean data but has **no way to turn the result into a shareable picture.** A chart is the natural last step of a wrangling session.
+- *"Leverage Existing Libraries — Vega-Lite handles charts."* The infrastructure is already paid for.
+- Astrolabe's **Chart Builder** (pick a mark, map columns → spec) is the *exact* beginner-friendly, no-JSON affordance Syto favors, and the live JSON editor is the power-user escape hatch.
+
+**The decision to make:** Is "produce a chart as the output of a workflow" *part of* doing the one
+thing well (wrangling ends in a usable artifact), or is it the BI/viz scope SOUL rejects?
+
+Two coherent resolutions:
+
+- **(A) Amend SOUL** to permit *single-chart output of a model* (not dashboards, not a chart library-as-product) — and integrate as a native feature (§6, Option B).
+- **(B) Keep it separate** — Astrolabe stays its own thing, or lives as a sibling `/tools/` mini-app that merely *shares code* with Syto (§6, Option C). The main app's non-goals stay intact.
+
+I recommend (A) with a tightly-scoped amendment, because the value lands precisely where Syto is
+currently weakest (no output artifact), and because doing it natively avoids the parallel-systems
+trap. But this is the user's call to make against SOUL.
+
+---
+
+## 4. Feature-by-Feature Reuse Map
+
+Legend: 🟢 already exists / strong reuse · 🟡 partial, needs adaptation · 🔴 net-new build
+
+| Astrolabe feature | Syto today | Verdict | Notes |
+|---|---|---|---|
+| **Vega-Lite rendering** | `charts.ts` + `vega-embed@7` render programmatic specs into DOM refs | 🟡 | Engine present; needs a *generic* "render this arbitrary spec" path + error surface. The hardcoded EDA specs don't help directly, but the rendering primitive does. |
+| **Dataset-reference resolution** (`{data:{name}}` → contents, recursing into layers) | none | 🔴 | New, but small and pure — and in Syto it resolves to a **Model's data**, not a separate dataset store. |
+| **Fit-mode transforms** (Original/Width/Height/Full via `"container"`) | none | 🔴 | Small, pure, copy-on-render spec rewrite. Directly portable. |
+| **JSON spec editor** | CodeMirror 6 (`CodeMirrorEditor.tsx`, `JsonEditorModal.tsx`) + lint infra (`linters/`) | 🟡 | Editor & lint plumbing exist. Missing: a **Vega-Lite schema service** for autocomplete + validation. (Note: Astrolabe's "minimap" and "VS Light/Dark/High-Contrast" editor themes are Monaco-isms; Syto is on CodeMirror — those exact settings don't carry over.) |
+| **Chart Builder** (mark + X/Y/Color/Size + field types → spec) | none | 🟡→🔴 | The single most valuable, most SOUL-aligned piece. Build it against a **Model's columns** using Syto's existing schema types. High reuse of the *dialog* pattern (registry + slide-panel/modal + debounced preview). |
+| **Column profiling / type inference** | `schema-engine.ts` (integer/float/date/datetime/boolean/json) | 🟢 | Syto's engine **supersedes** Astrolabe's (number/string/date/boolean). Astrolabe→Vega field-type mapping (numeric→Quantitative, date→Temporal, else Nominal) layers on top trivially. |
+| **Datasets library + manager modal** | Sources/Models already *are* the data library | 🔴 *(avoid)* | Do **not** build. Reuse Sources/Models. Building it = parallel systems. |
+| **Snippet library** (search/sort/CRUD, draft vs published, status, tags, storage monitor) | none | 🔴 | The genuinely new persistent entity. Only needed if going full snippet-manager (not recommended). Draft/Published has no analog in Syto's undo/redo model. |
+| **Settings** (editor/performance/formatting) | `ux-settings.ts` + settings dialog | 🟡 | System exists; add render-debounce + a couple of fields. Editor-theme/minimap fields are Monaco-shaped and mostly drop. |
+| **Import/Export envelope** (snippets+datasets JSON) | Workflow v2 export/import | 🔴 *(avoid)* | A second export format competing with workflow v2. If charts are part of a workflow, they belong *in* the workflow spec or alongside it — not in a rival envelope. |
+| **App shell / 3-pane layout** | Ribbon + sidebar + table + slide-panel | 🔴 *(avoid)* | Don't graft Astrolabe's shell. A chart view is a *mode/panel within* Syto's shell. |
+| **URL-hash routing** | Hash routing for source/model/dialog | 🟡 | Reusable, but Astrolabe's `#snippet-…`/`#datasets/…` scheme would **collide**; must namespace under Syto's existing scheme. |
+| **Keyboard shortcuts** | `EventRouter` owns Ctrl+S (save), Escape priority chain, etc. | 🟡 | **Collisions:** Astrolabe binds Ctrl+S (Publish) and Ctrl+K (Datasets). Syto already owns Ctrl+S. Must reconcile, not adopt verbatim. |
+| **Offline / PWA / installable** | `vite-plugin-pwa` already configured | 🟢 | Free. |
+| **i18n** | i18next, en/uk, namespaced | 🟢 | New strings go in a namespace; framework is there. |
+| **Toasts** | Notification system exists | 🟢 | Reuse. |
+
+**Reuse tally:** the *rendering, editing, persistence, settings, schema, i18n, PWA, and toast*
+substrate is largely present. The *data-model, shell, routing-scheme, and lifecycle* layers of
+Astrolabe are either redundant with Syto or actively conflicting and should be dropped.
+
+---
+
+## 5. Technical Friction Points (if ported verbatim)
+
+1. **Parallel data library.** Astrolabe datasets vs Syto Sources/Models — two stores, two
+   profilers, two "named data" concepts. Violates *No Parallel Systems*. (The fix: charts reference
+   Models.)
+2. **Parallel persistence + export.** A second IndexedDB store layout and a second JSON envelope
+   alongside workflow v2. Two backup formats for users to confuse.
+3. **Draft/Published has no home.** Syto's non-destructive model is *pipeline steps + undo/redo*,
+   not a per-document draft/published toggle. Astrolabe's central editing model would be a third,
+   unrelated state concept.
+4. **Shell mismatch.** Astrolabe's library·editor·preview triptych is a *whole app*. Syto's shell is
+   ribbon-driven with slide-panel dialogs. They don't compose; one must yield.
+5. **Routing & shortcut collisions.** Hash schemes overlap; Ctrl+S/Ctrl+K already bound.
+6. **Editor-feature gap.** Syto is on CodeMirror (no minimap, different theme model); Astrolabe's
+   settings assume Monaco. And neither today has a **Vega-Lite schema service** — that autocomplete/
+   validation is net-new work on either stack.
+7. **TopoJSON / arbitrary-JSON data.** Syto Sources are *tabular*. Astrolabe datasets include
+   TopoJSON and arbitrary JSON. Charting a Model covers the tabular case; map/topology charts would
+   be out of scope unless Sources grow a non-tabular kind.
+
+None of these are unsolvable — but every one of them is *work created by the port itself*, not by
+the user value. That's the signature of "harvest, don't port."
+
+---
+
+## 6. Integration Options
+
+### Option A — Full port (snippet manager inside Syto)
+Bring Astrolabe over more-or-less intact: snippet library, dataset manager, draft/published, its
+shell, its export.
+- **Pros:** Fastest way to "have Astrolabe." Feature-complete chart authoring.
+- **Cons:** Maximal parallel-systems debt (§5). Directly contradicts SOUL non-goals. Two data
+  libraries, two export formats, shell/routing/shortcut conflicts. **Not recommended.**
+
+### Option B — Harvest into a native "Visualize" feature ✅ *recommended*
+Add charting as the natural *output* step of a workflow, reusing Syto's own primitives:
+- A **"Chart" / "Visualize"** action on a **Model** opens a **Chart Builder** (Astrolabe's mark +
+  X/Y/Color/Size + field-type controls), populated from the Model's columns and `schema-engine`
+  types.
+- It produces a Vega-Lite spec rendered live via the existing `vega-embed`, using a ported
+  **reference-resolution + fit-mode** rendering contract where the named data resolves to the
+  **Model's rows**.
+- Power users get the **JSON spec editor** (CodeMirror, with a Vega-Lite schema service added) as the
+  escape hatch — consistent with *Beginner-Friendly, Not Beginner-Limited*.
+- The chart (its spec) is persisted **attached to the Model** (or to the workflow), not as a separate
+  snippet entity. Export rides along with workflow v2 (or a sibling field), not a rival envelope.
+- **Dropped from Astrolabe:** separate dataset library, draft/published, snippet search/sort/tags,
+  storage monitor, its shell, its import/export, its routing scheme.
+- **Pros:** No parallel systems. Lands value exactly where Syto is weak (no output artifact). Maximal
+  reuse of existing infra. Defensible against SOUL with a *narrow* amendment ("single-chart output of
+  a model," not dashboards/library).
+- **Cons:** Requires the SOUL decision (§3). Loses Astrolabe features that depend on the
+  snippet/dataset model (TopoJSON/URL datasets, multi-snippet library). Net-new: schema service,
+  builder dialog, render contract.
+
+### Option C — Sibling `/tools/` mini-app
+Port Astrolabe as a self-contained app under `/tools/astrolabe/`, sharing only *code* (vega render
+helpers, CodeMirror wrapper, i18n) with the main app — no AppStore/DialogStore coupling.
+- **Pros:** Keeps the main app's non-goals pristine (it's a separate utility, like other tools).
+  Lower conceptual conflict. Astrolabe keeps its own model.
+- **Cons:** Syto's `/tools/` layer is designed for *small, single-purpose* utilities; Astrolabe is a
+  full application — a stretch for that slot. Still carries Astrolabe's whole parallel data model,
+  just quarantined. "Integration" here means "co-located," not "unified" — limited synergy.
+
+---
+
+## 7. Recommendation
+
+1. **Make the SOUL call first (§3).** Decide whether single-chart *output of a model* is in scope.
+   If **no**, stop here or pursue Option C as a quarantined sibling. If **yes**, amend SOUL with a
+   tight scope statement and proceed to Option B.
+2. **Pursue Option B.** Harvest the three high-value, well-aligned pieces:
+   - the **Chart Builder** (bound to a Model, driven by `schema-engine` types),
+   - the **rendering contract** (reference-resolution + fit modes, resolving to Model data),
+   - the **schema-assisted JSON editor** (CodeMirror + a new Vega-Lite schema service).
+3. **Drop the parallel-systems pieces:** separate dataset library, draft/published, snippet
+   library + storage monitor, separate import/export envelope, Astrolabe's shell and routing scheme.
+4. **Reconcile, don't adopt,** the cross-cutting surfaces: fold settings into `ux-settings`,
+   namespace any new hash state under Syto's scheme, resolve the Ctrl+S/Ctrl+K shortcut collisions.
+
+This delivers the genuinely useful core of Astrolabe — turning cleaned data into a chart, with a
+beginner path and a power-user path — while staying true to *Do One Thing Well* and *No Parallel
+Systems*, and reusing the infrastructure Syto has already built.
+
+---
+
+## 8. Open Questions for the User
+
+- **SOUL scope:** Is "a chart as the output of a workflow" inside Syto's mission, or out? (Blocks everything.)
+- **Persistence model:** Should a chart spec live *on a Model*, *in the workflow v2 export*, or as a new top-level entity?
+- **Non-tabular data:** Do we ever need TopoJSON / arbitrary-JSON charts (maps), which Syto Sources can't currently hold? If not, that simplifies scope considerably.
+- **Editor depth:** Is full Vega-Lite schema autocomplete/validation in scope, or is a plain JSON editor + live error surface enough for v1?