astrolabe/docs/WHY-A-SEPARATE-REBUILD.md

Astrolabe → Syto Integration Analysis

Question: Can Astrolabe (a browser-based Vega-Lite snippet manager) be integrated into Syto's functionality? This document compares the Astrolabe specification (docs/spec/) against Syto in its current state, and recommends an integration path.

Short answer: Not as a wholesale port, and not as the "snippet manager" it is today — that framing collides with Syto's stated non-goals. But the valuable parts of Astrolabe (the Chart Builder, the generic spec→render pipeline, the schema-assisted JSON editor) map cleanly onto a Syto-native "chart a model" feature, and most of the supporting tech already exists in the codebase. The recommendation is harvest, don't port — and the framing decision needs a SOUL.md ruling first.

---

1. Executive Summary

Conceptual fit	Partial. Astrolabe and Syto are both local-first, browser-only, Vega-Lite-using, three-pane-ish workspaces. But Astrolabe's primary entity (a saved chart spec) is a thing Syto deliberately does not have.
Strategic fit	Conflicted. SOUL.md explicitly lists "Not a BI/visualization platform — charts are for exploration during wrangling, not final output" as a non-goal, and "Do One Thing Well." A snippet library is chart-authoring-as-product. This is the central tension and must be resolved before any code.
Technical fit	Good for the rendering/editing layer, poor for the data-model and shell layers. Syto already ships Vega-Lite, vega-embed, CodeMirror 6, IndexedDB persistence, a settings system, URL-hash routing, and a far stronger type/schema engine than Astrolabe's profiler. The friction is in the parallel systems a verbatim port would introduce.
Recommended path	Option B (harvest into a native "Visualize" feature). Reuse Astrolabe's Chart Builder and rendering contract; bind them to Syto Models instead of a new "dataset" entity; drop the snippet-as-primary-entity, the draft/published workflow, the separate dataset library, and the separate import/export envelope.

---

2. The Two Products Side by Side

Dimension	Astrolabe	Syto
Core artifact	A snippet = a saved Vega-Lite spec + metadata	A workflow = a declarative transform pipeline over a Source
Primary verb	Author & organize charts	Clean & reshape tabular data
Data unit	Dataset (named blob: JSON/CSV/TSV/TopoJSON, inline or URL)	Source (immutable imported table) → Model (derived table)
Persistence	Snippets (~5 MB tier) + Datasets (high-capacity tier), both local	Sources + Models in IndexedDB (lazy row data), prefs in localStorage
Editor	JSON editor w/ Vega-Lite schema autocomplete + live validation	CodeMirror 6 — used for transform JSON + the expression language
Rendering	Renders arbitrary user specs via reference-resolution + fit-mode transforms	Renders programmatically generated EDA specs (charts.ts)
Shell	3 panes: library · editor · preview, + modals	Ribbon + sidebar + data table + slide-panel/modal dialogs
Routing	URL hash: #snippet-<id>, #datasets/...	URL hash: active source/model/dialog
Export	One JSON envelope of all snippets + datasets	Workflow v2 JSON (transforms, topo-sorted)
Stack stance	Implementation-agnostic spec	Preact + Signals + Arquero + CSS Modules, fixed

The key observation: Astrolabe's "dataset" is conceptually Syto's "Source," and the thing you most want to chart in Syto — a cleaned, transformed Model — has no equivalent in Astrolabe at all. Astrolabe charts static blobs; Syto produces living, recomputed tables. A naive port would bolt a second, weaker data-library (Astrolabe datasets) next to Syto's existing one (Sources/Models), which directly violates SOUL's "No Parallel Systems" value.

---

3. The Strategic Tension (resolve this first)

This is not a technical blocker; it is a product-identity decision, and per project convention (SOUL.md is the arbiter for contract/design decisions) it must be settled before implementation.

What SOUL.md currently says:

• "Do One Thing Well… It's not trying to become a spreadsheet, a statistical package, a visualization tool, or a database. The EDA features… exist to help users understand their data before transforming it — not to replace dedicated analysis tools."
• "Not a BI/visualization platform: Charts are for exploration during wrangling, not final output. Dashboards and reporting are a separate concern."

A snippet manager — a personal, searchable, import/exportable library of saved charts — is squarely "charts as final output" and "a visualization tool." Porting Astrolabe as-is would contradict two written non-goals.

However, there is a reading that is fully aligned with the rest of SOUL:

• "Beginner-Friendly, Not Beginner-Limited" and "Progressive Complexity" — today a user can clean data but has no way to turn the result into a shareable picture. A chart is the natural last step of a wrangling session.
• "Leverage Existing Libraries — Vega-Lite handles charts." The infrastructure is already paid for.
• Astrolabe's Chart Builder (pick a mark, map columns → spec) is the exact beginner-friendly, no-JSON affordance Syto favors, and the live JSON editor is the power-user escape hatch.

The decision to make: Is "produce a chart as the output of a workflow" part of doing the one thing well (wrangling ends in a usable artifact), or is it the BI/viz scope SOUL rejects?

Two coherent resolutions:

• (A) Amend SOUL to permit single-chart output of a model (not dashboards, not a chart library-as-product) — and integrate as a native feature (§6, Option B).
• (B) Keep it separate — Astrolabe stays its own thing, or lives as a sibling /tools/ mini-app that merely shares code with Syto (§6, Option C). The main app's non-goals stay intact.

I recommend (A) with a tightly-scoped amendment, because the value lands precisely where Syto is currently weakest (no output artifact), and because doing it natively avoids the parallel-systems trap. But this is the user's call to make against SOUL.

---

4. Feature-by-Feature Reuse Map

Legend: 🟢 already exists / strong reuse · 🟡 partial, needs adaptation · 🔴 net-new build

Astrolabe feature	Syto today	Verdict	Notes
Vega-Lite rendering	charts.ts + vega-embed@7 render programmatic specs into DOM refs	🟡	Engine present; needs a generic "render this arbitrary spec" path + error surface. The hardcoded EDA specs don't help directly, but the rendering primitive does.
Dataset-reference resolution ({data:{name}} → contents, recursing into layers)	none	🔴	New, but small and pure — and in Syto it resolves to a Model's data, not a separate dataset store.
Fit-mode transforms (Original/Width/Height/Full via "container")	none	🔴	Small, pure, copy-on-render spec rewrite. Directly portable.
JSON spec editor	CodeMirror 6 (CodeMirrorEditor.tsx, JsonEditorModal.tsx) + lint infra (linters/)	🟡	Editor & lint plumbing exist. Missing: a Vega-Lite schema service for autocomplete + validation. (Note: Astrolabe's "minimap" and "VS Light/Dark/High-Contrast" editor themes are Monaco-isms; Syto is on CodeMirror — those exact settings don't carry over.)
Chart Builder (mark + X/Y/Color/Size + field types → spec)	none	🟡→🔴	The single most valuable, most SOUL-aligned piece. Build it against a Model's columns using Syto's existing schema types. High reuse of the dialog pattern (registry + slide-panel/modal + debounced preview).
Column profiling / type inference	schema-engine.ts (integer/float/date/datetime/boolean/json)	🟢	Syto's engine supersedes Astrolabe's (number/string/date/boolean). Astrolabe→Vega field-type mapping (numeric→Quantitative, date→Temporal, else Nominal) layers on top trivially.
Datasets library + manager modal	Sources/Models already are the data library	🔴 (avoid)	Do not build. Reuse Sources/Models. Building it = parallel systems.
Snippet library (search/sort/CRUD, draft vs published, status, tags, storage monitor)	none	🔴	The genuinely new persistent entity. Only needed if going full snippet-manager (not recommended). Draft/Published has no analog in Syto's undo/redo model.
Settings (editor/performance/formatting)	ux-settings.ts + settings dialog	🟡	System exists; add render-debounce + a couple of fields. Editor-theme/minimap fields are Monaco-shaped and mostly drop.
Import/Export envelope (snippets+datasets JSON)	Workflow v2 export/import	🔴 (avoid)	A second export format competing with workflow v2. If charts are part of a workflow, they belong in the workflow spec or alongside it — not in a rival envelope.
App shell / 3-pane layout	Ribbon + sidebar + table + slide-panel	🔴 (avoid)	Don't graft Astrolabe's shell. A chart view is a mode/panel within Syto's shell.
URL-hash routing	Hash routing for source/model/dialog	🟡	Reusable, but Astrolabe's #snippet-…/#datasets/… scheme would collide; must namespace under Syto's existing scheme.
Keyboard shortcuts	EventRouter owns Ctrl+S (save), Escape priority chain, etc.	🟡	Collisions: Astrolabe binds Ctrl+S (Publish) and Ctrl+K (Datasets). Syto already owns Ctrl+S. Must reconcile, not adopt verbatim.
Offline / PWA / installable	vite-plugin-pwa already configured	🟢	Free.
i18n	i18next, en/uk, namespaced	🟢	New strings go in a namespace; framework is there.
Toasts	Notification system exists	🟢	Reuse.

Reuse tally: the rendering, editing, persistence, settings, schema, i18n, PWA, and toast substrate is largely present. The data-model, shell, routing-scheme, and lifecycle layers of Astrolabe are either redundant with Syto or actively conflicting and should be dropped.

---

5. Technical Friction Points (if ported verbatim)

1. Parallel data library. Astrolabe datasets vs Syto Sources/Models — two stores, two profilers, two "named data" concepts. Violates No Parallel Systems. (The fix: charts reference Models.)
2. Parallel persistence + export. A second IndexedDB store layout and a second JSON envelope alongside workflow v2. Two backup formats for users to confuse.
3. Draft/Published has no home. Syto's non-destructive model is pipeline steps + undo/redo, not a per-document draft/published toggle. Astrolabe's central editing model would be a third, unrelated state concept.
4. Shell mismatch. Astrolabe's library·editor·preview triptych is a whole app. Syto's shell is ribbon-driven with slide-panel dialogs. They don't compose; one must yield.
5. Routing & shortcut collisions. Hash schemes overlap; Ctrl+S/Ctrl+K already bound.
6. Editor-feature gap. Syto is on CodeMirror (no minimap, different theme model); Astrolabe's settings assume Monaco. And neither today has a Vega-Lite schema service — that autocomplete/ validation is net-new work on either stack.
7. TopoJSON / arbitrary-JSON data. Syto Sources are tabular. Astrolabe datasets include TopoJSON and arbitrary JSON. Charting a Model covers the tabular case; map/topology charts would be out of scope unless Sources grow a non-tabular kind.

None of these are unsolvable — but every one of them is work created by the port itself, not by the user value. That's the signature of "harvest, don't port."

---

6. Integration Options

Option A — Full port (snippet manager inside Syto)

Bring Astrolabe over more-or-less intact: snippet library, dataset manager, draft/published, its shell, its export.

• Pros: Fastest way to "have Astrolabe." Feature-complete chart authoring.
• Cons: Maximal parallel-systems debt (§5). Directly contradicts SOUL non-goals. Two data libraries, two export formats, shell/routing/shortcut conflicts. Not recommended.

Option B — Harvest into a native "Visualize" feature ✅ recommended

Add charting as the natural output step of a workflow, reusing Syto's own primitives:

• A "Chart" / "Visualize" action on a Model opens a Chart Builder (Astrolabe's mark + X/Y/Color/Size + field-type controls), populated from the Model's columns and schema-engine types.
• It produces a Vega-Lite spec rendered live via the existing vega-embed, using a ported reference-resolution + fit-mode rendering contract where the named data resolves to the Model's rows.
• Power users get the JSON spec editor (CodeMirror, with a Vega-Lite schema service added) as the escape hatch — consistent with Beginner-Friendly, Not Beginner-Limited.
• The chart (its spec) is persisted attached to the Model (or to the workflow), not as a separate snippet entity. Export rides along with workflow v2 (or a sibling field), not a rival envelope.
• Dropped from Astrolabe: separate dataset library, draft/published, snippet search/sort/tags, storage monitor, its shell, its import/export, its routing scheme.
• Pros: No parallel systems. Lands value exactly where Syto is weak (no output artifact). Maximal reuse of existing infra. Defensible against SOUL with a narrow amendment ("single-chart output of a model," not dashboards/library).
• Cons: Requires the SOUL decision (§3). Loses Astrolabe features that depend on the snippet/dataset model (TopoJSON/URL datasets, multi-snippet library). Net-new: schema service, builder dialog, render contract.

Option C — Sibling /tools/ mini-app

Port Astrolabe as a self-contained app under /tools/astrolabe/, sharing only code (vega render helpers, CodeMirror wrapper, i18n) with the main app — no AppStore/DialogStore coupling.

• Pros: Keeps the main app's non-goals pristine (it's a separate utility, like other tools). Lower conceptual conflict. Astrolabe keeps its own model.
• Cons: Syto's /tools/ layer is designed for small, single-purpose utilities; Astrolabe is a full application — a stretch for that slot. Still carries Astrolabe's whole parallel data model, just quarantined. "Integration" here means "co-located," not "unified" — limited synergy.

---

7. Recommendation

1. Make the SOUL call first (§3). Decide whether single-chart output of a model is in scope. If no, stop here or pursue Option C as a quarantined sibling. If yes, amend SOUL with a tight scope statement and proceed to Option B.
2. Pursue Option B. Harvest the three high-value, well-aligned pieces:
   • the Chart Builder (bound to a Model, driven by schema-engine types),
   • the rendering contract (reference-resolution + fit modes, resolving to Model data),
   • the schema-assisted JSON editor (CodeMirror + a new Vega-Lite schema service).
3. Drop the parallel-systems pieces: separate dataset library, draft/published, snippet library + storage monitor, separate import/export envelope, Astrolabe's shell and routing scheme.
4. Reconcile, don't adopt, the cross-cutting surfaces: fold settings into ux-settings, namespace any new hash state under Syto's scheme, resolve the Ctrl+S/Ctrl+K shortcut collisions.

This delivers the genuinely useful core of Astrolabe — turning cleaned data into a chart, with a beginner path and a power-user path — while staying true to Do One Thing Well and No Parallel Systems, and reusing the infrastructure Syto has already built.

---

8. Open Questions for the User

• SOUL scope: Is "a chart as the output of a workflow" inside Syto's mission, or out? (Blocks everything.)
• Persistence model: Should a chart spec live on a Model, in the workflow v2 export, or as a new top-level entity?
• Non-tabular data: Do we ever need TopoJSON / arbitrary-JSON charts (maps), which Syto Sources can't currently hold? If not, that simplifies scope considerably.
• Editor depth: Is full Vega-Lite schema autocomplete/validation in scope, or is a plain JSON editor + live error surface enough for v1?