# 02 · Persistence Architecture

How Astrolabe stores data in the browser, and the rules that keep that storage testable, portable, and safe to evolve. This document is the implementation contract for the persistence layer. For the *behavioral* data model (what fields a Snippet or Dataset has, what the tiers hold), see [09 · Data Model & Persistence](../spec/09-data-model.md); this document covers *how the code is structured to implement it*.

---

## 1. The Infrastructure-Adapter Principle

**Rule: nothing outside `src/app/infrastructure/` ever touches `indexedDB`, `localStorage`, `window`, or `location` directly.** Every browser-storage interaction goes through a typed adapter module that exposes plain async functions returning domain objects.

```
src/
├── core/                    # portable engine — NO browser APIs, NO React
├── app/
│   ├── stores/              # Zustand stores; calls infrastructure, never IDB
│   ├── services/            # business logic; calls infrastructure, never IDB
│   └── infrastructure/      # the ONLY place that imports indexedDB/localStorage
│       ├── snippet-store.ts     # IndexedDB: snippets (metadata + drafts)
│       ├── dataset-store.ts     # IndexedDB: datasets (heavy payloads)
│       ├── settings-store.ts    # localStorage: UserSettings
│       └── prefs-store.ts       # localStorage: app/UI prefs (sort, layout)
```

### Why this boundary exists

- **Testability.** Stores and services depend on a small typed surface (`getSnippet(id): Promise<Snippet | null>`), not on the IndexedDB request API. Tests mock the adapter, not a browser global. The adapters themselves are tested directly against `fake-indexeddb` / a localStorage stub in Vitest.
- **Portability.** `src/core/` stays free of browser APIs so the spec/parse/transform logic can run in Node (tests, future CLI, SSR). The adapters are the seam where the portable core meets the browser.
- **Single place for migrations.** Schema upgrades and record migrations live in exactly one module per store. A reader looking for "how does v1 data become v2 data" has one file to open, not a scattered set of `if (record.someOldField)` checks across the UI.
- **Failure containment.** Quota errors, corrupt JSON, and missing keys are handled at the boundary and converted into typed results (or sane fallbacks), so the rest of the app never sees a raw `DOMException`.

> **Do:** `import { saveSnippet } from '@/app/infrastructure/snippet-store'`
> **Don't:** `indexedDB.open(...)` or `localStorage.getItem(...)` anywhere in a component, store, or service.

---

## 2. IndexedDB Wrapper

IndexedDB's native API is event-based (`onsuccess`/`onerror`) and verbose. The adapter wraps it into promises and exposes a tiny CRUD surface per object store. Define one shared helper and build typed stores on top of it.

### 2.1 Opening the database

Open with an explicit **version number** and an `onupgradeneeded` handler that creates/upgrades object stores. The version is a monotonically increasing integer; bump it whenever the *store layout* changes (a new object store, a new index). It is independent of per-record schema versions (§4).

```ts
// src/app/infrastructure/db.ts
const DB_NAME = 'astrolabe';
const DB_VERSION = 1;

let dbPromise: Promise<IDBDatabase> | null = null;

export function openDB(): Promise<IDBDatabase> {
  // Memoize: opening is idempotent and cheap to share across calls.
  if (dbPromise) return dbPromise;

  dbPromise = new Promise((resolve, reject) => {
    const req = indexedDB.open(DB_NAME, DB_VERSION);

    req.onupgradeneeded = (event) => {
      const db = req.result;
      const oldVersion = event.oldVersion;

      // Create stores idempotently — guard every create.
      if (!db.objectStoreNames.contains('snippets')) {
        db.createObjectStore('snippets', { keyPath: 'id' });
      }
      if (!db.objectStoreNames.contains('datasets')) {
        db.createObjectStore('datasets', { keyPath: 'id' });
      }

      // Per-version store-layout migrations go here, gated on oldVersion.
      // if (oldVersion < 2) { /* add index, split a store, ... */ }
      void oldVersion;
    };

    req.onsuccess = () => resolve(req.result);
    req.onerror = () => reject(req.error ?? new Error('Failed to open IndexedDB'));
  });

  return dbPromise;
}
```

### 2.2 Promise-wrapped CRUD helpers

Wrap a single IDB request and a whole transaction so callers write linear `async/await` code.

```ts
// src/app/infrastructure/db.ts (continued)
function wrap<T>(req: IDBRequest<T>): Promise<T> {
  return new Promise((resolve, reject) => {
    req.onsuccess = () => resolve(req.result);
    req.onerror = () => reject(req.error);
  });
}

async function tx<T>(
  store: string,
  mode: IDBTransactionMode,
  run: (s: IDBObjectStore) => IDBRequest<T>
): Promise<T> {
  const db = await openDB();
  return new Promise<T>((resolve, reject) => {
    const transaction = db.transaction(store, mode);
    const request = run(transaction.objectStore(store));
    transaction.oncomplete = () => resolve(request.result);
    transaction.onerror = () => reject(transaction.error);
    transaction.onabort = () => reject(transaction.error);
  });
}

export const get = <T>(store: string, key: IDBValidKey) =>
  tx<T | undefined>(store, 'readonly', (s) => s.get(key) as IDBRequest<T | undefined>);

export const getAll = <T>(store: string) =>
  tx<T[]>(store, 'readonly', (s) => s.getAll() as IDBRequest<T[]>);

export const put = <T>(store: string, value: T) =>
  tx<IDBValidKey>(store, 'readwrite', (s) => s.put(value as any));

export const del = (store: string, key: IDBValidKey) =>
  tx<undefined>(store, 'readwrite', (s) => s.delete(key) as IDBRequest<undefined>);
```

> **Do:** resolve on `transaction.oncomplete`, not on the request's `onsuccess` — the write is only durable once the transaction commits.
> **Don't:** hold an IndexedDB transaction open across an `await` to non-IDB work; transactions auto-close when the microtask queue drains and you'll get `TransactionInactiveError`.

---

## 3. Lazy Loading: Metadata vs Heavy Payloads

A snippet library can grow large, and **datasets can be megabytes each** (CSV text, parsed TopoJSON). Loading every dataset payload at startup just to render a list of names is wasteful and slow. The rule:

> **Store record metadata separately from large payloads. Load heavy data on demand. Treat `null` as "exists but not loaded yet" — distinct from absent.**

For Astrolabe this maps cleanly onto the two stores:

- **`snippets`** — snippet records are small (a spec is JSON text). They load eagerly as a set when the library opens.
- **`datasets`** — the `data` payload is the heavy part. The list view needs only the derived summary fields (`name`, `format`, `source`, `rowCount`, `columnCount`, `columns`, `size`, timestamps). Load `data` only when a snippet that references the dataset is actually previewed.

There are two ways to implement the split; pick per store:

1. **Two object stores** (`datasets` for metadata, `dataset-payloads` keyed by the same id for `data`) — strongest separation; a `getAll` on metadata never touches payload bytes.
2. **One store, lazy field** — keep `data` in the record but set it to `null` on the bulk list load and fetch it per-id on demand.

Astrolabe uses the **lazy-field** approach for datasets (one store, simpler), with `data === null` signalling "summary loaded, payload not yet."

```ts
// src/app/infrastructure/dataset-store.ts
import { get, getAll, put } from './db';
import { migrateDataset, type Dataset } from './dataset-migrations';

/** List view: returns every dataset's summary, payload nulled out. */
export async function loadDatasetSummaries(): Promise<Dataset[]> {
  const records = await getAll<Dataset>('datasets');
  return records.map((r) => ({ ...migrateDataset(r), data: null }));
}

/** Detail/preview: load (or return cached) full payload for one dataset. */
export async function ensureDatasetData(dataset: Dataset): Promise<Dataset['data']> {
  if (dataset.data !== null && dataset.data !== undefined) return dataset.data; // already loaded
  const record = await get<Dataset>('datasets', dataset.id);
  dataset.data = record?.data ?? null;
  return dataset.data;
}
```

> **Do:** use `null` for "not loaded" and a real value (including `''` or `[]`) for "loaded but empty." The distinction prevents a re-fetch loop.
> **Don't:** overwrite a stored payload with `null` on save. When persisting a record whose `data` is `null` (never loaded into memory), skip writing the payload field and leave the stored bytes intact — otherwise a list-load-then-save round-trip silently destroys data.

---

## 4. Per-Record Schema Versioning & Read-Time Migration

The IndexedDB **database version** (§2.1) governs *store layout*. A separate **per-record `version` field** governs the *shape of an individual record*. Both Snippet and Dataset records carry `version` (and `created` / `modified` timestamps). This lets record shapes evolve without forcing an `onupgradeneeded` database bump for every field rename.

Migrations are applied **on read** — when a record comes out of the store, run it through a migration function that upgrades it to the current shape before the app sees it. New writes always store the current version.

```ts
// src/app/infrastructure/snippet-migrations.ts
export const CURRENT_SNIPPET_VERSION = 2;

export function migrateSnippet(raw: any): Snippet {
  let r = { ...raw };
  const v = r.version ?? 1; // records written before versioning existed are v1

  if (v < 2) {
    // Example: a v1 snippet had a single `spec`; v2 splits draft from published.
    r.draftSpec = r.draftSpec ?? r.spec;
    r.tags = r.tags ?? [];
    r.datasetRefs = r.datasetRefs ?? [];
  }
  // if (v < 3) { ... }

  r.version = CURRENT_SNIPPET_VERSION;
  return r as Snippet;
}
```

```ts
// src/app/infrastructure/snippet-store.ts
export async function loadSnippets(): Promise<Snippet[]> {
  const records = await getAll<any>('snippets');
  return records.map(migrateSnippet); // upgrade every record at the boundary
}

export async function saveSnippet(s: Snippet): Promise<void> {
  await put('snippets', { ...s, version: CURRENT_SNIPPET_VERSION, modified: new Date().toISOString() });
}
```

### Rationale

- **Read-time migration is forgiving.** Old records sitting untouched in the store keep working; they upgrade lazily the next time they're loaded and re-saved. There is no big-bang migration step that can fail halfway.
- **Tolerate unknown fields.** A migration normalizes *missing/old* fields but must not strip fields it doesn't recognize — a record written by a *newer* build that downgraded must round-trip without data loss. Spread the original (`{ ...raw }`) and only fill in what's missing.
- **One function, well tested.** Each migration step is a pure function over a plain object — trivial to unit-test with fixture records from each historical version.

> **Do:** default `version` to the earliest shape (`1`) when the field is absent.
> **Don't:** branch on the presence of individual fields scattered through the app to detect "old data." Centralize that knowledge in the migration function.

---

## 5. localStorage Preferences (Settings & App/UI Prefs)

Small, frequently-read structured records live in `localStorage`, not IndexedDB: **UserSettings** (one record) and **app/UI preferences** (snippet sort, panel layout). Why split them from `UserSettings`? UI prefs change often (drag a panel divider, toggle a sort) and shouldn't force a rewrite of the whole settings blob on every interaction.

The pattern is **load-with-fallback, write-through on change.**

- **Load-with-fallback:** merge the parsed stored object over a complete `DEFAULTS` constant. Missing keys (a setting added in a later build) and malformed JSON silently fall back to defaults — the app always gets a fully-populated object and never `undefined`-crashes on a new field.
- **Write-through:** every update reads current, applies the change, and writes the whole record back immediately. No dirty-tracking, no flush step.
- **Environment-guarded:** `localStorage` is absent or throws in some test/SSR contexts; guard access and degrade to defaults rather than throwing.

```ts
// src/app/infrastructure/settings-store.ts
const KEY = 'astrolabe:settings';

export const CURRENT_SETTINGS_VERSION = 1;

export interface UserSettings {
  version: number;
  editor: { fontSize: number; theme: string; minimap: boolean; wordWrap: 'on' | 'off';
            lineNumbers: 'on' | 'off'; tabSize: number };
  performance: { renderDebounce: number };
  ui: { theme: 'light' | 'dark'; previewFitMode: 'default' | 'width' | 'height' | 'full' };
  formatting: { dateFormat: 'smart' | 'iso' | 'custom'; customDateFormat: string };
}

// Defaults must match the authoritative spec §07 table exactly — that is the
// contract; this is just where it's encoded.
const DEFAULTS: UserSettings = {
  version: CURRENT_SETTINGS_VERSION,
  editor: { fontSize: 12, theme: 'auto', minimap: false, wordWrap: 'on',
            lineNumbers: 'on', tabSize: 2 },
  performance: { renderDebounce: 1500 },
  ui: { theme: 'light', previewFitMode: 'default' },
  formatting: { dateFormat: 'smart', customDateFormat: '' },
};

// NOTE — editor.theme default is 'auto': the editor theme follows the app UI
// theme (light -> light editor theme, dark -> dark) via custom Monaco
// themes that match the app chrome, unless the user picks an explicit override.
// The explicit-override option set (custom themes; whether to include High
// Contrast or the stock Monaco themes) is still TBD — see spec §07's provisional
// editor-theme note. Resolve the `'auto'` sentinel to a concrete Monaco theme at
// editor-config time, keyed off the current UI theme.

function available(): boolean {
  try {
    return typeof localStorage !== 'undefined' && typeof localStorage.getItem === 'function';
  } catch {
    return false; // access itself can throw (e.g. blocked storage)
  }
}

export function loadSettings(): UserSettings {
  if (!available()) return structuredClone(DEFAULTS);
  try {
    const raw = localStorage.getItem(KEY);
    if (!raw) return structuredClone(DEFAULTS);
    const p = JSON.parse(raw);
    // Deep-merge each group over defaults so new keys fall back silently.
    return {
      version: CURRENT_SETTINGS_VERSION,
      editor: { ...DEFAULTS.editor, ...p.editor },
      performance: { ...DEFAULTS.performance, ...p.performance },
      ui: { ...DEFAULTS.ui, ...p.ui },
      formatting: { ...DEFAULTS.formatting, ...p.formatting },
    };
  } catch (err) {
    console.warn('[settings] failed to load, using defaults', err);
    return structuredClone(DEFAULTS);
  }
}

export function saveSettings(s: UserSettings): void {
  if (!available()) return;
  try {
    localStorage.setItem(KEY, JSON.stringify({ ...s, version: CURRENT_SETTINGS_VERSION }));
  } catch (err) {
    console.warn('[settings] failed to save', err);
  }
}
```

App/UI prefs follow the identical pattern under their own keys, e.g.:

```ts
// src/app/infrastructure/prefs-store.ts
const SORT_KEY = 'astrolabe:snippet-sort';
const LAYOUT_KEY = 'astrolabe:panel-layout';

const SORT_DEFAULTS = { sortBy: 'modified' as const, sortOrder: 'desc' as const };
// loadSort()/saveSort() and loadLayout()/saveLayout() mirror §5's guard+fallback shape.
```

> **Do:** keep a single complete `DEFAULTS` object as the source of truth and merge over it.
> **Don't:** read individual keys with bespoke `?? fallback` at each call site; one stale default and the shapes drift.

> **Testing:** exercise localStorage adapters against an **injected stub** (`vi.stubGlobal('localStorage', …)`), not the ambient global. Under Node + happy-dom a non-functional Node `localStorage` global shadows happy-dom's, so relying on the ambient one fails with `localStorage.clear is not a function`. Applies to every prefs/settings adapter test (settings-store today; dataset-payload/prefs stores later).

---

## 6. Storage Tiers, Budgets & Quota Monitoring

Astrolabe has three tiers with different capacities and risk profiles:

| Tier | Backing | Holds | Budget & behavior |
|------|---------|-------|-------------------|
| **Snippet store** | IndexedDB `snippets` | All snippet records | Practical budget ~**5 MB**. A storage monitor estimates usage and surfaces a warning as it fills. Snippets are user-authored and irreplaceable, so we fail **loudly**. |
| **Dataset store** | IndexedDB `datasets` | All dataset payloads | Separate, **high-capacity**; suited to large payloads. Lazily loaded (§3). |
| **Settings & prefs** | localStorage | `UserSettings` + app/UI prefs (§5) | Small; effectively unbounded for this use. |

Splitting snippets and datasets into separate stores means a few large datasets can't crowd out the snippet budget, and the snippet monitor can report a meaningful "how full is my library" number without summing dataset bytes.

### Estimating usage

Use the Storage Manager API where available, with a manual byte-sum fallback for the snippet tier so the ~5 MB budget is always reportable.

```ts
// src/app/infrastructure/storage-monitor.ts
export interface StorageReport {
  snippetBytes: number;   // estimated bytes used by the snippet tier
  snippetBudget: number;  // 5 MB practical budget
  ratio: number;          // snippetBytes / snippetBudget, clamped to >= 0
  warn: boolean;          // ratio crossed the warning threshold
}

const SNIPPET_BUDGET = 5 * 1024 * 1024;
const WARN_AT = 0.8;

export async function reportSnippetUsage(snippets: Snippet[]): Promise<StorageReport> {
  // Cheap, deterministic estimate: serialize the records we hold.
  const snippetBytes = snippets.reduce(
    (n, s) => n + new Blob([JSON.stringify(s)]).size,
    0
  );
  const ratio = snippetBytes / SNIPPET_BUDGET;
  const report: StorageReport = {
    snippetBytes,
    snippetBudget: SNIPPET_BUDGET,
    ratio,
    warn: ratio >= WARN_AT,
  };
  if (report.warn) {
    console.warn(
      `[storage] snippet tier ${(ratio * 100).toFixed(0)}% of ${SNIPPET_BUDGET} bytes`
    );
  }
  return report;
}
```

### Fail loudly, never silently lose data

When a write would exceed quota, IndexedDB rejects with a `QuotaExceededError`. The adapter must **propagate** this so the UI can tell the user to export and prune — it must never swallow the error and pretend the save succeeded.

```ts
export async function saveSnippet(s: Snippet): Promise<void> {
  try {
    await put('snippets', { ...s, version: CURRENT_SNIPPET_VERSION });
  } catch (err) {
    if (err instanceof DOMException && err.name === 'QuotaExceededError') {
      // Surface to the user via the store; do NOT silently drop the write.
      throw new StorageQuotaError('Snippet storage is full. Export and remove snippets to free space.');
    }
    throw err;
  }
}
```

> **Do:** surface quota warnings *before* the budget is hit (the 80% threshold) and hard errors loudly when a write fails.
> **Don't:** wrap a save in a bare `try/catch {}` that logs and returns — that turns "your work wasn't saved" into a silent data-loss bug. The only thing the adapter may safely swallow is a *read* failure, where falling back to defaults/empty is the correct behavior.

---

## 7. Checklist for Adding a New Persisted Entity

1. Define the record type with `id`, `created`, `modified`, and a `version` field.
2. Decide the tier: small + critical → IndexedDB store with a monitored budget; large payload → separate high-capacity store with lazy loading (§3); tiny + frequently changing → localStorage pref (§5).
3. Add the object store in `openDB`'s `onupgradeneeded`, guarded by `contains(...)`; bump `DB_VERSION` only if you changed store *layout*.
4. Add a `migrate<Entity>()` function and call it on every read.
5. Expose typed `load*/save*/ensure*` functions from one infrastructure module — and from *only* there.
6. If the tier has a budget, hook it into the storage monitor and propagate `QuotaExceededError`.
7. Test the adapter against `fake-indexeddb` / a localStorage stub; test the migration with fixtures from each historical version.