oleh/astrolabe
1
0
Fork 0
mirror of https://github.com/olehomelchenko/astrolabe.git synced 2026-08-08 02:02:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Initial scaffold: spec, architecture playbook, and M0 skeleton

Browse Source
This commit is contained in:
Oleh Omelchenko
2026-06-04 22:14:33 +03:00
commit 056644450c
51 changed files with 13754 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/06-type-inference.md
+346
View File
@@ -0,0 +1,346 @@
# Type Inference & Data Profiling
How Astrolabe looks at a tabular dataset and figures out, for each column, what
kind of data it holds — `number`, `string`, `date`, or `boolean` — and how it
rolls those facts up into the **profile** stored on a dataset record.
This is pure, portable logic. It lives in `src/core/`, touches no browser APIs
and no React, takes plain values in and returns plain data out, and is covered
by Vitest unit tests. Anything that needs a profile (the create form, the edit
flow, the detail panel) calls into this module; nothing here reaches back out.
---
## 1. Why infer types at all
A dataset is just rows of values. The UI wants to *describe* it without
re-parsing the payload every time: how many rows and columns, what the columns
are called, and roughly what each column contains. The inferred type drives the
small type indicator next to each column name in the dataset detail panel and
the meta line in the list. It is a **display hint**, not a contract — nothing
downstream coerces values based on it, and Vega-Lite does its own type handling
at render time. Because it is only a hint, a wrong guess is cheap, and the rules
below favour being simple and predictable over being clever.
We support exactly **four** inferred types:
| Type | Meaning |
| --------- | ---------------------------------------------------- |
| `number` | Every non-empty value is numeric. |
| `boolean` | Every non-empty value is `true`/`false` (any case). |
| `date` | Every non-empty value parses as a date. |
| `string` | The fallback — anything that isn't one of the above. |
There is deliberately no integer/float split, no datetime-vs-date distinction,
and no JSON type. Those distinctions add branches and edge cases without
changing what the user sees. Keep it at four.
---
## 2. Inferring one column
Given the values of a single column, decide its type.
### The shape of the algorithm
1. **Drop the empties.** Filter out `null`, `undefined`, and empty/whitespace-only
strings before doing anything. Empty cells carry no type signal — a column of
numbers with a few blanks is still a number column.
2. **All-empty → `string`.** If nothing survives the filter (the column is
entirely empty, or there are zero rows), default to `string`. There is no
evidence for any other type.
3. **Run the type checks in precedence order.** For each candidate type, ask:
*does **every** surviving value match this type?* The first candidate for
which the answer is yes wins. This is the **"all values match → that type,
else fall back"** rule: one stray value that doesn't fit knocks the column
down to the next candidate, and ultimately to `string`.
### Precedence order matters
The order of the checks is not arbitrary — it exists because the value-sets
overlap, and we want the most specific interpretation that fits.
1. **boolean** first. The strings `"true"`/`"false"` are not numbers and not
dates, so booleans never collide with the other checks — but putting them
first keeps a `0`/`1`-free true/false column out of `string`. (We do *not*
treat `0`/`1` as boolean; that's a number column.)
2. **number** second. `Number("2024")` is a perfectly good number, so a column
of bare years would read as `number` — which is the honest answer. Numbers
are checked before dates so that plain numeric columns never get
mis-classified as dates by an over-eager date parser.
3. **date** third. Date parsing is the loosest, most permissive check, so it
goes last among the positive checks. By the time we reach it we already know
the column isn't all-boolean and isn't all-numeric.
4. **string** is the fallback when no positive check matches every value.
> Mnemonic: **boolean → number → date → string**, narrowest evidence to widest.
### What counts as each type
- **numeric**: trim the string form; reject empty; `Number(trimmed)` must be
finite and not `NaN`. (Native `number` values pass directly.) Reject blank and
whitespace so `Number("") === 0` doesn't sneak through.
- **boolean**: native `boolean` values pass; otherwise the trimmed,
lower-cased string must be exactly `"true"` or `"false"`.
- **date**: guard *before* parsing. Require the trimmed value to look
date-shaped (a leading `YYYY-MM-DD` or `YYYY/MM/DD`, or `M/D/YYYY`) **and**
then confirm `Date.parse` returns a finite timestamp. The shape guard is
essential: `Date.parse` will happily accept `"42"` or `"March"` on some
engines, which would swallow number and string columns. Never rely on
`Date.parse` alone.
### Sketch
```ts
// src/core/type-inference.ts
export type ColumnType = 'number' | 'string' | 'date' | 'boolean';
const isEmpty = (v: unknown): boolean =>
v === null || v === undefined || (typeof v === 'string' && v.trim() === '');
const isNumeric = (v: unknown): boolean => {
if (typeof v === 'number') return Number.isFinite(v);
if (typeof v !== 'string') return false;
const t = v.trim();
if (t === '') return false;
const n = Number(t);
return !Number.isNaN(n) && Number.isFinite(n);
};
const isBoolean = (v: unknown): boolean => {
if (typeof v === 'boolean') return true;
if (typeof v !== 'string') return false;
const t = v.trim().toLowerCase();
return t === 'true' || t === 'false';
};
// Shape guard first, then confirm it actually parses.
const DATE_SHAPE = /^\d{4}[-/]\d{2}[-/]\d{2}|^\d{1,2}\/\d{1,2}\/\d{4}/;
const isDate = (v: unknown): boolean => {
if (typeof v !== 'string') return false;
const t = v.trim();
return DATE_SHAPE.test(t) && !Number.isNaN(Date.parse(t));
};
/**
* Infer one of four column types from a sample of column values.
* Empty cells are ignored; an all-empty column is `string`.
* Precedence: boolean → number → date → string.
*/
export function inferColumnType(values: readonly unknown[]): ColumnType {
const present = values.filter((v) => !isEmpty(v));
if (present.length === 0) return 'string';
if (present.every(isBoolean)) return 'boolean';
if (present.every(isNumeric)) return 'number';
if (present.every(isDate)) return 'date';
return 'string';
}
```
### Robustness notes
- **Mixed columns** fall through to `string` automatically — the `every` check
fails on the first non-conforming value, so a column of mostly-numbers with
one label is `string`, which is the safe, honest answer.
- **Whitespace** is trimmed in every check, so `" 42 "` reads as numeric and
`" "` is treated as empty.
- **Empty columns** (all cells blank, or a zero-row dataset) return `string` by
the all-empty rule — never throw, never guess.
- **Large columns**: see §4. `inferColumnType` itself just consumes whatever
array it's handed; the caller decides whether to sample.
### Do / Don't
- **Do** ignore empty cells before classifying.
- **Do** keep the precedence boolean → number → date → string.
- **Do** guard date detection with a shape regex before trusting `Date.parse`.
- **Don't** classify a column unless *every* present value matches — one
outlier means `string`.
- **Don't** add more types (integer, float, datetime, json). Four, no more.
- **Don't** let `Number("")`, `Date.parse("42")`, or `0`/`1` leak into the wrong
bucket.
---
## 3. Profiling a dataset
A **profile** is the set of derived summary fields stored on a dataset record so
the UI can describe it without re-parsing the payload. Per the data model, a
profiled dataset carries:
| Field | Type | Meaning |
| ------------- | --------------------------------- | -------------------------------------- |
| `rowCount` | `number \| null` | Data rows, or `null` when N/A. |
| `columnCount` | `number \| null` | Columns, or `null` when N/A. |
| `columns` | `string[]` | Column names, in order. |
| `columnTypes` | `Array<{ name; type }>` | Per-column inferred type (see §2). |
| `size` | `number` | Approximate payload size in bytes. |
`null` row/column counts and an empty `columns`/`columnTypes` are how the UI
shows **"N/A"** — see §3.2.
### 3.1 What gets profiled
Profiling applies only to **tabular inline data**:
- **JSON** that is an array of objects.
- **CSV** (comma-separated, header row).
- **TSV** (tab-separated, header row).
Everything else is **not profiled**:
- **URL datasets** — the library holds only the link, not the data, so there is
nothing to scan. Counts are `null` / N/A.
- **Non-tabular data** — a single JSON object, TopoJSON, or anything we can't
read as rows-of-columns. Counts are `null` / N/A.
For the not-profiled cases, `size` is still computed (it's just the byte length
of the stored payload), but `rowCount` and `columnCount` are `null`, and
`columns`/`columnTypes` are empty.
### 3.2 The algorithm
1. **Compute `size`** from the raw payload regardless of whether it's tabular —
byte length of the text (CSV/TSV) or of the JSON-serialized value.
2. **Decide if it's tabular.** Map `(format, parsed shape)` to a row set:
- `csv` / `tsv` → parse into rows-of-objects using the matching delimiter.
- `json` that is a non-empty **array of objects** → use it directly.
- anything else (`topojson`, a lone JSON object, an empty array) → not
tabular; return the N/A profile (`rowCount: null`, `columnCount: null`,
`columns: []`, `columnTypes: []`, plus `size`).
3. **Derive columns** from the union of keys across the rows (or the CSV/TSV
header), preserving first-seen order.
4. **Infer each column's type** by collecting that column's values across the
rows and calling `inferColumnType` (§2), sampling per §4.
5. **Assemble** `rowCount`, `columnCount`, `columns`, `columnTypes`, `size`.
### Sketch
```ts
// src/core/profile.ts
import { inferColumnType, type ColumnType } from './type-inference';
export interface DatasetProfile {
rowCount: number | null;
columnCount: number | null;
columns: string[];
columnTypes: Array<{ name: string; type: ColumnType }>;
size: number;
}
const NA = (size: number): DatasetProfile => ({
rowCount: null,
columnCount: null,
columns: [],
columnTypes: [],
size,
});
/** Profile a dataset payload. `rows` is the tabular form (CSV/TSV/JSON-array)
* already parsed to rows-of-objects, or null for non-tabular / URL data. */
export function profileData(
rows: ReadonlyArray<Record<string, unknown>> | null,
size: number,
): DatasetProfile {
if (!rows || rows.length === 0) return NA(size);
// Column order = first-seen order across all rows.
const columns: string[] = [];
const seen = new Set<string>();
for (const row of rows) {
for (const key of Object.keys(row)) {
if (!seen.has(key)) {
seen.add(key);
columns.push(key);
}
}
}
if (columns.length === 0) return NA(size);
const sample = sampleRows(rows);
const columnTypes = columns.map((name) => ({
name,
type: inferColumnType(sample.map((r) => r[name])),
}));
return {
rowCount: rows.length,
columnCount: columns.length,
columns,
columnTypes,
size,
};
}
```
Parsing CSV/TSV text and detecting the payload shape happen *upstream* of
`profileData`; this function takes already-parsed rows so it stays pure and
trivially testable. The caller passes `null` for URL and non-tabular datasets.
---
## 4. Sampling vs. full scan
`rowCount`/`columnCount`/`size` always reflect the **whole** dataset — they're
cheap (a length and a byte count). Only **type inference** has a per-value cost,
and it's the one place a huge dataset could hurt.
So: infer types from a **bounded sample** of rows, not the full column. A fixed
cap (e.g. the first ~200 rows) keeps profiling fast and predictable on large
pasted datasets while still being more than enough signal to classify a column.
```ts
const SAMPLE_SIZE = 200;
const sampleRows = <T>(rows: ReadonlyArray<T>): ReadonlyArray<T> =>
rows.length <= SAMPLE_SIZE ? rows : rows.slice(0, SAMPLE_SIZE);
```
Trade-off to be aware of: a column that is numeric for its first 200 rows but
turns to text later will be mis-typed as `number`. That's an accepted cost — the
type is a display hint, the mistake is cheap, and the speed win on large
datasets is worth it. Sampling the head (rather than randomly) keeps results
**deterministic**, which matters for tests and for not surprising the user when
the same paste profiles the same way twice.
### Do / Don't
- **Do** count rows/columns and size over the full payload.
- **Do** cap type-inference sampling at a fixed head slice for determinism.
- **Don't** randomly sample — non-deterministic profiles break tests and confuse
users.
- **Don't** scan every value of a million-row paste to guess a type.
---
## 5. Testing
Both functions are pure, so tests are plain input/output assertions in Vitest —
no mocks, no DOM, no fixtures beyond literal arrays.
Cover at least:
- **`inferColumnType`**: each type detected from a clean column; mixed columns
fall to `string`; empty/whitespace cells ignored; all-empty and zero-length →
`string`; precedence (a `["true","false"]` column is `boolean` not `string`; a
`["2024","2025"]` column is `number` not `date`); date shape guard rejects
`"42"` and `"hello"` even though one engine's `Date.parse` might accept them;
`0`/`1` are `number`, not `boolean`.
- **`profileData`**: a JSON-array dataset profiles fully; `null` rows (URL) and
an empty array (non-tabular) return the N/A profile but still carry `size`;
column order follows first-seen key order across ragged rows; sampling cap is
respected (a dataset longer than the cap still profiles, using only the head).
---
## Summary
- Four types only: **boolean → number → date → string**, checked in that order.
- **All present values must match** a type or the column falls through; empty
cells are ignored; an all-empty column is `string`.
- Guard date detection with a shape regex before trusting `Date.parse`.
- A **profile** carries `rowCount`, `columnCount`, `columns`, `columnTypes`,
`size`; URL and non-tabular datasets get a **null/N-A** profile (still sized).
- Counts and size scan the whole payload; **type inference samples the head** for
speed and determinism.
- All of it is **pure `src/core/` logic, unit-tested with Vitest**.