astrolabe/docs/architecture/06-type-inference.md

Type Inference & Data Profiling

How Astrolabe looks at a tabular dataset and figures out, for each column, what kind of data it holds — number, string, date, or boolean — and how it rolls those facts up into the profile stored on a dataset record.

This is pure, portable logic. It lives in src/core/, touches no browser APIs and no React, takes plain values in and returns plain data out, and is covered by Vitest unit tests. Anything that needs a profile (the create form, the edit flow, the detail panel) calls into this module; nothing here reaches back out.

---

1. Why infer types at all

A dataset is just rows of values. The UI wants to describe it without re-parsing the payload every time: how many rows and columns, what the columns are called, and roughly what each column contains. The inferred type drives the small type indicator next to each column name in the dataset detail panel and the meta line in the list. It is a display hint, not a contract — nothing downstream coerces values based on it, and Vega-Lite does its own type handling at render time. Because it is only a hint, a wrong guess is cheap, and the rules below favour being simple and predictable over being clever.

We support exactly four inferred types:

Type	Meaning
number	Every non-empty value is numeric.
boolean	Every non-empty value is true/false (any case).
date	Every non-empty value parses as a date.
string	The fallback — anything that isn't one of the above.

There is deliberately no integer/float split, no datetime-vs-date distinction, and no JSON type. Those distinctions add branches and edge cases without changing what the user sees. Keep it at four.

---

2. Inferring one column

Given the values of a single column, decide its type.

The shape of the algorithm

1. Drop the empties. Filter out null, undefined, and empty/whitespace-only strings before doing anything. Empty cells carry no type signal — a column of numbers with a few blanks is still a number column.
2. All-empty → string. If nothing survives the filter (the column is entirely empty, or there are zero rows), default to string. There is no evidence for any other type.
3. Run the type checks in precedence order. For each candidate type, ask: does every surviving value match this type? The first candidate for which the answer is yes wins. This is the "all values match → that type, else fall back" rule: one stray value that doesn't fit knocks the column down to the next candidate, and ultimately to string.

Precedence order matters

The order of the checks is not arbitrary — it exists because the value-sets overlap, and we want the most specific interpretation that fits.

1. boolean first. The strings "true"/"false" are not numbers and not dates, so booleans never collide with the other checks — but putting them first keeps a 0/1-free true/false column out of string. (We do not treat 0/1 as boolean; that's a number column.)
2. number second. Number("2024") is a perfectly good number, so a column of bare years would read as number — which is the honest answer. Numbers are checked before dates so that plain numeric columns never get mis-classified as dates by an over-eager date parser.
3. date third. Date parsing is the loosest, most permissive check, so it goes last among the positive checks. By the time we reach it we already know the column isn't all-boolean and isn't all-numeric.
4. string is the fallback when no positive check matches every value.

Mnemonic: boolean → number → date → string, narrowest evidence to widest.

What counts as each type

• numeric: trim the string form; reject empty; Number(trimmed) must be finite and not NaN. (Native number values pass directly.) Reject blank and whitespace so Number("") === 0 doesn't sneak through.
• boolean: native boolean values pass; otherwise the trimmed, lower-cased string must be exactly "true" or "false".
• date: guard before parsing. Require the trimmed value to look date-shaped (a leading YYYY-MM-DD or YYYY/MM/DD, or M/D/YYYY) and then confirm Date.parse returns a finite timestamp. The shape guard is essential: Date.parse will happily accept "42" or "March" on some engines, which would swallow number and string columns. Never rely on Date.parse alone.

Sketch

// src/core/type-inference.ts
export type ColumnType = 'number' | 'string' | 'date' | 'boolean';

const isEmpty = (v: unknown): boolean =>
  v === null || v === undefined || (typeof v === 'string' && v.trim() === '');

const isNumeric = (v: unknown): boolean => {
  if (typeof v === 'number') return Number.isFinite(v);
  if (typeof v !== 'string') return false;
  const t = v.trim();
  if (t === '') return false;
  const n = Number(t);
  return !Number.isNaN(n) && Number.isFinite(n);
};

const isBoolean = (v: unknown): boolean => {
  if (typeof v === 'boolean') return true;
  if (typeof v !== 'string') return false;
  const t = v.trim().toLowerCase();
  return t === 'true' || t === 'false';
};

// Shape guard first, then confirm it actually parses.
const DATE_SHAPE = /^\d{4}[-/]\d{2}[-/]\d{2}|^\d{1,2}\/\d{1,2}\/\d{4}/;
const isDate = (v: unknown): boolean => {
  if (typeof v !== 'string') return false;
  const t = v.trim();
  return DATE_SHAPE.test(t) && !Number.isNaN(Date.parse(t));
};

/**
 * Infer one of four column types from a sample of column values.
 * Empty cells are ignored; an all-empty column is `string`.
 * Precedence: boolean → number → date → string.
 */
export function inferColumnType(values: readonly unknown[]): ColumnType {
  const present = values.filter((v) => !isEmpty(v));
  if (present.length === 0) return 'string';

  if (present.every(isBoolean)) return 'boolean';
  if (present.every(isNumeric)) return 'number';
  if (present.every(isDate)) return 'date';
  return 'string';
}

Robustness notes

• Mixed columns fall through to string automatically — the every check fails on the first non-conforming value, so a column of mostly-numbers with one label is string, which is the safe, honest answer.
• Whitespace is trimmed in every check, so " 42 " reads as numeric and " " is treated as empty.
• Empty columns (all cells blank, or a zero-row dataset) return string by the all-empty rule — never throw, never guess.
• Large columns: see §4. inferColumnType itself just consumes whatever array it's handed; the caller decides whether to sample.

Do / Don't

• Do ignore empty cells before classifying.
• Do keep the precedence boolean → number → date → string.
• Do guard date detection with a shape regex before trusting Date.parse.
• Don't classify a column unless every present value matches — one outlier means string.
• Don't add more types (integer, float, datetime, json). Four, no more.
• Don't let Number(""), Date.parse("42"), or 0/1 leak into the wrong bucket.

---

3. Profiling a dataset

A profile is the set of derived summary fields stored on a dataset record so the UI can describe it without re-parsing the payload. Per the data model, a profiled dataset carries:

Field	Type	Meaning
rowCount	number | null	Data rows, or null when N/A.
columnCount	number | null	Columns, or null when N/A.
columns	string[]	Column names, in order.
columnTypes	Array<{ name; type }>	Per-column inferred type (see §2).
columnStats	Array<{ name; distinct; distinctCapped; numericExtent }>	Per-column cardinality + numeric range (see §3.3). Empty when N/A.
size	number	Approximate payload size in bytes.

null row/column counts and an empty columns/columnTypes/columnStats are how the UI shows "N/A" — see §3.2.

3.1 What gets profiled

Profiling applies to any tabular payload, whether pasted inline or fetched from a URL (the snapshot model stores a URL dataset's data locally, so it profiles through the same path as inline data — snapshotFromText shapes the fetched body, then computeDatasetProfile runs):

• JSON that is an array of objects.
• CSV (comma-separated, header row).
• TSV (tab-separated, header row).

Everything else is not profiled:

• Unfetched URL datasets — a URL reference with no snapshot yet (e.g. one migrated from an older record), so there is nothing to scan. Counts are null / N/A until it is refreshed.
• Non-tabular data — a single JSON object, TopoJSON, or anything we can't read as rows-of-columns. Counts are null / N/A.

For the not-profiled cases, size is still computed (it's just the byte length of the stored payload), but rowCount and columnCount are null, and columns/columnTypes are empty.

3.2 The algorithm

1. Compute size from the raw payload regardless of whether it's tabular — byte length of the text (CSV/TSV) or of the JSON-serialized value.
2. Decide if it's tabular. Map (format, parsed shape) to a row set:
   • csv / tsv → parse into rows-of-objects using the matching delimiter.
   • json that is a non-empty array of objects → use it directly.
   • anything else (topojson, a lone JSON object, an empty array) → not tabular; return the N/A profile (rowCount: null, columnCount: null, columns: [], columnTypes: [], columnStats: [], plus size).
3. Derive columns from the union of keys across the rows (or the CSV/TSV header), preserving first-seen order.
4. Infer each column's type and stats by collecting that column's values across the rows (sampling per §4) and calling inferColumnType (§2) plus deriving its cardinality + numeric extent (§3.3) — one pass per column over the same sample.
5. Assemble rowCount, columnCount, columns, columnTypes, columnStats, size.

Sketch

// src/core/profile.ts
import { inferColumnType, type ColumnType } from './type-inference';

export interface DatasetProfile {
  rowCount: number | null;
  columnCount: number | null;
  columns: string[];
  columnTypes: Array<{ name: string; type: ColumnType }>;
  columnStats: Array<{
    name: string;
    distinct: number; // capped at DISTINCT_CAP
    distinctCapped: boolean; // true ⇒ real cardinality ≥ DISTINCT_CAP
    numericExtent: { min: number; max: number } | null; // numeric columns only
  }>;
  size: number;
}

const NA = (size: number): DatasetProfile => ({
  rowCount: null,
  columnCount: null,
  columns: [],
  columnTypes: [],
  columnStats: [],
  size,
});

/** Profile a dataset payload. `rows` is the tabular form (CSV/TSV/JSON-array)
 *  already parsed to rows-of-objects, or null for non-tabular / URL data. */
export function profileData(
  rows: ReadonlyArray<Record<string, unknown>> | null,
  size: number,
): DatasetProfile {
  if (!rows || rows.length === 0) return NA(size);

  // Column order = first-seen order across all rows.
  const columns: string[] = [];
  const seen = new Set<string>();
  for (const row of rows) {
    for (const key of Object.keys(row)) {
      if (!seen.has(key)) {
        seen.add(key);
        columns.push(key);
      }
    }
  }
  if (columns.length === 0) return NA(size);

  const sample = sampleRows(rows);
  // One pass per column over the sample: type + stats (cardinality, numeric extent).
  const columnTypes = [];
  const columnStats = [];
  for (const name of columns) {
    const values = sample.map((r) => r[name]);
    const type = inferColumnType(values);
    columnTypes.push({ name, type });
    columnStats.push(columnStatsFor(name, values, type)); // see §3.3
  }

  return {
    rowCount: rows.length,
    columnCount: columns.length,
    columns,
    columnTypes,
    columnStats,
    size,
  };
}

Parsing CSV/TSV text and detecting the payload shape happen upstream of profileData; this function takes already-parsed rows so it stays pure and trivially testable. The caller passes null for URL and non-tabular datasets.

3.3 Column stats: cardinality + numeric extent

Alongside the display type, each column carries the two data-shape signals the Chart Builder needs for its data-aware Tier-B hints (chart-builder.ts builderWarnings) and for its default pre-population (smartDefaultEncodings prefers a low-cardinality category over a high-cardinality key, so the builder never opens on a degenerate chart; spec §06):

• distinct — the count of distinct non-empty values in the sample, counted only up to DISTINCT_CAP (50). Past the cap the exact number stops mattering — a legend or category axis with 50 entries is already unreadable — so counting stops and distinctCapped flags that the real cardinality is at least the cap. Powers the crowded-legend (discrete colour) and crowded-category warnings.
• numericExtent — { min, max } over the numeric values, for number columns only (null otherwise), using the same numeric test as inferColumnType so the two never disagree. Powers the negative-value Size guard (size encodes magnitude → negatives mislead; Draco hard.lp:56).

Both are derived in the same sample pass as type inference, so they are nearly free, and both are estimates (sampled from the head, like the type) — fine for soft advisories, not guarantees. URL / non-tabular data and datasets stored before this field carry empty columnStats, and the dependent hints simply skip.

---

4. Sampling vs. full scan

rowCount/columnCount/size always reflect the whole dataset — they're cheap (a length and a byte count). Only the per-value work — type inference and the §3.3 column stats — has a per-row cost, and it's the one place a huge dataset could hurt.

So: derive types and stats from a bounded sample of rows, not the full column. A fixed cap (e.g. the first ~200 rows) keeps profiling fast and predictable on large pasted datasets while still being more than enough signal to classify a column and estimate its cardinality / range.

const SAMPLE_SIZE = 200;
const sampleRows = <T>(rows: ReadonlyArray<T>): ReadonlyArray<T> =>
  rows.length <= SAMPLE_SIZE ? rows : rows.slice(0, SAMPLE_SIZE);

Trade-off to be aware of: a column that is numeric for its first 200 rows but turns to text later will be mis-typed as number. That's an accepted cost — the type is a display hint, the mistake is cheap, and the speed win on large datasets is worth it. Sampling the head (rather than randomly) keeps results deterministic, which matters for tests and for not surprising the user when the same paste profiles the same way twice.

Do / Don't

• Do count rows/columns and size over the full payload.
• Do cap type-inference and column-stats sampling at a fixed head slice for determinism (so distinct / numericExtent are estimates, like the type).
• Don't randomly sample — non-deterministic profiles break tests and confuse users.
• Don't scan every value of a million-row paste to guess a type.

---

5. Testing

Both functions are pure, so tests are plain input/output assertions in Vitest — no mocks, no DOM, no fixtures beyond literal arrays.

Cover at least:

• inferColumnType: each type detected from a clean column; mixed columns fall to string; empty/whitespace cells ignored; all-empty and zero-length → string; precedence (a ["true","false"] column is boolean not string; a ["2024","2025"] column is number not date); date shape guard rejects "42" and "hello" even though one engine's Date.parse might accept them; 0/1 are number, not boolean.
• profileData: a JSON-array dataset profiles fully; null rows (URL) and an empty array (non-tabular) return the N/A profile but still carry size; column order follows first-seen key order across ragged rows; sampling cap is respected (a dataset longer than the cap still profiles, using only the head).

---

Summary

• Four types only: boolean → number → date → string, checked in that order.
• All present values must match a type or the column falls through; empty cells are ignored; an all-empty column is string.
• Guard date detection with a shape regex before trusting Date.parse.
• A profile carries rowCount, columnCount, columns, columnTypes, size; URL and non-tabular datasets get a null/N-A profile (still sized).
• Counts and size scan the whole payload; type inference samples the head for speed and determinism.
• All of it is pure src/core/ logic, unit-tested with Vitest.