# Type Inference & Data Profiling

How Astrolabe looks at a tabular dataset and figures out, for each column, what
kind of data it holds — `number`, `string`, `date`, or `boolean` — and how it
rolls those facts up into the **profile** stored on a dataset record.

This is pure, portable logic. It lives in `src/core/`, touches no browser APIs
and no React, takes plain values in and returns plain data out, and is covered
by Vitest unit tests. Anything that needs a profile (the create form, the edit
flow, the detail panel) calls into this module; nothing here reaches back out.

---

## 1. Why infer types at all

A dataset is just rows of values. The UI wants to _describe_ it without
re-parsing the payload every time: how many rows and columns, what the columns
are called, and roughly what each column contains. The inferred type drives the
small type indicator next to each column name in the dataset detail panel and
the meta line in the list. It is a **display hint**, not a contract — nothing
downstream coerces values based on it, and Vega-Lite does its own type handling
at render time. Because it is only a hint, a wrong guess is cheap, and the rules
below favour being simple and predictable over being clever.

We support exactly **four** inferred types:

| Type      | Meaning                                              |
| --------- | ---------------------------------------------------- |
| `number`  | Every non-empty value is numeric.                    |
| `boolean` | Every non-empty value is `true`/`false` (any case).  |
| `date`    | Every non-empty value parses as a date.              |
| `string`  | The fallback — anything that isn't one of the above. |

There is deliberately no integer/float split, no datetime-vs-date distinction,
and no JSON type. Those distinctions add branches and edge cases without
changing what the user sees. Keep it at four.

---

## 2. Inferring one column

Given the values of a single column, decide its type.

### The shape of the algorithm

1. **Drop the empties.** Filter out `null`, `undefined`, and empty/whitespace-only
   strings before doing anything. Empty cells carry no type signal — a column of
   numbers with a few blanks is still a number column.
2. **All-empty → `string`.** If nothing survives the filter (the column is
   entirely empty, or there are zero rows), default to `string`. There is no
   evidence for any other type.
3. **Run the type checks in precedence order.** For each candidate type, ask:
   _does **every** surviving value match this type?_ The first candidate for
   which the answer is yes wins. This is the **"all values match → that type,
   else fall back"** rule: one stray value that doesn't fit knocks the column
   down to the next candidate, and ultimately to `string`.

### Precedence order matters

The order of the checks is not arbitrary — it exists because the value-sets
overlap, and we want the most specific interpretation that fits.

1. **boolean** first. The strings `"true"`/`"false"` are not numbers and not
   dates, so booleans never collide with the other checks — but putting them
   first keeps a `0`/`1`-free true/false column out of `string`. (We do _not_
   treat `0`/`1` as boolean; that's a number column.)
2. **number** second. `Number("2024")` is a perfectly good number, so a column
   of bare years would read as `number` — which is the honest answer. Numbers
   are checked before dates so that plain numeric columns never get
   mis-classified as dates by an over-eager date parser.
3. **date** third. Date parsing is the loosest, most permissive check, so it
   goes last among the positive checks. By the time we reach it we already know
   the column isn't all-boolean and isn't all-numeric.
4. **string** is the fallback when no positive check matches every value.

> Mnemonic: **boolean → number → date → string**, narrowest evidence to widest.

### What counts as each type

- **numeric**: trim the string form; reject empty; `Number(trimmed)` must be
  finite and not `NaN`. (Native `number` values pass directly.) Reject blank and
  whitespace so `Number("") === 0` doesn't sneak through.
- **boolean**: native `boolean` values pass; otherwise the trimmed,
  lower-cased string must be exactly `"true"` or `"false"`.
- **date**: guard _before_ parsing. Require the trimmed value to look
  date-shaped (a leading `YYYY-MM-DD` or `YYYY/MM/DD`, or `M/D/YYYY`) **and**
  then confirm `Date.parse` returns a finite timestamp. The shape guard is
  essential: `Date.parse` will happily accept `"42"` or `"March"` on some
  engines, which would swallow number and string columns. Never rely on
  `Date.parse` alone.

### Sketch

```ts
// src/core/type-inference.ts
export type ColumnType = 'number' | 'string' | 'date' | 'boolean';

const isEmpty = (v: unknown): boolean =>
  v === null || v === undefined || (typeof v === 'string' && v.trim() === '');

const isNumeric = (v: unknown): boolean => {
  if (typeof v === 'number') return Number.isFinite(v);
  if (typeof v !== 'string') return false;
  const t = v.trim();
  if (t === '') return false;
  const n = Number(t);
  return !Number.isNaN(n) && Number.isFinite(n);
};

const isBoolean = (v: unknown): boolean => {
  if (typeof v === 'boolean') return true;
  if (typeof v !== 'string') return false;
  const t = v.trim().toLowerCase();
  return t === 'true' || t === 'false';
};

// Shape guard first, then confirm it actually parses.
const DATE_SHAPE = /^\d{4}[-/]\d{2}[-/]\d{2}|^\d{1,2}\/\d{1,2}\/\d{4}/;
const isDate = (v: unknown): boolean => {
  if (typeof v !== 'string') return false;
  const t = v.trim();
  return DATE_SHAPE.test(t) && !Number.isNaN(Date.parse(t));
};

/**
 * Infer one of four column types from a sample of column values.
 * Empty cells are ignored; an all-empty column is `string`.
 * Precedence: boolean → number → date → string.
 */
export function inferColumnType(values: readonly unknown[]): ColumnType {
  const present = values.filter((v) => !isEmpty(v));
  if (present.length === 0) return 'string';

  if (present.every(isBoolean)) return 'boolean';
  if (present.every(isNumeric)) return 'number';
  if (present.every(isDate)) return 'date';
  return 'string';
}
```

### Robustness notes

- **Mixed columns** fall through to `string` automatically — the `every` check
  fails on the first non-conforming value, so a column of mostly-numbers with
  one label is `string`, which is the safe, honest answer.
- **Whitespace** is trimmed in every check, so `" 42 "` reads as numeric and
  `"  "` is treated as empty.
- **Empty columns** (all cells blank, or a zero-row dataset) return `string` by
  the all-empty rule — never throw, never guess.
- **Large columns**: see §4. `inferColumnType` itself just consumes whatever
  array it's handed; the caller decides whether to sample.

### Do / Don't

- **Do** ignore empty cells before classifying.
- **Do** keep the precedence boolean → number → date → string.
- **Do** guard date detection with a shape regex before trusting `Date.parse`.
- **Don't** classify a column unless _every_ present value matches — one
  outlier means `string`.
- **Don't** add more types (integer, float, datetime, json). Four, no more.
- **Don't** let `Number("")`, `Date.parse("42")`, or `0`/`1` leak into the wrong
  bucket.

---

## 3. Profiling a dataset

A **profile** is the set of derived summary fields stored on a dataset record so
the UI can describe it without re-parsing the payload. Per the data model, a
profiled dataset carries:

| Field         | Type                                                       | Meaning                                                            |
| ------------- | ---------------------------------------------------------- | ------------------------------------------------------------------ |
| `rowCount`    | `number \| null`                                           | Data rows, or `null` when N/A.                                     |
| `columnCount` | `number \| null`                                           | Columns, or `null` when N/A.                                       |
| `columns`     | `string[]`                                                 | Column names, in order.                                            |
| `columnTypes` | `Array<{ name; type }>`                                    | Per-column inferred type (see §2).                                 |
| `columnStats` | `Array<{ name; distinct; distinctCapped; numericExtent }>` | Per-column cardinality + numeric range (see §3.3). Empty when N/A. |
| `size`        | `number`                                                   | Approximate payload size in bytes.                                 |

`null` row/column counts and an empty `columns`/`columnTypes`/`columnStats` are
how the UI shows **"N/A"** — see §3.2.

### 3.1 What gets profiled

Profiling applies to any **tabular payload**, whether pasted inline or fetched
from a URL (the snapshot model stores a URL dataset's data locally, so it profiles
through the same path as inline data — `snapshotFromText` shapes the fetched body,
then `computeDatasetProfile` runs):

- **JSON** that is an array of objects.
- **CSV** (comma-separated, header row).
- **TSV** (tab-separated, header row).

Everything else is **not profiled**:

- **Unfetched URL datasets** — a URL reference with no snapshot yet (e.g. one
  migrated from an older record), so there is nothing to scan. Counts are `null` /
  N/A until it is refreshed.
- **Non-tabular data** — a single JSON object, TopoJSON, or anything we can't
  read as rows-of-columns. Counts are `null` / N/A.

For the not-profiled cases, `size` is still computed (it's just the byte length
of the stored payload), but `rowCount` and `columnCount` are `null`, and
`columns`/`columnTypes` are empty.

### 3.2 The algorithm

1. **Compute `size`** from the raw payload regardless of whether it's tabular —
   byte length of the text (CSV/TSV) or of the JSON-serialized value.
2. **Decide if it's tabular.** Map `(format, parsed shape)` to a row set:
   - `csv` / `tsv` → parse into rows-of-objects using the matching delimiter.
   - `json` that is a non-empty **array of objects** → use it directly.
   - anything else (`topojson`, a lone JSON object, an empty array) → not
     tabular; return the N/A profile (`rowCount: null`, `columnCount: null`,
     `columns: []`, `columnTypes: []`, `columnStats: []`, plus `size`).
3. **Derive columns** from the union of keys across the rows (or the CSV/TSV
   header), preserving first-seen order.
4. **Infer each column's type and stats** by collecting that column's values across
   the rows (sampling per §4) and calling `inferColumnType` (§2) plus deriving its
   cardinality + numeric extent (§3.3) — one pass per column over the same sample.
5. **Assemble** `rowCount`, `columnCount`, `columns`, `columnTypes`, `columnStats`,
   `size`.

### Sketch

```ts
// src/core/profile.ts
import { inferColumnType, type ColumnType } from './type-inference';

export interface DatasetProfile {
  rowCount: number | null;
  columnCount: number | null;
  columns: string[];
  columnTypes: Array<{ name: string; type: ColumnType }>;
  columnStats: Array<{
    name: string;
    distinct: number; // capped at DISTINCT_CAP
    distinctCapped: boolean; // true ⇒ real cardinality ≥ DISTINCT_CAP
    numericExtent: { min: number; max: number } | null; // numeric columns only
  }>;
  size: number;
}

const NA = (size: number): DatasetProfile => ({
  rowCount: null,
  columnCount: null,
  columns: [],
  columnTypes: [],
  columnStats: [],
  size,
});

/** Profile a dataset payload. `rows` is the tabular form (CSV/TSV/JSON-array)
 *  already parsed to rows-of-objects, or null for non-tabular / URL data. */
export function profileData(
  rows: ReadonlyArray<Record<string, unknown>> | null,
  size: number,
): DatasetProfile {
  if (!rows || rows.length === 0) return NA(size);

  // Column order = first-seen order across all rows.
  const columns: string[] = [];
  const seen = new Set<string>();
  for (const row of rows) {
    for (const key of Object.keys(row)) {
      if (!seen.has(key)) {
        seen.add(key);
        columns.push(key);
      }
    }
  }
  if (columns.length === 0) return NA(size);

  const sample = sampleRows(rows);
  // One pass per column over the sample: type + stats (cardinality, numeric extent).
  const columnTypes = [];
  const columnStats = [];
  for (const name of columns) {
    const values = sample.map((r) => r[name]);
    const type = inferColumnType(values);
    columnTypes.push({ name, type });
    columnStats.push(columnStatsFor(name, values, type)); // see §3.3
  }

  return {
    rowCount: rows.length,
    columnCount: columns.length,
    columns,
    columnTypes,
    columnStats,
    size,
  };
}
```

Parsing CSV/TSV text and detecting the payload shape happen _upstream_ of
`profileData`; this function takes already-parsed rows so it stays pure and
trivially testable. The caller passes `null` for URL and non-tabular datasets.

### 3.3 Column stats: cardinality + numeric extent

Alongside the display type, each column carries the two data-shape signals the
**Chart Builder** needs for its data-aware Tier-B hints (`chart-builder.ts`
`builderWarnings`) **and** for its default pre-population (`smartDefaultEncodings`
prefers a low-cardinality category over a high-cardinality key, so the builder never
opens on a degenerate chart; spec §06):

- **`distinct`** — the count of distinct non-empty values **in the sample**,
  counted only up to `DISTINCT_CAP` (50). Past the cap the exact number stops
  mattering — a legend or category axis with 50 entries is already unreadable — so
  counting stops and **`distinctCapped`** flags that the real cardinality is at
  least the cap. Powers the crowded-legend (discrete colour) and crowded-category
  warnings.
- **`numericExtent`** — `{ min, max }` over the numeric values, for `number`
  columns only (`null` otherwise), using the **same numeric test** as
  `inferColumnType` so the two never disagree. Powers the negative-value **Size**
  guard (size encodes magnitude → negatives mislead; Draco `hard.lp:56`).

Both are derived in the same sample pass as type inference, so they are nearly
free, and both are **estimates** (sampled from the head, like the type) — fine for
soft advisories, not guarantees. URL / non-tabular data and datasets stored before
this field carry empty `columnStats`, and the dependent hints simply skip.

---

## 4. Sampling vs. full scan

`rowCount`/`columnCount`/`size` always reflect the **whole** dataset — they're
cheap (a length and a byte count). Only the **per-value** work — type inference
and the §3.3 column stats — has a per-row cost, and it's the one place a huge
dataset could hurt.

So: derive types **and** stats from a **bounded sample** of rows, not the full
column. A fixed cap (e.g. the first ~200 rows) keeps profiling fast and
predictable on large pasted datasets while still being more than enough signal to
classify a column and estimate its cardinality / range.

```ts
const SAMPLE_SIZE = 200;
const sampleRows = <T>(rows: ReadonlyArray<T>): ReadonlyArray<T> =>
  rows.length <= SAMPLE_SIZE ? rows : rows.slice(0, SAMPLE_SIZE);
```

Trade-off to be aware of: a column that is numeric for its first 200 rows but
turns to text later will be mis-typed as `number`. That's an accepted cost — the
type is a display hint, the mistake is cheap, and the speed win on large
datasets is worth it. Sampling the head (rather than randomly) keeps results
**deterministic**, which matters for tests and for not surprising the user when
the same paste profiles the same way twice.

### Do / Don't

- **Do** count rows/columns and size over the full payload.
- **Do** cap type-inference **and column-stats** sampling at a fixed head slice for
  determinism (so `distinct` / `numericExtent` are estimates, like the type).
- **Don't** randomly sample — non-deterministic profiles break tests and confuse
  users.
- **Don't** scan every value of a million-row paste to guess a type.

---

## 5. Testing

Both functions are pure, so tests are plain input/output assertions in Vitest —
no mocks, no DOM, no fixtures beyond literal arrays.

Cover at least:

- **`inferColumnType`**: each type detected from a clean column; mixed columns
  fall to `string`; empty/whitespace cells ignored; all-empty and zero-length →
  `string`; precedence (a `["true","false"]` column is `boolean` not `string`; a
  `["2024","2025"]` column is `number` not `date`); date shape guard rejects
  `"42"` and `"hello"` even though one engine's `Date.parse` might accept them;
  `0`/`1` are `number`, not `boolean`.
- **`profileData`**: a JSON-array dataset profiles fully; `null` rows (URL) and
  an empty array (non-tabular) return the N/A profile but still carry `size`;
  column order follows first-seen key order across ragged rows; sampling cap is
  respected (a dataset longer than the cap still profiles, using only the head).

---

## Summary

- Four types only: **boolean → number → date → string**, checked in that order.
- **All present values must match** a type or the column falls through; empty
  cells are ignored; an all-empty column is `string`.
- Guard date detection with a shape regex before trusting `Date.parse`.
- A **profile** carries `rowCount`, `columnCount`, `columns`, `columnTypes`,
  `size`; URL and non-tabular datasets get a **null/N-A** profile (still sized).
- Counts and size scan the whole payload; **type inference samples the head** for
  speed and determinism.
- All of it is **pure `src/core/` logic, unit-tested with Vitest**.