Add Dataset Manager functionality with IndexedDB support

- Introduced a new modal for managing datasets, including options to create, view, and delete datasets.
- Implemented IndexedDB for persistent storage of datasets, allowing for efficient data retrieval and management.
- Added UI components for dataset details, including statistics and preview.
- Enhanced the app's JavaScript to handle dataset operations such as saving, updating, and deleting datasets.
- Integrated dataset reference resolution in the visualization editor to support dynamic data sources.
- Updated styles for the new modal and dataset management UI for improved user experience.
This commit is contained in:
2025-10-13 18:17:09 +03:00
parent 1734001d20
commit 615c2d7f98
7 changed files with 1415 additions and 26 deletions

View File

@@ -9,13 +9,26 @@ Instructions for Claude Code when working on this project.
## Architecture
- **Frontend-only**: HTML/CSS/JavaScript with CDN dependencies (Monaco Editor, Vega-Embed)
- **Storage**: localStorage with Phase 0 schema (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta)
- **Structure**: Three resizable panels (snippet library, Monaco editor, live preview)
- **No build tools**: Open `index.html` directly in browser
- **Storage**:
- **Snippets**: localStorage with Phase 0 schema (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta)
- **Datasets**: IndexedDB (unlimited size, multi-format: JSON/CSV/TSV/TopoJSON, inline & URL sources)
- **Structure**: Three resizable panels (snippet library, Monaco editor, live preview) + Dataset Manager modal
- **No build tools**: Open `index.html` directly in browser (needs local server for IndexedDB)
## Current Status
**Completed**: Phases 0-9 (All core functionality including import/export)
**Next**: Phase 10 - Dataset Management
**Completed**: Phases 0-10 (Core functionality + Dataset Management)
**Next**: Phase 11 - Advanced Dataset Features (optional enhancements)
### Key Features Implemented
- ✅ Snippet management with draft/published workflow
- ✅ Multi-field sorting and real-time search
- ✅ Storage monitoring and import/export
-**Dataset management with IndexedDB**
- Multi-format support (JSON, CSV, TSV, TopoJSON)
- Multi-source support (inline data, URL references)
- Automatic metadata calculation and URL fetching
- Dataset reference resolution in Vega-Lite specs
- Modal UI with button-group selectors
See `docs/dev-plan.md` for complete roadmap and technical details.

View File

@@ -189,31 +189,79 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu
---
### **Phase 10: Dataset Management - Part 1**
**Goal**: Separate dataset storage infrastructure
### **Phase 10: Dataset Management** ✅ **COMPLETE**
**Goal**: Separate dataset storage with multiple data sources and formats
- [ ] Implement dataset storage schema from Phase 0
- [ ] Create dataset CRUD operations
- [ ] Add dataset library panel/modal
- [ ] List all stored datasets with metadata
- [ ] Add/delete/rename datasets
- [ ] Display dataset size and row counts
**Deliverables**:
- IndexedDB-based dataset storage (separate from snippets, no 5MB localStorage limit)
- Full CRUD operations for datasets (create, read, update, delete)
- Modal-based Dataset Manager UI (accessible via header link and 📁 toggle button)
- Support for multiple data sources:
- **Inline data**: JSON, CSV, TSV, TopoJSON stored directly
- **URL data**: Remote data sources with format specification
- Automatic metadata calculation:
- Row count, column count, column names
- Data size in bytes
- Automatic URL fetching on dataset creation
- Refresh metadata button for URL datasets (🔄)
- Dataset list with informative metadata display
- Dataset details panel with:
- Editable name and comment
- Statistics display (rows, columns, size)
- Data preview (first 5 rows or URL info)
- Copy reference button (copies `"data": {"name": "dataset-name"}`)
- Delete button with confirmation
- Auto-resolve dataset references in Vega-Lite specs during rendering
- Format-aware rendering:
- JSON: `{ values: data }`
- CSV/TSV: `{ values: data, format: { type: 'csv'/'tsv' } }`
- TopoJSON: `{ values: data, format: { type: 'topojson' } }`
- URL: `{ url: "...", format: { type: '...' } }`
- Button-group UI for source/format selection (matches editor style)
- Graceful error handling for CORS and network failures
- Modal scrolling support for small viewports
**Deliverable**: Basic dataset storage separate from snippets
**Dataset Schema**:
```javascript
{
id: number,
name: string,
created: ISO string,
modified: ISO string,
data: array/string/object, // Inline data or URL string
format: 'json'|'csv'|'tsv'|'topojson',
source: 'inline'|'url',
comment: string,
rowCount: number|null,
columnCount: number|null,
columns: string[],
size: number|null // bytes
}
```
**Technical Implementation**:
- IndexedDB with keyPath 'id', indexes on 'name' (unique) and 'modified'
- Async/Promise-based DatasetStorage API
- Format-specific parsing for metadata calculation
- Vega-Lite's native format parsers used for rendering
- Metadata refresh fetches live data and updates statistics
- Modal resizes with viewport (max-height: 90vh)
---
### **Phase 11: Dataset Management - Part 2**
**Goal**: Reference datasets from specs
### **Phase 11: Advanced Dataset Features** _(Future)_
**Goal**: Enhanced dataset workflows
- [ ] Detect inline data in Vega-Lite specs
- [ ] "Extract to dataset" feature for inline data
- [ ] Replace inline data with dataset references
- [ ] Auto-resolve dataset references when rendering
- [ ] Update snippet UI to show linked datasets
- [ ] Handle missing dataset references gracefully
- [ ] Dataset usage tracking (which snippets reference which datasets)
- [ ] Import datasets from file upload
- [ ] Export individual datasets
- [ ] Dataset preview with table view
- [ ] Column type detection and display
**Deliverable**: Specs can reference shared datasets
**Deliverable**: Advanced dataset management and discovery
---
@@ -280,8 +328,9 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu
│ └── js/ # Modular JavaScript organization
│ ├── config.js # Global variables, settings, & sample data
│ ├── snippet-manager.js # Snippet storage, CRUD operations & localStorage wrapper
│ ├── dataset-manager.js # Dataset storage, CRUD operations & IndexedDB wrapper
│ ├── panel-manager.js # Panel resize, toggle & memory system
│ ├── editor.js # Monaco Editor initialization & Vega-Lite rendering
│ ├── editor.js # Monaco Editor initialization, dataset resolution & Vega-Lite rendering
│ └── app.js # Application initialization & event handlers
└── docs/
├── dev-plan.md # This development roadmap
@@ -311,15 +360,15 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu
## Current Status
**Completed**: Phases 0-9 (Storage, UI, editor, rendering, persistence, CRUD, organization, draft/published workflow, storage monitoring, import/export)
**Next**: Phase 10 - Dataset Management
**Completed**: Phases 0-10 (Storage, UI, editor, rendering, persistence, CRUD, organization, draft/published workflow, storage monitoring, import/export, dataset management)
**Next**: Phase 11 - Advanced Dataset Features (optional enhancements)
**See**: `CLAUDE.md` for concise current state summary
---
## Implemented Features
### Core Capabilities (Phases 0-9)
### Core Capabilities (Phases 0-10)
- Three-panel resizable layout with memory and persistence
- Monaco Editor v0.47.0 with Vega-Lite v5 schema validation
- Live Vega-Lite rendering with debounced updates and error display
@@ -334,6 +383,15 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu
- Storage monitoring with visual progress bar and warning states
- Export/Import functionality with format auto-detection
- Snippet size display (right-aligned, shown for ≥ 1 KB)
- **Dataset Management (Phase 10)**:
- IndexedDB storage for datasets (unlimited size)
- Multi-format support: JSON, CSV, TSV, TopoJSON
- Multi-source support: Inline data and URL references
- Modal-based Dataset Manager with full CRUD
- Automatic metadata calculation and display
- URL metadata fetching and refresh
- Dataset reference resolution in Vega-Lite specs
- Button-group UI for source/format selection
- Retro Windows 2000 aesthetic throughout
### Technical Implementation
@@ -348,4 +406,8 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu
- **Storage Calculation**: Blob API for accurate byte counting of snippet data
- **Flexbox Layout**: Scrollable snippet list with fixed metadata and storage monitor at bottom
- **Import/Export**: Format detection, field normalization, ID conflict resolution, additive merging
- **Size Display**: Per-snippet size calculation with conditional rendering (≥ 1 KB threshold)
- **Size Display**: Per-snippet size calculation with conditional rendering (≥ 1 KB threshold)
- **Dataset Storage**: IndexedDB with async/Promise-based API, unique name constraint
- **Dataset Resolution**: Async spec transformation before rendering, format-aware data injection
- **URL Metadata**: Fetch on creation with graceful CORS error handling
- **Modal UI**: Flexbox with overflow:auto, max-height responsive to viewport

View File

@@ -21,6 +21,7 @@
<div class="header-links">
<span class="header-link" id="import-link">Import</span>
<span class="header-link" id="export-link">Export</span>
<span class="header-link" id="datasets-link">Datasets</span>
<span class="header-link" id="help-link">Help</span>
<input type="file" id="import-file-input" accept=".json" style="display: none;" />
</div>
@@ -38,6 +39,9 @@
<button class="toggle-btn active" id="toggle-preview-panel" title="Toggle Preview Panel">
👁️
</button>
<button class="toggle-btn" id="toggle-datasets" title="Datasets">
📁
</button>
</div>
<div class="main-panels">
@@ -144,8 +148,138 @@
</div>
</div>
<!-- Dataset Manager Modal -->
<div id="dataset-modal" class="modal" style="display: none;">
<div class="modal-content">
<div class="modal-header">
<span class="modal-title">Dataset Manager</span>
<button class="modal-close" id="dataset-modal-close">×</button>
</div>
<div class="modal-body">
<!-- List View (default) -->
<div id="dataset-list-view" class="dataset-view">
<div class="dataset-list-header">
<button class="modal-btn primary" id="new-dataset-btn">New Dataset</button>
</div>
<div class="dataset-container">
<div class="dataset-list" id="dataset-list">
<!-- Dynamically populated by renderDatasetList() -->
</div>
<div class="dataset-details" id="dataset-details" style="display: none;">
<div class="dataset-detail-section">
<div class="dataset-detail-header">Name</div>
<input type="text" id="dataset-detail-name" class="dataset-input" placeholder="Dataset name..." />
<div class="dataset-detail-header">Comment</div>
<textarea id="dataset-detail-comment" class="dataset-textarea" placeholder="Add a comment..." rows="3"></textarea>
<div class="dataset-detail-header-row">
<span class="dataset-detail-header">Statistics</span>
<button class="dataset-refresh-btn" id="refresh-metadata-btn" style="display: none;" title="Refresh metadata from URL">🔄</button>
</div>
<div class="dataset-stats">
<div class="dataset-stat-item">
<span class="dataset-stat-label">Rows:</span>
<span id="dataset-detail-rows">0</span>
</div>
<div class="dataset-stat-item">
<span class="dataset-stat-label">Columns:</span>
<span id="dataset-detail-columns">0</span>
</div>
<div class="dataset-stat-item">
<span class="dataset-stat-label">Size:</span>
<span id="dataset-detail-size">0 B</span>
</div>
</div>
<div class="dataset-detail-header">Timestamps</div>
<div class="dataset-stats">
<div class="dataset-stat-item">
<span class="dataset-stat-label">Created:</span>
<span id="dataset-detail-created">-</span>
</div>
<div class="dataset-stat-item">
<span class="dataset-stat-label">Modified:</span>
<span id="dataset-detail-modified">-</span>
</div>
</div>
<div class="dataset-detail-header">Preview</div>
<pre id="dataset-preview" class="dataset-preview-box"></pre>
<div class="dataset-actions">
<button class="modal-btn" id="copy-reference-btn">Copy Reference</button>
<button class="modal-btn delete-btn" id="delete-dataset-btn">Delete</button>
</div>
</div>
</div>
</div>
</div>
<!-- Form View (for creating new datasets) -->
<div id="dataset-form-view" class="dataset-view" style="display: none;">
<div class="dataset-form">
<div class="dataset-form-header">Create New Dataset</div>
<div class="dataset-form-group">
<label class="dataset-form-label">Name *</label>
<input type="text" id="dataset-form-name" class="dataset-input" placeholder="Enter dataset name..." />
</div>
<div class="dataset-form-group">
<div class="dataset-toggle-row">
<div class="dataset-toggle-section">
<span class="dataset-toggle-label">Source:</span>
<div class="dataset-toggle-group">
<button class="dataset-toggle-btn active" data-source="inline" type="button">Inline</button>
<button class="dataset-toggle-btn" data-source="url" type="button">URL</button>
</div>
</div>
<div class="dataset-toggle-section">
<span class="dataset-toggle-label">Format:</span>
<div class="dataset-toggle-group">
<button class="dataset-toggle-btn active" data-format="json" type="button">JSON</button>
<button class="dataset-toggle-btn" data-format="csv" type="button">CSV</button>
<button class="dataset-toggle-btn" data-format="tsv" type="button">TSV</button>
<button class="dataset-toggle-btn" data-format="topojson" type="button">TopoJSON</button>
</div>
</div>
</div>
</div>
<div class="dataset-form-group" id="dataset-url-group" style="display: none;">
<label class="dataset-form-label">URL *</label>
<input type="text" id="dataset-form-url" class="dataset-input" placeholder="https://example.com/data.csv" />
</div>
<div class="dataset-form-group" id="dataset-data-group">
<label class="dataset-form-label">Data *</label>
<div class="dataset-format-hint" id="dataset-format-hint">
JSON array of objects: [{"col1": "value", "col2": 123}, ...]
</div>
<textarea id="dataset-form-data" class="dataset-textarea" placeholder='[{"col1": "value", "col2": 123}, ...]' rows="12"></textarea>
</div>
<div class="dataset-form-group">
<label class="dataset-form-label">Comment</label>
<textarea id="dataset-form-comment" class="dataset-textarea" placeholder="Optional description..." rows="3"></textarea>
</div>
<div class="dataset-form-error" id="dataset-form-error"></div>
<div class="dataset-form-actions">
<button class="modal-btn primary" id="save-dataset-btn">Save Dataset</button>
<button class="modal-btn" id="cancel-dataset-btn">Cancel</button>
</div>
</div>
</div>
</div>
</div>
</div>
<script src="src/js/config.js"></script>
<script src="src/js/snippet-manager.js"></script>
<script src="src/js/dataset-manager.js"></script>
<script src="src/js/panel-manager.js"></script>
<script src="src/js/editor.js"></script>
<script src="src/js/app.js"></script>

View File

@@ -117,6 +117,70 @@ document.addEventListener('DOMContentLoaded', function () {
});
}
// Dataset Manager
const datasetsLink = document.getElementById('datasets-link');
const toggleDatasetsBtn = document.getElementById('toggle-datasets');
const datasetModal = document.getElementById('dataset-modal');
const datasetModalClose = document.getElementById('dataset-modal-close');
const newDatasetBtn = document.getElementById('new-dataset-btn');
const cancelDatasetBtn = document.getElementById('cancel-dataset-btn');
const saveDatasetBtn = document.getElementById('save-dataset-btn');
const deleteDatasetBtn = document.getElementById('delete-dataset-btn');
const copyReferenceBtn = document.getElementById('copy-reference-btn');
// Open dataset manager
if (datasetsLink) {
datasetsLink.addEventListener('click', openDatasetManager);
}
if (toggleDatasetsBtn) {
toggleDatasetsBtn.addEventListener('click', openDatasetManager);
}
// Close dataset manager
if (datasetModalClose) {
datasetModalClose.addEventListener('click', closeDatasetManager);
}
// Close on overlay click
if (datasetModal) {
datasetModal.addEventListener('click', function (e) {
if (e.target === datasetModal) {
closeDatasetManager();
}
});
}
// New dataset button
if (newDatasetBtn) {
newDatasetBtn.addEventListener('click', showNewDatasetForm);
}
// Cancel dataset button
if (cancelDatasetBtn) {
cancelDatasetBtn.addEventListener('click', hideNewDatasetForm);
}
// Save dataset button
if (saveDatasetBtn) {
saveDatasetBtn.addEventListener('click', saveNewDataset);
}
// Delete dataset button
if (deleteDatasetBtn) {
deleteDatasetBtn.addEventListener('click', deleteCurrentDataset);
}
// Copy reference button
if (copyReferenceBtn) {
copyReferenceBtn.addEventListener('click', copyDatasetReference);
}
// Refresh metadata button
const refreshMetadataBtn = document.getElementById('refresh-metadata-btn');
if (refreshMetadataBtn) {
refreshMetadataBtn.addEventListener('click', refreshDatasetMetadata);
}
// View mode toggle buttons
document.getElementById('view-draft').addEventListener('click', () => {
switchViewMode('draft');

636
src/js/dataset-manager.js Normal file
View File

@@ -0,0 +1,636 @@
// Dataset management with IndexedDB
const DB_NAME = 'astrolabe-datasets';
const DB_VERSION = 1;
const STORE_NAME = 'datasets';
let db = null;
// Initialize IndexedDB
function initializeDatasetDB() {
return new Promise((resolve, reject) => {
const request = indexedDB.open(DB_NAME, DB_VERSION);
request.onerror = () => reject(request.error);
request.onsuccess = () => {
db = request.result;
resolve(db);
};
request.onupgradeneeded = (event) => {
const db = event.target.result;
// Create object store if it doesn't exist
if (!db.objectStoreNames.contains(STORE_NAME)) {
const objectStore = db.createObjectStore(STORE_NAME, { keyPath: 'id' });
objectStore.createIndex('name', 'name', { unique: true });
objectStore.createIndex('modified', 'modified', { unique: false });
}
};
});
}
// Generate unique ID
function generateDatasetId() {
return Date.now() + Math.random() * 1000;
}
// Calculate dataset statistics
function calculateDatasetStats(data, format, source) {
let rowCount = 0;
let columnCount = 0;
let columns = [];
let size = 0;
// For URL sources, we can't calculate stats without fetching
if (source === 'url') {
return { rowCount: null, columnCount: null, columns: [], size: null };
}
if (format === 'json' || format === 'topojson') {
if (!Array.isArray(data) || data.length === 0) {
return { rowCount: 0, columnCount: 0, columns: [], size: 0 };
}
rowCount = data.length;
const firstRow = data[0];
columns = typeof firstRow === 'object' ? Object.keys(firstRow) : [];
columnCount = columns.length;
size = new Blob([JSON.stringify(data)]).size;
} else if (format === 'csv' || format === 'tsv') {
// For CSV/TSV, data is stored as raw text
const lines = data.trim().split('\n');
rowCount = Math.max(0, lines.length - 1); // Subtract header row
if (lines.length > 0) {
const separator = format === 'csv' ? ',' : '\t';
columns = lines[0].split(separator).map(h => h.trim().replace(/^"|"$/g, ''));
columnCount = columns.length;
}
size = new Blob([data]).size;
}
return { rowCount, columnCount, columns, size };
}
// Dataset Storage API
const DatasetStorage = {
// Initialize database
async init() {
if (!db) {
await initializeDatasetDB();
}
return db;
},
// Create new dataset
async createDataset(name, data, format, source, comment = '') {
await this.init();
const now = new Date().toISOString();
const stats = calculateDatasetStats(data, format, source);
const dataset = {
id: generateDatasetId(),
name: name.trim(),
created: now,
modified: now,
data: data, // For inline: actual data, for URL: the URL string
format: format, // 'json', 'csv', 'tsv', or 'topojson'
source: source, // 'inline' or 'url'
comment: comment.trim(),
...stats
};
return new Promise((resolve, reject) => {
const transaction = db.transaction([STORE_NAME], 'readwrite');
const store = transaction.objectStore(STORE_NAME);
const request = store.add(dataset);
request.onsuccess = () => resolve(dataset);
request.onerror = () => reject(request.error);
});
},
// Get all datasets
async listDatasets() {
await this.init();
return new Promise((resolve, reject) => {
const transaction = db.transaction([STORE_NAME], 'readonly');
const store = transaction.objectStore(STORE_NAME);
const request = store.getAll();
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
},
// Get single dataset by ID
async getDataset(id) {
await this.init();
return new Promise((resolve, reject) => {
const transaction = db.transaction([STORE_NAME], 'readonly');
const store = transaction.objectStore(STORE_NAME);
const request = store.get(id);
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
},
// Get dataset by name
async getDatasetByName(name) {
await this.init();
return new Promise((resolve, reject) => {
const transaction = db.transaction([STORE_NAME], 'readonly');
const store = transaction.objectStore(STORE_NAME);
const index = store.index('name');
const request = index.get(name);
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
},
// Update dataset
async updateDataset(id, updates) {
await this.init();
const dataset = await this.getDataset(id);
if (!dataset) {
throw new Error('Dataset not found');
}
// Update fields
if (updates.name !== undefined) dataset.name = updates.name.trim();
if (updates.data !== undefined) {
dataset.data = updates.data;
if (updates.format !== undefined) dataset.format = updates.format;
if (updates.source !== undefined) dataset.source = updates.source;
// If metadata fields are explicitly provided, use them directly
if (updates.rowCount !== undefined) dataset.rowCount = updates.rowCount;
if (updates.columnCount !== undefined) dataset.columnCount = updates.columnCount;
if (updates.columns !== undefined) dataset.columns = updates.columns;
if (updates.size !== undefined) dataset.size = updates.size;
// Otherwise, calculate stats (for inline data)
if (updates.rowCount === undefined && updates.columnCount === undefined) {
const stats = calculateDatasetStats(updates.data, dataset.format, dataset.source);
Object.assign(dataset, stats);
}
}
if (updates.comment !== undefined) dataset.comment = updates.comment.trim();
dataset.modified = new Date().toISOString();
return new Promise((resolve, reject) => {
const transaction = db.transaction([STORE_NAME], 'readwrite');
const store = transaction.objectStore(STORE_NAME);
const request = store.put(dataset);
request.onsuccess = () => resolve(dataset);
request.onerror = () => reject(request.error);
});
},
// Delete dataset
async deleteDataset(id) {
await this.init();
return new Promise((resolve, reject) => {
const transaction = db.transaction([STORE_NAME], 'readwrite');
const store = transaction.objectStore(STORE_NAME);
const request = store.delete(id);
request.onsuccess = () => resolve(true);
request.onerror = () => reject(request.error);
});
},
// Check if dataset name exists
async nameExists(name, excludeId = null) {
const datasets = await this.listDatasets();
return datasets.some(d => d.name === name && d.id !== excludeId);
}
};
// Format bytes for display
function formatDatasetSize(bytes) {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
}
// Fetch URL data and calculate metadata
async function fetchURLMetadata(url, format) {
try {
const response = await fetch(url);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const contentLength = response.headers.get('content-length');
const text = await response.text();
let rowCount = 0;
let columnCount = 0;
let columns = [];
let size = contentLength ? parseInt(contentLength) : new Blob([text]).size;
// Parse based on format
if (format === 'json') {
const data = JSON.parse(text);
if (Array.isArray(data)) {
rowCount = data.length;
if (data.length > 0 && typeof data[0] === 'object') {
columns = Object.keys(data[0]);
columnCount = columns.length;
}
}
} else if (format === 'csv' || format === 'tsv') {
const lines = text.trim().split('\n');
rowCount = Math.max(0, lines.length - 1); // Subtract header
if (lines.length > 0) {
const separator = format === 'csv' ? ',' : '\t';
columns = lines[0].split(separator).map(h => h.trim().replace(/^"|"$/g, ''));
columnCount = columns.length;
}
} else if (format === 'topojson') {
// TopoJSON structure is complex, just note it exists
rowCount = null;
columnCount = null;
}
return { rowCount, columnCount, columns, size };
} catch (error) {
throw new Error(`Failed to fetch URL metadata: ${error.message}`);
}
}
// Render dataset list in modal
async function renderDatasetList() {
const datasets = await DatasetStorage.listDatasets();
const listContainer = document.getElementById('dataset-list');
if (datasets.length === 0) {
listContainer.innerHTML = '<div class="dataset-empty">No datasets yet. Click "New Dataset" to create one.</div>';
return;
}
// Sort by modified date (most recent first)
datasets.sort((a, b) => new Date(b.modified) - new Date(a.modified));
const html = datasets.map(dataset => {
let metaText;
if (dataset.source === 'url') {
// Show metadata if available, otherwise just URL and format
if (dataset.rowCount !== null && dataset.size !== null) {
metaText = `URL • ${dataset.rowCount} rows • ${dataset.format.toUpperCase()}${formatDatasetSize(dataset.size)}`;
} else {
metaText = `URL • ${dataset.format.toUpperCase()}`;
}
} else {
metaText = `${dataset.rowCount} rows • ${dataset.format.toUpperCase()}${formatDatasetSize(dataset.size)}`;
}
return `
<div class="dataset-item" data-dataset-id="${dataset.id}">
<div class="dataset-info">
<div class="dataset-name">${dataset.name}</div>
<div class="dataset-meta">${metaText}</div>
</div>
</div>
`;
}).join('');
listContainer.innerHTML = html;
// Attach click handlers
document.querySelectorAll('.dataset-item').forEach(item => {
item.addEventListener('click', function() {
const datasetId = parseFloat(this.dataset.datasetId);
selectDataset(datasetId);
});
});
}
// Select a dataset and show details
async function selectDataset(datasetId) {
const dataset = await DatasetStorage.getDataset(datasetId);
if (!dataset) return;
// Update selection state
document.querySelectorAll('.dataset-item').forEach(item => {
item.classList.remove('selected');
});
document.querySelector(`[data-dataset-id="${datasetId}"]`).classList.add('selected');
// Show details panel
const detailsPanel = document.getElementById('dataset-details');
detailsPanel.style.display = 'block';
// Show/hide refresh button for URL datasets
const refreshBtn = document.getElementById('refresh-metadata-btn');
if (dataset.source === 'url') {
refreshBtn.style.display = 'flex';
} else {
refreshBtn.style.display = 'none';
}
// Populate details
document.getElementById('dataset-detail-name').value = dataset.name;
document.getElementById('dataset-detail-comment').value = dataset.comment;
document.getElementById('dataset-detail-rows').textContent = dataset.rowCount !== null ? dataset.rowCount : 'N/A';
document.getElementById('dataset-detail-columns').textContent = dataset.columnCount !== null ? dataset.columnCount : 'N/A';
document.getElementById('dataset-detail-size').textContent = dataset.size !== null ? formatDatasetSize(dataset.size) : 'N/A';
document.getElementById('dataset-detail-created').textContent = new Date(dataset.created).toLocaleString();
document.getElementById('dataset-detail-modified').textContent = new Date(dataset.modified).toLocaleString();
// Show preview
let previewText;
if (dataset.source === 'url') {
previewText = `URL: ${dataset.data}\nFormat: ${dataset.format.toUpperCase()}`;
} else if (dataset.format === 'json' || dataset.format === 'topojson') {
const previewData = Array.isArray(dataset.data) ? dataset.data.slice(0, 5) : dataset.data;
previewText = JSON.stringify(previewData, null, 2);
} else if (dataset.format === 'csv' || dataset.format === 'tsv') {
const lines = dataset.data.split('\n');
previewText = lines.slice(0, 6).join('\n'); // Header + 5 rows
}
document.getElementById('dataset-preview').textContent = previewText;
// Store current dataset ID
window.currentDatasetId = datasetId;
}
// Open dataset manager modal
function openDatasetManager() {
const modal = document.getElementById('dataset-modal');
modal.style.display = 'flex';
renderDatasetList();
}
// Close dataset manager modal
function closeDatasetManager() {
const modal = document.getElementById('dataset-modal');
modal.style.display = 'none';
window.currentDatasetId = null;
}
// Update format hint and placeholder
function updateFormatHint(format) {
const hintEl = document.getElementById('dataset-format-hint');
const dataEl = document.getElementById('dataset-form-data');
if (format === 'json') {
hintEl.textContent = 'JSON array of objects: [{"col1": "value", "col2": 123}, ...]';
dataEl.placeholder = '[{"col1": "value", "col2": 123}, ...]';
} else if (format === 'csv') {
hintEl.textContent = 'CSV with header row: col1,col2\\nvalue1,123\\nvalue2,456';
dataEl.placeholder = 'col1,col2\nvalue1,123\nvalue2,456';
} else if (format === 'tsv') {
hintEl.textContent = 'TSV with header row: col1\\tcol2\\nvalue1\\t123\\nvalue2\\t456';
dataEl.placeholder = 'col1\tcol2\nvalue1\t123\nvalue2\t456';
} else if (format === 'topojson') {
hintEl.textContent = 'TopoJSON object: {"type": "Topology", "objects": {...}, "arcs": [...]}';
dataEl.placeholder = '{"type": "Topology", "objects": {...}}';
}
}
// Toggle between URL and inline data inputs
function toggleDataSource(source) {
const urlGroup = document.getElementById('dataset-url-group');
const dataGroup = document.getElementById('dataset-data-group');
if (source === 'url') {
urlGroup.style.display = 'block';
dataGroup.style.display = 'none';
} else {
urlGroup.style.display = 'none';
dataGroup.style.display = 'block';
}
}
// Show new dataset form
function showNewDatasetForm() {
document.getElementById('dataset-list-view').style.display = 'none';
document.getElementById('dataset-form-view').style.display = 'block';
document.getElementById('dataset-form-name').value = '';
document.getElementById('dataset-form-data').value = '';
document.getElementById('dataset-form-url').value = '';
document.getElementById('dataset-form-comment').value = '';
document.getElementById('dataset-form-error').textContent = '';
// Reset to inline data source and JSON format
document.querySelectorAll('[data-source]').forEach(btn => {
btn.classList.toggle('active', btn.dataset.source === 'inline');
});
document.querySelectorAll('[data-format]').forEach(btn => {
btn.classList.toggle('active', btn.dataset.format === 'json');
});
toggleDataSource('inline');
updateFormatHint('json');
// Add listeners if not already added
if (!window.datasetListenersAdded) {
// Source toggle button listeners
document.querySelectorAll('[data-source]').forEach(btn => {
btn.addEventListener('click', function () {
// Update active state
document.querySelectorAll('[data-source]').forEach(b => b.classList.remove('active'));
this.classList.add('active');
toggleDataSource(this.dataset.source);
});
});
// Format toggle button listeners
document.querySelectorAll('[data-format]').forEach(btn => {
btn.addEventListener('click', function () {
// Update active state
document.querySelectorAll('[data-format]').forEach(b => b.classList.remove('active'));
this.classList.add('active');
updateFormatHint(this.dataset.format);
});
});
window.datasetListenersAdded = true;
}
}
// Hide new dataset form
function hideNewDatasetForm() {
document.getElementById('dataset-list-view').style.display = 'block';
document.getElementById('dataset-form-view').style.display = 'none';
}
// Save new dataset
async function saveNewDataset() {
const name = document.getElementById('dataset-form-name').value.trim();
const source = document.querySelector('[data-source].active').dataset.source;
const format = document.querySelector('[data-format].active').dataset.format;
const comment = document.getElementById('dataset-form-comment').value.trim();
const errorEl = document.getElementById('dataset-form-error');
errorEl.textContent = '';
// Validation
if (!name) {
errorEl.textContent = 'Dataset name is required';
return;
}
let data;
let metadata = null;
if (source === 'url') {
const url = document.getElementById('dataset-form-url').value.trim();
if (!url) {
errorEl.textContent = 'URL is required';
return;
}
// Basic URL validation
try {
new URL(url);
} catch (error) {
errorEl.textContent = 'Invalid URL format';
return;
}
// Fetch metadata from URL
errorEl.textContent = 'Fetching data from URL...';
try {
metadata = await fetchURLMetadata(url, format);
errorEl.textContent = '';
} catch (error) {
errorEl.textContent = `Warning: ${error.message}. Dataset will be created without metadata.`;
// Continue anyway - URL might require CORS or auth
await new Promise(resolve => setTimeout(resolve, 2000)); // Show warning briefly
errorEl.textContent = '';
}
data = url; // Store the URL string
} else {
// Inline data
const dataText = document.getElementById('dataset-form-data').value.trim();
if (!dataText) {
errorEl.textContent = 'Data is required';
return;
}
// Basic validation of data format
try {
if (format === 'json' || format === 'topojson') {
const parsed = JSON.parse(dataText);
if (format === 'json' && !Array.isArray(parsed)) {
errorEl.textContent = 'JSON data must be an array of objects';
return;
}
if (format === 'json' && parsed.length === 0) {
errorEl.textContent = 'Data array cannot be empty';
return;
}
data = parsed; // Store as parsed JSON
} else if (format === 'csv' || format === 'tsv') {
const lines = dataText.trim().split('\n');
if (lines.length < 2) {
errorEl.textContent = `${format.toUpperCase()} must have at least a header row and one data row`;
return;
}
data = dataText; // Store as raw CSV/TSV string
}
} catch (error) {
errorEl.textContent = `Validation error: ${error.message}`;
return;
}
}
// Check if name already exists
if (await DatasetStorage.nameExists(name)) {
errorEl.textContent = 'A dataset with this name already exists';
return;
}
// Create dataset
try {
const dataset = await DatasetStorage.createDataset(name, data, format, source, comment);
// If we have metadata from URL fetch, update the dataset
if (metadata) {
await DatasetStorage.updateDataset(dataset.id, {
data: data,
...metadata
});
}
hideNewDatasetForm();
await renderDatasetList();
} catch (error) {
errorEl.textContent = `Failed to save dataset: ${error.message}`;
}
}
// Delete current dataset
async function deleteCurrentDataset() {
if (!window.currentDatasetId) return;
const dataset = await DatasetStorage.getDataset(window.currentDatasetId);
if (!dataset) return;
if (confirm(`Delete dataset "${dataset.name}"? This action cannot be undone.`)) {
await DatasetStorage.deleteDataset(window.currentDatasetId);
document.getElementById('dataset-details').style.display = 'none';
window.currentDatasetId = null;
await renderDatasetList();
}
}
// Copy dataset reference to clipboard
function copyDatasetReference() {
if (!window.currentDatasetId) return;
DatasetStorage.getDataset(window.currentDatasetId).then(dataset => {
const reference = `"data": {"name": "${dataset.name}"}`;
navigator.clipboard.writeText(reference).then(() => {
alert('Dataset reference copied to clipboard!');
});
});
}
// Refresh metadata for URL dataset
async function refreshDatasetMetadata() {
if (!window.currentDatasetId) return;
const dataset = await DatasetStorage.getDataset(window.currentDatasetId);
if (!dataset || dataset.source !== 'url') return;
const refreshBtn = document.getElementById('refresh-metadata-btn');
refreshBtn.disabled = true;
refreshBtn.textContent = '⏳';
try {
const metadata = await fetchURLMetadata(dataset.data, dataset.format);
// Update dataset with new metadata
await DatasetStorage.updateDataset(dataset.id, {
data: dataset.data,
...metadata
});
// Refresh the display
await selectDataset(dataset.id);
await renderDatasetList();
// Brief success indicator
refreshBtn.textContent = '✓';
setTimeout(() => {
refreshBtn.textContent = '🔄';
refreshBtn.disabled = false;
}, 1000);
} catch (error) {
alert(`Failed to refresh metadata: ${error.message}`);
refreshBtn.textContent = '🔄';
refreshBtn.disabled = false;
}
}

View File

@@ -1,3 +1,76 @@
// Resolve dataset references in a spec
async function resolveDatasetReferences(spec) {
// If spec has data.name, look it up
if (spec.data && spec.data.name && typeof spec.data.name === 'string') {
const datasetName = spec.data.name;
const dataset = await DatasetStorage.getDatasetByName(datasetName);
if (dataset) {
// Replace data reference with actual data in the format Vega-Lite expects
if (dataset.source === 'url') {
// For URL sources, pass the URL and format
spec.data = {
url: dataset.data,
format: { type: dataset.format }
};
} else {
// For inline sources
if (dataset.format === 'json') {
spec.data = { values: dataset.data };
} else if (dataset.format === 'csv') {
spec.data = {
values: dataset.data,
format: { type: 'csv' }
};
} else if (dataset.format === 'tsv') {
spec.data = {
values: dataset.data,
format: { type: 'tsv' }
};
} else if (dataset.format === 'topojson') {
spec.data = {
values: dataset.data,
format: { type: 'topojson' }
};
}
}
} else {
throw new Error(`Dataset "${datasetName}" not found`);
}
}
// Recursively resolve in layers (for layered specs)
if (spec.layer && Array.isArray(spec.layer)) {
for (let layer of spec.layer) {
await resolveDatasetReferences(layer);
}
}
// Recursively resolve in concat/hconcat/vconcat
if (spec.concat && Array.isArray(spec.concat)) {
for (let view of spec.concat) {
await resolveDatasetReferences(view);
}
}
if (spec.hconcat && Array.isArray(spec.hconcat)) {
for (let view of spec.hconcat) {
await resolveDatasetReferences(view);
}
}
if (spec.vconcat && Array.isArray(spec.vconcat)) {
for (let view of spec.vconcat) {
await resolveDatasetReferences(view);
}
}
// Recursively resolve in facet
if (spec.spec) {
await resolveDatasetReferences(spec.spec);
}
return spec;
}
// Render function that takes spec from editor
async function renderVisualization() {
const previewContainer = document.getElementById('vega-preview');
@@ -5,7 +78,10 @@ async function renderVisualization() {
try {
// Get current content from editor
const specText = editor.getValue();
const spec = JSON.parse(specText);
let spec = JSON.parse(specText);
// Resolve dataset references
spec = await resolveDatasetReferences(spec);
// Render with Vega-Embed (use global variable)
await window.vegaEmbed('#vega-preview', spec, {

View File

@@ -661,4 +661,408 @@ body {
.storage-fill.critical {
background: #ff0000;
}
/* Modal Styles */
.modal {
position: fixed;
top: 0;
left: 0;
width: 100vw;
height: 100vh;
background: rgba(0, 0, 0, 0.5);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
}
.modal-content {
background: #c0c0c0;
border: 2px outset #c0c0c0;
width: 90%;
max-width: 900px;
max-height: 90vh;
display: flex;
flex-direction: column;
box-shadow: 4px 4px 8px rgba(0, 0, 0, 0.3);
}
.modal-header {
background: #000080;
color: #ffffff;
padding: 4px 8px;
display: flex;
justify-content: space-between;
align-items: center;
height: 24px;
border-bottom: 2px solid #808080;
}
.modal-title {
font-size: 12px;
font-weight: bold;
}
.modal-close {
background: #c0c0c0;
border: 2px outset #c0c0c0;
color: #000000;
width: 20px;
height: 20px;
cursor: pointer;
font-size: 14px;
font-family: 'MS Sans Serif', Tahoma, sans-serif;
display: flex;
align-items: center;
justify-content: center;
padding: 0;
}
.modal-close:hover {
background: #d4d0c8;
}
.modal-close:active {
border: 2px inset #c0c0c0;
}
.modal-body {
flex: 1;
overflow: auto;
background: #ffffff;
border: 2px inset #c0c0c0;
margin: 8px;
min-height: 0;
}
/* Dataset Views */
.dataset-view {
min-height: 100%;
display: flex;
flex-direction: column;
}
/* List View */
.dataset-list-header {
padding: 8px;
background: #d4d0c8;
border-bottom: 2px solid #808080;
}
.dataset-container {
display: flex;
flex: 1;
overflow: hidden;
}
.dataset-list {
width: 300px;
overflow-y: auto;
border-right: 2px solid #808080;
background: #ffffff;
}
.dataset-item {
padding: 8px;
border-bottom: 1px solid #d0d0d0;
cursor: pointer;
background: #ffffff;
}
.dataset-item:hover {
background: #6a9ad5;
color: #ffffff;
}
.dataset-item.selected {
background: #316ac5;
color: #ffffff;
}
.dataset-name {
font-size: 12px;
font-weight: bold;
margin-bottom: 2px;
}
.dataset-meta {
font-size: 10px;
color: #606060;
}
.dataset-item.selected .dataset-meta,
.dataset-item:hover .dataset-meta {
color: inherit;
opacity: 0.9;
}
.dataset-empty {
padding: 32px;
text-align: center;
color: #808080;
font-style: italic;
font-size: 12px;
}
/* Dataset Details */
.dataset-details {
flex: 1;
overflow-y: auto;
background: #ffffff;
}
.dataset-detail-section {
padding: 16px;
}
.dataset-detail-header {
font-size: 11px;
font-weight: bold;
margin-bottom: 6px;
margin-top: 12px;
color: #000000;
}
.dataset-detail-header:first-child {
margin-top: 0;
}
.dataset-detail-header-row {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 6px;
margin-top: 12px;
}
.dataset-refresh-btn {
background: #c0c0c0;
border: 2px outset #c0c0c0;
color: #000000;
width: 24px;
height: 24px;
cursor: pointer;
font-size: 12px;
display: flex;
align-items: center;
justify-content: center;
padding: 0;
}
.dataset-refresh-btn:hover {
background: #d4d0c8;
}
.dataset-refresh-btn:active {
border: 2px inset #c0c0c0;
}
.dataset-refresh-btn:disabled {
opacity: 0.5;
cursor: default;
}
.dataset-input,
.dataset-textarea {
width: 100%;
font-family: 'MS Sans Serif', Tahoma, sans-serif;
font-size: 11px;
border: 2px inset #c0c0c0;
padding: 4px;
background: #ffffff;
}
.dataset-textarea {
resize: vertical;
}
.dataset-stats {
background: #f0f0f0;
border: 1px inset #c0c0c0;
padding: 8px;
font-size: 10px;
}
.dataset-stat-item {
display: flex;
justify-content: space-between;
margin-bottom: 4px;
}
.dataset-stat-item:last-child {
margin-bottom: 0;
}
.dataset-stat-label {
font-weight: bold;
color: #000000;
}
.dataset-preview-box {
background: #f0f0f0;
border: 2px inset #c0c0c0;
padding: 8px;
font-family: 'Courier New', monospace;
font-size: 10px;
overflow: auto;
max-height: 200px;
margin: 0;
}
.dataset-actions {
display: flex;
gap: 8px;
margin-top: 16px;
}
/* Dataset Form */
.dataset-form {
padding: 16px;
height: 100%;
overflow-y: auto;
}
.dataset-form-header {
font-size: 14px;
font-weight: bold;
margin-bottom: 16px;
color: #000000;
}
.dataset-form-group {
margin-bottom: 12px;
}
.dataset-form-label {
display: block;
font-size: 11px;
font-weight: bold;
margin-bottom: 4px;
color: #000000;
}
.dataset-toggle-row {
display: flex;
gap: 24px;
padding: 8px;
background: #f0f0f0;
border: 1px inset #c0c0c0;
flex-wrap: wrap;
}
.dataset-toggle-section {
display: flex;
align-items: center;
gap: 8px;
}
.dataset-toggle-label {
font-size: 10px;
color: #000000;
}
.dataset-toggle-group {
display: flex;
}
.dataset-toggle-btn {
background: #c0c0c0;
border: 1px solid #808080;
color: #000000;
padding: 2px 8px;
cursor: pointer;
font-size: 10px;
font-family: 'MS Sans Serif', Tahoma, sans-serif;
height: 20px;
box-sizing: border-box;
}
.dataset-toggle-btn:not(:first-child) {
border-left: none;
}
.dataset-toggle-btn:hover:not(.active) {
background: #d4d0c8;
}
.dataset-toggle-btn:active {
background: #316ac5;
color: #ffffff;
}
.dataset-toggle-btn.active {
background: #316ac5;
color: #ffffff;
border-top: 1px solid #0a246a;
border-left: 1px solid #0a246a;
border-bottom: 1px solid #4a7ac5;
border-right: 1px solid #4a7ac5;
}
.dataset-toggle-btn.active:not(:first-child) {
border-left: 1px solid #0a246a;
}
.dataset-format-hint {
font-size: 10px;
color: #606060;
font-style: italic;
margin-bottom: 4px;
padding: 4px;
background: #fffacd;
border: 1px solid #e0e0a0;
}
.dataset-form-error {
color: #ff0000;
font-size: 11px;
margin-bottom: 12px;
min-height: 16px;
}
.dataset-form-actions {
display: flex;
gap: 8px;
margin-top: 16px;
}
/* Modal Buttons */
.modal-btn {
background: #c0c0c0;
border: 2px outset #c0c0c0;
color: #000000;
padding: 6px 12px;
cursor: pointer;
font-size: 11px;
font-family: 'MS Sans Serif', Tahoma, sans-serif;
}
.modal-btn:hover {
background: #d4d0c8;
}
.modal-btn:active {
border: 2px inset #c0c0c0;
}
.modal-btn.primary {
background: #90ee90;
}
.modal-btn.primary:hover {
background: #a0ffa0;
}
.modal-btn.delete-btn {
background: #ff8080;
border: 2px outset #ff8080;
}
.modal-btn.delete-btn:hover {
background: #ff9999;
}
.modal-btn.delete-btn:active {
border: 2px inset #ff8080;
}