From 615c2d7f985455eb06e3b6551d8ea12b306ffa5d Mon Sep 17 00:00:00 2001 From: Oleh Omelchenko Date: Mon, 13 Oct 2025 18:17:09 +0300 Subject: [PATCH] Add Dataset Manager functionality with IndexedDB support - Introduced a new modal for managing datasets, including options to create, view, and delete datasets. - Implemented IndexedDB for persistent storage of datasets, allowing for efficient data retrieval and management. - Added UI components for dataset details, including statistics and preview. - Enhanced the app's JavaScript to handle dataset operations such as saving, updating, and deleting datasets. - Integrated dataset reference resolution in the visualization editor to support dynamic data sources. - Updated styles for the new modal and dataset management UI for improved user experience. --- CLAUDE.md | 23 +- docs/dev-plan.md | 102 ++++-- index.html | 134 ++++++++ src/js/app.js | 64 ++++ src/js/dataset-manager.js | 636 ++++++++++++++++++++++++++++++++++++++ src/js/editor.js | 78 ++++- src/styles.css | 404 ++++++++++++++++++++++++ 7 files changed, 1415 insertions(+), 26 deletions(-) create mode 100644 src/js/dataset-manager.js diff --git a/CLAUDE.md b/CLAUDE.md index 1778add..04714dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,13 +9,26 @@ Instructions for Claude Code when working on this project. ## Architecture - **Frontend-only**: HTML/CSS/JavaScript with CDN dependencies (Monaco Editor, Vega-Embed) -- **Storage**: localStorage with Phase 0 schema (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta) -- **Structure**: Three resizable panels (snippet library, Monaco editor, live preview) -- **No build tools**: Open `index.html` directly in browser +- **Storage**: + - **Snippets**: localStorage with Phase 0 schema (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta) + - **Datasets**: IndexedDB (unlimited size, multi-format: JSON/CSV/TSV/TopoJSON, inline & URL sources) +- **Structure**: Three resizable panels (snippet library, Monaco editor, live preview) + Dataset Manager modal +- **No build tools**: Open `index.html` directly in browser (needs local server for IndexedDB) ## Current Status -**Completed**: Phases 0-9 (All core functionality including import/export) -**Next**: Phase 10 - Dataset Management +**Completed**: Phases 0-10 (Core functionality + Dataset Management) +**Next**: Phase 11 - Advanced Dataset Features (optional enhancements) + +### Key Features Implemented +- ✅ Snippet management with draft/published workflow +- ✅ Multi-field sorting and real-time search +- ✅ Storage monitoring and import/export +- ✅ **Dataset management with IndexedDB** + - Multi-format support (JSON, CSV, TSV, TopoJSON) + - Multi-source support (inline data, URL references) + - Automatic metadata calculation and URL fetching + - Dataset reference resolution in Vega-Lite specs + - Modal UI with button-group selectors See `docs/dev-plan.md` for complete roadmap and technical details. diff --git a/docs/dev-plan.md b/docs/dev-plan.md index b8dc88c..9f431d5 100644 --- a/docs/dev-plan.md +++ b/docs/dev-plan.md @@ -189,31 +189,79 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu --- -### **Phase 10: Dataset Management - Part 1** -**Goal**: Separate dataset storage infrastructure +### **Phase 10: Dataset Management** ✅ **COMPLETE** +**Goal**: Separate dataset storage with multiple data sources and formats -- [ ] Implement dataset storage schema from Phase 0 -- [ ] Create dataset CRUD operations -- [ ] Add dataset library panel/modal -- [ ] List all stored datasets with metadata -- [ ] Add/delete/rename datasets -- [ ] Display dataset size and row counts +**Deliverables**: +- IndexedDB-based dataset storage (separate from snippets, no 5MB localStorage limit) +- Full CRUD operations for datasets (create, read, update, delete) +- Modal-based Dataset Manager UI (accessible via header link and 📁 toggle button) +- Support for multiple data sources: + - **Inline data**: JSON, CSV, TSV, TopoJSON stored directly + - **URL data**: Remote data sources with format specification +- Automatic metadata calculation: + - Row count, column count, column names + - Data size in bytes + - Automatic URL fetching on dataset creation +- Refresh metadata button for URL datasets (🔄) +- Dataset list with informative metadata display +- Dataset details panel with: + - Editable name and comment + - Statistics display (rows, columns, size) + - Data preview (first 5 rows or URL info) + - Copy reference button (copies `"data": {"name": "dataset-name"}`) + - Delete button with confirmation +- Auto-resolve dataset references in Vega-Lite specs during rendering +- Format-aware rendering: + - JSON: `{ values: data }` + - CSV/TSV: `{ values: data, format: { type: 'csv'/'tsv' } }` + - TopoJSON: `{ values: data, format: { type: 'topojson' } }` + - URL: `{ url: "...", format: { type: '...' } }` +- Button-group UI for source/format selection (matches editor style) +- Graceful error handling for CORS and network failures +- Modal scrolling support for small viewports -**Deliverable**: Basic dataset storage separate from snippets +**Dataset Schema**: +```javascript +{ + id: number, + name: string, + created: ISO string, + modified: ISO string, + data: array/string/object, // Inline data or URL string + format: 'json'|'csv'|'tsv'|'topojson', + source: 'inline'|'url', + comment: string, + rowCount: number|null, + columnCount: number|null, + columns: string[], + size: number|null // bytes +} +``` + +**Technical Implementation**: +- IndexedDB with keyPath 'id', indexes on 'name' (unique) and 'modified' +- Async/Promise-based DatasetStorage API +- Format-specific parsing for metadata calculation +- Vega-Lite's native format parsers used for rendering +- Metadata refresh fetches live data and updates statistics +- Modal resizes with viewport (max-height: 90vh) --- -### **Phase 11: Dataset Management - Part 2** -**Goal**: Reference datasets from specs +### **Phase 11: Advanced Dataset Features** _(Future)_ +**Goal**: Enhanced dataset workflows - [ ] Detect inline data in Vega-Lite specs - [ ] "Extract to dataset" feature for inline data -- [ ] Replace inline data with dataset references -- [ ] Auto-resolve dataset references when rendering - [ ] Update snippet UI to show linked datasets -- [ ] Handle missing dataset references gracefully +- [ ] Dataset usage tracking (which snippets reference which datasets) +- [ ] Import datasets from file upload +- [ ] Export individual datasets +- [ ] Dataset preview with table view +- [ ] Column type detection and display -**Deliverable**: Specs can reference shared datasets +**Deliverable**: Advanced dataset management and discovery --- @@ -280,8 +328,9 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu │ └── js/ # Modular JavaScript organization │ ├── config.js # Global variables, settings, & sample data │ ├── snippet-manager.js # Snippet storage, CRUD operations & localStorage wrapper +│ ├── dataset-manager.js # Dataset storage, CRUD operations & IndexedDB wrapper │ ├── panel-manager.js # Panel resize, toggle & memory system -│ ├── editor.js # Monaco Editor initialization & Vega-Lite rendering +│ ├── editor.js # Monaco Editor initialization, dataset resolution & Vega-Lite rendering │ └── app.js # Application initialization & event handlers └── docs/ ├── dev-plan.md # This development roadmap @@ -311,15 +360,15 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu ## Current Status -**Completed**: Phases 0-9 (Storage, UI, editor, rendering, persistence, CRUD, organization, draft/published workflow, storage monitoring, import/export) -**Next**: Phase 10 - Dataset Management +**Completed**: Phases 0-10 (Storage, UI, editor, rendering, persistence, CRUD, organization, draft/published workflow, storage monitoring, import/export, dataset management) +**Next**: Phase 11 - Advanced Dataset Features (optional enhancements) **See**: `CLAUDE.md` for concise current state summary --- ## Implemented Features -### Core Capabilities (Phases 0-9) +### Core Capabilities (Phases 0-10) - Three-panel resizable layout with memory and persistence - Monaco Editor v0.47.0 with Vega-Lite v5 schema validation - Live Vega-Lite rendering with debounced updates and error display @@ -334,6 +383,15 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu - Storage monitoring with visual progress bar and warning states - Export/Import functionality with format auto-detection - Snippet size display (right-aligned, shown for ≥ 1 KB) +- **Dataset Management (Phase 10)**: + - IndexedDB storage for datasets (unlimited size) + - Multi-format support: JSON, CSV, TSV, TopoJSON + - Multi-source support: Inline data and URL references + - Modal-based Dataset Manager with full CRUD + - Automatic metadata calculation and display + - URL metadata fetching and refresh + - Dataset reference resolution in Vega-Lite specs + - Button-group UI for source/format selection - Retro Windows 2000 aesthetic throughout ### Technical Implementation @@ -348,4 +406,8 @@ Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visu - **Storage Calculation**: Blob API for accurate byte counting of snippet data - **Flexbox Layout**: Scrollable snippet list with fixed metadata and storage monitor at bottom - **Import/Export**: Format detection, field normalization, ID conflict resolution, additive merging -- **Size Display**: Per-snippet size calculation with conditional rendering (≥ 1 KB threshold) \ No newline at end of file +- **Size Display**: Per-snippet size calculation with conditional rendering (≥ 1 KB threshold) +- **Dataset Storage**: IndexedDB with async/Promise-based API, unique name constraint +- **Dataset Resolution**: Async spec transformation before rendering, format-aware data injection +- **URL Metadata**: Fetch on creation with graceful CORS error handling +- **Modal UI**: Flexbox with overflow:auto, max-height responsive to viewport \ No newline at end of file diff --git a/index.html b/index.html index b6ff1ed..d783e27 100644 --- a/index.html +++ b/index.html @@ -21,6 +21,7 @@
Import Export + Datasets Help
@@ -38,6 +39,9 @@ +
@@ -144,8 +148,138 @@
+ + + + diff --git a/src/js/app.js b/src/js/app.js index d8e17de..2c13f7c 100644 --- a/src/js/app.js +++ b/src/js/app.js @@ -117,6 +117,70 @@ document.addEventListener('DOMContentLoaded', function () { }); } + // Dataset Manager + const datasetsLink = document.getElementById('datasets-link'); + const toggleDatasetsBtn = document.getElementById('toggle-datasets'); + const datasetModal = document.getElementById('dataset-modal'); + const datasetModalClose = document.getElementById('dataset-modal-close'); + const newDatasetBtn = document.getElementById('new-dataset-btn'); + const cancelDatasetBtn = document.getElementById('cancel-dataset-btn'); + const saveDatasetBtn = document.getElementById('save-dataset-btn'); + const deleteDatasetBtn = document.getElementById('delete-dataset-btn'); + const copyReferenceBtn = document.getElementById('copy-reference-btn'); + + // Open dataset manager + if (datasetsLink) { + datasetsLink.addEventListener('click', openDatasetManager); + } + if (toggleDatasetsBtn) { + toggleDatasetsBtn.addEventListener('click', openDatasetManager); + } + + // Close dataset manager + if (datasetModalClose) { + datasetModalClose.addEventListener('click', closeDatasetManager); + } + + // Close on overlay click + if (datasetModal) { + datasetModal.addEventListener('click', function (e) { + if (e.target === datasetModal) { + closeDatasetManager(); + } + }); + } + + // New dataset button + if (newDatasetBtn) { + newDatasetBtn.addEventListener('click', showNewDatasetForm); + } + + // Cancel dataset button + if (cancelDatasetBtn) { + cancelDatasetBtn.addEventListener('click', hideNewDatasetForm); + } + + // Save dataset button + if (saveDatasetBtn) { + saveDatasetBtn.addEventListener('click', saveNewDataset); + } + + // Delete dataset button + if (deleteDatasetBtn) { + deleteDatasetBtn.addEventListener('click', deleteCurrentDataset); + } + + // Copy reference button + if (copyReferenceBtn) { + copyReferenceBtn.addEventListener('click', copyDatasetReference); + } + + // Refresh metadata button + const refreshMetadataBtn = document.getElementById('refresh-metadata-btn'); + if (refreshMetadataBtn) { + refreshMetadataBtn.addEventListener('click', refreshDatasetMetadata); + } + // View mode toggle buttons document.getElementById('view-draft').addEventListener('click', () => { switchViewMode('draft'); diff --git a/src/js/dataset-manager.js b/src/js/dataset-manager.js new file mode 100644 index 0000000..77e89ba --- /dev/null +++ b/src/js/dataset-manager.js @@ -0,0 +1,636 @@ +// Dataset management with IndexedDB + +const DB_NAME = 'astrolabe-datasets'; +const DB_VERSION = 1; +const STORE_NAME = 'datasets'; + +let db = null; + +// Initialize IndexedDB +function initializeDatasetDB() { + return new Promise((resolve, reject) => { + const request = indexedDB.open(DB_NAME, DB_VERSION); + + request.onerror = () => reject(request.error); + request.onsuccess = () => { + db = request.result; + resolve(db); + }; + + request.onupgradeneeded = (event) => { + const db = event.target.result; + + // Create object store if it doesn't exist + if (!db.objectStoreNames.contains(STORE_NAME)) { + const objectStore = db.createObjectStore(STORE_NAME, { keyPath: 'id' }); + objectStore.createIndex('name', 'name', { unique: true }); + objectStore.createIndex('modified', 'modified', { unique: false }); + } + }; + }); +} + +// Generate unique ID +function generateDatasetId() { + return Date.now() + Math.random() * 1000; +} + +// Calculate dataset statistics +function calculateDatasetStats(data, format, source) { + let rowCount = 0; + let columnCount = 0; + let columns = []; + let size = 0; + + // For URL sources, we can't calculate stats without fetching + if (source === 'url') { + return { rowCount: null, columnCount: null, columns: [], size: null }; + } + + if (format === 'json' || format === 'topojson') { + if (!Array.isArray(data) || data.length === 0) { + return { rowCount: 0, columnCount: 0, columns: [], size: 0 }; + } + rowCount = data.length; + const firstRow = data[0]; + columns = typeof firstRow === 'object' ? Object.keys(firstRow) : []; + columnCount = columns.length; + size = new Blob([JSON.stringify(data)]).size; + } else if (format === 'csv' || format === 'tsv') { + // For CSV/TSV, data is stored as raw text + const lines = data.trim().split('\n'); + rowCount = Math.max(0, lines.length - 1); // Subtract header row + if (lines.length > 0) { + const separator = format === 'csv' ? ',' : '\t'; + columns = lines[0].split(separator).map(h => h.trim().replace(/^"|"$/g, '')); + columnCount = columns.length; + } + size = new Blob([data]).size; + } + + return { rowCount, columnCount, columns, size }; +} + +// Dataset Storage API +const DatasetStorage = { + // Initialize database + async init() { + if (!db) { + await initializeDatasetDB(); + } + return db; + }, + + // Create new dataset + async createDataset(name, data, format, source, comment = '') { + await this.init(); + + const now = new Date().toISOString(); + const stats = calculateDatasetStats(data, format, source); + + const dataset = { + id: generateDatasetId(), + name: name.trim(), + created: now, + modified: now, + data: data, // For inline: actual data, for URL: the URL string + format: format, // 'json', 'csv', 'tsv', or 'topojson' + source: source, // 'inline' or 'url' + comment: comment.trim(), + ...stats + }; + + return new Promise((resolve, reject) => { + const transaction = db.transaction([STORE_NAME], 'readwrite'); + const store = transaction.objectStore(STORE_NAME); + const request = store.add(dataset); + + request.onsuccess = () => resolve(dataset); + request.onerror = () => reject(request.error); + }); + }, + + // Get all datasets + async listDatasets() { + await this.init(); + + return new Promise((resolve, reject) => { + const transaction = db.transaction([STORE_NAME], 'readonly'); + const store = transaction.objectStore(STORE_NAME); + const request = store.getAll(); + + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); + }, + + // Get single dataset by ID + async getDataset(id) { + await this.init(); + + return new Promise((resolve, reject) => { + const transaction = db.transaction([STORE_NAME], 'readonly'); + const store = transaction.objectStore(STORE_NAME); + const request = store.get(id); + + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); + }, + + // Get dataset by name + async getDatasetByName(name) { + await this.init(); + + return new Promise((resolve, reject) => { + const transaction = db.transaction([STORE_NAME], 'readonly'); + const store = transaction.objectStore(STORE_NAME); + const index = store.index('name'); + const request = index.get(name); + + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); + }, + + // Update dataset + async updateDataset(id, updates) { + await this.init(); + + const dataset = await this.getDataset(id); + if (!dataset) { + throw new Error('Dataset not found'); + } + + // Update fields + if (updates.name !== undefined) dataset.name = updates.name.trim(); + if (updates.data !== undefined) { + dataset.data = updates.data; + if (updates.format !== undefined) dataset.format = updates.format; + if (updates.source !== undefined) dataset.source = updates.source; + + // If metadata fields are explicitly provided, use them directly + if (updates.rowCount !== undefined) dataset.rowCount = updates.rowCount; + if (updates.columnCount !== undefined) dataset.columnCount = updates.columnCount; + if (updates.columns !== undefined) dataset.columns = updates.columns; + if (updates.size !== undefined) dataset.size = updates.size; + + // Otherwise, calculate stats (for inline data) + if (updates.rowCount === undefined && updates.columnCount === undefined) { + const stats = calculateDatasetStats(updates.data, dataset.format, dataset.source); + Object.assign(dataset, stats); + } + } + if (updates.comment !== undefined) dataset.comment = updates.comment.trim(); + + dataset.modified = new Date().toISOString(); + + return new Promise((resolve, reject) => { + const transaction = db.transaction([STORE_NAME], 'readwrite'); + const store = transaction.objectStore(STORE_NAME); + const request = store.put(dataset); + + request.onsuccess = () => resolve(dataset); + request.onerror = () => reject(request.error); + }); + }, + + // Delete dataset + async deleteDataset(id) { + await this.init(); + + return new Promise((resolve, reject) => { + const transaction = db.transaction([STORE_NAME], 'readwrite'); + const store = transaction.objectStore(STORE_NAME); + const request = store.delete(id); + + request.onsuccess = () => resolve(true); + request.onerror = () => reject(request.error); + }); + }, + + // Check if dataset name exists + async nameExists(name, excludeId = null) { + const datasets = await this.listDatasets(); + return datasets.some(d => d.name === name && d.id !== excludeId); + } +}; + +// Format bytes for display +function formatDatasetSize(bytes) { + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`; + return `${(bytes / (1024 * 1024)).toFixed(2)} MB`; +} + +// Fetch URL data and calculate metadata +async function fetchURLMetadata(url, format) { + try { + const response = await fetch(url); + if (!response.ok) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); + } + + const contentLength = response.headers.get('content-length'); + const text = await response.text(); + + let rowCount = 0; + let columnCount = 0; + let columns = []; + let size = contentLength ? parseInt(contentLength) : new Blob([text]).size; + + // Parse based on format + if (format === 'json') { + const data = JSON.parse(text); + if (Array.isArray(data)) { + rowCount = data.length; + if (data.length > 0 && typeof data[0] === 'object') { + columns = Object.keys(data[0]); + columnCount = columns.length; + } + } + } else if (format === 'csv' || format === 'tsv') { + const lines = text.trim().split('\n'); + rowCount = Math.max(0, lines.length - 1); // Subtract header + if (lines.length > 0) { + const separator = format === 'csv' ? ',' : '\t'; + columns = lines[0].split(separator).map(h => h.trim().replace(/^"|"$/g, '')); + columnCount = columns.length; + } + } else if (format === 'topojson') { + // TopoJSON structure is complex, just note it exists + rowCount = null; + columnCount = null; + } + + return { rowCount, columnCount, columns, size }; + } catch (error) { + throw new Error(`Failed to fetch URL metadata: ${error.message}`); + } +} + +// Render dataset list in modal +async function renderDatasetList() { + const datasets = await DatasetStorage.listDatasets(); + const listContainer = document.getElementById('dataset-list'); + + if (datasets.length === 0) { + listContainer.innerHTML = '
No datasets yet. Click "New Dataset" to create one.
'; + return; + } + + // Sort by modified date (most recent first) + datasets.sort((a, b) => new Date(b.modified) - new Date(a.modified)); + + const html = datasets.map(dataset => { + let metaText; + if (dataset.source === 'url') { + // Show metadata if available, otherwise just URL and format + if (dataset.rowCount !== null && dataset.size !== null) { + metaText = `URL • ${dataset.rowCount} rows • ${dataset.format.toUpperCase()} • ${formatDatasetSize(dataset.size)}`; + } else { + metaText = `URL • ${dataset.format.toUpperCase()}`; + } + } else { + metaText = `${dataset.rowCount} rows • ${dataset.format.toUpperCase()} • ${formatDatasetSize(dataset.size)}`; + } + + return ` +
+
+
${dataset.name}
+
${metaText}
+
+
+ `; + }).join(''); + + listContainer.innerHTML = html; + + // Attach click handlers + document.querySelectorAll('.dataset-item').forEach(item => { + item.addEventListener('click', function() { + const datasetId = parseFloat(this.dataset.datasetId); + selectDataset(datasetId); + }); + }); +} + +// Select a dataset and show details +async function selectDataset(datasetId) { + const dataset = await DatasetStorage.getDataset(datasetId); + if (!dataset) return; + + // Update selection state + document.querySelectorAll('.dataset-item').forEach(item => { + item.classList.remove('selected'); + }); + document.querySelector(`[data-dataset-id="${datasetId}"]`).classList.add('selected'); + + // Show details panel + const detailsPanel = document.getElementById('dataset-details'); + detailsPanel.style.display = 'block'; + + // Show/hide refresh button for URL datasets + const refreshBtn = document.getElementById('refresh-metadata-btn'); + if (dataset.source === 'url') { + refreshBtn.style.display = 'flex'; + } else { + refreshBtn.style.display = 'none'; + } + + // Populate details + document.getElementById('dataset-detail-name').value = dataset.name; + document.getElementById('dataset-detail-comment').value = dataset.comment; + document.getElementById('dataset-detail-rows').textContent = dataset.rowCount !== null ? dataset.rowCount : 'N/A'; + document.getElementById('dataset-detail-columns').textContent = dataset.columnCount !== null ? dataset.columnCount : 'N/A'; + document.getElementById('dataset-detail-size').textContent = dataset.size !== null ? formatDatasetSize(dataset.size) : 'N/A'; + document.getElementById('dataset-detail-created').textContent = new Date(dataset.created).toLocaleString(); + document.getElementById('dataset-detail-modified').textContent = new Date(dataset.modified).toLocaleString(); + + // Show preview + let previewText; + if (dataset.source === 'url') { + previewText = `URL: ${dataset.data}\nFormat: ${dataset.format.toUpperCase()}`; + } else if (dataset.format === 'json' || dataset.format === 'topojson') { + const previewData = Array.isArray(dataset.data) ? dataset.data.slice(0, 5) : dataset.data; + previewText = JSON.stringify(previewData, null, 2); + } else if (dataset.format === 'csv' || dataset.format === 'tsv') { + const lines = dataset.data.split('\n'); + previewText = lines.slice(0, 6).join('\n'); // Header + 5 rows + } + document.getElementById('dataset-preview').textContent = previewText; + + // Store current dataset ID + window.currentDatasetId = datasetId; +} + +// Open dataset manager modal +function openDatasetManager() { + const modal = document.getElementById('dataset-modal'); + modal.style.display = 'flex'; + renderDatasetList(); +} + +// Close dataset manager modal +function closeDatasetManager() { + const modal = document.getElementById('dataset-modal'); + modal.style.display = 'none'; + window.currentDatasetId = null; +} + +// Update format hint and placeholder +function updateFormatHint(format) { + const hintEl = document.getElementById('dataset-format-hint'); + const dataEl = document.getElementById('dataset-form-data'); + + if (format === 'json') { + hintEl.textContent = 'JSON array of objects: [{"col1": "value", "col2": 123}, ...]'; + dataEl.placeholder = '[{"col1": "value", "col2": 123}, ...]'; + } else if (format === 'csv') { + hintEl.textContent = 'CSV with header row: col1,col2\\nvalue1,123\\nvalue2,456'; + dataEl.placeholder = 'col1,col2\nvalue1,123\nvalue2,456'; + } else if (format === 'tsv') { + hintEl.textContent = 'TSV with header row: col1\\tcol2\\nvalue1\\t123\\nvalue2\\t456'; + dataEl.placeholder = 'col1\tcol2\nvalue1\t123\nvalue2\t456'; + } else if (format === 'topojson') { + hintEl.textContent = 'TopoJSON object: {"type": "Topology", "objects": {...}, "arcs": [...]}'; + dataEl.placeholder = '{"type": "Topology", "objects": {...}}'; + } +} + +// Toggle between URL and inline data inputs +function toggleDataSource(source) { + const urlGroup = document.getElementById('dataset-url-group'); + const dataGroup = document.getElementById('dataset-data-group'); + + if (source === 'url') { + urlGroup.style.display = 'block'; + dataGroup.style.display = 'none'; + } else { + urlGroup.style.display = 'none'; + dataGroup.style.display = 'block'; + } +} + +// Show new dataset form +function showNewDatasetForm() { + document.getElementById('dataset-list-view').style.display = 'none'; + document.getElementById('dataset-form-view').style.display = 'block'; + document.getElementById('dataset-form-name').value = ''; + document.getElementById('dataset-form-data').value = ''; + document.getElementById('dataset-form-url').value = ''; + document.getElementById('dataset-form-comment').value = ''; + document.getElementById('dataset-form-error').textContent = ''; + + // Reset to inline data source and JSON format + document.querySelectorAll('[data-source]').forEach(btn => { + btn.classList.toggle('active', btn.dataset.source === 'inline'); + }); + document.querySelectorAll('[data-format]').forEach(btn => { + btn.classList.toggle('active', btn.dataset.format === 'json'); + }); + toggleDataSource('inline'); + updateFormatHint('json'); + + // Add listeners if not already added + if (!window.datasetListenersAdded) { + // Source toggle button listeners + document.querySelectorAll('[data-source]').forEach(btn => { + btn.addEventListener('click', function () { + // Update active state + document.querySelectorAll('[data-source]').forEach(b => b.classList.remove('active')); + this.classList.add('active'); + toggleDataSource(this.dataset.source); + }); + }); + + // Format toggle button listeners + document.querySelectorAll('[data-format]').forEach(btn => { + btn.addEventListener('click', function () { + // Update active state + document.querySelectorAll('[data-format]').forEach(b => b.classList.remove('active')); + this.classList.add('active'); + updateFormatHint(this.dataset.format); + }); + }); + + window.datasetListenersAdded = true; + } +} + +// Hide new dataset form +function hideNewDatasetForm() { + document.getElementById('dataset-list-view').style.display = 'block'; + document.getElementById('dataset-form-view').style.display = 'none'; +} + +// Save new dataset +async function saveNewDataset() { + const name = document.getElementById('dataset-form-name').value.trim(); + const source = document.querySelector('[data-source].active').dataset.source; + const format = document.querySelector('[data-format].active').dataset.format; + const comment = document.getElementById('dataset-form-comment').value.trim(); + const errorEl = document.getElementById('dataset-form-error'); + + errorEl.textContent = ''; + + // Validation + if (!name) { + errorEl.textContent = 'Dataset name is required'; + return; + } + + let data; + let metadata = null; + + if (source === 'url') { + const url = document.getElementById('dataset-form-url').value.trim(); + if (!url) { + errorEl.textContent = 'URL is required'; + return; + } + // Basic URL validation + try { + new URL(url); + } catch (error) { + errorEl.textContent = 'Invalid URL format'; + return; + } + + // Fetch metadata from URL + errorEl.textContent = 'Fetching data from URL...'; + try { + metadata = await fetchURLMetadata(url, format); + errorEl.textContent = ''; + } catch (error) { + errorEl.textContent = `Warning: ${error.message}. Dataset will be created without metadata.`; + // Continue anyway - URL might require CORS or auth + await new Promise(resolve => setTimeout(resolve, 2000)); // Show warning briefly + errorEl.textContent = ''; + } + + data = url; // Store the URL string + } else { + // Inline data + const dataText = document.getElementById('dataset-form-data').value.trim(); + if (!dataText) { + errorEl.textContent = 'Data is required'; + return; + } + + // Basic validation of data format + try { + if (format === 'json' || format === 'topojson') { + const parsed = JSON.parse(dataText); + if (format === 'json' && !Array.isArray(parsed)) { + errorEl.textContent = 'JSON data must be an array of objects'; + return; + } + if (format === 'json' && parsed.length === 0) { + errorEl.textContent = 'Data array cannot be empty'; + return; + } + data = parsed; // Store as parsed JSON + } else if (format === 'csv' || format === 'tsv') { + const lines = dataText.trim().split('\n'); + if (lines.length < 2) { + errorEl.textContent = `${format.toUpperCase()} must have at least a header row and one data row`; + return; + } + data = dataText; // Store as raw CSV/TSV string + } + } catch (error) { + errorEl.textContent = `Validation error: ${error.message}`; + return; + } + } + + // Check if name already exists + if (await DatasetStorage.nameExists(name)) { + errorEl.textContent = 'A dataset with this name already exists'; + return; + } + + // Create dataset + try { + const dataset = await DatasetStorage.createDataset(name, data, format, source, comment); + + // If we have metadata from URL fetch, update the dataset + if (metadata) { + await DatasetStorage.updateDataset(dataset.id, { + data: data, + ...metadata + }); + } + + hideNewDatasetForm(); + await renderDatasetList(); + } catch (error) { + errorEl.textContent = `Failed to save dataset: ${error.message}`; + } +} + +// Delete current dataset +async function deleteCurrentDataset() { + if (!window.currentDatasetId) return; + + const dataset = await DatasetStorage.getDataset(window.currentDatasetId); + if (!dataset) return; + + if (confirm(`Delete dataset "${dataset.name}"? This action cannot be undone.`)) { + await DatasetStorage.deleteDataset(window.currentDatasetId); + document.getElementById('dataset-details').style.display = 'none'; + window.currentDatasetId = null; + await renderDatasetList(); + } +} + +// Copy dataset reference to clipboard +function copyDatasetReference() { + if (!window.currentDatasetId) return; + + DatasetStorage.getDataset(window.currentDatasetId).then(dataset => { + const reference = `"data": {"name": "${dataset.name}"}`; + navigator.clipboard.writeText(reference).then(() => { + alert('Dataset reference copied to clipboard!'); + }); + }); +} + +// Refresh metadata for URL dataset +async function refreshDatasetMetadata() { + if (!window.currentDatasetId) return; + + const dataset = await DatasetStorage.getDataset(window.currentDatasetId); + if (!dataset || dataset.source !== 'url') return; + + const refreshBtn = document.getElementById('refresh-metadata-btn'); + refreshBtn.disabled = true; + refreshBtn.textContent = '⏳'; + + try { + const metadata = await fetchURLMetadata(dataset.data, dataset.format); + + // Update dataset with new metadata + await DatasetStorage.updateDataset(dataset.id, { + data: dataset.data, + ...metadata + }); + + // Refresh the display + await selectDataset(dataset.id); + await renderDatasetList(); + + // Brief success indicator + refreshBtn.textContent = '✓'; + setTimeout(() => { + refreshBtn.textContent = '🔄'; + refreshBtn.disabled = false; + }, 1000); + } catch (error) { + alert(`Failed to refresh metadata: ${error.message}`); + refreshBtn.textContent = '🔄'; + refreshBtn.disabled = false; + } +} diff --git a/src/js/editor.js b/src/js/editor.js index 2ea6129..a61b308 100644 --- a/src/js/editor.js +++ b/src/js/editor.js @@ -1,3 +1,76 @@ +// Resolve dataset references in a spec +async function resolveDatasetReferences(spec) { + // If spec has data.name, look it up + if (spec.data && spec.data.name && typeof spec.data.name === 'string') { + const datasetName = spec.data.name; + const dataset = await DatasetStorage.getDatasetByName(datasetName); + + if (dataset) { + // Replace data reference with actual data in the format Vega-Lite expects + if (dataset.source === 'url') { + // For URL sources, pass the URL and format + spec.data = { + url: dataset.data, + format: { type: dataset.format } + }; + } else { + // For inline sources + if (dataset.format === 'json') { + spec.data = { values: dataset.data }; + } else if (dataset.format === 'csv') { + spec.data = { + values: dataset.data, + format: { type: 'csv' } + }; + } else if (dataset.format === 'tsv') { + spec.data = { + values: dataset.data, + format: { type: 'tsv' } + }; + } else if (dataset.format === 'topojson') { + spec.data = { + values: dataset.data, + format: { type: 'topojson' } + }; + } + } + } else { + throw new Error(`Dataset "${datasetName}" not found`); + } + } + + // Recursively resolve in layers (for layered specs) + if (spec.layer && Array.isArray(spec.layer)) { + for (let layer of spec.layer) { + await resolveDatasetReferences(layer); + } + } + + // Recursively resolve in concat/hconcat/vconcat + if (spec.concat && Array.isArray(spec.concat)) { + for (let view of spec.concat) { + await resolveDatasetReferences(view); + } + } + if (spec.hconcat && Array.isArray(spec.hconcat)) { + for (let view of spec.hconcat) { + await resolveDatasetReferences(view); + } + } + if (spec.vconcat && Array.isArray(spec.vconcat)) { + for (let view of spec.vconcat) { + await resolveDatasetReferences(view); + } + } + + // Recursively resolve in facet + if (spec.spec) { + await resolveDatasetReferences(spec.spec); + } + + return spec; +} + // Render function that takes spec from editor async function renderVisualization() { const previewContainer = document.getElementById('vega-preview'); @@ -5,7 +78,10 @@ async function renderVisualization() { try { // Get current content from editor const specText = editor.getValue(); - const spec = JSON.parse(specText); + let spec = JSON.parse(specText); + + // Resolve dataset references + spec = await resolveDatasetReferences(spec); // Render with Vega-Embed (use global variable) await window.vegaEmbed('#vega-preview', spec, { diff --git a/src/styles.css b/src/styles.css index 3b6d33d..2a181b4 100644 --- a/src/styles.css +++ b/src/styles.css @@ -661,4 +661,408 @@ body { .storage-fill.critical { background: #ff0000; +} + +/* Modal Styles */ +.modal { + position: fixed; + top: 0; + left: 0; + width: 100vw; + height: 100vh; + background: rgba(0, 0, 0, 0.5); + display: flex; + align-items: center; + justify-content: center; + z-index: 1000; +} + +.modal-content { + background: #c0c0c0; + border: 2px outset #c0c0c0; + width: 90%; + max-width: 900px; + max-height: 90vh; + display: flex; + flex-direction: column; + box-shadow: 4px 4px 8px rgba(0, 0, 0, 0.3); +} + +.modal-header { + background: #000080; + color: #ffffff; + padding: 4px 8px; + display: flex; + justify-content: space-between; + align-items: center; + height: 24px; + border-bottom: 2px solid #808080; +} + +.modal-title { + font-size: 12px; + font-weight: bold; +} + +.modal-close { + background: #c0c0c0; + border: 2px outset #c0c0c0; + color: #000000; + width: 20px; + height: 20px; + cursor: pointer; + font-size: 14px; + font-family: 'MS Sans Serif', Tahoma, sans-serif; + display: flex; + align-items: center; + justify-content: center; + padding: 0; +} + +.modal-close:hover { + background: #d4d0c8; +} + +.modal-close:active { + border: 2px inset #c0c0c0; +} + +.modal-body { + flex: 1; + overflow: auto; + background: #ffffff; + border: 2px inset #c0c0c0; + margin: 8px; + min-height: 0; +} + +/* Dataset Views */ +.dataset-view { + min-height: 100%; + display: flex; + flex-direction: column; +} + +/* List View */ +.dataset-list-header { + padding: 8px; + background: #d4d0c8; + border-bottom: 2px solid #808080; +} + +.dataset-container { + display: flex; + flex: 1; + overflow: hidden; +} + +.dataset-list { + width: 300px; + overflow-y: auto; + border-right: 2px solid #808080; + background: #ffffff; +} + +.dataset-item { + padding: 8px; + border-bottom: 1px solid #d0d0d0; + cursor: pointer; + background: #ffffff; +} + +.dataset-item:hover { + background: #6a9ad5; + color: #ffffff; +} + +.dataset-item.selected { + background: #316ac5; + color: #ffffff; +} + +.dataset-name { + font-size: 12px; + font-weight: bold; + margin-bottom: 2px; +} + +.dataset-meta { + font-size: 10px; + color: #606060; +} + +.dataset-item.selected .dataset-meta, +.dataset-item:hover .dataset-meta { + color: inherit; + opacity: 0.9; +} + +.dataset-empty { + padding: 32px; + text-align: center; + color: #808080; + font-style: italic; + font-size: 12px; +} + +/* Dataset Details */ +.dataset-details { + flex: 1; + overflow-y: auto; + background: #ffffff; +} + +.dataset-detail-section { + padding: 16px; +} + +.dataset-detail-header { + font-size: 11px; + font-weight: bold; + margin-bottom: 6px; + margin-top: 12px; + color: #000000; +} + +.dataset-detail-header:first-child { + margin-top: 0; +} + +.dataset-detail-header-row { + display: flex; + justify-content: space-between; + align-items: center; + margin-bottom: 6px; + margin-top: 12px; +} + +.dataset-refresh-btn { + background: #c0c0c0; + border: 2px outset #c0c0c0; + color: #000000; + width: 24px; + height: 24px; + cursor: pointer; + font-size: 12px; + display: flex; + align-items: center; + justify-content: center; + padding: 0; +} + +.dataset-refresh-btn:hover { + background: #d4d0c8; +} + +.dataset-refresh-btn:active { + border: 2px inset #c0c0c0; +} + +.dataset-refresh-btn:disabled { + opacity: 0.5; + cursor: default; +} + +.dataset-input, +.dataset-textarea { + width: 100%; + font-family: 'MS Sans Serif', Tahoma, sans-serif; + font-size: 11px; + border: 2px inset #c0c0c0; + padding: 4px; + background: #ffffff; +} + +.dataset-textarea { + resize: vertical; +} + +.dataset-stats { + background: #f0f0f0; + border: 1px inset #c0c0c0; + padding: 8px; + font-size: 10px; +} + +.dataset-stat-item { + display: flex; + justify-content: space-between; + margin-bottom: 4px; +} + +.dataset-stat-item:last-child { + margin-bottom: 0; +} + +.dataset-stat-label { + font-weight: bold; + color: #000000; +} + +.dataset-preview-box { + background: #f0f0f0; + border: 2px inset #c0c0c0; + padding: 8px; + font-family: 'Courier New', monospace; + font-size: 10px; + overflow: auto; + max-height: 200px; + margin: 0; +} + +.dataset-actions { + display: flex; + gap: 8px; + margin-top: 16px; +} + +/* Dataset Form */ +.dataset-form { + padding: 16px; + height: 100%; + overflow-y: auto; +} + +.dataset-form-header { + font-size: 14px; + font-weight: bold; + margin-bottom: 16px; + color: #000000; +} + +.dataset-form-group { + margin-bottom: 12px; +} + +.dataset-form-label { + display: block; + font-size: 11px; + font-weight: bold; + margin-bottom: 4px; + color: #000000; +} + +.dataset-toggle-row { + display: flex; + gap: 24px; + padding: 8px; + background: #f0f0f0; + border: 1px inset #c0c0c0; + flex-wrap: wrap; +} + +.dataset-toggle-section { + display: flex; + align-items: center; + gap: 8px; +} + +.dataset-toggle-label { + font-size: 10px; + color: #000000; +} + +.dataset-toggle-group { + display: flex; +} + +.dataset-toggle-btn { + background: #c0c0c0; + border: 1px solid #808080; + color: #000000; + padding: 2px 8px; + cursor: pointer; + font-size: 10px; + font-family: 'MS Sans Serif', Tahoma, sans-serif; + height: 20px; + box-sizing: border-box; +} + +.dataset-toggle-btn:not(:first-child) { + border-left: none; +} + +.dataset-toggle-btn:hover:not(.active) { + background: #d4d0c8; +} + +.dataset-toggle-btn:active { + background: #316ac5; + color: #ffffff; +} + +.dataset-toggle-btn.active { + background: #316ac5; + color: #ffffff; + border-top: 1px solid #0a246a; + border-left: 1px solid #0a246a; + border-bottom: 1px solid #4a7ac5; + border-right: 1px solid #4a7ac5; +} + +.dataset-toggle-btn.active:not(:first-child) { + border-left: 1px solid #0a246a; +} + +.dataset-format-hint { + font-size: 10px; + color: #606060; + font-style: italic; + margin-bottom: 4px; + padding: 4px; + background: #fffacd; + border: 1px solid #e0e0a0; +} + +.dataset-form-error { + color: #ff0000; + font-size: 11px; + margin-bottom: 12px; + min-height: 16px; +} + +.dataset-form-actions { + display: flex; + gap: 8px; + margin-top: 16px; +} + +/* Modal Buttons */ +.modal-btn { + background: #c0c0c0; + border: 2px outset #c0c0c0; + color: #000000; + padding: 6px 12px; + cursor: pointer; + font-size: 11px; + font-family: 'MS Sans Serif', Tahoma, sans-serif; +} + +.modal-btn:hover { + background: #d4d0c8; +} + +.modal-btn:active { + border: 2px inset #c0c0c0; +} + +.modal-btn.primary { + background: #90ee90; +} + +.modal-btn.primary:hover { + background: #a0ffa0; +} + +.modal-btn.delete-btn { + background: #ff8080; + border: 2px outset #ff8080; +} + +.modal-btn.delete-btn:hover { + background: #ff9999; +} + +.modal-btn.delete-btn:active { + border: 2px inset #ff8080; } \ No newline at end of file