mirror of
https://github.com/olehomelchenko/astrolabe-nvc.git
synced 2025-12-21 21:22:23 +00:00
docs: move from phase-based development
This commit is contained in:
177
CHANGELOG.md
Normal file
177
CHANGELOG.md
Normal file
@@ -0,0 +1,177 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to Astrolabe will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
---
|
||||
|
||||
## [1.0.0] - 2025-10-15
|
||||
|
||||
### Initial Release
|
||||
|
||||
Complete feature set for lightweight Vega-Lite snippet management.
|
||||
|
||||
#### Core Features
|
||||
- Three-panel resizable layout with drag handles and panel memory
|
||||
- Monaco Editor v0.47.0 with Vega-Lite v5 schema validation and autocomplete
|
||||
- Live Vega-Lite preview with debounced rendering
|
||||
- Draft/published workflow for safe experimentation
|
||||
- Auto-save system with 1-second debounce
|
||||
- Multi-field sorting (Modified/Created/Name) with direction toggle
|
||||
- Real-time search across snippet name, comment, and spec content
|
||||
- Snippet size display for large snippets (≥1KB)
|
||||
|
||||
#### Storage & Data Management
|
||||
- localStorage-based snippet storage with 5MB monitoring
|
||||
- IndexedDB-based dataset library (unlimited size)
|
||||
- Multi-format dataset support: JSON, CSV, TSV, TopoJSON
|
||||
- Multi-source dataset support: inline data and URL references
|
||||
- Automatic dataset reference resolution in Vega-Lite specs
|
||||
- Intelligent format auto-detection with confidence scoring
|
||||
- Storage usage monitor with visual warnings (90% and 95% thresholds)
|
||||
|
||||
#### Dataset Features
|
||||
- Full CRUD operations for datasets
|
||||
- Automatic metadata calculation (rows, columns, size, types)
|
||||
- URL dataset fetching with CORS error handling
|
||||
- Metadata refresh for URL datasets
|
||||
- Bidirectional snippet ↔ dataset linking with usage tracking
|
||||
- Extract inline data from specs to dataset library
|
||||
- Table preview with type detection (number 🔢, date 📅, boolean ✓, text 🔤)
|
||||
- On-demand URL preview loading with session cache
|
||||
- Create new snippet from dataset with auto-generated spec
|
||||
|
||||
#### Import/Export
|
||||
- Export all snippets to JSON with auto-generated filename
|
||||
- Import snippets with format auto-detection and field mapping
|
||||
- Export/import datasets with format-specific file extensions
|
||||
- URL dataset export fetches and downloads live content
|
||||
- Additive import (no overwrites) with ID conflict resolution
|
||||
- Automatic "imported" tag for external snippets
|
||||
|
||||
#### User Experience
|
||||
- Cross-platform keyboard shortcuts (Cmd/Ctrl+Shift+N, Cmd/Ctrl+K, Cmd/Ctrl+S, Escape)
|
||||
- Toast notification system (error, success, warning, info)
|
||||
- Comprehensive tooltips with keyboard hints
|
||||
- Enhanced Help modal with 6 sections (About, Features, Getting Started, Shortcuts, Storage, Privacy)
|
||||
- Retro Windows 2000 aesthetic throughout UI
|
||||
|
||||
#### Settings & Customization
|
||||
- Configurable editor options (font size 10-18px, tab size, minimap, word wrap, line numbers)
|
||||
- Performance tuning (render debounce delay 300-3000ms)
|
||||
- Date formatting options (smart/relative, locale, ISO, custom with tokens)
|
||||
- Theme selection (Light, Dark Experimental)
|
||||
- Automatic editor theme synchronization with UI theme
|
||||
- Settings persistence in localStorage
|
||||
|
||||
#### URL State Management
|
||||
- Hash-based routing for snippets and datasets
|
||||
- Browser back/forward navigation support
|
||||
- Page reload preserves selected snippet or dataset
|
||||
- Shareable URLs for specific snippets or datasets
|
||||
- Modal state persistence in URL
|
||||
|
||||
#### Technical Implementation
|
||||
- Vanilla JavaScript (no frameworks, no build tools)
|
||||
- AMD loader conflict resolution between Monaco and Vega
|
||||
- Recursive dataset reference extraction
|
||||
- Format-aware data injection for rendering
|
||||
- Blob API for accurate storage size calculation
|
||||
- Component-based CSS architecture with theming support
|
||||
|
||||
---
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- (Bugfixes will be listed here)
|
||||
|
||||
### Added
|
||||
- (New features will be listed here)
|
||||
|
||||
### Changed
|
||||
- (Improvements and refinements will be listed here)
|
||||
|
||||
### Removed
|
||||
- (Removed features will be listed here)
|
||||
|
||||
---
|
||||
|
||||
## Release Notes
|
||||
|
||||
### v1.0.0 - Feature-Complete MVP
|
||||
|
||||
Astrolabe v1.0 represents a complete, production-ready implementation of a browser-based Vega-Lite snippet manager. The application is fully functional with no known critical bugs.
|
||||
|
||||
**Development Timeline**: October 2024 - October 2025
|
||||
|
||||
**Key Accomplishments**:
|
||||
- Zero external dependencies (beyond CDN libraries)
|
||||
- No build tools required
|
||||
- 100% local-first architecture
|
||||
- Comprehensive dataset management system
|
||||
- Professional UX with keyboard shortcuts and notifications
|
||||
|
||||
**Known Limitations**:
|
||||
- Experimental dark theme has minor visibility issues
|
||||
- No cross-device synchronization (future feature)
|
||||
- Storage limited to browser (no cloud backup)
|
||||
- Primarily tested in Chrome/Chromium browsers
|
||||
|
||||
**Browser Compatibility**:
|
||||
- Chrome/Chromium: Fully tested and supported
|
||||
- Firefox: Likely compatible, not exhaustively tested
|
||||
- Safari: Likely compatible, not exhaustively tested
|
||||
- IE11: Not supported (requires modern ES6+ features)
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Guide
|
||||
|
||||
### From Pre-Release to v1.0
|
||||
|
||||
No migration required. All features are additive and backward-compatible with any data created during development.
|
||||
|
||||
**Data Compatibility**:
|
||||
- Snippets in localStorage: No changes to schema
|
||||
- Datasets in IndexedDB: No changes to schema
|
||||
- Settings in localStorage: New settings added with defaults
|
||||
|
||||
**New Settings** (automatically applied):
|
||||
- `editor.fontSize`: Default 14px
|
||||
- `editor.tabSize`: Default 2
|
||||
- `editor.minimap`: Default true
|
||||
- `editor.wordWrap`: Default 'on'
|
||||
- `editor.lineNumbers`: Default 'on'
|
||||
- `dateFormat`: Default 'smart'
|
||||
- `renderDebounceDelay`: Default 1500ms
|
||||
- `theme`: Default 'light'
|
||||
|
||||
---
|
||||
|
||||
## Future Roadmap
|
||||
|
||||
Planned features based on user feedback and enhancement proposals:
|
||||
|
||||
### Short-term (Maintenance)
|
||||
- Cross-browser compatibility testing and fixes
|
||||
- Dark theme visibility improvements
|
||||
- Performance optimization for large datasets
|
||||
- Additional keyboard shortcuts
|
||||
|
||||
### Medium-term (Enhancements)
|
||||
- Advanced tagging system with tag filtering
|
||||
- Snippet templates and starter library
|
||||
- Bulk operations (delete multiple, export selected)
|
||||
- Drag-and-drop import for snippets and datasets
|
||||
- Snippet duplication with customizable naming
|
||||
|
||||
### Long-term (Major Features)
|
||||
- Authentication and user accounts
|
||||
- Cloud synchronization
|
||||
- Snippet sharing via URL
|
||||
- Public snippet gallery (optional)
|
||||
- Collaborative editing
|
||||
|
||||
See [GitHub Issues](https://github.com/olehomelchenko/astrolabe-nvc/issues) for active feature requests and bug reports.
|
||||
71
CLAUDE.md
71
CLAUDE.md
@@ -10,58 +10,57 @@ Instructions for Claude Code when working on this project.
|
||||
|
||||
- **Frontend-only**: HTML/CSS/JavaScript with CDN dependencies (Monaco Editor, Vega-Embed)
|
||||
- **Storage**:
|
||||
- **Snippets**: localStorage with Phase 0 schema (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta)
|
||||
- **Snippets**: localStorage (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta)
|
||||
- **Datasets**: IndexedDB (unlimited size, multi-format: JSON/CSV/TSV/TopoJSON, inline & URL sources)
|
||||
- **Structure**: Three resizable panels (snippet library, Monaco editor, live preview) + Dataset Manager modal
|
||||
- **Deployment**: Web app files in `/web` folder; `/docs` folder contains development documentation only
|
||||
- **Deployment**: Web app files in `/web` folder; `/project-docs` folder contains development documentation
|
||||
- **No build tools**: Open `web/index.html` directly in browser (needs local server for IndexedDB)
|
||||
|
||||
## Current Status
|
||||
|
||||
**Completed**: Phases 0-13 (Core functionality + Dataset Management + Advanced Dataset Features + Polish & UX Refinements)
|
||||
**In Progress**: GitHub Pages deployment preparation
|
||||
**Next**: Phase 14 - Advanced Snippet Features or additional refinements
|
||||
**Version**: 1.0.0 (Feature-complete MVP)
|
||||
**Deployment**: Live at astrolabe-viz.com
|
||||
**Mode**: Maintenance and iterative improvements
|
||||
|
||||
### Key Features Implemented
|
||||
- ✅ Snippet management with draft/published workflow
|
||||
- ✅ Multi-field sorting and real-time search
|
||||
- ✅ Storage monitoring and import/export
|
||||
- ✅ **Dataset management with IndexedDB**
|
||||
- Multi-format support (JSON, CSV, TSV, TopoJSON)
|
||||
- Multi-source support (inline data, URL references)
|
||||
- Automatic metadata calculation and URL fetching
|
||||
### Core Capabilities
|
||||
- Snippet management with draft/published workflow
|
||||
- Dataset library (IndexedDB) with multi-format support (JSON, CSV, TSV, TopoJSON)
|
||||
- Dataset reference resolution in Vega-Lite specs
|
||||
- Modal UI with button-group selectors
|
||||
- ✅ **Advanced Dataset Features (Phase 12)**
|
||||
- Bidirectional snippet ↔ dataset linking with usage tracking
|
||||
- User settings and theme system (Light, Dark Experimental)
|
||||
- Import/export functionality for snippets and datasets
|
||||
- Real-time search and multi-field sorting
|
||||
- Cross-platform keyboard shortcuts
|
||||
- Toast notification system
|
||||
- Storage monitoring with visual warnings
|
||||
- URL state management (shareable links, browser navigation)
|
||||
- Bidirectional snippet ↔ dataset linking
|
||||
- Table preview with type detection
|
||||
- Extract inline data to datasets
|
||||
- Import/Export datasets with auto-format detection
|
||||
- Table preview with type detection (🔢📅🔤✓)
|
||||
- On-demand URL preview loading with caching
|
||||
- ✅ **Polish & UX Features (Phase 13)**
|
||||
- Cross-platform keyboard shortcuts (Cmd/Ctrl+Shift+N, Cmd/Ctrl+K, Cmd/Ctrl+S, Escape)
|
||||
- Toast notification system (error, success, warning, info)
|
||||
- Comprehensive tooltips on all interactive elements
|
||||
- Enhanced Help modal with 6 sections (About, Features, Getting Started, Shortcuts, Storage, Privacy)
|
||||
- Data persistence warnings
|
||||
- **User Settings System**
|
||||
- Configurable editor options (font size 10-18px, tab size, minimap, word wrap, line numbers)
|
||||
- Performance tuning (render debounce delay 300-3000ms)
|
||||
- Date formatting options (smart/relative, locale, ISO, custom with tokens)
|
||||
- UI theme selection (Light, Dark Experimental)
|
||||
- **Theme System**
|
||||
- Light theme (Windows 2000 classic aesthetic)
|
||||
- Experimental dark theme with CSS variables for theming
|
||||
- Automatic editor theme synchronization with UI theme
|
||||
|
||||
See `docs/dev-plan.md` for complete roadmap and technical details.
|
||||
See `project-docs/architecture.md` for complete technical details.
|
||||
|
||||
## Development Mode
|
||||
|
||||
Development is **iterative** based on:
|
||||
- Bug reports and fixes
|
||||
- User feedback and feature requests
|
||||
- Performance optimization
|
||||
- Cross-browser compatibility improvements
|
||||
- Code quality enhancements
|
||||
|
||||
When implementing changes:
|
||||
- Treat each issue/enhancement as an independent task
|
||||
- Group related bugfixes in a single commit
|
||||
- Update CHANGELOG.md for user-facing changes
|
||||
- Test thoroughly across different browsers when possible
|
||||
- Maintain backward compatibility with existing data
|
||||
|
||||
## Development Principles
|
||||
|
||||
- **Iterative**: Each phase produces working, testable functionality
|
||||
- **Lean**: No frameworks, no build step, minimal dependencies
|
||||
- **Maintainable**: Clean code organization with logical separation of concerns
|
||||
- **Simple**: Favor code removal over addition; avoid over-engineering
|
||||
- **Local-first**: All data stored in browser, no server dependencies
|
||||
|
||||
## General coding instructions
|
||||
Astrolabe is a project with minimalistic philosophy; it tries to avoid external dependencies and complexity, if possible.
|
||||
|
||||
32
README.md
32
README.md
@@ -75,10 +75,11 @@ A lightweight, browser-based snippet manager for Vega-Lite visualizations. Organ
|
||||
|
||||
## Documentation
|
||||
|
||||
- **`docs/dev-plan.md`** – Complete development roadmap, architecture decisions, and implementation details (13 phases)
|
||||
- **`docs/features-list.md`** – Feature matrix and implementation status
|
||||
- **`docs/storage-examples.md`** – Data structure examples and storage schema
|
||||
- **`CLAUDE.md`** – Project context and current status (for AI assistants)
|
||||
- **`project-docs/architecture.md`** – Technical reference: data schemas, implementation patterns, and system architecture
|
||||
- **`project-docs/features-list.md`** – Complete feature inventory and code organization
|
||||
- **`project-docs/storage-examples.md`** – Data structure examples and storage schema
|
||||
- **`CHANGELOG.md`** – Version history and release notes
|
||||
- **`CLAUDE.md`** – Project context and development guidelines (for AI assistants)
|
||||
|
||||
## Known Limitations & Feedback
|
||||
|
||||
@@ -133,7 +134,28 @@ This project uses MIT license.
|
||||
|
||||
## Future Roadmap
|
||||
|
||||
- Authentication & Cloud Sync (optional backend integration) - will not be done via AI editors as it poses high risk of messing up with people's personal data. Contributors welcome for this milestone!
|
||||
Planned improvements based on user feedback:
|
||||
|
||||
**Short-term** (Maintenance & Polish):
|
||||
- Cross-browser compatibility testing and fixes
|
||||
- Dark theme visibility improvements
|
||||
- Performance optimization for large datasets
|
||||
- Additional keyboard shortcuts and UX refinements
|
||||
|
||||
**Medium-term** (Feature Enhancements):
|
||||
- Advanced tagging system with tag filtering
|
||||
- Snippet templates and starter library
|
||||
- Bulk operations (delete multiple, export selected)
|
||||
- Drag-and-drop import for snippets and datasets
|
||||
|
||||
**Long-term** (Major Features):
|
||||
- Authentication & cloud sync (optional backend integration)
|
||||
- **Note**: Will not be implemented via AI editors due to security concerns with personal data. Contributors welcome for this milestone!
|
||||
- Snippet sharing via URL
|
||||
- Public snippet gallery (optional)
|
||||
- Collaborative editing
|
||||
|
||||
See [CHANGELOG.md](CHANGELOG.md) for detailed roadmap and [GitHub Issues](https://github.com/olehomelchenko/astrolabe-nvc/issues) for active feature requests and bug reports.
|
||||
|
||||
---
|
||||
|
||||
|
||||
564
project-docs/architecture.md
Normal file
564
project-docs/architecture.md
Normal file
@@ -0,0 +1,564 @@
|
||||
# Astrolabe Architecture
|
||||
|
||||
**A lightweight, browser-based snippet manager for Vega-Lite visualizations**
|
||||
|
||||
> **Technical reference for developers**
|
||||
> For project overview, see [../README.md](../README.md)
|
||||
> For development guidelines, see [../CLAUDE.md](../CLAUDE.md)
|
||||
> For feature inventory, see [features-list.md](features-list.md)
|
||||
|
||||
## Project Vision
|
||||
|
||||
Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visualization specifications. It emphasizes efficient snippet management with a clean, resizable three-panel interface: snippet library, Monaco editor with schema-aware autocomplete, and live preview.
|
||||
|
||||
## Design Principles
|
||||
|
||||
- **Local-first**: All data stored in browser (localStorage for snippets, IndexedDB for datasets)
|
||||
- **Minimal dependencies**: Vanilla JavaScript, no build tools, direct CDN imports
|
||||
- **Developer-friendly**: Full JSON schema support, syntax validation, and intellisense
|
||||
- **Version-aware**: Draft/published workflow for safe experimentation
|
||||
- **Dataset management**: Separate storage for datasets with reference system
|
||||
- **Extensible**: Clean architecture for future cloud sync and authentication
|
||||
|
||||
---
|
||||
|
||||
## Technical Stack
|
||||
|
||||
- **Frontend**: Vanilla JavaScript (ES6+), HTML5, CSS3
|
||||
- **Editor**: Monaco Editor v0.47.0 (via CDN)
|
||||
- **Visualization**: Vega-Embed v6 (includes Vega v5 & Vega-Lite v5)
|
||||
- **Storage**: localStorage (snippets) + IndexedDB (datasets)
|
||||
- **Architecture**: Modular script organization with logical file separation
|
||||
- **Backend**: None (frontend-only application)
|
||||
|
||||
---
|
||||
|
||||
## Data Schemas
|
||||
|
||||
### Snippet Schema
|
||||
|
||||
Snippets are stored in localStorage with full draft/published version control.
|
||||
|
||||
```javascript
|
||||
{
|
||||
id: number, // Unique identifier (timestamp + random)
|
||||
name: string, // User-editable name (default: ISO datetime)
|
||||
created: ISO string, // Creation timestamp
|
||||
modified: ISO string, // Last modification timestamp
|
||||
spec: string, // Published Vega-Lite spec (JSON)
|
||||
draftSpec: string|null, // Working draft spec (null = no changes)
|
||||
comment: string, // User notes
|
||||
tags: string[], // Organizational tags
|
||||
datasetRefs: string[], // Referenced dataset names
|
||||
meta: object // Extensibility field
|
||||
}
|
||||
```
|
||||
|
||||
**Key Design Decisions**:
|
||||
- ID generation: `Date.now() + Math.floor(Math.random() * 10000)` for uniqueness
|
||||
- Auto-naming: ISO datetime format (YYYY-MM-DD_HH-MM-SS) for default names
|
||||
- Draft/published workflow: separate `spec` and `draftSpec` fields for safe experimentation
|
||||
- Dataset tracking: `datasetRefs` array automatically populated by reference extraction
|
||||
|
||||
### Dataset Schema
|
||||
|
||||
Datasets are stored in IndexedDB for unlimited size support.
|
||||
|
||||
```javascript
|
||||
{
|
||||
id: number, // Unique identifier (timestamp + random)
|
||||
name: string, // Unique dataset name (enforced by index)
|
||||
created: ISO string, // Creation timestamp
|
||||
modified: ISO string, // Last modification timestamp
|
||||
data: any, // Inline data (array/object) or URL string
|
||||
format: string, // 'json'|'csv'|'tsv'|'topojson'
|
||||
source: string, // 'inline'|'url'
|
||||
comment: string, // User notes
|
||||
rowCount: number|null, // Calculated row count
|
||||
columnCount: number|null, // Calculated column count
|
||||
columns: string[], // Column names
|
||||
size: number|null, // Data size in bytes
|
||||
meta: object // Extensibility field
|
||||
}
|
||||
```
|
||||
|
||||
**Key Design Decisions**:
|
||||
- Unique name constraint: IndexedDB index prevents duplicate dataset names
|
||||
- Computed metadata: automatic calculation of size, rowCount, columns for datasets
|
||||
- Multi-format: JSON, CSV, TSV, TopoJSON with format-specific parsing
|
||||
- Multi-source: Inline data storage or URL references with automatic fetching
|
||||
|
||||
### Settings Schema
|
||||
|
||||
User preferences stored in localStorage separately from snippets.
|
||||
|
||||
```javascript
|
||||
{
|
||||
sortBy: string, // 'modified'|'created'|'name'
|
||||
sortOrder: string, // 'asc'|'desc'
|
||||
panelWidths: number[], // Panel width percentages [left, center, right]
|
||||
panelVisibility: boolean[], // Panel visibility states
|
||||
panelMemory: number[], // Remembered widths for hidden panels
|
||||
editor: {
|
||||
fontSize: number, // 10-18px
|
||||
tabSize: number, // 2, 4, 8
|
||||
minimap: boolean, // Show/hide minimap
|
||||
wordWrap: string, // 'on'|'off'
|
||||
lineNumbers: string // 'on'|'off'
|
||||
},
|
||||
dateFormat: string, // 'smart'|'locale'|'iso'|'custom'
|
||||
customDateFormat: string, // Token-based format (e.g., 'YYYY-MM-DD')
|
||||
renderDebounceDelay: number, // 300-3000ms
|
||||
theme: string, // 'light'|'dark'
|
||||
meta: object // Extensibility field
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### LocalStorage Structure
|
||||
|
||||
```
|
||||
astrolabe:snippets # JSON array of all snippets
|
||||
astrolabe:settings # User preferences and UI state
|
||||
```
|
||||
|
||||
**Limits**:
|
||||
- 5 MB total shared across all localStorage data
|
||||
- Storage monitor tracks usage with visual warnings at 90% and 95%
|
||||
|
||||
### IndexedDB Structure
|
||||
|
||||
**Database**: `astrolabeDB`
|
||||
**Version**: 1
|
||||
**Object Store**: `datasets`
|
||||
- **keyPath**: `id`
|
||||
- **Indexes**:
|
||||
- `name` (unique) - prevents duplicate dataset names
|
||||
- `modified` - for efficient sorting
|
||||
|
||||
**Benefits**:
|
||||
- Effectively unlimited storage (hundreds of MB typical)
|
||||
- Better performance for large datasets
|
||||
- Structured query support via indexes
|
||||
|
||||
---
|
||||
|
||||
## Code Organization
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
web/
|
||||
├── index.html # Main HTML structure
|
||||
├── src/
|
||||
│ ├── js/
|
||||
│ │ ├── config.js # Global variables, settings API, utilities
|
||||
│ │ ├── snippet-manager.js # Snippet CRUD, storage, search, sort
|
||||
│ │ ├── dataset-manager.js # Dataset CRUD, IndexedDB operations
|
||||
│ │ ├── panel-manager.js # Layout resizing and persistence
|
||||
│ │ ├── editor.js # Monaco and Vega library integration
|
||||
│ │ ├── user-settings.js # Settings management
|
||||
│ │ └── app.js # Application initialization and events
|
||||
│ └── styles.css # Retro Windows 2000 aesthetic
|
||||
└── (CDN libraries loaded at runtime)
|
||||
```
|
||||
|
||||
### Module Responsibilities
|
||||
|
||||
**config.js** (~200 lines)
|
||||
- Global state variables (`currentSnippetId`, `currentViewMode`, etc.)
|
||||
- Settings API (load, save, get, set, validate)
|
||||
- Utility functions (date formatting, Toast notifications, URLState)
|
||||
- Sample data for first-time users
|
||||
|
||||
**snippet-manager.js** (~1100 lines)
|
||||
- SnippetStorage wrapper for localStorage operations
|
||||
- Full CRUD operations (create, read, update, delete, duplicate)
|
||||
- Search and multi-field sorting
|
||||
- Draft/published workflow logic
|
||||
- Dataset reference extraction (recursive)
|
||||
- Import/export functionality
|
||||
- Storage monitoring and size calculation
|
||||
- Auto-save system with debouncing
|
||||
|
||||
**dataset-manager.js** (~1200 lines)
|
||||
- DatasetStorage wrapper for IndexedDB operations
|
||||
- Full CRUD operations with async/Promise API
|
||||
- Format detection (JSON, CSV, TSV, TopoJSON)
|
||||
- Auto-detection system (URL/format/confidence scoring)
|
||||
- Metadata calculation (rows, columns, size, types)
|
||||
- URL fetching with CORS handling
|
||||
- Table preview rendering with type detection
|
||||
- Import/export with format conversion
|
||||
- Extract inline data to dataset
|
||||
- Usage tracking (snippet ↔ dataset linking)
|
||||
|
||||
**panel-manager.js** (~200 lines)
|
||||
- Drag-to-resize implementation
|
||||
- Panel show/hide toggle logic
|
||||
- Panel memory system (remembers sizes when hidden)
|
||||
- Proportional width redistribution
|
||||
- localStorage persistence for layout state
|
||||
|
||||
**editor.js** (~150 lines)
|
||||
- Monaco Editor initialization with Vega-Lite schema
|
||||
- AMD loader conflict resolution (Monaco vs Vega)
|
||||
- Debounced rendering (1.5s for edits, immediate for selection)
|
||||
- Dataset reference resolution before rendering
|
||||
- Error display in preview panel
|
||||
- Format-aware data injection (JSON/CSV/TSV/TopoJSON/URL)
|
||||
|
||||
**user-settings.js** (~300 lines)
|
||||
- Settings validation and defaults
|
||||
- Editor configuration management
|
||||
- Theme system (light/dark)
|
||||
- Date formatting engine
|
||||
- Performance tuning options
|
||||
- Settings modal UI logic
|
||||
|
||||
**app.js** (~250 lines)
|
||||
- Application initialization sequence
|
||||
- Event listener registration
|
||||
- Monaco editor setup
|
||||
- URL state management (hashchange listener)
|
||||
- Keyboard shortcut handlers
|
||||
- Modal management
|
||||
|
||||
**styles.css** (~280 lines)
|
||||
- Windows 2000 aesthetic (classic gray, beveled borders)
|
||||
- Component-based architecture (base classes + modifiers)
|
||||
- CSS variables for theming
|
||||
- Responsive layout with flexbox
|
||||
- Toast notification animations
|
||||
- Modal styles with viewport constraints
|
||||
|
||||
---
|
||||
|
||||
## Technical Implementation Patterns
|
||||
|
||||
### State Management
|
||||
|
||||
**Synchronous Flags**:
|
||||
- `isUpdatingEditor`: Prevents auto-save during programmatic editor updates
|
||||
- `currentViewMode`: Tracks draft vs published view ('draft' or 'published')
|
||||
- `currentSnippetId`: Selected snippet for metadata/editing
|
||||
- `currentDatasetId`: Selected dataset in modal
|
||||
|
||||
**State Flow**:
|
||||
1. User selects snippet → `selectSnippet(id)` called
|
||||
2. Flag `isUpdatingEditor = true` set
|
||||
3. Editor content updated programmatically
|
||||
4. Flag `isUpdatingEditor = false` cleared
|
||||
5. Editor change listener ignores update (flag was true)
|
||||
6. Manual edits trigger auto-save (flag is false)
|
||||
|
||||
### Auto-Save System
|
||||
|
||||
**Debouncing Strategy**:
|
||||
- Editor changes: 1.5s debounce for rendering
|
||||
- Draft spec changes: 1s debounce for auto-save
|
||||
- Name/comment changes: 1s debounce for auto-save
|
||||
- Search input: 300ms debounce for filtering
|
||||
|
||||
**Implementation**:
|
||||
```javascript
|
||||
let autoSaveTimeout = null;
|
||||
|
||||
function autoSaveDraft() {
|
||||
clearTimeout(autoSaveTimeout);
|
||||
autoSaveTimeout = setTimeout(() => {
|
||||
const content = editor.getValue();
|
||||
SnippetStorage.updateDraft(currentSnippetId, content);
|
||||
updateSnippetList(); // Refresh status lights
|
||||
}, 1000);
|
||||
}
|
||||
```
|
||||
|
||||
### Dataset Resolution
|
||||
|
||||
**Recursive Reference Extraction**:
|
||||
Vega-Lite specs can have datasets at multiple levels (layers, concat, facet). The resolution algorithm:
|
||||
|
||||
1. Parse JSON spec
|
||||
2. Recursively search for `data: { name: "..." }` patterns
|
||||
3. Extract all unique dataset names
|
||||
4. Update `datasetRefs` array in snippet metadata
|
||||
5. Before rendering, resolve each reference to actual data
|
||||
|
||||
**Resolution Logic**:
|
||||
```javascript
|
||||
async function resolveDatasetReferences(spec) {
|
||||
const specObj = JSON.parse(spec);
|
||||
|
||||
async function resolve(node) {
|
||||
if (node.data?.name) {
|
||||
const dataset = await DatasetStorage.getByName(node.data.name);
|
||||
if (dataset) {
|
||||
node.data = formatDataForVega(dataset);
|
||||
}
|
||||
}
|
||||
// Recurse into layers, concat, facet, etc.
|
||||
if (node.layer) await Promise.all(node.layer.map(resolve));
|
||||
if (node.concat) await Promise.all(node.concat.map(resolve));
|
||||
// ... other composite types
|
||||
}
|
||||
|
||||
await resolve(specObj);
|
||||
return JSON.stringify(specObj);
|
||||
}
|
||||
```
|
||||
|
||||
### URL State Management
|
||||
|
||||
**Hash-based Routing**:
|
||||
- Snippet selection: `#snippet-123456`
|
||||
- Dataset modal: `#datasets`
|
||||
- Dataset selection: `#datasets/dataset-123456`
|
||||
- New dataset form: `#datasets/new`
|
||||
|
||||
**URLState Utility**:
|
||||
```javascript
|
||||
const URLState = {
|
||||
parse() {
|
||||
const hash = window.location.hash.slice(1); // Remove '#'
|
||||
if (hash.startsWith('snippet-')) return { type: 'snippet', id: hash };
|
||||
if (hash === 'datasets') return { type: 'datasets' };
|
||||
if (hash.startsWith('datasets/dataset-')) return { type: 'dataset', id: hash };
|
||||
if (hash === 'datasets/new') return { type: 'new-dataset' };
|
||||
return null;
|
||||
},
|
||||
|
||||
update(state, replace = false) {
|
||||
const hash = formatState(state); // e.g., 'snippet-123456'
|
||||
if (replace) {
|
||||
history.replaceState(null, '', '#' + hash);
|
||||
} else {
|
||||
window.location.hash = hash;
|
||||
}
|
||||
},
|
||||
|
||||
clear() {
|
||||
history.replaceState(null, '', window.location.pathname);
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Shareable links to specific snippets/datasets
|
||||
- Browser back/forward navigation works naturally
|
||||
- Page refresh preserves user context
|
||||
- Multi-tab workflows supported
|
||||
|
||||
### Type Detection Algorithm
|
||||
|
||||
**Column Type Inference** (for table preview):
|
||||
|
||||
1. Sample first 100 values in column
|
||||
2. Try parsing each value as number/date/boolean
|
||||
3. Calculate success rate for each type
|
||||
4. If ≥80% match a type, classify as that type
|
||||
5. Otherwise, classify as text
|
||||
|
||||
**Type Detection**:
|
||||
```javascript
|
||||
function detectColumnType(values) {
|
||||
const sample = values.slice(0, 100).filter(v => v != null);
|
||||
const total = sample.length;
|
||||
|
||||
let numberCount = 0, dateCount = 0, booleanCount = 0;
|
||||
|
||||
for (const val of sample) {
|
||||
if (!isNaN(Number(val))) numberCount++;
|
||||
if (Date.parse(val)) dateCount++;
|
||||
if (['true', 'false', 'yes', 'no'].includes(String(val).toLowerCase())) booleanCount++;
|
||||
}
|
||||
|
||||
if (numberCount / total >= 0.8) return 'number';
|
||||
if (dateCount / total >= 0.8) return 'date';
|
||||
if (booleanCount / total >= 0.8) return 'boolean';
|
||||
return 'text';
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Debouncing
|
||||
|
||||
All user input is debounced to prevent excessive operations:
|
||||
- **Rendering**: 1.5s prevents rapid re-renders while editing
|
||||
- **Auto-save**: 1s reduces localStorage write frequency
|
||||
- **Search**: 300ms balances responsiveness and performance
|
||||
|
||||
### Lazy Loading
|
||||
|
||||
- **Monaco Editor**: Loaded asynchronously from CDN on page load
|
||||
- **Vega libraries**: Loaded after Monaco to avoid AMD conflicts
|
||||
- **URL dataset previews**: Fetched on-demand only when "Load Preview" clicked
|
||||
- **Table rendering**: Only first 20 rows displayed (full data in memory)
|
||||
|
||||
### Storage Optimization
|
||||
|
||||
- **Snippets**: Stored as JSON strings in localStorage (5MB limit)
|
||||
- **Datasets**: Stored in IndexedDB (no practical limit)
|
||||
- **URL datasets**: Data not persisted, only URL and metadata stored
|
||||
- **Preview cache**: In-memory only, cleared on page reload
|
||||
|
||||
### Memory Management
|
||||
|
||||
- **Editor content**: Monaco handles memory internally
|
||||
- **Vega charts**: Previous chart cleared before new render
|
||||
- **Preview cache**: Limited to session lifetime, not persisted
|
||||
- **Event listeners**: Properly cleaned up on modal close
|
||||
|
||||
---
|
||||
|
||||
## Extension Points
|
||||
|
||||
### Adding New Data Formats
|
||||
|
||||
1. Add format to `DatasetStorage.detectFormat()`
|
||||
2. Implement parser in `DatasetStorage.calculateMetadata()`
|
||||
3. Add format option to dataset form UI
|
||||
4. Update `formatDataForVega()` in editor.js for rendering
|
||||
|
||||
### Adding New Settings
|
||||
|
||||
1. Define default in `DEFAULT_SETTINGS` (config.js)
|
||||
2. Add validation in `validateSettings()`
|
||||
3. Add UI control in settings modal
|
||||
4. Apply setting where needed (e.g., editor configuration)
|
||||
|
||||
### Cloud Sync Integration
|
||||
|
||||
Future architecture for authentication and sync:
|
||||
|
||||
1. Add authentication module (OAuth or email/password)
|
||||
2. Design minimal REST API for CRUD operations
|
||||
3. Implement sync logic:
|
||||
- Upload: localStorage → server on changes
|
||||
- Download: server → localStorage on login
|
||||
- Conflict resolution: timestamp-based or manual
|
||||
4. Add sync status indicator in UI
|
||||
5. Maintain local-first architecture (offline-capable)
|
||||
|
||||
---
|
||||
|
||||
## Known Technical Constraints
|
||||
|
||||
### Browser Compatibility
|
||||
|
||||
- **Monaco Editor**: Requires modern browser (ES6+, modules)
|
||||
- **IndexedDB**: Requires browser support (all modern browsers)
|
||||
- **localStorage**: 5 MB limit varies by browser
|
||||
- **CSS Grid/Flexbox**: IE11 not supported
|
||||
|
||||
### Storage Limits
|
||||
|
||||
- **Snippets**: 5 MB localStorage limit (shared across all snippets)
|
||||
- **Datasets**: IndexedDB quota varies (typically 50%+ of available disk)
|
||||
- **URL datasets**: Subject to CORS restrictions
|
||||
- **Session storage**: Not used (all persistence is long-term)
|
||||
|
||||
### Security Considerations
|
||||
|
||||
- **No authentication**: All data accessible to anyone with browser access
|
||||
- **XSS risk**: User-provided specs executed by Vega (trusted content assumed)
|
||||
- **CORS**: URL datasets must have proper CORS headers
|
||||
- **localStorage**: Readable by any script on same origin
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Local Development
|
||||
|
||||
1. Clone repository
|
||||
2. Run local web server: `python -m http.server` or `npx http-server`
|
||||
3. Open `web/index.html` in browser
|
||||
4. Use browser DevTools for debugging
|
||||
5. Test with multiple browsers (Chrome, Firefox, Safari)
|
||||
|
||||
### Testing Checklist
|
||||
|
||||
- **Snippet CRUD**: Create, edit, duplicate, delete
|
||||
- **Draft/Publish**: Toggle modes, publish, revert
|
||||
- **Dataset CRUD**: Create inline/URL, edit, delete
|
||||
- **Search/Sort**: Real-time filtering, multi-field sorting
|
||||
- **Import/Export**: Round-trip data integrity
|
||||
- **Storage**: Approach 5MB limit and verify warnings
|
||||
- **URL State**: Refresh page, use back/forward buttons
|
||||
- **Keyboard Shortcuts**: Test all shortcuts on Mac/Windows
|
||||
- **Responsive**: Resize panels, hide/show panels
|
||||
- **Error Handling**: Invalid JSON, network failures, CORS errors
|
||||
|
||||
### Code Style Guidelines
|
||||
|
||||
- **Vanilla JS**: No frameworks, no transpilation
|
||||
- **ES6+**: Use modern syntax (arrow functions, const/let, template literals)
|
||||
- **Comments**: Explain "why", not "what"
|
||||
- **Naming**: Descriptive variable names, camelCase for JS
|
||||
- **Functions**: Small, single-purpose functions
|
||||
- **Error Handling**: Try/catch with user-friendly messages
|
||||
- **No external builds**: Direct CDN imports, no webpack/babel
|
||||
|
||||
---
|
||||
|
||||
## Migration Path
|
||||
|
||||
### From localStorage to IndexedDB (Snippets)
|
||||
|
||||
If snippet storage needs to exceed 5 MB in the future:
|
||||
|
||||
1. Create new IndexedDB object store: `snippets`
|
||||
2. Write migration script: `localStorage → IndexedDB`
|
||||
3. Update `SnippetStorage` to use IndexedDB API
|
||||
4. Keep settings in localStorage (small size)
|
||||
5. Run migration on app load (one-time)
|
||||
6. Show migration progress to user
|
||||
|
||||
### Adding Authentication
|
||||
|
||||
Architecture for future cloud sync:
|
||||
|
||||
1. Add login modal (email/password or OAuth)
|
||||
2. Store auth token in localStorage
|
||||
3. Add sync button in header
|
||||
4. Implement conflict resolution UI
|
||||
5. Keep local-first (offline mode still works)
|
||||
6. Add "Last synced" indicator
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Monaco Editor Not Loading
|
||||
|
||||
- Check CDN availability (loader.js, editor.main.js)
|
||||
- Verify AMD loader not conflicting (temporary `window.define` removal)
|
||||
- Check browser console for errors
|
||||
|
||||
### Vega Rendering Failures
|
||||
|
||||
- Verify spec is valid JSON
|
||||
- Check dataset references are resolved
|
||||
- Look for CORS errors on URL datasets
|
||||
- Verify Vega-Lite schema compatibility
|
||||
|
||||
### Storage Issues
|
||||
|
||||
- Check localStorage quota (5 MB limit)
|
||||
- Verify IndexedDB support in browser
|
||||
- Clear browser cache if storage corrupted
|
||||
- Use export to back up before clearing
|
||||
|
||||
### Performance Problems
|
||||
|
||||
- Reduce render debounce delay in settings
|
||||
- Check for large datasets in snippets (move to dataset library)
|
||||
- Verify browser DevTools not throttling
|
||||
- Test in incognito mode (extensions disabled)
|
||||
@@ -1,13 +1,14 @@
|
||||
# Astrolabe - Current Features List
|
||||
# Astrolabe - Feature Reference
|
||||
|
||||
> **Purpose**: Comprehensive inventory of all implemented features for code review and optimization
|
||||
> **Created**: 2025-10-15
|
||||
> **Updated**: 2025-10-16
|
||||
> **Status**: Phases 0-12 Complete, Phase 13 In Progress (Keyboard Shortcuts, Notifications, Tooltips, Help Documentation)
|
||||
> **Comprehensive inventory of all implemented features for code review and maintenance**
|
||||
> Version 1.0.0 | Updated 2025-10-19
|
||||
>
|
||||
> For technical details, see [architecture.md](architecture.md)
|
||||
> For data structures, see [storage-examples.md](storage-examples.md)
|
||||
|
||||
---
|
||||
|
||||
## 🔭 **ASTROLABE - Current Features Inventory**
|
||||
## Feature Inventory
|
||||
|
||||
### **1. Layout & UI Structure**
|
||||
- Three-panel resizable layout (Snippet Library | Editor | Preview)
|
||||
@@ -201,7 +202,7 @@
|
||||
|
||||
---
|
||||
|
||||
### **13. Advanced Dataset Features (Phase 12)**
|
||||
### **13. Advanced Dataset Features**
|
||||
- **Dataset Dependencies & Linking**:
|
||||
- Recursive dataset reference extraction from Vega-Lite specs
|
||||
- Bidirectional snippet ↔ dataset linking with clickable links
|
||||
@@ -293,7 +294,7 @@ src/
|
||||
|
||||
---
|
||||
|
||||
### **13. Polish & UX Refinements (Phase 13)** ⏳ _In Progress_
|
||||
### **14. Polish & UX Refinements**
|
||||
|
||||
**Keyboard Shortcuts**:
|
||||
- Cmd/Ctrl+Shift+N: Create new snippet
|
||||
@@ -329,16 +330,31 @@ src/
|
||||
|
||||
---
|
||||
|
||||
## 📝 **Next Steps**
|
||||
## Potential Enhancements
|
||||
|
||||
### Phase 13 In Progress! Keyboard shortcuts, notifications, tooltips, and help documentation complete.
|
||||
Based on user feedback and feature requests, potential improvements include:
|
||||
|
||||
**Remaining Phase 13 Tasks**:
|
||||
- [ ] Loading states for rendering
|
||||
- [ ] Empty states (no snippets, no datasets)
|
||||
- [ ] Improved visual design polish
|
||||
- [ ] Cross-browser testing
|
||||
**User Experience**:
|
||||
- Loading states for rendering
|
||||
- Enhanced empty states (no snippets, no datasets)
|
||||
- Visual design refinements
|
||||
- Additional keyboard shortcuts
|
||||
|
||||
**Future Phases**:
|
||||
- **Phase 14**: Advanced Snippet Features (tagging system, templates, bulk operations)
|
||||
- **Phase 15**: Authentication & Backend (cloud sync, sharing)
|
||||
**Features**:
|
||||
- Advanced tagging system with tag filtering
|
||||
- Snippet templates and starter library
|
||||
- Bulk operations (delete multiple, export selected)
|
||||
- Drag-and-drop import
|
||||
|
||||
**Technical**:
|
||||
- Cross-browser compatibility testing and fixes
|
||||
- Performance optimization for large datasets
|
||||
- Dark theme visibility improvements
|
||||
- Code refactoring and modularization
|
||||
|
||||
**Future Major Features**:
|
||||
- Authentication and cloud sync
|
||||
- Snippet sharing via URL
|
||||
- Collaborative editing
|
||||
|
||||
See [CHANGELOG.md](../CHANGELOG.md) for planned roadmap and [GitHub Issues](https://github.com/olehomelchenko/astrolabe-nvc/issues) for active feature requests.
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
# Astrolabe - Storage Examples
|
||||
|
||||
> **Data structure examples and schema documentation**
|
||||
>
|
||||
> For technical architecture, see [architecture.md](architecture.md)
|
||||
> For feature inventory, see [features-list.md](features-list.md)
|
||||
|
||||
Snippet example:
|
||||
---
|
||||
|
||||
## Snippet Schema Example
|
||||
```json
|
||||
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user