From f93eaa473b4f8f7efb7018b1961969601040d029 Mon Sep 17 00:00:00 2001 From: Oleh Omelchenko Date: Sun, 19 Oct 2025 01:49:53 +0300 Subject: [PATCH] docs: move from phase-based development --- CHANGELOG.md | 177 ++++++ CLAUDE.md | 75 ++- README.md | 32 +- project-docs/architecture.md | 564 ++++++++++++++++++ .../dev-plan-archive.md} | 0 project-docs/features-list.md | 52 +- project-docs/storage-examples.md | 9 +- 7 files changed, 847 insertions(+), 62 deletions(-) create mode 100644 CHANGELOG.md create mode 100644 project-docs/architecture.md rename project-docs/{dev-plan.md => archive/dev-plan-archive.md} (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..90a888b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,177 @@ +# Changelog + +All notable changes to Astrolabe will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). + +--- + +## [1.0.0] - 2025-10-15 + +### Initial Release + +Complete feature set for lightweight Vega-Lite snippet management. + +#### Core Features +- Three-panel resizable layout with drag handles and panel memory +- Monaco Editor v0.47.0 with Vega-Lite v5 schema validation and autocomplete +- Live Vega-Lite preview with debounced rendering +- Draft/published workflow for safe experimentation +- Auto-save system with 1-second debounce +- Multi-field sorting (Modified/Created/Name) with direction toggle +- Real-time search across snippet name, comment, and spec content +- Snippet size display for large snippets (≥1KB) + +#### Storage & Data Management +- localStorage-based snippet storage with 5MB monitoring +- IndexedDB-based dataset library (unlimited size) +- Multi-format dataset support: JSON, CSV, TSV, TopoJSON +- Multi-source dataset support: inline data and URL references +- Automatic dataset reference resolution in Vega-Lite specs +- Intelligent format auto-detection with confidence scoring +- Storage usage monitor with visual warnings (90% and 95% thresholds) + +#### Dataset Features +- Full CRUD operations for datasets +- Automatic metadata calculation (rows, columns, size, types) +- URL dataset fetching with CORS error handling +- Metadata refresh for URL datasets +- Bidirectional snippet ↔ dataset linking with usage tracking +- Extract inline data from specs to dataset library +- Table preview with type detection (number 🔢, date 📅, boolean ✓, text 🔤) +- On-demand URL preview loading with session cache +- Create new snippet from dataset with auto-generated spec + +#### Import/Export +- Export all snippets to JSON with auto-generated filename +- Import snippets with format auto-detection and field mapping +- Export/import datasets with format-specific file extensions +- URL dataset export fetches and downloads live content +- Additive import (no overwrites) with ID conflict resolution +- Automatic "imported" tag for external snippets + +#### User Experience +- Cross-platform keyboard shortcuts (Cmd/Ctrl+Shift+N, Cmd/Ctrl+K, Cmd/Ctrl+S, Escape) +- Toast notification system (error, success, warning, info) +- Comprehensive tooltips with keyboard hints +- Enhanced Help modal with 6 sections (About, Features, Getting Started, Shortcuts, Storage, Privacy) +- Retro Windows 2000 aesthetic throughout UI + +#### Settings & Customization +- Configurable editor options (font size 10-18px, tab size, minimap, word wrap, line numbers) +- Performance tuning (render debounce delay 300-3000ms) +- Date formatting options (smart/relative, locale, ISO, custom with tokens) +- Theme selection (Light, Dark Experimental) +- Automatic editor theme synchronization with UI theme +- Settings persistence in localStorage + +#### URL State Management +- Hash-based routing for snippets and datasets +- Browser back/forward navigation support +- Page reload preserves selected snippet or dataset +- Shareable URLs for specific snippets or datasets +- Modal state persistence in URL + +#### Technical Implementation +- Vanilla JavaScript (no frameworks, no build tools) +- AMD loader conflict resolution between Monaco and Vega +- Recursive dataset reference extraction +- Format-aware data injection for rendering +- Blob API for accurate storage size calculation +- Component-based CSS architecture with theming support + +--- + +## [Unreleased] + +### Fixed +- (Bugfixes will be listed here) + +### Added +- (New features will be listed here) + +### Changed +- (Improvements and refinements will be listed here) + +### Removed +- (Removed features will be listed here) + +--- + +## Release Notes + +### v1.0.0 - Feature-Complete MVP + +Astrolabe v1.0 represents a complete, production-ready implementation of a browser-based Vega-Lite snippet manager. The application is fully functional with no known critical bugs. + +**Development Timeline**: October 2024 - October 2025 + +**Key Accomplishments**: +- Zero external dependencies (beyond CDN libraries) +- No build tools required +- 100% local-first architecture +- Comprehensive dataset management system +- Professional UX with keyboard shortcuts and notifications + +**Known Limitations**: +- Experimental dark theme has minor visibility issues +- No cross-device synchronization (future feature) +- Storage limited to browser (no cloud backup) +- Primarily tested in Chrome/Chromium browsers + +**Browser Compatibility**: +- Chrome/Chromium: Fully tested and supported +- Firefox: Likely compatible, not exhaustively tested +- Safari: Likely compatible, not exhaustively tested +- IE11: Not supported (requires modern ES6+ features) + +--- + +## Upgrade Guide + +### From Pre-Release to v1.0 + +No migration required. All features are additive and backward-compatible with any data created during development. + +**Data Compatibility**: +- Snippets in localStorage: No changes to schema +- Datasets in IndexedDB: No changes to schema +- Settings in localStorage: New settings added with defaults + +**New Settings** (automatically applied): +- `editor.fontSize`: Default 14px +- `editor.tabSize`: Default 2 +- `editor.minimap`: Default true +- `editor.wordWrap`: Default 'on' +- `editor.lineNumbers`: Default 'on' +- `dateFormat`: Default 'smart' +- `renderDebounceDelay`: Default 1500ms +- `theme`: Default 'light' + +--- + +## Future Roadmap + +Planned features based on user feedback and enhancement proposals: + +### Short-term (Maintenance) +- Cross-browser compatibility testing and fixes +- Dark theme visibility improvements +- Performance optimization for large datasets +- Additional keyboard shortcuts + +### Medium-term (Enhancements) +- Advanced tagging system with tag filtering +- Snippet templates and starter library +- Bulk operations (delete multiple, export selected) +- Drag-and-drop import for snippets and datasets +- Snippet duplication with customizable naming + +### Long-term (Major Features) +- Authentication and user accounts +- Cloud synchronization +- Snippet sharing via URL +- Public snippet gallery (optional) +- Collaborative editing + +See [GitHub Issues](https://github.com/olehomelchenko/astrolabe-nvc/issues) for active feature requests and bug reports. diff --git a/CLAUDE.md b/CLAUDE.md index 657cb82..773915d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,58 +10,57 @@ Instructions for Claude Code when working on this project. - **Frontend-only**: HTML/CSS/JavaScript with CDN dependencies (Monaco Editor, Vega-Embed) - **Storage**: - - **Snippets**: localStorage with Phase 0 schema (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta) + - **Snippets**: localStorage (id, name, created, modified, spec, draftSpec, comment, tags, datasetRefs, meta) - **Datasets**: IndexedDB (unlimited size, multi-format: JSON/CSV/TSV/TopoJSON, inline & URL sources) - **Structure**: Three resizable panels (snippet library, Monaco editor, live preview) + Dataset Manager modal -- **Deployment**: Web app files in `/web` folder; `/docs` folder contains development documentation only +- **Deployment**: Web app files in `/web` folder; `/project-docs` folder contains development documentation - **No build tools**: Open `web/index.html` directly in browser (needs local server for IndexedDB) ## Current Status -**Completed**: Phases 0-13 (Core functionality + Dataset Management + Advanced Dataset Features + Polish & UX Refinements) -**In Progress**: GitHub Pages deployment preparation -**Next**: Phase 14 - Advanced Snippet Features or additional refinements +**Version**: 1.0.0 (Feature-complete MVP) +**Deployment**: Live at astrolabe-viz.com +**Mode**: Maintenance and iterative improvements -### Key Features Implemented -- ✅ Snippet management with draft/published workflow -- ✅ Multi-field sorting and real-time search -- ✅ Storage monitoring and import/export -- ✅ **Dataset management with IndexedDB** - - Multi-format support (JSON, CSV, TSV, TopoJSON) - - Multi-source support (inline data, URL references) - - Automatic metadata calculation and URL fetching - - Dataset reference resolution in Vega-Lite specs - - Modal UI with button-group selectors -- ✅ **Advanced Dataset Features (Phase 12)** - - Bidirectional snippet ↔ dataset linking with usage tracking - - Extract inline data to datasets - - Import/Export datasets with auto-format detection - - Table preview with type detection (🔢📅🔤✓) - - On-demand URL preview loading with caching -- ✅ **Polish & UX Features (Phase 13)** - - Cross-platform keyboard shortcuts (Cmd/Ctrl+Shift+N, Cmd/Ctrl+K, Cmd/Ctrl+S, Escape) - - Toast notification system (error, success, warning, info) - - Comprehensive tooltips on all interactive elements - - Enhanced Help modal with 6 sections (About, Features, Getting Started, Shortcuts, Storage, Privacy) - - Data persistence warnings - - **User Settings System** - - Configurable editor options (font size 10-18px, tab size, minimap, word wrap, line numbers) - - Performance tuning (render debounce delay 300-3000ms) - - Date formatting options (smart/relative, locale, ISO, custom with tokens) - - UI theme selection (Light, Dark Experimental) - - **Theme System** - - Light theme (Windows 2000 classic aesthetic) - - Experimental dark theme with CSS variables for theming - - Automatic editor theme synchronization with UI theme +### Core Capabilities +- Snippet management with draft/published workflow +- Dataset library (IndexedDB) with multi-format support (JSON, CSV, TSV, TopoJSON) +- Dataset reference resolution in Vega-Lite specs +- User settings and theme system (Light, Dark Experimental) +- Import/export functionality for snippets and datasets +- Real-time search and multi-field sorting +- Cross-platform keyboard shortcuts +- Toast notification system +- Storage monitoring with visual warnings +- URL state management (shareable links, browser navigation) +- Bidirectional snippet ↔ dataset linking +- Table preview with type detection +- Extract inline data to datasets -See `docs/dev-plan.md` for complete roadmap and technical details. +See `project-docs/architecture.md` for complete technical details. + +## Development Mode + +Development is **iterative** based on: +- Bug reports and fixes +- User feedback and feature requests +- Performance optimization +- Cross-browser compatibility improvements +- Code quality enhancements + +When implementing changes: +- Treat each issue/enhancement as an independent task +- Group related bugfixes in a single commit +- Update CHANGELOG.md for user-facing changes +- Test thoroughly across different browsers when possible +- Maintain backward compatibility with existing data ## Development Principles -- **Iterative**: Each phase produces working, testable functionality - **Lean**: No frameworks, no build step, minimal dependencies - **Maintainable**: Clean code organization with logical separation of concerns - **Simple**: Favor code removal over addition; avoid over-engineering +- **Local-first**: All data stored in browser, no server dependencies ## General coding instructions Astrolabe is a project with minimalistic philosophy; it tries to avoid external dependencies and complexity, if possible. diff --git a/README.md b/README.md index c70af8d..7eba34d 100644 --- a/README.md +++ b/README.md @@ -75,10 +75,11 @@ A lightweight, browser-based snippet manager for Vega-Lite visualizations. Organ ## Documentation -- **`docs/dev-plan.md`** – Complete development roadmap, architecture decisions, and implementation details (13 phases) -- **`docs/features-list.md`** – Feature matrix and implementation status -- **`docs/storage-examples.md`** – Data structure examples and storage schema -- **`CLAUDE.md`** – Project context and current status (for AI assistants) +- **`project-docs/architecture.md`** – Technical reference: data schemas, implementation patterns, and system architecture +- **`project-docs/features-list.md`** – Complete feature inventory and code organization +- **`project-docs/storage-examples.md`** – Data structure examples and storage schema +- **`CHANGELOG.md`** – Version history and release notes +- **`CLAUDE.md`** – Project context and development guidelines (for AI assistants) ## Known Limitations & Feedback @@ -133,7 +134,28 @@ This project uses MIT license. ## Future Roadmap -- Authentication & Cloud Sync (optional backend integration) - will not be done via AI editors as it poses high risk of messing up with people's personal data. Contributors welcome for this milestone! +Planned improvements based on user feedback: + +**Short-term** (Maintenance & Polish): +- Cross-browser compatibility testing and fixes +- Dark theme visibility improvements +- Performance optimization for large datasets +- Additional keyboard shortcuts and UX refinements + +**Medium-term** (Feature Enhancements): +- Advanced tagging system with tag filtering +- Snippet templates and starter library +- Bulk operations (delete multiple, export selected) +- Drag-and-drop import for snippets and datasets + +**Long-term** (Major Features): +- Authentication & cloud sync (optional backend integration) + - **Note**: Will not be implemented via AI editors due to security concerns with personal data. Contributors welcome for this milestone! +- Snippet sharing via URL +- Public snippet gallery (optional) +- Collaborative editing + +See [CHANGELOG.md](CHANGELOG.md) for detailed roadmap and [GitHub Issues](https://github.com/olehomelchenko/astrolabe-nvc/issues) for active feature requests and bug reports. --- diff --git a/project-docs/architecture.md b/project-docs/architecture.md new file mode 100644 index 0000000..66fc8f1 --- /dev/null +++ b/project-docs/architecture.md @@ -0,0 +1,564 @@ +# Astrolabe Architecture + +**A lightweight, browser-based snippet manager for Vega-Lite visualizations** + +> **Technical reference for developers** +> For project overview, see [../README.md](../README.md) +> For development guidelines, see [../CLAUDE.md](../CLAUDE.md) +> For feature inventory, see [features-list.md](features-list.md) + +## Project Vision + +Astrolabe is a focused tool for managing, editing, and previewing Vega-Lite visualization specifications. It emphasizes efficient snippet management with a clean, resizable three-panel interface: snippet library, Monaco editor with schema-aware autocomplete, and live preview. + +## Design Principles + +- **Local-first**: All data stored in browser (localStorage for snippets, IndexedDB for datasets) +- **Minimal dependencies**: Vanilla JavaScript, no build tools, direct CDN imports +- **Developer-friendly**: Full JSON schema support, syntax validation, and intellisense +- **Version-aware**: Draft/published workflow for safe experimentation +- **Dataset management**: Separate storage for datasets with reference system +- **Extensible**: Clean architecture for future cloud sync and authentication + +--- + +## Technical Stack + +- **Frontend**: Vanilla JavaScript (ES6+), HTML5, CSS3 +- **Editor**: Monaco Editor v0.47.0 (via CDN) +- **Visualization**: Vega-Embed v6 (includes Vega v5 & Vega-Lite v5) +- **Storage**: localStorage (snippets) + IndexedDB (datasets) +- **Architecture**: Modular script organization with logical file separation +- **Backend**: None (frontend-only application) + +--- + +## Data Schemas + +### Snippet Schema + +Snippets are stored in localStorage with full draft/published version control. + +```javascript +{ + id: number, // Unique identifier (timestamp + random) + name: string, // User-editable name (default: ISO datetime) + created: ISO string, // Creation timestamp + modified: ISO string, // Last modification timestamp + spec: string, // Published Vega-Lite spec (JSON) + draftSpec: string|null, // Working draft spec (null = no changes) + comment: string, // User notes + tags: string[], // Organizational tags + datasetRefs: string[], // Referenced dataset names + meta: object // Extensibility field +} +``` + +**Key Design Decisions**: +- ID generation: `Date.now() + Math.floor(Math.random() * 10000)` for uniqueness +- Auto-naming: ISO datetime format (YYYY-MM-DD_HH-MM-SS) for default names +- Draft/published workflow: separate `spec` and `draftSpec` fields for safe experimentation +- Dataset tracking: `datasetRefs` array automatically populated by reference extraction + +### Dataset Schema + +Datasets are stored in IndexedDB for unlimited size support. + +```javascript +{ + id: number, // Unique identifier (timestamp + random) + name: string, // Unique dataset name (enforced by index) + created: ISO string, // Creation timestamp + modified: ISO string, // Last modification timestamp + data: any, // Inline data (array/object) or URL string + format: string, // 'json'|'csv'|'tsv'|'topojson' + source: string, // 'inline'|'url' + comment: string, // User notes + rowCount: number|null, // Calculated row count + columnCount: number|null, // Calculated column count + columns: string[], // Column names + size: number|null, // Data size in bytes + meta: object // Extensibility field +} +``` + +**Key Design Decisions**: +- Unique name constraint: IndexedDB index prevents duplicate dataset names +- Computed metadata: automatic calculation of size, rowCount, columns for datasets +- Multi-format: JSON, CSV, TSV, TopoJSON with format-specific parsing +- Multi-source: Inline data storage or URL references with automatic fetching + +### Settings Schema + +User preferences stored in localStorage separately from snippets. + +```javascript +{ + sortBy: string, // 'modified'|'created'|'name' + sortOrder: string, // 'asc'|'desc' + panelWidths: number[], // Panel width percentages [left, center, right] + panelVisibility: boolean[], // Panel visibility states + panelMemory: number[], // Remembered widths for hidden panels + editor: { + fontSize: number, // 10-18px + tabSize: number, // 2, 4, 8 + minimap: boolean, // Show/hide minimap + wordWrap: string, // 'on'|'off' + lineNumbers: string // 'on'|'off' + }, + dateFormat: string, // 'smart'|'locale'|'iso'|'custom' + customDateFormat: string, // Token-based format (e.g., 'YYYY-MM-DD') + renderDebounceDelay: number, // 300-3000ms + theme: string, // 'light'|'dark' + meta: object // Extensibility field +} +``` + +--- + +## Storage Architecture + +### LocalStorage Structure + +``` +astrolabe:snippets # JSON array of all snippets +astrolabe:settings # User preferences and UI state +``` + +**Limits**: +- 5 MB total shared across all localStorage data +- Storage monitor tracks usage with visual warnings at 90% and 95% + +### IndexedDB Structure + +**Database**: `astrolabeDB` +**Version**: 1 +**Object Store**: `datasets` +- **keyPath**: `id` +- **Indexes**: + - `name` (unique) - prevents duplicate dataset names + - `modified` - for efficient sorting + +**Benefits**: +- Effectively unlimited storage (hundreds of MB typical) +- Better performance for large datasets +- Structured query support via indexes + +--- + +## Code Organization + +### File Structure + +``` +web/ +├── index.html # Main HTML structure +├── src/ +│ ├── js/ +│ │ ├── config.js # Global variables, settings API, utilities +│ │ ├── snippet-manager.js # Snippet CRUD, storage, search, sort +│ │ ├── dataset-manager.js # Dataset CRUD, IndexedDB operations +│ │ ├── panel-manager.js # Layout resizing and persistence +│ │ ├── editor.js # Monaco and Vega library integration +│ │ ├── user-settings.js # Settings management +│ │ └── app.js # Application initialization and events +│ └── styles.css # Retro Windows 2000 aesthetic +└── (CDN libraries loaded at runtime) +``` + +### Module Responsibilities + +**config.js** (~200 lines) +- Global state variables (`currentSnippetId`, `currentViewMode`, etc.) +- Settings API (load, save, get, set, validate) +- Utility functions (date formatting, Toast notifications, URLState) +- Sample data for first-time users + +**snippet-manager.js** (~1100 lines) +- SnippetStorage wrapper for localStorage operations +- Full CRUD operations (create, read, update, delete, duplicate) +- Search and multi-field sorting +- Draft/published workflow logic +- Dataset reference extraction (recursive) +- Import/export functionality +- Storage monitoring and size calculation +- Auto-save system with debouncing + +**dataset-manager.js** (~1200 lines) +- DatasetStorage wrapper for IndexedDB operations +- Full CRUD operations with async/Promise API +- Format detection (JSON, CSV, TSV, TopoJSON) +- Auto-detection system (URL/format/confidence scoring) +- Metadata calculation (rows, columns, size, types) +- URL fetching with CORS handling +- Table preview rendering with type detection +- Import/export with format conversion +- Extract inline data to dataset +- Usage tracking (snippet ↔ dataset linking) + +**panel-manager.js** (~200 lines) +- Drag-to-resize implementation +- Panel show/hide toggle logic +- Panel memory system (remembers sizes when hidden) +- Proportional width redistribution +- localStorage persistence for layout state + +**editor.js** (~150 lines) +- Monaco Editor initialization with Vega-Lite schema +- AMD loader conflict resolution (Monaco vs Vega) +- Debounced rendering (1.5s for edits, immediate for selection) +- Dataset reference resolution before rendering +- Error display in preview panel +- Format-aware data injection (JSON/CSV/TSV/TopoJSON/URL) + +**user-settings.js** (~300 lines) +- Settings validation and defaults +- Editor configuration management +- Theme system (light/dark) +- Date formatting engine +- Performance tuning options +- Settings modal UI logic + +**app.js** (~250 lines) +- Application initialization sequence +- Event listener registration +- Monaco editor setup +- URL state management (hashchange listener) +- Keyboard shortcut handlers +- Modal management + +**styles.css** (~280 lines) +- Windows 2000 aesthetic (classic gray, beveled borders) +- Component-based architecture (base classes + modifiers) +- CSS variables for theming +- Responsive layout with flexbox +- Toast notification animations +- Modal styles with viewport constraints + +--- + +## Technical Implementation Patterns + +### State Management + +**Synchronous Flags**: +- `isUpdatingEditor`: Prevents auto-save during programmatic editor updates +- `currentViewMode`: Tracks draft vs published view ('draft' or 'published') +- `currentSnippetId`: Selected snippet for metadata/editing +- `currentDatasetId`: Selected dataset in modal + +**State Flow**: +1. User selects snippet → `selectSnippet(id)` called +2. Flag `isUpdatingEditor = true` set +3. Editor content updated programmatically +4. Flag `isUpdatingEditor = false` cleared +5. Editor change listener ignores update (flag was true) +6. Manual edits trigger auto-save (flag is false) + +### Auto-Save System + +**Debouncing Strategy**: +- Editor changes: 1.5s debounce for rendering +- Draft spec changes: 1s debounce for auto-save +- Name/comment changes: 1s debounce for auto-save +- Search input: 300ms debounce for filtering + +**Implementation**: +```javascript +let autoSaveTimeout = null; + +function autoSaveDraft() { + clearTimeout(autoSaveTimeout); + autoSaveTimeout = setTimeout(() => { + const content = editor.getValue(); + SnippetStorage.updateDraft(currentSnippetId, content); + updateSnippetList(); // Refresh status lights + }, 1000); +} +``` + +### Dataset Resolution + +**Recursive Reference Extraction**: +Vega-Lite specs can have datasets at multiple levels (layers, concat, facet). The resolution algorithm: + +1. Parse JSON spec +2. Recursively search for `data: { name: "..." }` patterns +3. Extract all unique dataset names +4. Update `datasetRefs` array in snippet metadata +5. Before rendering, resolve each reference to actual data + +**Resolution Logic**: +```javascript +async function resolveDatasetReferences(spec) { + const specObj = JSON.parse(spec); + + async function resolve(node) { + if (node.data?.name) { + const dataset = await DatasetStorage.getByName(node.data.name); + if (dataset) { + node.data = formatDataForVega(dataset); + } + } + // Recurse into layers, concat, facet, etc. + if (node.layer) await Promise.all(node.layer.map(resolve)); + if (node.concat) await Promise.all(node.concat.map(resolve)); + // ... other composite types + } + + await resolve(specObj); + return JSON.stringify(specObj); +} +``` + +### URL State Management + +**Hash-based Routing**: +- Snippet selection: `#snippet-123456` +- Dataset modal: `#datasets` +- Dataset selection: `#datasets/dataset-123456` +- New dataset form: `#datasets/new` + +**URLState Utility**: +```javascript +const URLState = { + parse() { + const hash = window.location.hash.slice(1); // Remove '#' + if (hash.startsWith('snippet-')) return { type: 'snippet', id: hash }; + if (hash === 'datasets') return { type: 'datasets' }; + if (hash.startsWith('datasets/dataset-')) return { type: 'dataset', id: hash }; + if (hash === 'datasets/new') return { type: 'new-dataset' }; + return null; + }, + + update(state, replace = false) { + const hash = formatState(state); // e.g., 'snippet-123456' + if (replace) { + history.replaceState(null, '', '#' + hash); + } else { + window.location.hash = hash; + } + }, + + clear() { + history.replaceState(null, '', window.location.pathname); + } +}; +``` + +**Benefits**: +- Shareable links to specific snippets/datasets +- Browser back/forward navigation works naturally +- Page refresh preserves user context +- Multi-tab workflows supported + +### Type Detection Algorithm + +**Column Type Inference** (for table preview): + +1. Sample first 100 values in column +2. Try parsing each value as number/date/boolean +3. Calculate success rate for each type +4. If ≥80% match a type, classify as that type +5. Otherwise, classify as text + +**Type Detection**: +```javascript +function detectColumnType(values) { + const sample = values.slice(0, 100).filter(v => v != null); + const total = sample.length; + + let numberCount = 0, dateCount = 0, booleanCount = 0; + + for (const val of sample) { + if (!isNaN(Number(val))) numberCount++; + if (Date.parse(val)) dateCount++; + if (['true', 'false', 'yes', 'no'].includes(String(val).toLowerCase())) booleanCount++; + } + + if (numberCount / total >= 0.8) return 'number'; + if (dateCount / total >= 0.8) return 'date'; + if (booleanCount / total >= 0.8) return 'boolean'; + return 'text'; +} +``` + +--- + +## Performance Considerations + +### Debouncing + +All user input is debounced to prevent excessive operations: +- **Rendering**: 1.5s prevents rapid re-renders while editing +- **Auto-save**: 1s reduces localStorage write frequency +- **Search**: 300ms balances responsiveness and performance + +### Lazy Loading + +- **Monaco Editor**: Loaded asynchronously from CDN on page load +- **Vega libraries**: Loaded after Monaco to avoid AMD conflicts +- **URL dataset previews**: Fetched on-demand only when "Load Preview" clicked +- **Table rendering**: Only first 20 rows displayed (full data in memory) + +### Storage Optimization + +- **Snippets**: Stored as JSON strings in localStorage (5MB limit) +- **Datasets**: Stored in IndexedDB (no practical limit) +- **URL datasets**: Data not persisted, only URL and metadata stored +- **Preview cache**: In-memory only, cleared on page reload + +### Memory Management + +- **Editor content**: Monaco handles memory internally +- **Vega charts**: Previous chart cleared before new render +- **Preview cache**: Limited to session lifetime, not persisted +- **Event listeners**: Properly cleaned up on modal close + +--- + +## Extension Points + +### Adding New Data Formats + +1. Add format to `DatasetStorage.detectFormat()` +2. Implement parser in `DatasetStorage.calculateMetadata()` +3. Add format option to dataset form UI +4. Update `formatDataForVega()` in editor.js for rendering + +### Adding New Settings + +1. Define default in `DEFAULT_SETTINGS` (config.js) +2. Add validation in `validateSettings()` +3. Add UI control in settings modal +4. Apply setting where needed (e.g., editor configuration) + +### Cloud Sync Integration + +Future architecture for authentication and sync: + +1. Add authentication module (OAuth or email/password) +2. Design minimal REST API for CRUD operations +3. Implement sync logic: + - Upload: localStorage → server on changes + - Download: server → localStorage on login + - Conflict resolution: timestamp-based or manual +4. Add sync status indicator in UI +5. Maintain local-first architecture (offline-capable) + +--- + +## Known Technical Constraints + +### Browser Compatibility + +- **Monaco Editor**: Requires modern browser (ES6+, modules) +- **IndexedDB**: Requires browser support (all modern browsers) +- **localStorage**: 5 MB limit varies by browser +- **CSS Grid/Flexbox**: IE11 not supported + +### Storage Limits + +- **Snippets**: 5 MB localStorage limit (shared across all snippets) +- **Datasets**: IndexedDB quota varies (typically 50%+ of available disk) +- **URL datasets**: Subject to CORS restrictions +- **Session storage**: Not used (all persistence is long-term) + +### Security Considerations + +- **No authentication**: All data accessible to anyone with browser access +- **XSS risk**: User-provided specs executed by Vega (trusted content assumed) +- **CORS**: URL datasets must have proper CORS headers +- **localStorage**: Readable by any script on same origin + +--- + +## Development Workflow + +### Local Development + +1. Clone repository +2. Run local web server: `python -m http.server` or `npx http-server` +3. Open `web/index.html` in browser +4. Use browser DevTools for debugging +5. Test with multiple browsers (Chrome, Firefox, Safari) + +### Testing Checklist + +- **Snippet CRUD**: Create, edit, duplicate, delete +- **Draft/Publish**: Toggle modes, publish, revert +- **Dataset CRUD**: Create inline/URL, edit, delete +- **Search/Sort**: Real-time filtering, multi-field sorting +- **Import/Export**: Round-trip data integrity +- **Storage**: Approach 5MB limit and verify warnings +- **URL State**: Refresh page, use back/forward buttons +- **Keyboard Shortcuts**: Test all shortcuts on Mac/Windows +- **Responsive**: Resize panels, hide/show panels +- **Error Handling**: Invalid JSON, network failures, CORS errors + +### Code Style Guidelines + +- **Vanilla JS**: No frameworks, no transpilation +- **ES6+**: Use modern syntax (arrow functions, const/let, template literals) +- **Comments**: Explain "why", not "what" +- **Naming**: Descriptive variable names, camelCase for JS +- **Functions**: Small, single-purpose functions +- **Error Handling**: Try/catch with user-friendly messages +- **No external builds**: Direct CDN imports, no webpack/babel + +--- + +## Migration Path + +### From localStorage to IndexedDB (Snippets) + +If snippet storage needs to exceed 5 MB in the future: + +1. Create new IndexedDB object store: `snippets` +2. Write migration script: `localStorage → IndexedDB` +3. Update `SnippetStorage` to use IndexedDB API +4. Keep settings in localStorage (small size) +5. Run migration on app load (one-time) +6. Show migration progress to user + +### Adding Authentication + +Architecture for future cloud sync: + +1. Add login modal (email/password or OAuth) +2. Store auth token in localStorage +3. Add sync button in header +4. Implement conflict resolution UI +5. Keep local-first (offline mode still works) +6. Add "Last synced" indicator + +--- + +## Troubleshooting + +### Monaco Editor Not Loading + +- Check CDN availability (loader.js, editor.main.js) +- Verify AMD loader not conflicting (temporary `window.define` removal) +- Check browser console for errors + +### Vega Rendering Failures + +- Verify spec is valid JSON +- Check dataset references are resolved +- Look for CORS errors on URL datasets +- Verify Vega-Lite schema compatibility + +### Storage Issues + +- Check localStorage quota (5 MB limit) +- Verify IndexedDB support in browser +- Clear browser cache if storage corrupted +- Use export to back up before clearing + +### Performance Problems + +- Reduce render debounce delay in settings +- Check for large datasets in snippets (move to dataset library) +- Verify browser DevTools not throttling +- Test in incognito mode (extensions disabled) diff --git a/project-docs/dev-plan.md b/project-docs/archive/dev-plan-archive.md similarity index 100% rename from project-docs/dev-plan.md rename to project-docs/archive/dev-plan-archive.md diff --git a/project-docs/features-list.md b/project-docs/features-list.md index c047c4b..00e8374 100644 --- a/project-docs/features-list.md +++ b/project-docs/features-list.md @@ -1,13 +1,14 @@ -# Astrolabe - Current Features List +# Astrolabe - Feature Reference -> **Purpose**: Comprehensive inventory of all implemented features for code review and optimization -> **Created**: 2025-10-15 -> **Updated**: 2025-10-16 -> **Status**: Phases 0-12 Complete, Phase 13 In Progress (Keyboard Shortcuts, Notifications, Tooltips, Help Documentation) +> **Comprehensive inventory of all implemented features for code review and maintenance** +> Version 1.0.0 | Updated 2025-10-19 +> +> For technical details, see [architecture.md](architecture.md) +> For data structures, see [storage-examples.md](storage-examples.md) --- -## 🔭 **ASTROLABE - Current Features Inventory** +## Feature Inventory ### **1. Layout & UI Structure** - Three-panel resizable layout (Snippet Library | Editor | Preview) @@ -201,7 +202,7 @@ --- -### **13. Advanced Dataset Features (Phase 12)** +### **13. Advanced Dataset Features** - **Dataset Dependencies & Linking**: - Recursive dataset reference extraction from Vega-Lite specs - Bidirectional snippet ↔ dataset linking with clickable links @@ -293,7 +294,7 @@ src/ --- -### **13. Polish & UX Refinements (Phase 13)** ⏳ _In Progress_ +### **14. Polish & UX Refinements** **Keyboard Shortcuts**: - Cmd/Ctrl+Shift+N: Create new snippet @@ -329,16 +330,31 @@ src/ --- -## 📝 **Next Steps** +## Potential Enhancements -### Phase 13 In Progress! Keyboard shortcuts, notifications, tooltips, and help documentation complete. +Based on user feedback and feature requests, potential improvements include: -**Remaining Phase 13 Tasks**: -- [ ] Loading states for rendering -- [ ] Empty states (no snippets, no datasets) -- [ ] Improved visual design polish -- [ ] Cross-browser testing +**User Experience**: +- Loading states for rendering +- Enhanced empty states (no snippets, no datasets) +- Visual design refinements +- Additional keyboard shortcuts -**Future Phases**: -- **Phase 14**: Advanced Snippet Features (tagging system, templates, bulk operations) -- **Phase 15**: Authentication & Backend (cloud sync, sharing) +**Features**: +- Advanced tagging system with tag filtering +- Snippet templates and starter library +- Bulk operations (delete multiple, export selected) +- Drag-and-drop import + +**Technical**: +- Cross-browser compatibility testing and fixes +- Performance optimization for large datasets +- Dark theme visibility improvements +- Code refactoring and modularization + +**Future Major Features**: +- Authentication and cloud sync +- Snippet sharing via URL +- Collaborative editing + +See [CHANGELOG.md](../CHANGELOG.md) for planned roadmap and [GitHub Issues](https://github.com/olehomelchenko/astrolabe-nvc/issues) for active feature requests. diff --git a/project-docs/storage-examples.md b/project-docs/storage-examples.md index a2d43d6..bca375f 100644 --- a/project-docs/storage-examples.md +++ b/project-docs/storage-examples.md @@ -1,6 +1,13 @@ +# Astrolabe - Storage Examples +> **Data structure examples and schema documentation** +> +> For technical architecture, see [architecture.md](architecture.md) +> For feature inventory, see [features-list.md](features-list.md) -Snippet example: +--- + +## Snippet Schema Example ```json {