Skip to content

Data Manager

The Data Manager is the central hub for loading, managing, and organizing your optimization results. This is typically the first page you'll use when working with the OptiScope Dash App.

Overview

The Data Manager allows you to: - Upload optimization result files - Connect to existing databases - View and manage loaded data sources - Export selected results - Delete unwanted data sources

Loading Data

Upload Files

You can upload optimization results in various formats:

Supported Formats: - CSV files (.csv) - JSON files (.json) - SQLite databases (.db, .sqlite) - DuckDB databases (.duckdb)

How to Upload: 1. Click the upload area or drag and drop files 2. The app automatically detects the file format 3. Files are processed and added to your session

Data Loading Behavior:

  • Database Files (.db, .sqlite, .duckdb):
  • Copied to a temporary file on the server
  • The app connects to this temporary copy

  • Efficient for large datasets (data is queried as needed)

  • Text/Data Files (CSV, JSON):

  • Content is read and ingested into the session database
  • Data is fully loaded into the application's active storage
  • Best for smaller to medium-sized datasets

Connect to Database

For existing databases on your filesystem:

  1. Enter the absolute path to your database file
  2. Optionally provide a custom label
  3. Click Mount Database

Benefits: - No file copying (connects directly to the original file) - Ideal for large databases or shared resources - Changes are reflected immediately

Connection String Format: 

# SQLite
/path/to/your/database.db
sqlite:///path/to/your/database.db

# DuckDB
/path/to/your/database.duckdb

Managing Data Sources

View Loaded Sources

Each loaded data source is displayed as a card showing: - Label: Name of the data source - Number of Results: Count of optimization results - Storage Location: Where the data is stored - Storage Type: Type of storage backend (memory, database, etc.) - Status: Valid or Invalid

Actions

For each data source, you can:

  • Select (checkbox): Select for batch operations
  • Info (ℹī¸): View detailed information about the source
  • Rename (✏ī¸): Change the display name
  • Delete (🗑ī¸): Remove the data source from the session

Batch Operations

Select All / Deselect All: Quickly select or deselect all data sources

Export Selected: - Export to SQLite (.db) - Export to DuckDB (.duckdb) - Creates a downloadable database file containing all selected results

Clear Selected: Remove all selected data sources from the session

Storage Information

The Storage Overview card displays: - Primary Storage: Main storage backend information - Secondary Storage: Additional storage details (if configured) - Cache: Cache storage information - Size: Storage size in MB - Result Count: Number of stored results

Best Practices

File Organization

  • Use descriptive labels for your data sources
  • Group related results together
  • Regularly clean up unused data sources

Performance

  • For large datasets (>100MB), use database formats (SQLite/DuckDB)
  • For quick exploration, CSV/JSON files work well
  • Connect directly to databases instead of uploading when possible

Data Persistence

  • Data loaded in the app is session-specific
  • Export important results before closing the app
  • Use database connections for persistent storage

Troubleshooting

File Upload Issues

  • File not recognized: Ensure the file format is supported
  • Upload fails: Check file size and format validity
  • Data not appearing: Refresh the page or check the logs

Database Connection Issues

  • Path not found: Verify the absolute path is correct
  • Permission denied: Ensure you have read access to the database file
  • Invalid database: Check that the database file is not corrupted
  • Path: /data_manager
  • Category: Data
  • Icon: Database import icon
  • Requires Data: No (always accessible)