Skip to content

Storage Backends

OptiScope provides a flexible storage system that allows you to manage optimization results across different backends. This is useful for handling large datasets, sharing results between users, or caching frequently accessed data.

Storage Manager

The StorageManager is the central component that coordinates storage operations. It supports:

  • Primary Storage: The main backend where data is stored (e.g., a database or file system).
  • Secondary Storage: An optional backup or archival backend.
  • Caching: An in-memory cache to speed up access to frequently used results.

Creating a Storage Manager

You can create a storage manager using the create_storage_manager helper function.

from optiscope.storage import create_storage_manager

# Create a manager with file system storage and in-memory caching
manager = create_storage_manager(
    primary_type="filesystem",
    primary_config={"base_path": "./results_store"},
    use_cache=True,
    cache_size_mb=200
)

Available Backends

File System Storage

Stores results as files on the local disk. This is simple and effective for single-user scenarios.

  • Type: "filesystem"
  • Config: base_path (directory to store files)

Database Storage

Stores results in a relational database using SQLAlchemy. This is ideal for multi-user environments or when you need to query metadata efficiently.

  • Type: "database"
  • Config: connection_string (e.g., sqlite:///results.db, postgresql://user:pass@localhost/db)

Note: You need to install the database extras to use this backend: pip install optiscope[database]

Memory Storage

Stores results in RAM. This is fast but volatile (data is lost when the program ends). It is primarily used for caching or temporary testing.

  • Type: "memory"
  • Config: max_memory_mb (optional limit)

Key Features

Caching

When caching is enabled, the StorageManager keeps recently accessed results in memory. This significantly improves performance when you need to access the same result multiple times.

Fallback and Replication

  • Fallback: If the primary storage fails (e.g., database connection lost), the manager can automatically try to load from the secondary storage.
  • Replication: When saving a result, you can choose to replicate it to the secondary storage for backup.
# Save to primary and replicate to secondary
manager.save_result("run_001", result, replicate=True)

Migration

You can move results between backends using migrate_result.

# Move a result from cache to primary storage
manager.migrate_result("run_001", from_backend="cache", to_backend="primary")

Example Workflow

from optiscope.storage import create_storage_manager
from optiscope.core import OptimizationResult

# 1. Setup storage
manager = create_storage_manager(
    primary_type="filesystem",
    primary_config={"base_path": "./data"},
    use_cache=True
)

# 2. Save a result
result = OptimizationResult(...)
manager.save_result("experiment_1", result)

# 3. Load a result (checks cache first, then filesystem)
loaded_result = manager.load_result("experiment_1")

# 4. List available results
keys = manager.list_results()
print(f"Stored results: {keys}")

# 5. Cleanup
manager.close()