Storage

Storage abstraction layer with multiple backend support (local filesystem, fsspec, obstore), configuration-based registration, and Arrow table import/export with CSV format support.

Write and read formats

Row-oriented writes accept only json and newline-delimited jsonl. Arrow-table writes accept only parquet, arrow-ipc, and csv. The pipeline raises StorageCapabilityError for a mismatched format before encoding or storage I/O, so it cannot write one payload type under another format label. Read APIs retain the full format set because they decode all five formats into Arrow tables.

JSONL reads use PyArrow's native JSON reader. Its type inference applies to the result, including conversion of date-like strings to Arrow timestamps. This avoids Python per-line decoding and Table.from_pylist() copies. The reader block is sized to the payload, so individual rows may exceed PyArrow's default 1 MiB block. It does not make load_from_storage() bounded-memory: that API reads the complete object payload before decoding it.

Parquet Batch Streaming

The stream_arrow_sync() and stream_arrow_async() backend methods stream Parquet files in file and row-group order. They accept a keyword-only batch_size (default 65_536) which controls the maximum rows in each record batch. Each read is restricted to one Parquet row group, so the I/O bound is one row group rather than one record batch. Choose the Parquet row-group size when writing files according to the memory bound required while reading them.

These methods intentionally support only file_format="parquet" and raise StorageCapabilityError for any other format. Use the regular Arrow read APIs for CSV, Arrow IPC, JSON, and JSONL payloads. Closing a sync generator or calling aclose() on its async iterator closes the active storage reader.

Resolve portable storage addresses

Use backend.resolve_uri(path) when another component needs the address of an object. The method accepts the same backend-relative path as the read and write methods. It does not perform storage I/O, require the target to exist, or change the backend's configured path prefixes.

For example, fsspec and obstore both resolve data.parquet against s3://bucket/prefix with base_path="workspaces" as s3://bucket/prefix/workspaces/data.parquet. Local, fsspec-file, and obstore-file backends return the equivalent absolute filesystem path. Consumers should use this method instead of joining base_path, base_uri, or store_uri themselves.

The returned address is unsigned. Use sign_sync() or sign_async() when a caller needs a time-limited access grant. resolve_uri() does not sign, percent-encode, or validate the address, and its argument is an object key rather than an already-qualified remote URI.

Pipelines

class sqlspec.storage.SyncStoragePipeline

Bases: _StoragePipelineBase

Pipeline coordinating storage registry operations and telemetry.

write_rows(rows, destination, *, format_hint=None, storage_options=None)

Write dictionary rows to storage using cached serializers.

Return type:

StorageTelemetry

write_arrow(table, destination, *, format_hint=None, storage_options=None, compression=None)

Write an Arrow table to storage using zero-copy buffers.

Return type:

StorageTelemetry

read_arrow(source, *, file_format, storage_options=None)

Read an artifact from storage and decode it into an Arrow table.

Return type:

tuple[Table, StorageTelemetry]

stream_read(source, *, chunk_size=None, storage_options=None)

Stream bytes from an artifact.

Return type:

Iterator[bytes]

allocate_staging_artifacts(requests)

Allocate staging metadata for upcoming loads.

Return type:

list[StagedArtifact]

cleanup_staging_artifacts(artifacts, *, ignore_errors=True)

Delete staged artifacts best-effort.

Return type:

None

class sqlspec.storage.AsyncStoragePipeline

Bases: _StoragePipelineBase

Async variant of the storage pipeline leveraging async-capable backends when available.

async stream_read_async(source, *, chunk_size=None, storage_options=None)

Stream bytes from an artifact asynchronously.

Return type:

AsyncIterator[bytes]

Registry

class sqlspec.storage.StorageRegistry

Bases: object

Global storage registry for named backend configurations.

Allows registering named storage backends that can be accessed from anywhere in your application. Backends are automatically selected based on URI scheme unless explicitly overridden.

__init__()

register_alias(alias, uri, *, backend=None, base_path='', **kwargs)

Register a named alias for a storage configuration.

Parameters:

• alias (str) -- Unique alias name

• uri (str) -- Storage URI

• backend (str | None) -- Force specific backend ("local", "fsspec", "obstore") instead of auto-detection

• base_path (str) -- Base path to prepend to all operations

• **kwargs (Any) -- Backend-specific configuration options

Return type:

None

get(uri_or_alias, *, backend=None, **kwargs)

Get backend instance using URI-first routing with automatic backend selection.

Parameters:

• uri_or_alias (str | Path) -- URI to resolve directly OR named alias

• backend (str | None) -- Force specific backend ("local", "fsspec", "obstore") instead of auto-selection

• **kwargs (Any) -- Additional backend-specific configuration options

Return type:

ObjectStoreProtocol

Returns:

Backend instance with automatic backend selection

Raises:

ImproperConfigurationError -- If alias not found or invalid input

is_alias_registered(alias)

Check if a named alias is registered.

Return type:

bool

list_aliases()

List all registered aliases.

Return type:

list[str]

clear_cache(uri_or_alias=None)

Clear resolved backend cache.

Return type:

None

clear()

Clear all aliases and instances.

Return type:

None

clear_instances()

Clear only cached instances, keeping aliases.

Return type:

None

clear_aliases()

Clear only aliases, keeping cached instances.

Return type:

None

Configuration Types

class sqlspec.storage.StorageCapabilities

Bases: TypedDict

Runtime-evaluated driver storage capabilities.

class sqlspec.storage.PartitionStrategyConfig

Bases: TypedDict

Configuration for partition fan-out strategies.

class sqlspec.storage.StorageLoadRequest

Bases: TypedDict

Request describing a staging allocation.

class sqlspec.storage.StagedArtifact

Bases: TypedDict

Metadata describing a staged artifact managed by the pipeline.

class sqlspec.storage.StorageTelemetry

Bases: TypedDict

Telemetry payload for storage bridge operations.

class sqlspec.storage.StorageBridgeJob

Bases: NamedTuple

Handle representing a storage bridge operation.

job_id: str

Alias for field number 0

status: str

Alias for field number 1

telemetry: StorageTelemetry

Alias for field number 2

static __new__(_cls, job_id: str, status: str, telemetry: StorageTelemetry)

Create new instance of StorageBridgeJob(job_id, status, telemetry)

Backends

class sqlspec.storage.backends.base.ObjectStoreBase

Bases: object

Base class for storage backends.

All synchronous methods follow the *_sync naming convention for consistency with their async counterparts.

abstractmethod resolve_uri(path)

Resolve a backend-relative path to its unsigned address.

Return type:

str

abstractmethod read_bytes_sync(path, **kwargs)

Read bytes from storage synchronously.

Return type:

bytes

abstractmethod write_bytes_sync(path, data, **kwargs)

Write bytes to storage synchronously.

Return type:

None

abstractmethod stream_read_sync(path, chunk_size=None, **kwargs)

Stream bytes from storage synchronously.

Return type:

Iterator[bytes]

abstractmethod read_text_sync(path, encoding='utf-8', **kwargs)

Read text from storage synchronously.

Return type:

str

abstractmethod write_text_sync(path, data, encoding='utf-8', **kwargs)

Write text to storage synchronously.

Return type:

None

abstractmethod list_objects_sync(prefix='', recursive=True, **kwargs)

List objects in storage synchronously.

Return type:

list[str]

abstractmethod exists_sync(path, **kwargs)

Check if object exists in storage synchronously.

Return type:

bool

abstractmethod delete_sync(path, **kwargs)

Delete object from storage synchronously.

Return type:

None

abstractmethod copy_sync(source, destination, **kwargs)

Copy object within storage synchronously.

Return type:

None

abstractmethod move_sync(source, destination, **kwargs)

Move object within storage synchronously.

Return type:

None

abstractmethod glob_sync(pattern, **kwargs)

Find objects matching pattern synchronously.

Return type:

list[str]

abstractmethod get_metadata_sync(path, **kwargs)

Get object metadata from storage synchronously.

Return type:

dict[str, object]

abstractmethod is_object_sync(path)

Check if path points to an object synchronously.

Return type:

bool

abstractmethod is_path_sync(path)

Check if path points to a directory synchronously.

Return type:

bool

abstractmethod read_arrow_sync(path, **kwargs)

Read Arrow table from storage synchronously.

Return type:

Table

abstractmethod write_arrow_sync(path, table, **kwargs)

Write Arrow table to storage synchronously.

Return type:

None

abstractmethod stream_arrow_sync(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches from storage synchronously.

Return type:

Iterator[RecordBatch]

abstractmethod async read_bytes_async(path, **kwargs)

Read bytes from storage asynchronously.

Return type:

bytes

abstractmethod async write_bytes_async(path, data, **kwargs)

Write bytes to storage asynchronously.

Return type:

None

abstractmethod async read_text_async(path, encoding='utf-8', **kwargs)

Read text from storage asynchronously.

Return type:

str

abstractmethod async write_text_async(path, data, encoding='utf-8', **kwargs)

Write text to storage asynchronously.

Return type:

None

abstractmethod async stream_read_async(path, chunk_size=None, **kwargs)

Stream bytes from storage asynchronously.

Return type:

AsyncIterator[bytes]

abstractmethod async list_objects_async(prefix='', recursive=True, **kwargs)

List objects in storage asynchronously.

Return type:

list[str]

abstractmethod async exists_async(path, **kwargs)

Check if object exists in storage asynchronously.

Return type:

bool

abstractmethod async delete_async(path, **kwargs)

Delete object from storage asynchronously.

Return type:

None

abstractmethod async copy_async(source, destination, **kwargs)

Copy object within storage asynchronously.

Return type:

None

abstractmethod async move_async(source, destination, **kwargs)

Move object within storage asynchronously.

Return type:

None

abstractmethod async get_metadata_async(path, **kwargs)

Get object metadata from storage asynchronously.

Return type:

dict[str, object]

abstractmethod async read_arrow_async(path, **kwargs)

Read Arrow table from storage asynchronously.

Return type:

Table

abstractmethod async write_arrow_async(path, table, **kwargs)

Write Arrow table to storage asynchronously.

Return type:

None

abstractmethod stream_arrow_async(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches from storage asynchronously.

Return type:

AsyncIterator[RecordBatch]

class sqlspec.storage.backends.local.LocalStore

Bases: object

Simple local file system storage backend.

Provides file system operations without requiring fsspec or obstore. Supports file:// URIs and regular file paths.

All synchronous methods use the *_sync suffix for consistency with async methods.

__init__(uri='', **kwargs)

Initialize local storage backend.

Parameters:

• uri (str) -- File URI or path

• **kwargs (Any) -- Additional options including:

• base_path (-) -- Subdirectory relative to URI path. If relative, it's combined with the URI path. If absolute, it takes precedence (backward compatible).

• file (The _sphinx_paramlinks_sqlspec.storage.backends.local.LocalStore.URI may be a) -- // path (Windows style like file:///C:/path is supported).

• provided (When _sphinx_paramlinks_sqlspec.storage.backends.local.LocalStore.both URI and base_path are) --

  • file:///home/user/storage + base_path="subdir" -> /home/user/storage/subdir

  • file:///home/user/storage + base_path="/other" -> /other (absolute takes precedence)

• combined (they _sphinx_paramlinks_sqlspec.storage.backends.local.LocalStore.are) --

  • file:///home/user/storage + base_path="subdir" -> /home/user/storage/subdir

  • file:///home/user/storage + base_path="/other" -> /other (absolute takes precedence)

resolve_uri(path)

Resolve a backend-relative path to an absolute filesystem path.

Parameters:

path (str | Path) -- The same backend-relative path accepted by read and write methods.

Return type:

str

Returns:

The absolute filesystem path. The target does not need to exist.

read_bytes_sync(path, **kwargs)

Read bytes from file synchronously.

Return type:

bytes

write_bytes_sync(path, data, **kwargs)

Write bytes to file synchronously.

Return type:

None

read_text_sync(path, encoding='utf-8', **kwargs)

Read text from file synchronously.

Return type:

str

write_text_sync(path, data, encoding='utf-8', **kwargs)

Write text to file synchronously.

Return type:

None

stream_read_sync(path, chunk_size=None, **kwargs)

Stream bytes from file synchronously.

Yields:

Chunks of bytes from the file, with size determined by chunk_size (default -- 65536 bytes).

list_objects_sync(prefix='', recursive=True, **kwargs)

List objects in directory synchronously.

Parameters:

• prefix (str) -- Optional prefix that may look like a directory or filename filter.

• recursive (bool) -- Whether to walk subdirectories.

• **kwargs (Any) -- Additional backend-specific options (currently unused).

• directory (When _sphinx_paramlinks_sqlspec.storage.backends.local.LocalStore.list_objects_sync.the prefix resembles a)

• names. (Paths _sphinx_paramlinks_sqlspec.storage.backends.local.LocalStore.list_objects_sync.outside base_path are returned with their absolute)

Return type:

list[str]

exists_sync(path, **kwargs)

Check if file exists synchronously.

Return type:

bool

delete_sync(path, **kwargs)

Delete file or directory synchronously.

Return type:

None

copy_sync(source, destination, **kwargs)

Copy file or directory synchronously.

Return type:

None

move_sync(source, destination, **kwargs)

Move file or directory synchronously.

Return type:

None

glob_sync(pattern, **kwargs)

Find files matching pattern synchronously.

Supports both relative and absolute patterns by adjusting where the glob search begins.

Return type:

list[str]

get_metadata_sync(path, **kwargs)

Get file metadata synchronously.

Return type:

dict[str, object]

is_object_sync(path)

Check if path points to a file synchronously.

Return type:

bool

is_path_sync(path)

Check if path points to a directory synchronously.

Return type:

bool

read_arrow_sync(path, **kwargs)

Read Arrow table from file synchronously.

Return type:

Table

write_arrow_sync(path, table, **kwargs)

Write Arrow table to file synchronously.

Return type:

None

stream_arrow_sync(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches from files matching pattern synchronously.

Yields:

Arrow record batches from matching files.

property supports_signing: bool

Whether this backend supports URL signing.

Local file storage does not support URL signing. Local files are accessed directly via file:// URIs.

Returns:

Always False for local storage.

sign_sync(paths, expires_in=3600, for_upload=False)

Generate signed URL(s).

Overloads:

• self, paths (str), expires_in (int), for_upload (bool) → str

• self, paths (list[str]), expires_in (int), for_upload (bool) → list[str]

Raises:

• NotImplementedError -- Local file storage does not require URL signing.

• Local files are accessed directly via file -- // URIs.

async read_bytes_async(path, **kwargs)

Read bytes from file asynchronously.

Return type:

bytes

async write_bytes_async(path, data, **kwargs)

Write bytes to file asynchronously.

Return type:

None

async read_text_async(path, encoding='utf-8', **kwargs)

Read text from file asynchronously.

Return type:

str

async write_text_async(path, data, encoding='utf-8', **kwargs)

Write text to file asynchronously.

Return type:

None

async stream_read_async(path, chunk_size=None, **kwargs)

Stream bytes from file asynchronously.

Uses AsyncThreadedBytesIterator to offload blocking file I/O to a thread pool, ensuring the event loop is not blocked during read operations.

Parameters:

• path (str | Path) -- Path to the file to read.

• chunk_size (int | None) -- Size of chunks to yield (default: 65536 bytes).

• **kwargs (Any) -- Additional arguments (unused).

Return type:

AsyncIterator[bytes]

Returns:

AsyncIterator yielding chunks of bytes.

async list_objects_async(prefix='', recursive=True, **kwargs)

List objects asynchronously.

Return type:

list[str]

async exists_async(path, **kwargs)

Check if file exists asynchronously.

Return type:

bool

async delete_async(path, **kwargs)

Delete file asynchronously.

Return type:

None

async copy_async(source, destination, **kwargs)

Copy file asynchronously.

Return type:

None

async move_async(source, destination, **kwargs)

Move file asynchronously.

Return type:

None

async get_metadata_async(path, **kwargs)

Get file metadata asynchronously.

Return type:

dict[str, object]

async read_arrow_async(path, **kwargs)

Read Arrow table asynchronously.

Uses async_() to offload blocking PyArrow I/O to thread pool.

Return type:

Table

async write_arrow_async(path, table, **kwargs)

Write Arrow table asynchronously.

Uses async_() to offload blocking PyArrow I/O to thread pool.

Return type:

None

stream_arrow_async(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches asynchronously.

Parameters:

• pattern (str) -- Glob pattern to match files.

• file_format (Literal['parquet']) -- Storage format. Only Parquet supports bounded batch streaming.

• batch_size (int) -- Maximum number of rows in each yielded record batch.

• **kwargs (Any) -- Additional arguments passed to stream_arrow_sync().

Return type:

AsyncIterator[RecordBatch]

Returns:

AsyncIterator yielding Arrow record batches.

async sign_async(paths, expires_in=3600, for_upload=False)

Generate signed URL(s) asynchronously.

Overloads:

• self, paths (str), expires_in (int), for_upload (bool) → str

• self, paths (list[str]), expires_in (int), for_upload (bool) → list[str]

class sqlspec.storage.backends.fsspec.FSSpecBackend

Bases: object

Storage backend using fsspec.

Implements ObjectStoreProtocol using fsspec for various protocols including HTTP, HTTPS, FTP, and cloud storage services.

All synchronous methods use the *_sync suffix for consistency with async methods.

__init__(uri, **kwargs)

Initialize the fsspec-backed storage backend.

Parameters:

• uri (str) -- Filesystem URI (protocol://path).

• **kwargs (Any) -- Additional fsspec configuration options, including an optional base_path.

• URIs (For _sphinx_paramlinks_sqlspec.storage.backends.fsspec.FSSpecBackend.cloud) -- // URIs, we derive a default base_path from the URI path when no explicit base_path is provided. When both URI and base_path are provided, they are combined (base_path is appended to URI-derived path).

resolve_uri(path)

Resolve a backend-relative path to an unsigned address.

Parameters:

path (str | Path) -- The same backend-relative path accepted by read and write methods.

Return type:

str

Returns:

An absolute filesystem path for file or a protocol-qualified URI for other filesystems. The target does not need to exist.

read_bytes_sync(path, **kwargs)

Read bytes from an object synchronously.

Return type:

bytes

write_bytes_sync(path, data, **kwargs)

Write bytes to an object synchronously.

Return type:

None

read_text_sync(path, encoding='utf-8', **kwargs)

Read text from an object synchronously.

Return type:

str

write_text_sync(path, data, encoding='utf-8', **kwargs)

Write text to an object synchronously.

Return type:

None

exists_sync(path, **kwargs)

Check if an object exists synchronously.

Return type:

bool

delete_sync(path, **kwargs)

Delete an object synchronously.

Return type:

None

copy_sync(source, destination, **kwargs)

Copy an object synchronously.

Return type:

None

move_sync(source, destination, **kwargs)

Move an object synchronously.

Return type:

None

read_arrow_sync(path, **kwargs)

Read an Arrow table from storage synchronously.

Return type:

Table

write_arrow_sync(path, table, **kwargs)

Write an Arrow table to storage synchronously.

Return type:

None

list_objects_sync(prefix='', recursive=True, **kwargs)

List objects with optional prefix synchronously.

Return type:

list[str]

glob_sync(pattern, **kwargs)

Find objects matching a glob pattern synchronously.

Return type:

list[str]

is_object_sync(path)

Check if path points to an object synchronously.

Return type:

bool

is_path_sync(path)

Check if path points to a prefix (directory-like) synchronously.

Return type:

bool

get_metadata_sync(path, **kwargs)

Get object metadata synchronously.

Return type:

dict[str, object]

property supports_signing: bool

Whether this backend supports URL signing.

FSSpec backends do not support URL signing. Use ObStoreBackend for S3, GCS, or Azure if you need signed URLs.

Returns:

Always False for fsspec backends.

sign_sync(paths, expires_in=3600, for_upload=False)

Generate signed URL(s).

Overloads:

• self, paths (str), expires_in (int), for_upload (bool) → str

• self, paths (list[str]), expires_in (int), for_upload (bool) → list[str]

Raises:

NotImplementedError -- fsspec backends do not support URL signing. Use obstore backend for S3, GCS, or Azure if you need signed URLs.

stream_read_sync(path, chunk_size=None, **kwargs)

Stream bytes from storage synchronously.

Yields:

Chunks of bytes from the file, with size determined by chunk_size (default -- 65536 bytes).

stream_arrow_sync(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches from storage synchronously.

Parameters:

• pattern (str) -- The glob pattern to match.

• file_format (Literal['parquet']) -- Storage format. Only Parquet supports bounded batch streaming.

• batch_size (int) -- Maximum number of rows in each yielded record batch.

• **kwargs (Any) -- Additional arguments passed to PyArrow batch iteration.

Yields:

Arrow record batches from matching files.

async read_bytes_async(path, **kwargs)

Read bytes from storage asynchronously.

Return type:

bytes

async write_bytes_async(path, data, **kwargs)

Write bytes to storage asynchronously.

Return type:

None

async stream_read_async(path, chunk_size=None, **kwargs)

Stream bytes from storage asynchronously.

Uses AsyncThreadedBytesIterator to read chunks of the file in a thread pool, ensuring the event loop is not blocked while avoiding buffering the entire file into memory.

Parameters:

• path (str | Path) -- Path to the file to read.

• chunk_size (int | None) -- Size of chunks to yield (default: 65536 bytes).

• **kwargs (Any) -- Additional arguments passed to fs.open.

Return type:

AsyncIterator[bytes]

Returns:

AsyncIterator yielding chunks of bytes.

stream_arrow_async(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches from storage asynchronously.

Parameters:

• pattern (str) -- The glob pattern to match.

• file_format (Literal['parquet']) -- Storage format. Only Parquet supports bounded batch streaming.

• batch_size (int) -- Maximum number of rows in each yielded record batch.

• **kwargs (Any) -- Additional arguments passed to PyArrow batch iteration.

Return type:

AsyncIterator[RecordBatch]

Returns:

AsyncIterator yielding Arrow record batches.

async read_text_async(path, encoding='utf-8', **kwargs)

Read text from storage asynchronously.

Return type:

str

async write_text_async(path, data, encoding='utf-8', **kwargs)

Write text to storage asynchronously.

Return type:

None

async list_objects_async(prefix='', recursive=True, **kwargs)

List objects in storage asynchronously.

Return type:

list[str]

async exists_async(path, **kwargs)

Check if object exists in storage asynchronously.

Return type:

bool

async delete_async(path, **kwargs)

Delete object from storage asynchronously.

Return type:

None

async copy_async(source, destination, **kwargs)

Copy object in storage asynchronously.

Return type:

None

async move_async(source, destination, **kwargs)

Move object in storage asynchronously.

Return type:

None

async get_metadata_async(path, **kwargs)

Get object metadata from storage asynchronously.

Return type:

dict[str, object]

async sign_async(paths, expires_in=3600, for_upload=False)

Generate signed URL(s) asynchronously.

Overloads:

• self, paths (str), expires_in (int), for_upload (bool) → str

• self, paths (list[str]), expires_in (int), for_upload (bool) → list[str]

async read_arrow_async(path, **kwargs)

Read Arrow table from storage asynchronously.

Uses async_() with storage limiter to offload blocking PyArrow I/O to thread pool.

Return type:

Table

async write_arrow_async(path, table, **kwargs)

Write Arrow table to storage asynchronously.

Uses async_() with storage limiter to offload blocking PyArrow I/O to thread pool.

Return type:

None

class sqlspec.storage.backends.obstore.ObStoreBackend

Bases: object

Object storage backend using obstore.

Implements ObjectStoreProtocol using obstore's Rust-based implementation for storage operations. Supports AWS S3, Google Cloud Storage, Azure Blob Storage, local filesystem, and HTTP endpoints.

All synchronous methods use the *_sync suffix for consistency with async methods.

Implementation Details & Invariants:

• LocalStore Paths: For LocalStore, the base_path is already included in the store root (combined with the URI path; if base_path is absolute, Path division will use it directly). Hence, we use an empty prefix when resolving paths for LocalStore, whereas cloud stores use base_path as a prefix.

• Native Streaming: Uses obstore's native streaming yielding Buffer objects, which are converted to bytes.

• Seekable Streams: PyArrow's ParquetFile reads through obstore's seekable open_reader interface without draining the object into memory.

• Thread Offloading: Uses async_() with a storage limiter to offload blocking PyArrow serialization/parsing to a thread pool, preventing event loop blocking.

__init__(uri, **kwargs)

Initialize obstore backend.

Parameters:

• uri (str) -- Storage URI. Supported formats:

• file (-) -- ///absolute/path - Local filesystem

• s3 (-) -- //bucket/prefix - AWS S3

• gs (-) -- //bucket/prefix - Google Cloud Storage

• az (-) -- //container/prefix - Azure Blob Storage

• memory (-) -- // - In-memory storage (for testing)

• **kwargs (Any) -- Additional options:

• base_path (-) -- For local files (file://), this is combined with

• example (the _sphinx_paramlinks_sqlspec.storage.backends.obstore.ObStoreBackend.URI path to form the storage root. For)

• uri="file -- ///data" + base_path="uploads" → /data/uploads If base_path is absolute, it overrides the URI path (backward compat). For cloud storage, base_path is used as an object key prefix.

• options (- _sphinx_paramlinks_sqlspec.storage.backends.obstore.ObStoreBackend.Other obstore configuration)

classmethod from_config(config)

Create backend from configuration dictionary.

Return type:

ObStoreBackend

resolve_uri(path)

Resolve a backend-relative path to an unsigned address.

Parameters:

path (str | Path) -- The same backend-relative path accepted by read and write methods.

Return type:

str

Returns:

An absolute filesystem path for local stores or a protocol-qualified URI for remote stores. The target does not need to exist.

read_bytes_sync(path, **kwargs)

Read bytes using obstore synchronously.

Return type:

bytes

write_bytes_sync(path, data, **kwargs)

Write bytes using obstore synchronously.

Return type:

None

read_text_sync(path, encoding='utf-8', **kwargs)

Read text using obstore synchronously.

Return type:

str

write_text_sync(path, data, encoding='utf-8', **kwargs)

Write text using obstore synchronously.

Return type:

None

list_objects_sync(prefix='', recursive=True, **kwargs)

List objects using obstore synchronously.

Return type:

list[str]

exists_sync(path, **kwargs)

Check if object exists using obstore synchronously.

Return type:

bool

delete_sync(path, **kwargs)

Delete object using obstore synchronously.

Return type:

None

copy_sync(source, destination, **kwargs)

Copy object using obstore synchronously.

Return type:

None

move_sync(source, destination, **kwargs)

Move object using obstore synchronously.

Return type:

None

glob_sync(pattern, **kwargs)

Find objects matching pattern synchronously.

Lists all objects and filters them client-side using the pattern.

Return type:

list[str]

get_metadata_sync(path, **kwargs)

Get object metadata using obstore synchronously.

Return type:

dict[str, object]

is_object_sync(path)

Check if path is an object using obstore synchronously.

Return type:

bool

is_path_sync(path)

Check if path is a prefix/directory using obstore synchronously.

Return type:

bool

read_arrow_sync(path, **kwargs)

Read Arrow table using obstore synchronously.

Return type:

Table

write_arrow_sync(path, table, **kwargs)

Write Arrow table using obstore synchronously.

Return type:

None

stream_read_sync(path, chunk_size=None, **kwargs)

Stream bytes using obstore's native streaming synchronously.

Uses obstore's sync streaming iterator which yields chunks without loading the entire file into memory, for both local and remote backends.

Yields:

Chunks of bytes from the file, with size determined by chunk_size (default -- 65536 bytes).

stream_arrow_sync(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches using obstore's native streaming synchronously.

For each matching file, PyArrow reads through obstore's seekable reader.

Yields:

Arrow record batches in file and row-group order.

property supports_signing: bool

Whether this backend supports URL signing.

Only S3, GCS, and Azure backends support pre-signed URLs. Local file storage does not support URL signing.

Returns:

True if the protocol supports signing, False otherwise.

sign_sync(paths, expires_in=3600, for_upload=False)

Generate signed URL(s) for the object(s).

Overloads:

• self, paths (str), expires_in (int), for_upload (bool) → str

• self, paths (list[str]), expires_in (int), for_upload (bool) → list[str]

Parameters:

• paths (str | list[str]) -- Single object path or list of paths to sign.

• expires_in (int) -- URL expiration time in seconds (default: 3600, max: 604800 = 7 days).

• for_upload (bool) -- Whether the URL is for upload (PUT) vs download (GET).

Returns:

Single signed URL string if paths is a string, or list of signed URLs if paths is a list. Preserves input type for convenience.

async read_bytes_async(path, **kwargs)

Read bytes from storage asynchronously.

Return type:

bytes

async write_bytes_async(path, data, **kwargs)

Write bytes to storage asynchronously.

Return type:

None

async stream_read_async(path, chunk_size=None, **kwargs)

Stream bytes from storage asynchronously.

Uses obstore's native async streaming to yield chunks of bytes without buffering the entire file into memory.

Return type:

AsyncIterator[bytes]

async list_objects_async(prefix='', recursive=True, **kwargs)

List objects in storage asynchronously.

Return type:

list[str]

async read_text_async(path, encoding='utf-8', **kwargs)

Read text from storage asynchronously.

Return type:

str

async write_text_async(path, data, encoding='utf-8', **kwargs)

Write text to storage asynchronously.

Return type:

None

async exists_async(path, **kwargs)

Check if object exists in storage asynchronously.

Return type:

bool

async delete_async(path, **kwargs)

Delete object from storage asynchronously.

Return type:

None

async copy_async(source, destination, **kwargs)

Copy object in storage asynchronously.

Return type:

None

async move_async(source, destination, **kwargs)

Move object in storage asynchronously.

Return type:

None

async get_metadata_async(path, **kwargs)

Get object metadata from storage asynchronously.

Return type:

dict[str, object]

async read_arrow_async(path, **kwargs)

Read Arrow table from storage asynchronously.

Uses async_() with storage limiter to offload blocking PyArrow I/O to thread pool.

Return type:

Table

async write_arrow_async(path, table, **kwargs)

Write Arrow table to storage asynchronously.

Uses async_() with storage limiter to offload blocking PyArrow serialization to thread pool, preventing event loop blocking.

Return type:

None

stream_arrow_async(pattern, *, file_format='parquet', batch_size=65536, **kwargs)

Stream Arrow record batches from storage asynchronously.

Parameters:

• pattern (str) -- Glob pattern to match files.

• file_format (Literal['parquet']) -- Storage format. Only Parquet supports bounded batch streaming.

• batch_size (int) -- Maximum number of rows in each yielded record batch.

• **kwargs (Any) -- Additional arguments passed to stream_arrow_sync().

Return type:

AsyncIterator[RecordBatch]

Returns:

AsyncIterator yielding Arrow record batches.

async sign_async(paths, expires_in=3600, for_upload=False)

Generate signed URL(s) asynchronously.

Overloads:

• self, paths (str), expires_in (int), for_upload (bool) → str

• self, paths (list[str]), expires_in (int), for_upload (bool) → list[str]

Parameters:

• paths (str | list[str]) -- Single object path or list of paths to sign.

• expires_in (int) -- URL expiration time in seconds (default: 3600, max: 604800 = 7 days).

• for_upload (bool) -- Whether the URL is for upload (PUT) vs download (GET).

Returns:

Single signed URL string if paths is a string, or list of signed URLs if paths is a list. Preserves input type for convenience.

Module Functions

sqlspec.storage.create_storage_bridge_job(status, telemetry)

Create a storage bridge job handle with a unique identifier.

Return type:

StorageBridgeJob

sqlspec.storage.get_storage_bridge_diagnostics()

Return aggregated storage bridge + serializer cache metrics.

Return type:

dict[str, float]

sqlspec.storage.get_storage_bridge_metrics()

Return aggregated storage bridge metrics.

Return type:

dict[str, int]

sqlspec.storage.reset_storage_bridge_metrics()

Reset aggregated storage bridge metrics.

Return type:

None

sqlspec.storage.resolve_storage_path(path, base_path='', protocol='file', strip_file_scheme=True)

Resolve path relative to base_path with protocol-specific handling.

Parameters:

• path (str | Path) -- Path to resolve.

• base_path (str) -- Base path to prepend if path is relative.

• protocol (str) -- Storage protocol.

• strip_file_scheme (bool) -- Whether to strip file:// prefixes.

Return type:

str

Returns:

Resolved path string suitable for the storage backend.