aistore.pytorch.iter_dataset

Iterable Dataset for AIS

Module Contents

Classes

Name	Description
`AISIterDataset`	An iterable-style dataset that iterates over objects in AIS and yields

API

class aistore.pytorch.iter_dataset.AISIterDataset(
    ais_source_list: typing.Union[aistore.sdk.AISSource, typing.List[aistore.sdk.AISSource]],
    prefix_map: typing.Dict[aistore.sdk.AISSource, typing.Union[str, typing.List[str]]] = {},
    etl_name: str = None,
    show_progress: bool = False,
    partition_sources_by_worker: bool = False
)

Bases: AISBaseIterDataset

An iterable-style dataset that iterates over objects in AIS and yields samples represented as a tuple of object_name (str) and object_content (bytes). If etl_name is provided, that ETL must already exist on the AIStore cluster.

Parameters:

ais_source_list

Union[AISSource, List[AISSource]]

Single or list of AISSource objects to load data

prefix_map

Dict(AISSource, Union[str, List[str]])Defaults to {}

Map of AISSource objects to list of prefixes that only allows

etl_name

strDefaults to None

Optional ETL on the AIS cluster to apply to each object

show_progress

boolDefaults to False

Enables console dataset reading progress indicator

partition_sources_by_worker

boolDefaults to False

When True, distributes sources across DataLoader workers so each worker only lists its share, avoiding duplicate paged listing calls. Most effective when ais_source_list has at least as many sources as workers. Defaults to False.

aistore.pytorch.iter_dataset.AISIterDataset.__iter__()