auto_dataloader¶

ignite.distributed.auto.auto_dataloader(dataset, **kwargs)[source]¶

Helper method to create a dataloader adapted for non-distributed and distributed configurations (supporting all available backends from available_backends()).

Internally, we create a dataloader with provided kwargs while applying the following updates:

batch size is scaled by world size: batch_size / world_size if larger or equal world size.
number of workers is scaled by number of local processes: num_workers / nprocs if larger or equal world size.
if no sampler provided by user, torch DistributedSampler is setup.
if a sampler is provided by user, it is wrapped by DistributedProxySampler.
if the default device is ‘cuda’, pin_memory is automatically set to True.

Warning

Custom batch sampler is not adapted for distributed configuration. Please, make sure that provided batch sampler is compatible with distributed configuration.

Examples:

import ignite.distribted as idist

train_loader = idist.auto_dataloader(
    train_dataset,
    batch_size=32,
    num_workers=4,
    shuffle=True,
    pin_memory="cuda" in idist.device().type,
    drop_last=True,
)

Parameters

dataset (torch.utils.data.dataset.Dataset) – input torch dataset
kwargs (Any) – keyword arguments for torch DataLoader.

Returns

torch DataLoader or XLA MpDeviceLoader for XLA devices

Return type

Union[torch.utils.data.dataloader.DataLoader, _MpDeviceLoader]