When the Dask array inside a Data object is computed the compute result is currently not cached. This means that when the data is accessed again, the computation (which may involve a reading from disk) is repeated. This is great for keeping resident memory down, but not so good for speed performance. Sometimes, speed needs to be prioritised over memory, particularly when data is being accessed from remote stores.
This behaviour should be controllable in two ways:
- Provide a
persistkeyword argument toData.computewhich leaves the Dask graph in a persisted state. - Provide a configuration function
cf.persist_datathat controls whether or not to persist the data of every Dask graph computation.
Note that accessing the array attribute always trigger a Dask graph computation under the hood.
Currently, the only way to do this is in two stages:
>>> # Current (== 3.19.0) behaviour >>> d.persist(inplace=True) >>> a = d.compute() >>> a = d.array
Proposed new options:
>>> # New ( > 3.19.0) behaviour >>> a = d.compute(persist=True) >>> with cf.persist_data(True): ... a = d.compute() ... >>> cf.persist_data(True) >>> a = d.compute(persist=False) # Override environment setting