DiFfRG.parallel#
Running an analysis over many simulations at once, and remembering the result.
A scan is a folder full of runs, and what is usually wanted from it is one number per run: a condensate, the position of a shock, the outcome of a fit. Two things make that slow, and this module addresses both.
The work is done run by run in the notebook process. sim_map spreads it over a pool of workers instead. The function is shipped with cloudpickle, so a function defined in a notebook cell works, and only its (small) return value travels back.
The same analysis is re-run every time a plot is redrawn. sim_map remembers what it computed for each file, in this process and in a cache file, so a second pass over an unchanged scan costs nothing.
The cache is keyed on the file (its path, its mtime and its size) and on the code of the function: an edited analysis recomputes on its own, a rerun simulation is picked up on its own, and neither needs the cache to be cleared by hand. See sim_map.
Functions#
|
The number of worker processes used when none is given. |
|
The process pool used by pmap and sim_map, created on first use. |
Stops the worker processes, if any are running. |
|
|
[func(item) for item in items], evaluated over the process pool. |
|
The file sim_map stores its results in, optionally moving it. |
Writes the cache out, merging in whatever another session has added. |
|
|
Forgets every remembered result. |
|
func(sim) for every simulation, over the pool, remembered per file. |
|
read_meta for many files at once, over the pool. |
Module Contents#
- DiFfRG.parallel.default_workers() int#
The number of worker processes used when none is given.
The number of cores this process may actually run on, which under a batch system is the allocation rather than the size of the machine.
- DiFfRG.parallel.get_executor(workers=None)#
The process pool used by pmap and sim_map, created on first use.
Deliberately not loky’s get_reusable_executor: that one is a singleton shared with whatever else uses it – an adaptive runner, most likely – and asking it for a different number of workers tears down its pool.
- Parameters:
workers (int, optional) – The number of worker processes. Defaults to default_workers().
- Returns:
The pool.
- Return type:
loky.ProcessPoolExecutor
- DiFfRG.parallel.shutdown_executor()#
Stops the worker processes, if any are running.
- DiFfRG.parallel.pmap(func, items, workers=None) list#
[func(item) for item in items], evaluated over the process pool.
- Parameters:
func (callable) – What to apply. Shipped with cloudpickle, so a function defined in a notebook cell, a lambda or a closure all work.
items (iterable) – Its arguments, one per call.
workers (int, optional) – The number of worker processes. Defaults to default_workers().
- Returns:
The results, in the order of items.
- Return type:
- DiFfRG.parallel.cache_file(path=None) str#
The file sim_map stores its results in, optionally moving it.
Defaults to $XDG_CACHE_HOME/DiFfRG/sim_map.pkl. The entries are keyed on the absolute path of the simulation file, so one cache serves every project.
- DiFfRG.parallel.cache_save()#
Writes the cache out, merging in whatever another session has added.
Called at the end of every sim_map, so there is usually no reason to call it directly.
- DiFfRG.parallel.cache_clear(disk=True)#
Forgets every remembered result.
Only needed when the cache is in the way – an analysis whose result depends on something other than the simulation file and its own code, say. Editing the analysis or rerunning a simulation invalidates the affected entries by itself.
- Parameters:
disk (bool, optional) – Whether to delete the cache file as well.
- DiFfRG.parallel.sim_map(func, sims, workers=None, cache=True) list#
func(sim) for every simulation, over the pool, remembered per file.
Meant for the observables an analysis reduces a run to; the result has to be something small, since it travels back from a worker process and is kept in the cache. The whole point is that the run itself does not travel: a SimulationData is sent as the path it reads, and the worker reads the frames it needs.
- Parameters:
- Returns:
func’s value for each simulation, in order.
- Return type:
- DiFfRG.parallel.read_metas(paths, workers=None, cache=True) dict#
read_meta for many files at once, over the pool.
Files that cannot be read are left out of the result rather than raising. What has been read before, and has not been written to since, is taken from memory: redrawing a plot then does not read the folder again. That memory is bounded, and holds the few thousand most recently used files.