lenskit.parallel

LensKit parallel computation support.

lenskit.parallel.initialize(*, processes=None, threads=None, backend_threads=None, child_threads=None)

Set up and configure LensKit parallelism. This only needs to be called if you want to control when and how parallelism is set up; components using parallelism will call ensure_init(), which will call this function with its default arguments if it has not been called.

Parameters:

• processes (int | None) – The number of processes to use for multiprocessing evaluations (see LK_NUM_PROCS)

• threads (int | None) – The number of threads to use for parallel model training and similar operations (see LK_NUM_THREADS).

• backend_threads (int | None) – The number of threads underlying computational engines should use (see LK_NUM_BACKEND_THREADS).

• child_threads (int | None) – The number of threads backends are allowed to use in the worker processes in multiprocessing operations (see LK_NUM_CHILD_THREADS).

lenskit.parallel.get_parallel_config()

Ensure that parallelism is configured and return the configuration.

Return type:

ParallelConfig

lenskit.parallel.ensure_parallel_init()

Make sure LensKit parallelism is configured, and configure with defaults if it is not.

Components using parallelism or intensive computations should call this function before they begin training.

lenskit.parallel.invoker(model, func, n_jobs=None, progress=None)

Get an appropriate invoker for performing operations on model.

Parameters:

• model (M) – The model object on which to perform operations.

• func (Callable[[M, A], R]) – The function to call. The function must be pickleable.

• n_jobs (int | None) – The number of processes to use for parallel operations. If None, will call proc_count() with a maximum default process count of 4.

• progress (Progress | None) –

  A progress bar to use to report status. It should have the following states:

  • dispatched

  • in-progress

  • finished

  One can be created with invoke_progress()

Returns:

An invoker to perform operations on the model.

Return type:

ModelOpInvoker

lenskit.parallel.invoke_progress(logger=None, label=None, total=None, unit=None)

Create a progress bar for parallel tasks. It is populated with the correct state of tasks for invoker().

See make_progress() for details on parameter meanings.

Parameters:

• logger (str | Logger | None)

• label (str | None)

• total (int | None)

• unit (str | None)

Return type:

Progress

class lenskit.parallel.ModelOpInvoker

Bases: ABC, Generic[A, R]

Interface for invoking operations on a model, possibly in parallel. The operation invoker is configured with a model and a function to apply, and applies that function to the arguments supplied in map. Child process invokers also route logging messages to the parent process, so logging works even with multiprocessing.

An invoker is a context manager that calls shutdown() when exited.

abstract map(tasks)

Apply the configured function to the model and iterables. This is like map(), except it supplies the invoker’s model as the first object to func.

Parameters:

• iterables – Iterables of arguments to provide to the function.

• tasks (Iterable[A])

Returns:

An iterable of the results.

Return type:

iterable

Modules

chunking	Support for computing chunks for parallel computation.
config
invoker(model, func[, n_jobs, progress])	Get an appropriate invoker for performing operations on model.
pool
sequential
serialize	Serialization utilities for parallel processing.
worker