lenskit.parallel#

LensKit parallel computation support.

lenskit.parallel.initialize(config=None, *, processes=None, threads=None, backend_threads=None, child_threads=None)#

Set up and configure LensKit parallelism. This only needs to be called if you want to control when and how parallelism is set up; components using parallelism will call ensure_init(), which will call this function with its default arguments if it has not been called.

See also

Configuring Parallelism

Parameters:

  • processes (int | None) – The number of processes to use for multiprocessing evaluations (see LK_NUM_PROCS)

  • threads (int | None) – The number of threads to use for parallel model training and similar operations (see LK_NUM_THREADS).

  • backend_threads (int | None) – The number of threads underlying computational engines should use (see LK_NUM_BACKEND_THREADS).

  • child_threads (int | None) – The number of threads backends are allowed to use in the worker processes in multiprocessing operations (see LK_NUM_CHILD_THREADS).

  • config (ParallelConfig | None)

lenskit.parallel.get_parallel_config()#

Ensure that parallelism is configured and return the configuration.

Return type:

ParallelConfig

lenskit.parallel.ensure_parallel_init()#

Make sure LensKit parallelism is configured, and configure with defaults if it is not.

Components using parallelism or intensive computations should call this function before they begin training.

lenskit.parallel.invoker(model, func, n_jobs=None, *, worker_parallel=None)#

Get an appropriate invoker for performing operations on model.

Parameters:

  • model (M) – The model object on which to perform operations.

  • func (Callable[[M, A], R]) – The function to call. The function must be pickleable.

  • n_jobs (int | None) – The number of processes to use for parallel operations. If None, will call proc_count() with a maximum default process count of 4.

  • worker_parallel (ParallelConfig | None) – A parallel configuration for subprocess workers.

Returns:

An invoker to perform operations on the model.

Return type:

ModelOpInvoker

class lenskit.parallel.ModelOpInvoker#

Bases: ABC, Generic[A, R]

Interface for invoking operations on a model, possibly in parallel. The operation invoker is configured with a model and a function to apply, and applies that function to the arguments supplied in map. Child process invokers also route logging messages to the parent process, so logging works even with multiprocessing.

An invoker is a context manager that calls shutdown() when exited.

abstract map(tasks)#

Apply the configured function to the model and iterables. This is like map(), except it supplies the invoker’s model as the first object to func.

Parameters:

  • iterables – Iterables of arguments to provide to the function.

  • tasks (Iterable[A])

Returns:

An iterable of the results.

Return type:

iterable

lenskit.parallel.multiprocess_executor(n_jobs=None, sp_config=None)#

Construct a ProcessPoolExecutor configured for LensKit work.

Parameters:
Return type:

ProcessPoolExecutor

Modules

chunking

Support for computing chunks for parallel computation.

config

invoker(model, func[, n_jobs, worker_parallel])

Get an appropriate invoker for performing operations on model.

pool

sequential

serialize

Serialization utilities for parallel processing.

worker