lenskit.parallel.ray#

Support for parallelism with Ray.

Functions

ensure_cluster()

inference_worker_cpus()

init_cluster(*[, num_cpus, proc_slots, ...])

Initialize or connect to a Ray cluster, with the LensKit options.

is_ray_worker()

Determine whether the current process is running on a Ray worker.

ray_active()

Query whether Ray is active.

ray_available()

Check if Ray is available.

ray_supported()

Check if this Ray setup is supported by LensKit.

training_worker_cpus()

Classes

RayOpInvoker(model, func)

lenskit.parallel.ray.init_cluster(*, num_cpus=None, proc_slots=None, resources=None, worker_parallel=None, limit_slots=True, **kwargs)#

Initialize or connect to a Ray cluster, with the LensKit options.

The resulting cluster can be used by an invoker, or it can be used directly. The Ray invoker uses batching, though, so it only works well with many small tasks.

Parameters:

  • num_cpus (int | None) – The total number of CPUs to allow. Defaults to :fun:`effective_cpu_count`.

  • proc_slots (int | None) – The number of “process slots” for LensKit parallel operations. Defaults to the LensKit process count. These slots are recorded as the lk_process resource on the Ray cluster.

  • resources (dict[str, float] | None) – Additional custom resources to register in the Ray cluster.

  • worker_parallel (ParallelConfig | None) – Parallel processing configuration for worker processes. If None, uses the default.

  • limit_slots (bool) – False to disable the LensKit slot interface.

  • kwargs – Other options to pass to ray.init().

Stability:

Experimental

lenskit.parallel.ray.ray_supported()#

Check if this Ray setup is supported by LensKit.

Return type:

bool

lenskit.parallel.ray.ray_available()#

Check if Ray is available.

Return type:

bool

lenskit.parallel.ray.ray_active()#

Query whether Ray is active.

Return type:

bool

lenskit.parallel.ray.is_ray_worker()#

Determine whether the current process is running on a Ray worker.

Return type:

bool

class lenskit.parallel.ray.RayOpInvoker(model, func)#

Bases: ModelOpInvoker[A, R], Generic[M, A, R]

Parameters:

  • model (M)

  • func (InvokeOp[M, A, R])

map(tasks)#

Apply the configured function to the model and iterables. This is like map(), except it supplies the invoker’s model as the first object to func.

Parameters:

  • iterables – Iterables of arguments to provide to the function.

  • tasks (Iterable[A])

Returns:

An iterable of the results.

Return type:

iterable

shutdown()#

Shut down this invoker.