lenskit.data.ItemListCollection#

class lenskit.data.ItemListCollection(key, *, index=True)#

Bases: Generic[KL], ABC

A collection of item lists. This protocol defines read access to the collection; see ItemListCollector for the ability to add new lists.

An item list collection consists of a sequence of item lists with associated keys following a fixed schema. Item list collections support iteration (in order) and lookup by key. They are used to represent a variety of things, including test data and the results of a batch run.

The key schema can be specified either by a list of field names, or by providing a named tuple class (created by either namedtuple() or NamedTuple) defining the key schema. Schemas should not be nested: field values must be scalars, not tuples or lists. Keys should also be hashable.

This protocol and its implementations exist, instead of using raw dictionaries or lists, to consistently handle some of the nuances of multi-valued keys, and different collections having different key fields. For example, if a run produces item lists with both user IDs and sequence numbers, but your test data is only indexed by user ID, the projected lookup capabilities make it easy to find the test data to go with an item list in the run.

Item list collections support indexing by position, like a list, returning a tuple of the key and list; iterating over an item list collection similarly produces (key, list) pairs (so an item list collection is a Sequence of key/list pairs).

If the item list is _indexed_ (constructed with index=True), it also supports lookup by _key_ with lookup(). The key can be supplied as either a tuple or an instance of the key type. If more than one item with the same key is inserted into the collection, then the _last_ one is returned (just like a dictionary), but the others remain in the underlying list when it is iterated.

Note

Constructing an item list collection yields a ListILC.

Parameters:

  • key (type[KL] | Sequence[str]) – The type (a NamedTuple class) or list of field names specifying the key schema.

  • index (bool) – Whether or not to index lists by key to facilitate fast lookups.

__init__(key)#
Parameters:

key (type[KL] | Sequence[str])

Methods

__init__(key)

empty(key, *[, index])

Create a new empty, mutable item list collection.

from_df(df, key, *others)

Create an item list collection from a data frame.

from_dict()

Create an item list collection from a dictionary.

items()

Iterate over item lists and keys.

keys()

Iterate over keys.

lists()

Iterate over item lists without keys.

load_parquet()

Load this item list from a Parquet file.

lookup()

Look up a list by key.

lookup_projected(key)

Look up an item list using a projected key.

record_batches([batch_size, columns])

Get the item list collection as Arrow record batches (in native layout).

save_parquet(path, *[, layout, batch_size, ...])

Save this item list collection to a Parquet file.

to_arrow(*[, batch_size])

Convert this item list collection to an Arrow table.

to_df()

Convert this item list collection to a data frame.

Attributes

key_fields

The names of the key fields.

key_type

The type of collection keys.

list_schema

Get the schema for the lists in this ILC.
static empty(key, *, index=True)#

Create a new empty, mutable item list collection.

Parameters:
Return type:

MutableItemListCollection[K]

static from_dict(data: Mapping[tuple[int | str | bytes | integer[Any] | str_ | bytes_ | object_, ...] | int | str | bytes | integer[Any] | str_ | bytes_ | object_, ItemList], key: type[K]) → ItemListCollection[K]#
static from_dict(data: Mapping[tuple[int | str | bytes | integer[Any] | str_ | bytes_ | object_, ...] | int | str | bytes | integer[Any] | str_ | bytes_ | object_, ItemList], key: Sequence[str] | str | None = None) → ItemListCollection[tuple[int | str | bytes | integer[Any] | str_ | bytes_ | object_, ...]]

Create an item list collection from a dictionary.

See also

lenskit.data.collection.ListILC.from_dict()

static from_df(df, key, *others)#

Create an item list collection from a data frame.

See also

lenskit.data.collection.ListILC.from_df()

Note

Keys with empty item lists will be silently excluded from the output data.

Parameters:

  • df (DataFrame) – The data frame to convert.

  • key (type[K] | Sequence[str | AliasedColumn] | str | AliasedColumn) – The key type or field(s). Can be specified as a single column name (or AliasedColumn).

  • others (str | AliasedColumn) – Other columns to consider; primarily used to pass additional aliased columns to normalize other clumnes like the item ID.

Return type:

MutableItemListCollection[Any]

to_df()#

Convert this item list collection to a data frame.

Warning

If this item list collection has any keys with empty lists, those lists will be excluded from the output.

Return type:

DataFrame

to_arrow(*, batch_size=5000)#

Convert this item list collection to an Arrow table.

The resulting table has one row per item list, with the item list contents an items column of a structured list type. This preserves empty item lists for higher-fidelity data storage.

Parameters:

batch_size (int) – The Arrow record batch size.

Return type:

Table

save_parquet(path, *, layout='native', batch_size=5000, compression='zstd', mkdir=True)#

Save this item list collection to a Parquet file. This supports two types of Parquet files: “native” collections store one row per list, with the item list contents in a repeated structure column named items; this layout fully preserves the item list collection, including empty item lists. The “flat” layout is easier to work with in software such as Pandas, but cannot store empty item lists.

Parameters:

  • layout (Literal['native', 'flat']) – The table layout to use.

  • batch_size (int) – The Arrow record batch size.

  • compression (Literal['zstd', 'gzip', 'snappy', 'lz4'] | None) – The compression scheme to use.

  • mkdir (bool) – Whether to create the parent directories if they don’t exist.

  • path (PathLike[str])

Return type:

None

classmethod load_parquet(path: PathLike[str] | list[PathLike[str]], *, layout: Literal['native'] = 'native') → ItemListCollection#
classmethod load_parquet(path: PathLike[str] | list[PathLike[str]], key: type[K] | Sequence[str | AliasedColumn] | str | AliasedColumn, *, layout: Literal['flat']) → ItemListCollection

Load this item list from a Parquet file.

Parameters:

  • path – Path to the Parquet file to load.

  • key – The key to use (only when loading tabular layout).

  • layout – The layout to use, either LensKit native layout or a flat tabular layout.

record_batches(batch_size=5000, columns=None)#

Get the item list collection as Arrow record batches (in native layout).

Parameters:
Return type:

Generator[RecordBatch, None, None]

property key_fields: tuple[str]#

The names of the key fields.

property key_type: type[KL]#

The type of collection keys.

abstract property list_schema: dict[str, DataType]#

Get the schema for the lists in this ILC.

abstract lookup(key: tuple) → ItemList | None#
abstract lookup(*key: int | str | bytes | integer[Any] | str_ | bytes_ | object_, **kwkey: int | str | bytes | integer[Any] | str_ | bytes_ | object_) → ItemList | None

Look up a list by key. If multiple lists have the same key, this returns the last (like a dictionary).

This method can be called with the key tuple as a single argument (and this can be either the actual named tuple, or an ordinary tuple of the same length), or with the individual key fields as positional or named arguments.

Parameters:

key – The key tuple or key tuple fields.

lookup_projected(key)#

Look up an item list using a projected key. A projected key is a key that may have additional fields beyond those defined by this collection, that are ignored for the purposes of lookup.

Parameters:

key (tuple) – The key. Must be a named tuple (e.g. a key obtained from another item list collection).

Returns:

The item list with the specified key, projected to this collection’s key fields, or None if no such list exists.

Return type:

ItemList | None

abstract items()#

Iterate over item lists and keys.

Return type:

Iterator[tuple[KL, ItemList]]

lists()#

Iterate over item lists without keys.

Return type:

Iterator[ItemList]

keys()#

Iterate over keys.

Return type:

Iterator[KL]