Prediction Accuracy Metrics¶
lenskit.metrics.predict module containins prediction accuracy metrics.
rmse(predictions, truth, missing='error')¶
Compute RMSE (root mean squared error).
the root mean squared approximation error
mae(predictions, truth, missing='error')¶
Compute MAE (mean absolute error).
the mean absolute approximation error
Working with Missing Data¶
LensKit rating predictors do not report predictions when their core model is unable
to predict. For example, a nearest-neighbor recommender will not score an item if
it cannot find any suitable neighbors. Following the Pandas convention, these items
are given a score of NaN (when Pandas implements better missing data handling, it will
use that, so use
pandas.Series.notna(), not the
However, this causes problems when computing predictive accuracy: recommenders are not being tested on the same set of items. If a recommender only scores the easy items, for example, it could do much better than a recommender that is willing to attempt more difficult items.
A good solution to this is to use a fallback predictor so that every item has a
prediction. In LensKit,
this functionality; it wraps a sequence of recommenders, and for each item, uses
the first one that generates a score.
You set it up like this:
cf = ItemItem(20) base = Bias(damping=5) algo = Fallback(cf, base)