Evaluating Recommender Output

LensKit’s evaluation support is based on post-processing the output of recommenders and predictors. The batch utilities provide support for generating these outputs.

We generally recommend using Jupyter notebooks for evaluation.

When writing recommender system evaluation results for publication, it’s important to be precise about how exactly your metrics are being computed [TDV21]; to aid with that, each metric function’s documentation includes a mathematical definition of the metric.

Evaluation Topics

• Prediction Accuracy Metrics
  • Metric Functions
    • rmse()
    • mae()
  • Convenience Functions
    • user_metric()
    • global_metric()
  • Working with Missing Data
• Top-N Evaluation
  • Top-N Analysis
    • RecListAnalysis
      • RecListAnalysis.add_metric()
      • RecListAnalysis.compute()
  • Metrics
    • Classification Metrics
      • precision()
      • recall()
      • hit()
    • Ranked List Metrics
      • recip_rank()
    • Utility Metrics
      • ndcg()
      • dcg()
      • _dcg()
    • Writing a Metric
      • bulk_impl()

Saving and Loading Outputs

In our own experiments, we typically store the output of recommendation runs in LensKit experiments in CSV or Parquet files, along with whatever parameters are relevant from the configuration.