amazingdd/daisyrec

This is the repository of our article published in RecSys 2020 "Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison"

Overview

daisyRec is a Python toolkit developed for benchmarking top-N recommendation task. The name DAISY stands for multi-Dimension fAir comparIson for recommender SYstem.

The figure below shows the overall framework of DaisyRec-v2.0.

This repository is used for publishing. If you are interested in details of our experiments ranking results, try to reach this repo file.

We really appreciate these repositories to help us improve the code efficiency:

• RecSys2019_DeepLearning_Evaluation
• RecBole

How to Run

Make sure you have a CUDA enviroment to accelarate since the deep-learning models could be based on it.

1. Install from pip

pip install daisyRec

2. Clone from github

git clone https://github.com/AmazingDD/daisyRec.git && cd daisyRec

• Example codes are listed in run_examples, try to refer them and find out how to use daisy; you can also implement these codes by moving them into daisyRec/.

• The GUI Command Generator for test.py and tune.py, which can assist you to quikly write arguments and run the fair comparison experiments, is now available here.

  The generated command will be like this:

  python tune.py --param1=20 --param2=30 ....
  python test.py --param1=20 --param2=30 ....
  

  We highly recommend you to implement the code with our GUI firstly!

Documentation

The documentation of DaisyRec is available here, which provides detailed explanations for all arguments.

Implemented Algorithms

Models in daisyRec only take triples <user, item, rating> into account, so FM-related models will be specialized accrodingly. Below are the algorithms implemented in daisyRec. More baselines will be added later.

Model	Publication
MostPop	A re-visit of the popularity baseline in recommender systems
ItemKNN	Item-based top-N recommendation algorithms
EASE	Embarrassingly Shallow Autoencoders for Sparse Data
PureSVD	Top-n recommender system via matrix completion
SLIM	SLIM: Sparse Linear Methods for Top-N Recommender Systems
MF	Matrix factorization techniques for recommender systems
FM	Factorization Machines
NeuMF	Neural Collaborative Filtering
NFM	Neural Factorization Machines for Sparse Predictive Analytics
NGCF	Neural Graph Collaborative Filtering
Multi-VAE	Variational Autoencoders for Collaborative Filtering
Item2Vec	Item2vec: neural item embedding for collaborative filtering
LightGCN	LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation

Datasets

You can download experiment data, and put them into the data folder. All data are available in links below:

• MovieLens-100K / 1M / 10M / 20M
• Netflix Prize Data
• Last.fm
• Book Crossing
• Epinions
• CiteULike
• Amazon-Book/Electronic/Clothing/Music (ratings only)
• Yelp Challenge

Cite

Please cite both of the following papers if you use DaisyRec in a research paper in any way (e.g., code and ranking results):

@inproceedings{sun2020are,
  title={Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison},
  author={Sun, Zhu and Yu, Di and Fang, Hui and Yang, Jie and Qu, Xinghua and Zhang, Jie and Geng, Cong},
  booktitle={Proceedings of the 14th ACM Conference on Recommender Systems},
  year={2020}
}

@article{sun2022daisyrec,
  title={DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation},
  author={Sun, Zhu and Fang, Hui and Yang, Jie and Qu, Xinghua and Liu, Hongyang and Yu, Di and Ong, Yew-Soon and Zhang, Jie},
  journal={arXiv preprint arXiv:2206.10848},
  year={2022}
}

daisyrec's Issues

the parameter test_method='ufo' in daisy.utils.splitter.split_test()

In your fuction split_test(), parameter test_method='ufo' means : split by ratio in user level.
I think the 'ufo' should let 20% of each user's interaction records be the test set. In your fuction, let 20% of users' interactions as test set, so that user' embedding in the test set are not trained...

about ctr prediction metric

This is a good project!
But can u add some ctr prediction metrics such as 'AUC' or 'F1'？
I think add it is easy, and it will be more helpful.

Why can MRR be bigger than 1?

Hello, I have read your paper and find it is a fantastic work. However, I find mrr metric is bigger than 1 in your result such as ML1M. But in my view, it can not be greater than 1. Can you clear up my confusion, thanks!

Qidong Liu
Best wishes!

Paper availability?

Hi,

I'm interested in reading the associated paper "Are We Evaluating Rigorously?" but I can't seem to find it anywhere. Is it available openly?

Thanks

Bug in CDAE model about out_activation

I notice that in your CDAERecommender.py (line 69-70), no matter the out_activation is sigmoid or relu, it is always set as nn.Sigmoid(). I am not sure if this is a type error?

https://github.com/AmazingDD/daisyRec/blob/dev/daisy/model/CDAERecommender.py

Is DaisyRec going to have session-based recommenders?

First of all, I really appreciate your work and this library. It's what exactly I've looking for..!

I see your paper ends with "diving into more diverse recommendation tasks (e.g., session-aware)".

I just wondered that this library is going to include session based recommenders such as GRURec(https://arxiv.org/abs/1511.06939)...

The problem with the <UNK> in the oitems variable returned by the skip-gram function in the Item2Vec model

According to item2idx, we can know that the index corresponding to is max_item_num. Meanwhile, oitems list contains a large number of index entries. The largest index of the lookup table is max_item_num-1, which causes out-of-bounds. So how can this problem be solveed?

Yelp dataset statistics

Hi! I want to compare my model's results with your benchmark on Yelp dataset, but the dataset I found has different number of interactions/user/items with the numbers you report in the paper.

I downloaded the dataset from the site and kaggle and both sources has 160,585 items, 2,189,457 users and 8,635,403 interactions (as on the site). But you report 174,567 items , 1,326,101 users, and 5,261,669 interactions for origin Yelp in the paper (table 1). Papers says you considered all interactions with rating >=1 for the Yelp dataset and your code corresponds to this (no filtering by rating).

Could you please tell, where to get or how to prepare the data to get identical dataset and compare my model with your baselines?

Best regards!

JSONDecode Error

I'm on the dev branch, so that may be the issue, but I'm just trying to get a decent baseline for the EASE algorithm. Using the recommended command generator, I tried to run the command python tune.py --optimization_metric=ndcg --hyperopt_trail=20 --algo_name=ease --dataset=ml-100k --prepro=origin --topk=50 --epochs=50 --test_size=0.2 --val_size=0.1 --cand_num=1000 --test_method=tsbr --val_method=tsbr --tune_pack='{}' but then the output threw an error:

11 Sep 20:03 INFO - {'gpu': '0', 'seed': 2022, 'reproducibility': True, 'state': None, 'optimization_metric': 'ndcg', 'hyperopt_trail': 20, 'tune_testset': False, 'tune_pack': "'{}'", 'algo_name': 'ease', 'val_method': 'tsbr', 'test_method': 'tsbr', 'fold_num': 1, 'val_size': 0.1, 'test_size': 0.2, 'topk': 50, 'cand_num': 1000, 'sample_method': 'uniform', 'sample_ratio': 0, 'num_ng': 4, 'batch_size': 256, 'loss_type': 'BPR', 'init_method': 'default', 'optimizer': 'default', 'early_stop': False, 'data_path': 'data/', 'res_path': None, 'dataset': 'ml-100k', 'prepro': 'origin', 'level': 'ui', 'UID_NAME': 'user', 'IID_NAME': 'item', 'INTER_NAME': 'rating', 'TID_NAME': 'timestamp', 'binary_inter': True, 'positive_threshold': None, 'metrics': ['recall', 'mrr', 'ndcg', 'hit', 'precision'], 'reg': 200.0}
Traceback (most recent call last):
  File "daisyRec\tune.py", line 106, in <module>
    param_dict = json.loads(config['tune_pack'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

ModuleNotFoundError: No module named '...'

I followed the instructions to run the project, but immediately ran into ModuleNotFoundError: No module named 'optuna'. Did I miss an install somewhere?

If I didn't miss anything, which version of optuna should I use, and can this be included in the requirements.txt?