fzzying/bt-hollywood

Movie Rating with Collaborative Filtering

bt-hollywood

Movie Rating with Collaborative Filtering

Environment

Customized cloudera/quickstart image: upgrade python to 2.7.11 The customized image can be downloaded by

docker pull zhou42/movierating:v1

Files

DataDownload.py download data from movielens website and store the data at the local file system MovieRating.py load the file in the HDFS file system and build the latent factor CF model recommenderSystem.py make recommendations to the new user

Steps to run the program

• First step

spark-submit --driver-memory 4g DataDownload.py 

The movielens data will be downloaded to the local file system (in a new folder "./datasets")

• Second step

hadoop fs -put datasets
hadoop fs -mkdir models
hadoop fs -mkdir checkpoint

Put the local folder "./datasets" into the HDFS; make a new folder in HDFS to store the final model trained; checkpoint is used to avoid stackover flow

• Third step

spark-submit --driver-memory 4g MovieRating.py

Note the size of driver memory can be further increased depending on the OS memory size Train the model; choose the parameters according the results on validation set; result is as below