Movie Rating with Collaborative Filtering
Customized cloudera/quickstart image: upgrade python to 2.7.11 The customized image can be downloaded by
docker pull zhou42/movierating:v1
DataDownload.py download data from movielens website and store the data at the local file system MovieRating.py load the file in the HDFS file system and build the latent factor CF model recommenderSystem.py make recommendations to the new user
- First step
spark-submit --driver-memory 4g DataDownload.py
The movielens data will be downloaded to the local file system (in a new folder "./datasets")
- Second step
hadoop fs -put datasets
hadoop fs -mkdir models
hadoop fs -mkdir checkpoint
Put the local folder "./datasets" into the HDFS; make a new folder in HDFS to store the final model trained; checkpoint is used to avoid stackover flow
- Third step
spark-submit --driver-memory 4g MovieRating.py
Note the size of driver memory can be further increased depending on the OS memory size
Train the model; choose the parameters according the results on validation set;
result is as below