This repository hosts the code for Cayce, a trend spotter for mobile video games. Every day, Cayce web scrapes the data on top grossing apps from the Apple and Android app stores, detects the trending apps using outlier detection machine learning algorithms, and sends a slack message about one trending app from each app store. Cayce is named after the main character in the book “Pattern Recognition” by William Gibson, who has an innate ability to detect the hottest trends. We focused our trend spotting on seven “soft launch” countries, which are countries where developers test launch their apps before releasing them worldwide. If we are able to detect trending apps in these soft launch countries, we will be able to spot the biggest hits before they hit the major markets.
We web scraped the data using two open-source scrapers found on GitHub: google-play-scraper and app-store-scraper. We wrote Node.js scripts to interact with these scrapers and download the top free, grossing, and trending apps from each store for the seven soft launch countries. Using BASH scripts, we run the Node.js scripts each day and save the output as JSON files. Using Python and PyMongo, we add the attributes of country, chart, date, and rank to these JSON files and upload the data into MongoDB.
We then detect the trending apps using Outlier Detection algorithms. Outlier detection is used when we want to separate normal observations from outliers, similar to novelty detection, but we do not have a clean, labeled data set which tells which observations the outliers are. We tested four algorithms: Elliptic Envelope, Isolation Forest, Local Outlier Factor, and One Class Support Vector Machine. We chose Isolation Forest because it makes no assumption about the underlying distribution of the non-outlier data, is well-suited for high-dimensional data, and allows us to set a parameter called “contamination” that limits the number of observations that are classified as outliers to only the most extreme.
Our outlier detection script was written in Python using Pandas and Scikit-Learn. For each store, we query the past five days of data from MongoDB, transform the data frame to contain the changes in ranks between each of the five days, and feed the data frame into an Isolation Forest model. Of the apps that are labeled as outliers, we eliminate apps that are either currently on the top 100 grossing chart in the US or were sent in a slack message on a previous day. We then select the app that had the highest change in rank in the past five days. We then send a message about that app containing the details of the app title, the release date, the countries it is currently on the top grossing charts in and its ranks in those countries, the developer, the store URL, and the icon of the app. We automated all of these scripts using CRONTAB on a Google Cloud Server.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent