- Environment: conda create --name cloud python=3.7
- pip install kafka-python
- pip install flask
- pip install yfinance
- pip install googlefinance
- Download kafka from source, install java-8, and check docs for installation.
- Download Spark 2.4.5 version, 220MB file size with Hadoop 2.6.
- sudo apt install scala
- pip install pyspark==2.4.6
- https://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm
- https://askubuntu.com/questions/1133216/downgrading-java-11-to-java-8
- pip install redis
- sudo apt-get install redis-server
- Install npm
- Followed by all the packages here https://github.com/LuQQiu/DataPipeline/tree/master/Node.js