Real Time News Scraping and Recommendation System is a web application that provides users with the latest news from various sources such as CNN, BBC, and The New York Times. This system utilizes a robust news pipeline to scrape the most recent updates from different news outlets and tailors the news feed to each user's preferences.
- Python 3.8.18
- npm 10.1
- Node.js v20.9
- Nodemon 3.0.1
- Conda env -- tapNews
- News Pipeline: MongoDB, Redis, RabbitMQ, TF-IDF
- News Topic Modeling: TensorFlow, CNN, NLP
- News Recommendation: NLP
- Web Server: React, Node.js, RPC, SOA, JWT
- News Pipeline: The system continually monitors, fetches, deduplicates, and updates the database with the latest news, ensuring real-time information.
- News Topic Modeling: Utilizing TensorFlow, CNN, and NLP, the system trains a pipeline to model news topics, enhancing the relevance of recommended news articles.
- News Recommendations: Users receive personalized news recommendations based on their interests and preferences.
- Web Application: The ReactJS frontend offers a user-friendly interface, while the Node.js backend processes data, facilitates JSON-RPC communication.
- Backend Service: The server monitors, fetches, and updates the database with real-time news. It manages data from the frontend, TensorFlow serving, and training processes, connecting seamlessly with the database for efficient data processing.
- backend-server <-> web-server server
- JSON-RPC
- Jayson
- web-server: server <-> client
- React
- Express
To get started with the Real Time News Scraping and Recommendation System, follow these steps:
- install npm
npm install -g [email protected]
- install node.js
npm install -g node
node -v 20.9.0
- install nodemon
npm install -g [email protected]
- clone the repo
git clone https://github.com/minnnh/Real_Time_News_Platform
- install the requirements
pip install -r requirements.txt
- run the system
sudo ./backend_launcher.sh
sudo ./news_pipeline_launcher.sh
sudo ./news_recommendation_launcher.sh
sudo ./news_topic_modeling_launcher.sh
sudo ./click_log_processor.sh
sudo ./web_client_launcher.sh
sudo ./web_server_launcher.sh
- backend-server
- common
- pipeline
- topic modeling
- recommendation
- recommendation service
- conmmon
- backend-server
- web-client
- web-server
- web-server server
- middleware
- models (mongoose)
- config
- passport
- routes
- app
- web-server client
- auth
- base
- signup
- login
- backend-server
- common
- pipeline
- topic modeling
- recommendation
- web-server server
- login
- signup
- news
- web-server client
- login
- signup
- logout
- news panel