minnnh/real_time_news_platform

Real_Time_News_Scraping_and_Recommendation_System

Real Time News Scraping and Recommendation System

Real Time News Scraping and Recommendation System is a web application that provides users with the latest news from various sources such as CNN, BBC, and The New York Times. This system utilizes a robust news pipeline to scrape the most recent updates from different news outlets and tailors the news feed to each user's preferences.

Demo

Environment

• Python 3.8.18
• npm 10.1
• Node.js v20.9
• Nodemon 3.0.1
• Conda env -- tapNews

Built with

• News Pipeline: MongoDB, Redis, RabbitMQ, TF-IDF
• News Topic Modeling: TensorFlow, CNN, NLP
• News Recommendation: NLP
• Web Server: React, Node.js, RPC, SOA, JWT

How It Works

• News Pipeline: The system continually monitors, fetches, deduplicates, and updates the database with the latest news, ensuring real-time information.
• News Topic Modeling: Utilizing TensorFlow, CNN, and NLP, the system trains a pipeline to model news topics, enhancing the relevance of recommended news articles.
• News Recommendations: Users receive personalized news recommendations based on their interests and preferences.
• Web Application: The ReactJS frontend offers a user-friendly interface, while the Node.js backend processes data, facilitates JSON-RPC communication.
• Backend Service: The server monitors, fetches, and updates the database with real-time news. It manages data from the frontend, TensorFlow serving, and training processes, connecting seamlessly with the database for efficient data processing.

Connections

• backend-server <-> web-server server
  • JSON-RPC
  • Jayson
• web-server: server <-> client
  • React
  • Express

Getting Started

To get started with the Real Time News Scraping and Recommendation System, follow these steps:

Prerequisites

• install npm

npm install -g [email protected]

• install node.js

npm install -g node

node -v 20.9.0

• install nodemon

npm install -g [email protected]

Installation

1. clone the repo

git clone https://github.com/minnnh/Real_Time_News_Platform

2. install the requirements

pip install -r requirements.txt

3. run the system

sudo ./backend_launcher.sh

sudo ./news_pipeline_launcher.sh

sudo ./news_recommendation_launcher.sh

sudo ./news_topic_modeling_launcher.sh

sudo ./click_log_processor.sh  

sudo ./web_client_launcher.sh

sudo ./web_server_launcher.sh

Todo List

coding

• backend-server
• common
• pipeline
• topic modeling
• recommendation
  • recommendation service
  • conmmon
  • backend-server
  • web-client
  • web-server
• web-server server
  • middleware
  • models (mongoose)
  • config
  • passport
  • routes
  • app
• web-server client
  • auth
  • base
  • signup
  • login

testing

• backend-server
• common
• pipeline
• topic modeling
• recommendation
• web-server server
  • login
  • signup
  • news
• web-server client
  • login
  • signup
  • logout
  • news panel