lunasnowflake/cryptocurrency

CryptoCurrency

This repository showcases the development of an end-to-end cryptocurrency dashboard orchestrated on AWS infrastructure using MageAI. The dashboard captures real-time OHLC (Open, High, Low, Close) data via an API and stores it in AWS PostgreSQL. Additionally, it integrates web scraping and NLP BERT sentiment analysis to quantify public sentiments, enriching stakeholders' comprehension of market dynamics.

1. Data Collection

   • Use the "Coindesk" source to collect cryptocurrency price data.
   • Utilize the "Price Requester (Mage.ai-Python)" to fetch the prices from Coindesk.
   • Store the collected data in the "AWS S3" bucket.

2. Data Processing

   • Set up a "Kafka Server (AWS EC2)" to handle data streaming.
   • Create a "Producer" to send the collected price data to the Kafka server.
   • Implement a "Consumer" to receive the data from the Kafka server.

3. Data Storage and Analysis

   • Store the collected prices in the "AWS DB (Prices)" for further analysis.
   • Use the "AWS Crawler" to extract relevant information from the collected data.
   • Set up a "Forecast (Python)" module to predict future cryptocurrency prices.
   • Visualize the forecasted data using "PowerBI".
   • Store the forecasted prices in the "AWS DB (Forecasted)".

4. News Scrapping and Sentiment Analysis

   • Implement a "News Scrapping (Mage.ai-Python)" module to gather news articles related to cryptocurrencies from "Bing News" and "Coin MarketCap".
   • Perform sentiment analysis using "Sentiment Analysis (Hugging-Face)".
   • Store the sentiment analysis results in the "AWS DB (Sentiments)".

5. Project Integration

   • Combine all the collected and analyzed data to create a comprehensive view of the cryptocurrency market.
   • Use "PQ - Power Query" to clean and transform the data for further analysis or visualization.

Features

• Real-time Data Capture: Utilizes an API to fetch real-time OHLC data for cryptocurrencies.
• Storage in AWS PostgreSQL: Stores the collected data securely in AWS PostgreSQL for further analysis.
• Sentiment Analysis: Integrates web scraping and NLP BERT sentiment analysis to quantify public sentiments related to cryptocurrencies.
• Machine Learning Forecasting: Develops and trains a machine learning forecasting model, updated daily through MageAI, offering accurate cryptocurrency forecasts.
• Dynamic Reporting with PowerBI: Engineers seamless data flow from PostgreSQL to PowerBI, creating a dynamic report showcasing market scenarios, trends, sentiments, and forecasts.
• Advanced Analysis Capabilities: Implements sophisticated DAX measures in PowerBI, equipping stakeholders with advanced statistical and technical analysis capabilities for informed decision-making in the volatile cryptocurrency landscape.

How to Use

• Clone the repository to your local machine.
• Set up AWS infrastructure with PostgreSQL.
• Configure MageAI for data orchestration.
• Run the provided scripts for data collection, sentiment analysis, and machine learning forecasting.
• Integrate the generated data with PowerBI for dynamic reporting.
• Explore the dashboard and leverage advanced analysis capabilities for insightful decision-making.
• Read Doc.txt

cd crypto_prices-Mage.ai
mage start

💻 Tech Stack

Languages

Frameworks & Libraries (ML)

Others

About Me

Imagine a curious soul frolicking in the realm of Data Science and Development – that's me! With a heart brimming with joy, I dance through projects ranging from coding escapades to the magic of AI and DevOps. Technology isn't just my playground; it's my vibrant canvas where every line of code is a brushstroke of pure delight!
I am open to any suggestions, connect with me anywhere! Also, I would appreciate it if I could get a 🌟

🌐 Socials