yiqin0209/hpdp

🌟 Hit star button to save this repo in your profile

About Us

The information on this Github is part of the materials for the subject High Performance Data Processing (SECP3133). This folder contains general big data information as well as big data case studies using Malaysian datasets. This case study was created by a Bachelor of Computer Science (Data Engineering), Universiti Teknologi Malaysia student.

🔥 Important things

1. Essential Preparations for a Successful Start in High-Performance Data Processing Class
2. Course Information
3. Student information
4. Exercise

Activity

📄👍 Turnitin Report

No	Module	Description	File	Submission
1	Task 1	Portfolio Creation
2	Task 2	AWS Certification
3.	Assignment 1	Data analysis using Google Sheets
4.	Assignment 2	Exploratory data analysis
5.	Assignment 3	EDA Big Data
6.	Course	Feature Engineering
7.	Assignment 4	Feature Engineering
8.	Assignment 5	Automated Feature Engineering tools
9.	Assignment 6	Mastering Big Data Handling
10.	Assignment 7	Comparison between libraries

Lab

No	Module	Description	File
1.	Lab 1	Understanding Your Data
2.	Lab 2	EDA Big Data
3.	Lab 3	Feature Engineering

Notes

1. Data

No.	Content	File
1.	Charting Your Path in Data and Machine Learning
2.	Navigating the Data Science Landscape
3.	Database Types
4.	Navigating the Database Landscape
5.	The Data Journey: From Raw to Refined
6.	The Data to MLOps Journey: An End-to-End Process
7.	Data Platforms Architecture: Governance and Operations
8.	Creating Data Products to Monetize Data
9.	Revolutionizing Data and Machine Learning with DataOps and MLOps
10.	Data Science for Beginners - A Curriculum

2. Case Study

No.	Content	File
1.	Unveiling Instagram's Engagement Magic through Machine Learning
2.	Unlocking Spotify's Musical Enchantment with Machine Learning
3.	Netflix
4.	Big data in healthcare: management, analysis and future prospects
5.	Introduction to Big Data Computing for Geospatial Applications
6.	40 Stats and Real-Life Examples of How Companies Use Big Data
7.	Leveraging High Performance Data Processing: Insights from Industry Leaders

3. Big Data Management

No.	Section	Content
1	Introduction to Big Data Management	A. What is Big Data? B. The Importance of Managing Big Data C. Why Big Data Management is Important? D. The History of Big Data
2	Understanding Big Data	A. Defining Big Data B. Characteristics of Big Data C. Sources of Big Data D. Challenges in Dealing with Big Data E. The workflow of Big Data Management
3	The Role of Big Data Management	A. Managing big data B. Benefits of Effective Big Data Management C. Risks of Ignoring Big Data Management D. Industries Benefiting from Big Data Management
4	Data Collection and Storage	A. Data Collection Methods    1. Traditional Data Sources    2. Emerging Data Sources B. Data Storage and Warehousing C. Data Security and Privacy Concerns
5	Data Processing and Analysis	A. Data Preprocessing B. Data Analysis Tools and Techniques C. Real-time Data Processing D. Machine Learning in Big Data Analysis
6	Data Integration and Governance	A. Data Integration Strategies B. Data Governance Best Practices C. Ensuring Data Quality
7	Data Visualization	A. Importance of Data Visualization B. Data Visualization Tools C. Creating Effective Data Visualizations
8	Case Studies	A. Successful Big Data Management Implementations B. Lessons Learned from Failed Big Data Projects C. Real-world Examples
9	Challenges in Big Data Management	A. Scalability Issues B. Security Concerns C. Compliance and Regulatory Challenges
10	Future Trends in Big Data Management	A. The Evolving Landscape of Big Data B. Predictions for the Future C. Preparing for the Next Generation of Big Data
11	Best Practices in Big Data Management	A. Key Takeaways B. Actionable Strategies C. Tips for Effective Big Data Management
12	Other References

Dataset

• Awesome Public Datasets
• Portal Data Terbuka Malaysia
• Department of Statistics Malaysia
• data.world
• Dataportal.asia
• knoema
• The World Bank
• Dataset Search - Google
• UCI Machine Learning Repository
• Kaggle datasets
• Awesome-public-datasets
• Datahub.io
• Earthdata
• CERN Open Data Portal

Contribution 🛠️

Please create an Issue for any improvements, suggestions or errors in the content.

You can also contact me using Linkedin for any other queries or feedback.