mahmoudparsian/data-algorithms-with-spark

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

Data Algorithms with Spark by Mahmoud Parsian

"... This book will be a great resource for both readers looking to implement existing algorithms in a scalable fashion and readers who are developing new, custom algorithms using Spark. ..." Dr. Matei Zaharia Original Creator of Apache Spark FOREWORD by Dr. Matei Zaharia

---

Data Algorithms with Spark by Mahmoud Parsian

Foreword by Dr. Matei Zaharia (Original Creator of Apache Spark)

Author: Mahmoud Parsian

Goal of this book: Data Algorithms with Spark

Story of this book: Data Algorithms with Spark

---

• Mahmoud Parsian's Author Page @Amazon

• Mahmoud Parsian's Author Page @LinkedIn

• This new O'Reilly book is the successor Edition of Data Algorithms (published by O'Reilly)

• This book uses PySpark (much simpler and readable)

• Published date: April 8, 2022

• @OReillyMedia: Data Algorithms with Spark, By @mahmoudparsian

• Autor Contact: [ Email ] [ Mahmoud Parsian @LinkedIn ][ Mahmoud Parsian @GitHub ]

---

Github Chapter Solutions

• This GitHub repository will host all source code and scripts for Data Algorithms with Spark

• Chapter solutions are provided in PySpark and Scala

  • PySpark solutions are provided by Mahmoud Parsian
  • Scala solutions are provided by Deepak Kumar and Biman Mandal

---

Software:

All programs are tested with the following software:

Spark	Python	Scala	Java
Apache Spark 3.4.0	Python 3.10.5	Scala 2.13	Java 11

---

Table of Contents

Chapter	Title
Glossary	Glossary of Big Data, MapReduce, Spark
Chapter 1	Introduction to Data Algorithms
Chapter 2	Transformations in Action
Chapter 3	Mapper Transformations
Chapter 4	Reductions in Spark
Chapter 5	Partitioning Data
Chapter 6	Graph Algorithms
Chapter 7	Interacting with External Data Sources
Chapter 8	Ranking Algorithms
Chapter 9	Fundamental Data Design Patterns
Chapter 10	Common Data Design Patterns
Chapter 11	Join Design Patterns
Chapter 12	Feature Engineering in PySpark

---

Bonus Chapters

Bonus Chapter	Title / Description
Glossary	Glossary of Big Data, MapReduce, Spark
Word Count	Solutions for Word Count using RDDs and DataFrames
Anagrams	Find words, which are anagrams
Lambda Expressions	Using Lambda Expressions in PySpark programs
TF-IDF	Term Frequency - Inverse Document Frequency
K-mers	K-mers for DNA Sequences
Correlation	All vs. All Correlation
Mapping Partitions	mapPartitions() Complete Example
UDF	User-Defined Function Examples
DataFrames Transformations	Examples on Creation and Transformation of DataFrames
DataFrames Tutorials	DataFrames Tutorials: from collections and CSV text files
Join Operations	Examples on join of RDDs and DataFrames
PySpark Tutorial 101	Examples on using PySpark RDDs and DataFrames
Physical Data Partitioning	Tutorial of Physical Data Partitioning
Monoids and Combiners	Monoid as a Design Principle

---

---

---

data-algorithms-with-spark's Issues

Missing data

I have come here from the link from chapter 1 of the book and is looking for the FASTA data.

Ref: https://github.com/mahmoudparsian/data-algorithms-with-spark/tree/master/code/chap02

Missing Files On Github

Hi,

You miss files for PageRank chapter on github. Data Files, and also pagerank_2.py

Thanks

Please upload the code and data for the release chapters

Though chapters have been released, the related code and data is not available, please do release them as well

Code for chapter 9 is missing on github

Hi,

code for chapter 9 is missing entirely, but you point here from the book, saying that we can find code here - https://github.com/mahmoudparsian/data-algorithms-with-spark/tree/master/code/chap09

Can you add please?

Like your book though. Thank you :)