mahmoudparsian/pyspark-tutorial

PySpark-Tutorial provides basic algorithms using PySpark

PySpark Tutorial

• PySpark is the Python API for Spark.

• The purpose of PySpark tutorial is to provide basic distributed algorithms using PySpark.

• PySpark supports two types of Data Abstractions:

  • RDDs
  • DataFrames

• PySpark Interactive Mode: has an interactive shell ($SPARK_HOME/bin/pyspark) for basic testing and debugging and is not supposed to be used for production environment.

• PySpark Batch Mode: you may use $SPARK_HOME/bin/spark-submit command for running PySpark programs (may be used for testing and production environemtns)

---

Glossary: big data, MapReduce, Spark

---

Basics of PySpark with Examples

---

PySpark Examples and Tutorials

• PySpark Examples: RDDs
• PySpark Examples: DataFramess
• DNA Base Counting
• Classic Word Count
• Find Frequency of Bigrams
• Join of Two Relations R(K, V₁), S(K, V₂)
• Basic Mapping of RDD Elements
• How to add all RDD elements together
• How to multiply all RDD elements together
• Find Top-N and Bottom-N
• Find average by using combineByKey()
• How to filter RDD elements
• How to find average
• Cartesian Product: rdd1.cartesian(rdd2)
• Sort By Key: sortByKey() ascending/descending
• How to Add Indices
• Map Partitions: mapPartitions() by Examples
• Monoid: Design Principle

---

Books

Data Algorithms with Spark

Data Algorithms

PySpark Algorithms

---

Miscellaneous

Download, Install Spark and Run PySpark

How to Minimize the Verbosity of Spark

---

PySpark Tutorial and References...

• Getting started with PySpark - Part 1
• Getting started with PySpark - Part 2
• A really really fast introduction to PySpark
• PySpark
• Basic Big Data Manipulation with PySpark
• Working in Pyspark: Basics of Working with Data and RDDs

---

Questions/Comments

• View Mahmoud Parsian's profile on LinkedIn
• Please send me an email: [email protected]
• Twitter: @mahmoudparsian

Thank you!

best regards,
Mahmoud Parsian

---

---

pyspark-tutorial's Issues

pyspark v3.1.1 removed pyspark.streaming.kafka?

why stop() function doesn't work correctly?

`conf = (SparkConf()
#.setMaster("local")
.setMaster("spark://master:7077")
.setAppName("My_APP")
.set("spark.executor.cores", '1')
.set("spark.scheduler.mode", "FAIR")
)

sess = SparkSession.builder.config(conf=conf).getOrCreate()
sess.stop()
DataFrame = sess.sql("show tables")
print(DataFrame)
print(sess)`

This is the program output：
DataFrame[database: string, tableName: string, isTemporary: boolean] <pyspark.sql.session.SparkSession object at 0x7fe3b8e709b0>

Why is sparksesion not closed and what can I do to close it? Thank you