awesome-spark/spark-gotchas

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

Spark Gotchas

Table of Contents

• Introduction
  • Version compatibility
• Understanding Spark Architecture
  • Lack of Global Shared State and Spark Closures
  • Distributed Processing and its Scope
  • Feature Parity and Architecture of the Guest Languages
• Spark Application Deployment
  • Dynamic allocation
    • Yet Another Dynamic Resource Allocation YA(D)RN
• Spark Application Building
• RDD actions and Transformations by Example
  • Be Smart About groupByKey
    • What Exactly Is Wrong With groupByKey
    • How Not to Optimize
    • Not All groupBy Methods Are Equal
    • When to Use groupByKey and When to Avoid It
    • Hidden groupByKey
  • Immutability of a Data Structure Does Not Imply Immutability of the Data
• Spark SQL and Dataset API
  • Non-lazy Evaluation in DataFrameReader.load
    • Explicit Schema
    • Sampling for Schema Inference
    • PySpark Specific Considerations
  • DataFrame Schema Nullablility
    • nullability by Reflection
    • Marking StructFields Excplicitly as Nullable
    • Impact of Nullable
  • Reading Data Using JDBC Source
    • Parallelizing Reads
    • MySQL: Dates, Timestamps and Lies
  • Window Functions
    • Understanding Window Functions
    • Window Definitions
    • Example Usage
    • Requirements and Performance Considerations
• Data Preparation
  • DataFrame Metadata
    • Metadata in ML pipelines
    • Setting custom column metadata
• Iterative Algorithms
  • Iterative Applications and Lineage
    • Checkpointing
    • "Flat Transformations"
    • Truncating Lineage in Dataset API
  • Controling Number of Partitions in Iterative Applications
• PySpark Applications
• Serialization
  • JVM
    • Java Serialization
    • Kryo Serialization
  • PySpark Serialization
    • Python Serializers Characteristics
    • PySpark Serialization Strategies
    • Configuring PySpark Serialization
    • PySpark and Kryo
  • SerDe During JVM - Guest Communication

License

This work, excluding code examples, is licensed under Creative Commons Attribution-ShareAlike 4.0 International license.

Accompanying code and code snippets are licensed under MIT license.

spark-gotchas's Issues

Add recipes section.

This section could contain small reusable snippets for operations like adding row number, sliding, grouping etc.

Explain semantics of StructField.nullable

In particular

• It is not enforced.
• It is used for execution optimizations.
• Reflection and Product types (Option as a way to indicate nullable fields`).

This can work as a supporting material for #37.

Need help related with Spark

Hi,

I need your help on one of the questions on spark. Sorry, I did not get any way to contact you apart from this. As you have already answered similar kind of question on stackoveflow, I thought you would be able to help me on this. Here is the link to my question : question
This is the usecase which I am trying to solve : usecase

Compare time / size ratio for pickle / marshal

Be smart about groupBy(Key)

• What is wrong with groupBy*.
• Why not all gropuBy* are equal.
• Why naive implementations will only make things worse.
• When groupBy* is the right choice.

Small deployments and spark.dynamicAllocation.maxExecutors

I mean following issue in particular: https://awesome-spark.slack.com/archives/D0TAP9J1H/p1459408921000142

Explain why we should avoid summary metadata in Spark.

Describe non-lazy evaluation in DataFrameReader

WIP: https://github.com/awesome-spark/spark-gotchas/blob/04-non-lazy-evaluation/04_dataframe_by_example.md

Serialization When and why ?

Maybe we can also discuss serialization. There is lots discussion about its benifits, why it's not possible with Python.

Add introduction to window functions.

Ignored.

Repository description and README

It would be nice to have some description and / or README (beyond TOC). Any ideas?

Describe general mechanism and limitations of predicate pushdown