Repository for spark vs pandas benchmarking
4 benchmarks are performed for comparing pandas vs spark.
- Time taken to load data in dataframe
- SQL 1: select max(close) from data
- SQL 2: select count(distinct volume) from data
- SQL 3: select sum(high) from data group by date
2 benchmarks performed on pandas
- Sort data of a column for different data sizes
- Find sum of a column for different data sizes