milesgranger/pontem

Treat Spark like pandas.

pontem

Treat PySpark DataFrames like pandas.

This is currently just a hobby project, not suitable for use.

Turn somethinig like this:

# Pure PySpark API; df is type pyspark.sql.DataFrame
def multiply(n):
    return udf(lambda col: col * n, FloatType())
df = df.withColumn('new_col', df.select(multiply(2)(df['other_col'])))

...into this:

# Using pontem.core.DataFrame object.
df['new_col'] = df['other_col'] * 2

pontem's Issues

Implement tests

Series implementation

Error in floordiv when zeros present

Overloading is complete for pontem.Series objects, with an exception that unorderable types: int() < NoneType() error occurs when zeros are present.

To replicate:

import pontem as pt
series = pt.Series(sc=sc, data=range(0, 10))
series //= series

DataFrame implementation