wdm0006/dummyrdd

A pure python mock of pyspark's RDD

dummyrdd's Issues

Fix "git-pandas" error on CONTRIBUTING.md

Need documentation on S3 use

It's not clear that TinyS3 is needed, also not obvious without digging through code on how to set the AWS keys. Would also be nice if it supported profiles like boto, but that seems to be a limitation of TinyS3

Remote unnecessary list conversion on RDD.map

from:
def map(self, f, preservesPartitioning=False): data = list(map(f, self._jrdd)) return RDD(data, self.ctx)

to:
def map(self, f, preservesPartitioning=False): data = map(f, self._jrdd) return RDD(data, self.ctx)

Add unit tests on Context class

Remove toolz from requirements (not needed anymore)

Add unit tests on SparkContext class

DataFrames support

Do you know about anyone working on the support for Spark DataFrames? Are there any public plans on doing so?

I may be interested in collaborating with it.

Add unit tests on SQLContext class

Add unit tests on RDD class

Coverage badge not updated

I still see coverage as "69%":
https://github.com/wdm0006/DummyRDD/

On coveralls it already says "78%":
https://coveralls.io/github/wdm0006/DummyRDD?branch=master

Any idea of what is happening?

`RDD.mapPartitions` should provide `iteratro` or `list` instead of concrete element

RDD.mapPartitions in the mock works like RDD.map but shouldn't.

import dummy_spark
rdd = dummy_spark.RDD([1, 2, 3], None)
_ = rdd.map(print) # should print 1\n2\n3\n
1
2
3
# OK!
_ = rdd.mapPartitions(print) # should print [1, 2, 3] or <iterator object>
1
2
3
# FAIL!

NewAPIHadoopRDD Implimentation

cc: @htssouza

I use the NewAPIHadoopRDD method in the spark context for the ElasticSearch-Hadoop connector, and am working on how I can mock that out here. My thinking is that there are 3 ways to do it:

1. Actually implement a java caller in py4j or something similar
2. Hack in support for the ES-Hadoop connector functionality in pure python to serve just my needs
3. Add in a way to pass a custom implementation for NewAPIHadoopRDD as a patch, so users can set that functionality at runtime in their applications.

1 is definitely best, but most difficult, and 2 is definitely a bit too specific to just what I'm using this for, 3 is really flexible, but that does break the drop-in compatibility with pyspark we've maintained thus-far.

I'm thinking of basically just doing a string comparison on the input classes and if elasticsearch is in them, I'll implement 2, otherwise raise a not implemented error, and then have a method for setting a patched functionality as a fallback. If there are any other classes/drivers people are using we can try to implement them similarly.

Thoughts?

Add Travis-CI "button" on README

Guide here:
https://docs.travis-ci.com/user/status-images/

Define&Include LICENSE