TDigest __repr__ not in line with constructor

[isarn]

TDigest __repr__ not in line with constructor

[erikerlandson]

I added #25 and published it as 0.5.2, thanks!

[JonathanTaws]

I think this also causes an issue with Spark broadcasting feature.

from random import gauss, randint
from isarnproject.sketches.spark.tdigest import *
data = spark.createDataFrame([[randint(1,10),gauss(0,1)] for x in range(1000)])
udf1 = tdigestIntUDF("_1", maxDiscrete = 25)
udf2 = tdigestDoubleUDF("_2", compression = 0.5)
agg = data.agg(udf1, udf2).first()

td = agg[0]

td_broadcast = spark.sparkContext.broadcast(td)
td_broadcast.value

Results in:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/spark/python/pyspark/broadcast.py", line 146, in value
    self._value = self.load_from_path(self._path)
  File "/usr/lib/spark/python/pyspark/broadcast.py", line 123, in load_from_path
    return self.load(f)
  File "/usr/lib/spark/python/pyspark/broadcast.py", line 129, in load
    return pickle.load(file)
TypeError: __init__() takes 5 positional arguments but 6 were given

[erikerlandson]

interesting, I'm sure I can make it conform to a parsable constructor expression

[JonathanTaws]

interesting, I'm sure I can make it conform to a parsable constructor expression

I believe removing the nclusters from the __repr__ would achieve this:

def __repr__(self): 
     return "TDigest(%s, %s, %s, %s)" % \ 
         (repr(self.compression), repr(self.maxDiscrete), repr(self._cent), repr(self._mass)) 

[erikerlandson]

Closing with #23 - thanks @JonathanTaws !

[JonathanTaws]

While testing with the new release, I found that it's still not working with the __repr__ change - __reduce__ also needs to be updated. I submitted a new PR here: #25. Sorry for not testing this thoroughly!

[JonathanTaws]

Thanks, all working properly now.