Regarding ApproxQuantile

[titicaca]

Regarding ApproxQuantile

[titicaca]

You can set this param approxQuantileRelativeError, which is the Relative Error for Approximate Quantile Calculation (0 <= value <= 1), default is 0 for calculating the exact value, which would be expensive for large datasets.

[titicaca]

Furthermore, ApproxQuantile is used to calculating the threshold of anomaly score for finding anomaly points according to the given param contamination. Regarding ApproxQuantile function, you can find more detail in the spark api doc:

https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html#DataFrame.approxQuantile

[tahmedge]

Hi, thanks for your reply. I am still bit confused.

Suppose, the contamination value is 0.1 . In this case, what could be the best value for ApproxQuantile ?

[titicaca]

It depends on what the relative error of the quantile value you can accept. The smaller the relative error is, the more expensive and more memory required for calculation. You can try with different values from small to large, so that it can fit your memory consume.

[tahmedge]

Thanks a lot.