Diff Failure on AWS Glue

[g-research]

Diff Failure on AWS Glue

[EnricoMi]

Minor optimization:

Instead of

a_frame.diff(b_frame).where("diff != 'N'").count() == 0

you could do

not a_frame.diff(b_frame).where("diff != 'N'").isEmpty()

Because the former makes Spark compute the entire diff in order to return the count.
The latter makes Spark stop computing the diff once a single non-N row is found.

[EnricoMi]

It looks like your a_frame and b_frame are single-partition dataframes. This means that Spark has all the data in a single executor, which runs out of memory with sufficiently many rows. Add more workers does not help then.

Try adding a .repartition(100) behind load().

[bobhaffner]

Thanks, @EnricoMi ! I'll try out your suggestions

[bobhaffner]

Hello! I tried not a_frame.diff(b_frame).where("diff != 'N'").isEmpty() and I tried a few Ns for the .repartition(), but no luck unfortunately. Any other suggestions are much appreciated!

[EnricoMi]

Can you go to the Spark UI -> SQL tab -> click on the job that fails. If you could save that HTML page or screenshot it, that would help a lot. That page looks like this:

Also useful is the Executors tab and the Stages tab (click on the stage that fails).

Does .count() behind .load() work at all for over 10m rows?

[bobhaffner]

So I stumbled across something...

Setting a fetchsize (eg.option("fetchsize","100000")) in the JDBC config allows me to successfully perform the subsquent diff on two 21M dataframes

[EnricoMi]

So looks like this is unrelated to spark-extension and solved. Closing.