org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStageX grows beyond 64 KB

[moj-analytical-services]

org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStageX grows beyond 64 KB

[mamonu]

Some proposed solutions for this from SO:

1. In the config:
   spark = SparkSession.builder.master('local').appName('blah')\ .config('spark.sql.codegen.wholeStage', 'false')\ ## <-- add this line .getOrCreate()

this is the local version so probably have a look if this can be defined in Glue AWS.

2. adding extra "checkpoints" in code.

[mamonu]

looking this a bit more today it seems that when spark.sql.codegen.wholeStage is on then its slightly faster but nothing would break if we just turned it off.

https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-whole-stage-codegen.html

[RobinL]

What's weird about this is there is a persist on df_gammas which should prevent this.

The problem is happening because the whole of the 'generate gammas' code and the 'compute expectation' is being bundled into a single project statement.

Solutions:

• the df -> rdd -> df trick might work:

https://stackoverflow.com/questions/50891509/apache-spark-codegen-stage-grows-beyond-64-kb

• split the job out into a 'compute gammas and persist to s3', and 'read from s3 and compute expectation' . since we're already repartition to 200 when we write out df_e this might actually not slow things down much