The memory could not be written unhandled exception crash python.exe

[EtienneT]

The memory could not be written unhandled exception crash python.exe

[EtienneT]

Testing more this morning, I am pretty sure polars has not enough memory to work with, but I would expect to receive an out of memory exception in python and then I can deal with that. But right now polars makes python.exe crash completely without any chance to catch the exception.

[EtienneT]

Ok I think I found the culprit:

.melt(id_vars=['Node', 'Date', 'Hour', 'Type'])
.group_by(['Node', 'Date', 'Hour', 'Type'])
.agg(
    (pl.col('value').cast(pl.Float64).product() * 0.5).alias('Score')
)

This part of the code was functionaly equivalent to:

.select(
    pl.col('Node'), pl.col('TradingDate'), pl.col('Hour'), pl.col('BidType'),
    (pl.reduce(lambda a, b: a * b, pl.col('^.*_pdf$').cast(pl.Float64)) * 0.5).alias('Score')
)

I just didn't know the existence of pl.reduce.

Now parameters that made my code crash don't make it crash anymore.

But... the problem of polars making python crash completely is still there.

[ritchie46]

But... the problem of polars making python crash completely is still there.

That's not something we can easily solve. Windows probably overcommits memory here and starts crashing when we write to it. If we couldn't allocate it we would have gotten a panic.

[ritchie46]

I will close this as this is not actionable by us. Going out of memory isn't something we claim to prevent.