Possible memory leak when processing parquet files

[marcboeker]

Possible memory leak when processing parquet files

[marcboeker]

Hi @goober
thanks for bringing this up and also providing a repo with code to reproduce this. I haven't had the time to look into it, but two things that come into my mind.

1. In the beginning of writing this driver I have encountered a similar behaviour and thought, that there is a memory leak. It has turned out, that Go is very greedy about memory and the GC only releases it once he has to as other resources are requesting the memory. This could take some time. Have you tried to close the DB connection and wait for some time afterwards (while the program is still running) and observe the memory usage? Maybe the memory will be freed after some period of time.

2. Have you tried to run your queries using the Python DuckDB driver and compared it to go-duckdb? Maybe this is not a memory leak on the Go/CGO side but more in the DuckDB or Parquet part?

Once I have more time, I'll dig deeper into it.

[goober]

I have updated the repo with a second application based on Python that should process the parquet file in the same way as the golang application. Even though it does not release all memory that I would have expected, it releases something as you can see in the image:

I have also let the golang application run for a while to see if any memory is released, but unfortunately it keeps the memory at the same level.

I saw that another issue, that could be related, has been opened against the duckdb repository recently but in their case it was a NodeJs based application. I add it here for reference as well. duckdb/duckdb#6607

[goober]

@marcboeker I have found some memory leaks when the result is being processed in chunks. See linked PR #83 for a fix.