pegeler/methylmallet

High performance gene methylation data processing

Disclaimer

Comments and posts on this site are my own and do not necessarily reflect the views of my employer or any organization of which I am a member.

methylmallet's Issues

Works Well!

Works, takes a bit more time than the pure Python method, but MUCH thinner on ram.

doesn't clean up 'sorted_test_files' in /working when complete

Otherwise exceptionally well done!

Should we merge the `open-all` branch?

The open-all branch uses a different approach. Instead of appending data one column at a time, it opens all files simultaneously. On small datasets (~4 files), it is actually much slower. However, I think it will scale more gracefully when we start reading in datasets into the hundreds or thousands. It'd like to run some tests on the datasets of various sizes to find the break-even point.

That said, the number of files in this dataset is already pushing up against the soft limit of the number of files that can be opened at one time.

Another thing to consider is using this approach but opening and seeking each file as needed. The code to do so is written but commented out. Sounds too slow, though I haven't tried it.