GWAS Plugin Initialization

[Bobsimonoff]

GWAS Plugin Initialization

[Bobsimonoff]

I reviewed the Perl code. I decided to take the following approach since I do not know Perl well enough to modify it:

1. cleaned up the GWAS file (removed the almost 40k of rows that do not comply with the rs regex check
2. broke the catalog file into multiple pieces using an algorithm that keeps associated rsids in the same file where associated means rs1, rs2 and rs3 need to go in the same file if one study includes rs1 and rs2 but another includes rs2 and rs3
3. Ran the files in parallel
4. uncompressed the processed files, combined them, and recompressed them

This cur my initial run time down to a couple of hours.

[nakib103]

Hello @Bobsimonoff ,

Thanks again for looking into the plugin.

Yes, as you mentioned, the plugin does not use buffer and read the whole (filtered) file into memory and thus became a memory extensive operation. I will update the mechanism to reduce memory usage. On the contrary, reducing run time would not be as easy with large file size. The silver lining is that creating the processed file is not supposed to run frequently.

I will update you here once we have some optimisation in place. In the meantime, you can continue file split method that you are currently using.

Thank you for notifying us about the issue.

Best regards,
Nakib

[nakib103]

Hello @Bobsimonoff,

We have added improvement on the plugin,

• memory: load only one line from the file into memory instead of the whole file.
• runtime: the long run time was caused by database queries to retrieve variant location information. By default, now this is retrieved from the GWAS data file instead. But, you can still use database, if you want, by providing db=1 option.

This features will be available in the next Ensembl release 112. I will close this issue but if you face further problems feel free to open a new one.

Best regards,
Nakib