Handle massive queries

[kechrislab]

Handle massive queries

[smahaffey]

Thank you @JoseCorCab. We will talk about creating a cached search that you can save/load/filter later. That's a good idea for large searches you can do more with the results without rerunning them. I think you might be able to accomplish this by running the searches repeatedly and joining the tables into a larger table that you can write to a file. You could do this ahead of time off of the cluster and write the table to a text file or .rData file that you can read when you start running on the cluster. This would effectively accomplish what you are requesting if I'm understanding correctly. If you have trouble with any of the queries timing out please provide an example and I will look into it. The data transfer can sometimes take extra time but any of the individual queries should be relatively quick. If they aren't quick then I need to also look at that problem.

For a custom query you should be able to submit one. Please look at the documentation for search_multimir(query). This should allow you to create your own queries and submit them. I understand the desire to create a local copy from those results that you can further query. I don't think it's technically challenging to implement this. I will bring this up as well. Thank you.

[smahaffey]

One other idea based on another request is maybe we can offer the option to store the result with a unique ID on the server so you could just retreive the results. However I think the transfer of a large query should usually be the slower part of the query.

[JoseCorCab]

Hello, thanks for the suggestion. My problem arises because a R loop is taken as a unique job in the cluster and the login node has a time limit for each job. I will try to make small individual queries (inside of a R script) inside a bash loop. In this way each query is a different job and it shouldn't break.

One other idea based on another request is maybe we can offer the option to store the result with a unique ID on the server so you could just retreive the results. However I think the transfer of a large query should usually be the slower part of the query.

I think that this in not a good option because it will overload your server and this can handicap other users.

Thanks for everything!!