Retriever experiences massive slowdown with postgreSQL

[dmcglinn]

Retriever experiences massive slowdown with postgreSQL

[bendmorris]

In order to check whether tables/databases exist (to indicate whether a dataset has been downloaded already) the Retriever executes a lot of little queries. If the database connection is slow, this could cause a significant slowdown. There's probably a more efficient way to do this.

[ethanwhite]

I see this slow down as well, but it's not nearly as slow as it is in Dan's case (though 10 seconds is still a long time to wait for something to pop up these days). Dan, can you try changing the port for Postgres back to the default 5432 and see if that improves things.

[ethanwhite]

Looks like this ought to give us everything we need in a single query:

SELECT * FROM pg_tables WHERE schemaname NOT LIKE 'pg_%';

[ethanwhite]

Or better yet, just grab what we need:

SELECT schemaname, tablename FROM pg_tables WHERE schemaname NOT LIKE 'pg_%';

[bendmorris]

I just changed it to a one-time query as Ethan suggested - could someone who can reproduce this problem go ahead and test the latest version of master to see if it's been resolved?

[dmcglinn]

I'll test it out later today and report back

Dan
-------- Original Message --------
Subject: Re: [retriever] Retriever experiences massive slowdown with
postgreSQL (#42)
From: Ben Morris
[email protected]
To: Dan McGlinn [email protected]
Date: Friday, May 04, 2012 9:10:22 AM

I just changed it to a one-time query as Ethan suggested - could someone who can reproduce this problem go ahead and test the latest version of master to see if it's been resolved?

---

Reply to this email directly or view it on GitHub:
#42 (comment)

Daniel J. McGlinn, PhD
Postdoctoral Researcher
Utah State University
Department of Biology, BNR 132
Logan, UT 84322-5305
http://mcglinn.web.unc.edu/
cell: 405-612-1780

[ethanwhite]

Yeah, that did it. This is in master so it will be fixed in the next release.

Dan - it would be great to test on your machine if can build master from source, but you can't just download the .exe at the moment. At the very least we'll have you do an early test of the new .exe next week before we formally release. Thanks for testing things out and reporting the problem.