Super Weird issues re transactions and connection pools

[tortoise]

Super Weird issues re transactions and connection pools

[abondar]

Hi
Currently I can't go all out investigating it, will look further into in on weekend.
From what I managed to see right now:

postgresql just hangs around and waits for something

Problem seems to be that connections in pool ends and test hangs up waiting for connection to be released. So the problem is why they are not released by previous tests?

some operations, such as .update use the connection very differently than .filter Why do we have code sections that reserve a single connection? and some that dont?

The thing is .filter doesn't make a query itself, it just prepares values for it. Actual query is made by executor class, and it was meant that all execute methods in executor should reserve single connection for all of a query. You can see that it is reserved in execute_select the same way it is reserved in execute_update. That was made exactly because of problems with pool I had in the initial stage of project, when recursive operations just took all of the connections from pool and it hang up the same way, waiting for connection to be released, never aquiring it, because upper level query was waiting for recursive queries to be done.

I think bug should be somewhere there, may be some part of code doesn't honour this idea and just takes fresh connection from the pool, so because of it subsequent query in a transaction doesn't see the prior changes made in the transaction.

Also - may be we can see this problem because green running multiply tests concurrently and different tests hang up on middle because all connections are already taken, but even if it is this way, the bug is still there because query should be able to finish itself within one connection and releasing it after finish

[grigi]

I already checked green and single process, definitely not that.

Yes, I was talking of how execute_select reserved a single connection, and how UpdateQuery does not.
Also the whole transaction set was re-worked recently, and it is likely that due to changes in behaviour there we broke pooling (or rather I broke it unwittingly).

I feel that the grabbing of a single connection seems... over-complicating things. Likely we don't understand the real issue?
We probably need tests that operate on the backends directly, so we can determine if the problem is in our client implementation, or the executor, or the queryset?

[grigi]

Since we essentially removed the known broken pool implementation (and we still retain some of the benefit, such as persistent connection re-use) I'm closing this issue.