Broken ActiveRecord connections after a timeout

[collimarco]

Broken ActiveRecord connections after a timeout

[GBIDEV]

This is more likely a Heroku issue you need to bring up with Heroku Support. They may have better logs to identify the issue. Maybe start a Stack Overflow thread as well to get a wider response from the online community.

Depending on your plan with Heroku, you may be able to get them to assist with the restart of the dyno or you can create your own trigger to restart dynos daily as depending on your plan, they may have moved your DB during that period which caused the issue. I have experienced this before and as we were on a Hobby plan, they did not help us, so we added our own restart task daily so that we work on our timezones and not theirs. (Add Heroku Schedular and add a task "heroku ps:restart --app xxxxx")

rack-timeout threw the error, Rack::Timeout::RequestTimeoutException:, which confirms it caught it, but it could be your worker thread or any other puma or background task kept on retrying or attempting to complete the task but due to DB connectivity could not commit the transaction or query.

[dentarg]

@collimarco Please consider the options that have been suggested in #49 ( statement_timeout, RACK_TIMEOUT_TERM_ON_TIMEOUT)

[mtsmfm]

Without term on timeout, not only connections but also almost every code can be broken completely.
I described the detail in #169 and am proposing to set term on timeout by default.

[wuputah]

Is there any way to fix this inside this gem?

No, but you can use the "TERM on timeout" feature.

Should we report this to Rails / ActiveRecord? Why the connection was not healed automatically after some time?

It's been reported but it's not considered something they can fix. You can opt out of using prepared statements (and this is actually recommended by Heroku to turn this off) which does avoid this particular issue but you can still run into cases where the connection can enter a bad state.

Is there any way to tell Heroku to restart the dyno in this case (when it detects high rates of 500 errors)?

No -- there's infinite causes for a 500 and while restarting might work in some cases, the solution most apps use here would be to monitor the 500 rate to page a human to address the root cause of the 500s.