Adding resource while script runs in background causes trace error

[geopython]

Adding resource while script runs in background causes trace error

[tomkralidis]

The hard truth of this issue is that it's a reality of using SQLite3 as the DB. I think the workaround here would be to use PostgreSQL or something similar?

[justb4]

Possibly related: we find problems that the database is locked for writing when for some reason one of the remote services keeps holding the connection, sometimes for over 5 mins.

What I see in models.py is that a DB.session.commit() is done after all runs have completed where for each run a DB.session.add() is done. In general that can take quite a while. I've also noticed that adding/updating resources on the hour may cause failures.

So the root cause is I think the long transaction/lock (during entire models.py run).

A possible fix would be to have multiple short-lived transactions, one for each run just after completion of the run. I don't know what the performance impact would be, but this would make editing IMO at least less failure-prone.

[justb4]

Added a fix as described in my comment above: tested: managing (add/delete) Resources during healthcheck run is not a problem anymore in our case. Please test. Also added some print output showing Run timings and status.

[justb4]

@samperd does the fix work for you? You can test by adding resources while GHC is running checks.

[samperd]

@justb4 Thanks for applying this fix. I will work towards checking out this fix.

[justb4]

@samperd any news? Can we hopefully close the issue?

[justb4]

We assume the above fix works, at least for us it did. Closing issue, pls reopen if problem still exists.