Comments (5)
@jeffwillette thanks for raising the issue.
@mihran113 @alberttorosyan please take a look at this whenever you can.
from aim.
It appears to be a problem with sqlite. the run_metadata.sqlite database is showing that it is locked even though there should be no process which is writing to it. This must somehow be a result of the crash.
I left the forever hang go for a while and it ended in this error with this triggering many more exceptions which end in the same error:
Traceback (most recent call last):
File "/c2/jeff/anaconda3/envs/set-ssl/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/c2/jeff/anaconda3/envs/set-ssl/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked
I tried manually unlockin/backing up/dumping the database which I read should unlock it, but it doesn't. I moved trhe database to another filesystem, and I was able to read from it using sqlite3, but once I move it back to the location in the .aim folder, it is still locked.
This means it must have something to do with the filesystem, which is a NFS. The thing is, there is no process which is actively connected to the database, so something must have persisted from the crash to keep it locked, but I cannot find what it is. Any ideas?
from aim.
Seems related to #1865.
Also sqlite recommends not running a db on an NFS (https://www.sqlite.org/faq.html, https://www.sqlite.org/howtocorrupt.html). If you google around for this topic, it comes up with "don't do it" almost everywhere.
from aim.
Hey @jeffwillette! Thanks for the additional input. Looking into this issue now.
Is there any additional output when you run aim up --log-level DEBUG
?
On a separate note, do you recall when the crash happened? It could be a separate issue or somehow related to this one.
from aim.
@alberttorosyan, I think it was only the trace I posted above. I am almost certain this issue comes down to NFS and sqlite clashing with each other, but I wasn't sure how to proceed so I just had to start over and delete the old repo (lucky there was nothing crucial in there).
The crash happened, right before the problem came up. Servers unexpectedly lost power in a power outage and when I got back and tried to fire up aim again, I was confronted with this error.
I think this might be quite dangerous for those running on an NFS. If anyone runs into this problem in the future, the only way I was able to get the sqlite database to unlock was to copy the file to a non-NFS drive and then I was open to open the db manually to inspect the tables. So if any important information were in there, I guess the whole aim repo could be copied to the drive and it should theoretically work again.
Anyway, if this is determined to solely be a sqlite/NFS issue, then feel free to close the issue.
from aim.
Related Issues (20)
- Configure axes to automatically share the same data-defined axis length HOT 2
- Revamp the design of app cards
- Passing a bearer token
- aim v4.0 offline support HOT 2
- No module named 'aim._sdk.run_status_watcher'
- Failed to run Aim Tracking Server HOT 1
- could we start UI from a remote repo: aim://host:port?
- How can I display `hash` column at the first in aim-v4.x?
- Random run initialization error HOT 3
- Run merging and aggregation
- server-client connection error HOT 2
- LOG.old files HOT 2
- Upgrade node version and react version HOT 7
- Can't edit or see existing tags on the Tags page HOT 4
- Having problems using with fairseq HOT 4
- Potential security issue HOT 5
- When aim server is killed, the client use run.close() will be hang
- cannot import aimstack without aimos package HOT 1
- SQLAlchemy support for MySQL HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aim.