runs stuck in progress

[Laiaborrell]

runs stuck in progress

[mihran113]

Hey @Laiaborrell! Thanks a lot for the report, that seems kinda strange, as there's no scenario that runs can reactivate by themselves. My only guess is that the runs were tryed to be deleted, and something went wrong in the process of deletion, that's why it's showing that the runs are not found. As aim stores data about runs in 2 dbs (sqlite and rocksdb). I think that rocksdb portions of the data were removed, and the data in sqlite is still there.
You can check if that's the case by checking if the ./aim/meta/chunks/{run_hash} directory still exists or not.

[Laiaborrell]

Hey @mihran113, thanks for your reply!! I checked and the hashes for the runs are still in the chunks folder:

I did not try to delete any of the files either :/
It is weird for me because they appear as active and the run time keeps increasing (8 days now), but the gpu were the process was training has been stopped... Also, the chunk folders' files were also last updated three days ago, when their training finished