sqlite3.OperationalError: database is locked

[n8henrie]

sqlite3.OperationalError: database is locked

[ozgeneral]

@n8henrie I had the same problem, and probably related information: I was using a lot of concurrent calls.

I found the process that has locked the file and it seems to be:

oezgen             637   0.0  0.2  4681680  26532   ??  S     5Nov19 183:07.43 /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/78.0.3904.70/Helpers/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper --type=utility --field-trial-handle=1718379636,13217150702388057384,3641183978531146804,131072 --lang=en-US --service-sandbox-type=network --service-request-channel-token=5198092952403259345

I think Google Chrome was also using the cookies for the same pages
So I'm not sure what in particular caused the problem but for those who ended up here and looking for quick fix (on mac):

 fuser /Users/<USERNAME_HERE>/Library/Application\ Support/Google/Chrome/Default/Cookies
/Users/oezgen/Library/Application Support/Google/Chrome/Default/Cookies: <SOME_PID>
 kill -9 <SOME_PID>

[n8henrie]

Huh, thanks for the report. Can I ask about your use case? Are you retrieving cookies for thousands of URLs at once? Or doing so over time (i.e. a long-running process)?

Are you browsing while requesting the cookies? I know Chrome does lock the file at times; I generally recommend quitting the application prior to using pycookiecheat to avoid issues here.

[FuhuXia]

I have a list of 30k URLs. I am processing them one by one over time. The whole process usually take 1+ hour. Sometimes it finishes fine from beginning to end, sometimes it has this glitch 2-3 times during the process.

I occasionally browse the site to keep session live. The glitch happen on its own, not when I am browsing.

[n8henrie]

Yes, I have also run into a similar intermittent error when trying to access the cookie file while Chrome is running. I don't know of a better solution than not running while browsing, since Chrome seems to lock the database on occasion.

Perhaps there are some sqlite flags to open with read-only access that bypass locking concerns?

[stale]

This issue has been automatically marked as stale because it has not had recent activity or it has not had a response to a question in 14 days. It will be closed automatically if no further activity occurs within 7 days. Thank you for your contributions.

[n8henrie]

Looks like this may be out of my control:

https://stackoverflow.com/questions/7857755/is-it-possible-to-open-a-locked-sqlite-database-in-read-only-mode

Unclear if adding ?mode=ro may help: https://docs.python.org/3.8/library/sqlite3.html#sqlite3.connect

[n8henrie]

Please see if 230574e helps

[n8henrie]

Bump.

This issue is pending a response and is blocked until I have more info. To keep the issue list groomed, I will often close these stalled issues if they go a prolonged time without a response. In this case, I will give it another week or so.

[stale]

This issue has been automatically marked as stale because it has not had recent activity or it has not had a response to a question in 14 days. It will be closed automatically if no further activity occurs within 7 days. Thank you for your contributions.

[n8henrie]

Thanks for chiming in. Would you mind trying the dev branch (which includes the commit linked above to open in read-only mode) and see if you still have this issue?

[ozgeneral]

@n8henrie thanks for the quick fix (and sorry for the late reply from my side)

I haven't had this problem since my last comment, although I am still using a lot of concurrent calls. I'm not sure how I triggered it before...

I will start using dev branch with pycookiecheat and post more details here if I encounter the same issue again.

[n8henrie]

Sounds good, thanks for the update.

[NightMachinery]

@n8henrie commented on Jan 26, 2020, 5:53 PM GMT+3:30:

Thanks for chiming in. Would you mind trying the dev branch (which includes the commit linked above to open in read-only mode) and see if you still have this issue?

Is this commit still in the dev branch? I just installed git+https://github.com/n8henrie/pycookiecheat.git@dev, and I still get

Traceback (most recent call last):
  File "<string>", line 7, in <module>
  File "/Users/evar/anaconda/lib/python3.7/site-packages/pycookiecheat/pycookiecheat.py", line 248, in chrome_cookies
    ) in conn.execute("PRAGMA table_info(cookies)"):
sqlite3.OperationalError: database is locked

[n8henrie]

Comparing the commit I linked above: 230574e with the current dev branch:

pycookiecheat/src/pycookiecheat/pycookiecheat.py

Line 234 in e8e9893

conn = sqlite3.connect("file:{}?mode=ro".format(cookie_file), uri=True)

it looks like it is.

So I guess that commit didn't prevent the issue, though it still seems like a good idea (since there's no reason my library should be modifying anything at this point).

So I think there is some other process, likely Chrome or a leftover background process from Chrome but possibly something else, that has the db locked.