Comments (10)
Sorry to bug you @peter-wangxu but I am at the stage where I need to make this work or move on but don't want to spend time rewriting to use an mqtt if this can be made to work with a small change.
from persist-queue.
Sorry to bug you @peter-wangxu but I am at the stage where I need to make this work or move on but don't want to spend time rewriting to use an mqtt if this can be made to work with a small change.
not very big change, i could try to give a version this weekend.
from persist-queue.
That would be absolutely amazing!! Thank you so much!!
I don't see an option to buy you a coffee/beer, I definitely owe you one after this!
from persist-queue.
Hey @peter-wangxu! I finally got a chance to set it up, the only thing blocking me from testing is that it's saying MySQLQueue object has no attribute total, this is when I try to do qsize. Is that still available, from what I can read it looks like it is but maybe I am doing something wrong.
I see it now, self.total +=1 was part of the put before and now it's not (would need it for the get as well). Did you want me to do a pull request to fix this or did you want to?
I have made the fix in my testing env, and testing now. Will updated
UPDATE: I must still be missing something cause it's still complaining.
So I figured out what is missing, but I'll have to dig in tomorrow to figure out how to fix it.
self.total = self._count()
def _count(self):
sql = 'SELECT COUNT({}) FROM {} WHERE status < ?'.format(
self._key_column, self._table_name
)
row = self._getter.execute(sql, (AckStatus.unack,)).fetchone()
return row[0] if row else 0
from persist-queue.
Hey @peter-wangxu sorry to bother but i tried a few things and can't seem to get it working.
from persist-queue.
Thanks so much for the updates, something still appears to be of though cause the first process is putting records on the queue but my second process isn't recognizing the records are there after running a qsize
from persist-queue.
Here is the code snippet, for some reason it's not updating once the loop starts and I use my other process to put items into the queue.
import os, requests, json, sys, persistqueue, time
job_queue = persistqueue.MySQLQueue(os.getenv("DOCKERENV_MARIAHOST"), "pyqueue", "Password", "pyQueue", int(os.getenv("DOCKERENV_MARIAPORT")))
job_queue_p1 = persistqueue.MySQLQueue(os.getenv("DOCKERENV_MARIAHOST"), "pyqueue", "Password", "pyQueue_p1", int(os.getenv("DOCKERENV_MARIAPORT")))
total_queue = job_queue.qsize() + job_queue_p1.qsize()
if __name__ == '__main__':
while True:
total_queue = job_queue.qsize() + job_queue_p1.qsize()
print(total_queue)
time.sleep(10)
from persist-queue.
Found it.
Here is someone else with the issue.
https://stackoverflow.com/questions/60276355/why-python-script-loop-is-not-fetching-updated-data-from-db
I updated my code to count the records from the db instead of queue size, and without conn.commit() the count wasn't updating either but as soon as I do conn.commit() after execute it started working properly.
from persist-queue.
Found it. Here is someone else with the issue. https://stackoverflow.com/questions/60276355/why-python-script-loop-is-not-fetching-updated-data-from-db
I updated my code to count the records from the db instead of queue size, and without conn.commit() the count wasn't updating either but as soon as I do conn.commit() after execute it started working properly.
Thanks for the investigation. i was busy with other stuff, will continue work on this later this weekend
from persist-queue.
Thanks for the investigation. i was busy with other stuff, will continue work on this later this weekend
No stress at all, I had time as was needing to solve it so I figured I would dig into it. I really appreciate all your help on this!
from persist-queue.
Related Issues (20)
- qsize not changing HOT 1
- pypi release HOT 3
- SQLite objects created in a thread can only be used in that same thread HOT 2
- File is not getting cleared after get() function HOT 13
- Connection not die after kill signal. HOT 1
- Queue gets corrupted after power cut HOT 9
- sqlite vacuum not working HOT 8
- Queue items not removed after each get() HOT 3
- Priority queue HOT 4
- What if data is already serialized HOT 2
- What if data is already serialized Part II ? HOT 1
- Get() hangs when used with pqid HOT 1
- FILOSQLiteAckQueue is not exported from __init__ HOT 2
- AttributeError: module 'persistqueue' has no attribute 'MySQLQueue' HOT 3
- [BUG] "No sqlite3 module found, sqlite3 based queues are not available" during load, despite sqlite3 being present... and also works despite? HOT 3
- Error while get queue element by id
- Feature Request: allow `before` and `after` controls for `put` to fine tune placement within queue
- Feature Request: Add separate metadata block from ID
- Provide async api
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from persist-queue.