Comments (7)
Backblaze has many servers and the client maintains connections to not cause thrashing. Then they are closed later, but client can see them in this CLOSE_WAIT state, it's not really harmful. You can increase your open files limit using ulimit
.
Perhaps we can try to collect those sockets more aggresively, I'm going to look into that.
from b2-sdk-python.
ok, I looked into it and it might be that you are holding on to the response objects and you are not getting the garbage collector sweep them up (which would finish closing the socket). Can you please check if that's the case? If it's not, can you provide a small code sample that demonstrates the leak? It might be something on the sdk side as well and if it is we'll try to fix it.
from b2-sdk-python.
Is there a method in the sdk to scan these connections and close them? If so, I'm not really using it.
Anyway, follow the important snippets of my code.
Remembering that this runs in a daemon, so the tendency is that the process never ends.
I've already run other tests where the execution is unique, and in the end, all connections are closed normally.
The problem really only happens, because my process never ends.
from b2-sdk-python.
Imagine the following, I have a table that keeps the information of each client (bucket name, key id and app keys)
This daemon is "listening" to this table every minute to find out if there is a backup scheduled for that time.
So when there is a list of backups to be made, my code gathers everything that each client needs and sends it file by file.
At the end of each run, we can see that the connections are not closed.
I even increased the ulimit of the machine, but that only postpones the problem.
Before the too many open files error happened in 5 days, now it takes longer to happen, but it still happens.
from b2-sdk-python.
Please provide pip freeze
- out of which most important are b2sdk
and requests
versions.
Ideally, please provide a minimal example to reproduce this problem.
I'm continuously running python processes using b2sdk and have not seen zombie connections (neither ss
, lsof
seem to show them). There are few (<5) connections in *_WAIT
state tops. In my scenario multiple files (~10) are uploaded every other minute. In yours, do you have "bursts" of uploads, i.e. how many files do you upload at a time?
from b2-sdk-python.
Perhaps the request.response object is not being read, so urllib keeps not the connection, but the response object, which in turn occupies the connection?
I kind of remember a bug like that being fixed a while ago. @MuriloBianco are you using the newest version of the sdk?
from b2-sdk-python.
Guys, first of all, thank you all for your support.
Some time ago we updated all the libs in the environment.
I followed all the services for a whole month and the problem stopped happening.
Thank you everyone and sorry for not responding sooner.
from b2-sdk-python.
Related Issues (20)
- Remove hard dependency from tqdm HOT 3
- Refactor path handling code to use pathlib instead of strings HOT 1
- `UnboundSourceBytes` can break on upload retry
- download_file_by_name incompatible with documentation [MY MISTAKE - PLEASE DELETE]
- some directory exclusion patterns that worked in 1.21.0 don't work in 1.22.0 HOT 2
- delete_file_version needs bypassGovernance flag HOT 1
- passing compiled regex results in AttributeError HOT 2
- large file missing key ID HOT 10
- sync fails when encryption settings provider returns write mode b2sdk.v2.SSE_NONE HOT 8
- b2sdk v2 delete_bucket() throws v3.BucketIdNotFound exception
- b2sdk UploadTokenUsedConcurrently
- b2sdk.v2.SqliteAccountInfo does not raise an error when it fails to locate an account file HOT 6
- Invalid escape sequences in docstrings
- Tests no longer shipped in tarball HOT 6
- test_upload_url_concurrency not in test/ directory HOT 4
- lazy-fixture is not compatible with pytest 8 HOT 7
- Locale test fails HOT 1
- Progress Listener Test fails if tqdm is not installed HOT 1
- "TypeError: Argument of type 'Bucket' cannot be assigned to parameter of type 'Bucket' in b2sdk.v2" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from b2-sdk-python.