Comments (11)
I'm afraid that the current batch implementation does not deal well with any methods whose return values depend on the response payload (as opposed to just being able to update the relevant blobs / buckets from those responses when the batch completes).
Adding support for such methods would be a large undertaking.
from python-storage.
Could the issue be left open and marked as a future enhancement request?
from python-storage.
@bits01 I have added this issue to our feature request project
from python-storage.
Thank you. Should the issue be re-opened then? It's currently marked closed
.
from python-storage.
I think, it can be done in fashion of: googleapis/google-cloud-python#8618. But first it's better to deal with 8618
from python-storage.
@IlyaFaer, what is the issue that needs to be dealt with first? googleapis/google-cloud-python#8616 doesn't seem related.
from python-storage.
@tseaver, sorry, mistake, 8618. I'm waiting for review in it, and if changes will be accepted, we could use something alike in this issue
from python-storage.
Just to be clear: at this moment the most reliable way to copy a lots of files from one bucket to another (retry strategy, batch requests and object rewrite) in python is subprocess.run(["gsutil", "cp",...])
?
I was comparing contributors here to gsutil repo, and looks like are totally different and isolated teams, why?
from python-storage.
Noted during review: Blob.rewrite
returns a 3-tuple, (rewrite_token, bytes_rewritten, object_size)
, in order to permit tracking progress on the rewrite (it may only complete partially).
from python-storage.
This is also an issue when trying to use batch()
to copy blobs while preserving the ACL of the original blob.
from python-storage.
Closing; We've added clarifications on the limited Batch support in the python client, see details in #1045
from python-storage.
Related Issues (20)
- Need to be able to download objects without storage.bucket.get permission same as gsutil HOT 1
- BlobWriter should abort multipart upload during exception handling HOT 7
- docs: add object retention samples
- Add http.client.RemoteDisconnected to list of retryable exceptions HOT 1
- Add 503 to list of retriable HTTP response codes HOT 4
- Warning: a recent release failed HOT 1
- Discussion: Contribution Idea - Python Code Sample for Handling Large JSON Files on Google Cloud HOT 3
- Sign blob URL using workload identity instead of common service account credentials HOT 8
- blob.upload_from_string get error Caused by SSLError(SSLEOFError HOT 2
- Support Storage Control Quickstart HOT 2
- `Blob.content_type` is `None` when created `from_string()` HOT 1
- match_glob keyword argument on google.cloud.storage.Client().list_blobs() has disappeared HOT 2
- Blob Writer's close function causes latency > 15s under load. HOT 6
- FR: Support HNS enablement in bucket metadata
- Can't set Cache-Control on GCS object HOT 2
- media_link & self_link in blob do not update when client option "api_endpoint" is set HOT 2
- Bypass 8MB limit to allow file to be uploaded in single request HOT 1
- Retry batch delete blob on 503
- Some unit tests require real credentials files
- Support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-storage.