'Blob.exists()' does not work within batch context

[googleapis]

'Blob.exists()' does not work within batch context

[tseaver]

@ludwigschubert Under the covers, Blob.exists() makes a GET request to the blob's resource URL and converts a 404 response into a False, 20x into a True. This implementation does not fit well with the current Batch design, which fakes the responses inside the context manager, and then applies the sub-responses to the individual targets at exit. See the comment in Blob.exists.

[davidbernat]

To be clear: when this was moved to "feature request" by @frankyn , the implied statement is:

there is no way to use blob.exists() within a batch context

Is this correct?

Thanks!

[davidbernat]

For @ludwigschubert and those who arrive via Google.

I found a simple fix for this: wrap each call for blob.exists() in a try/catch.

def to_parallelize_catch_exceptions(f, on_error_value, *args):
    try:
        return f(*args)
    except:
        return on_error_value

def parallelize(blobs, function, args_lists=None, on_error_value=None, n_threads=50):
    if args_lists is None:
        args = [(function, on_error_value, blob) for i, blobs in enumerate(stubs)]
   else:
        args = [(function, on_error_value, blob, *args_lists[i]) for i, stub in enumerate(blobs)]
    with multiprocessing.Pool(min(n_threads, len(blobs))) as p:
       values = p.starmap(to_parallelize_catch_exceptions, args)
   return values

def blob_exists(blob):
    blob.exists()

blobs = [....]
with client.batch():
    parallelize(blobs, blob_exists, on_error_value=False)

Of course, there are two layers of abstraction above which are not expressly necessary, but I've copy-pasted my internal code that handles the parallelization for all the cloud client functions. Feel free to comment or DM me for more information as necessary. Cheers.

[tseaver]

Possible implementation at googleapis/google-cloud-python#8618

[cojenco]

Thank you folks for providing above workarounds!

The current batch design does not support library methods whose return values depend on the response payload. In this case, the error handling and conversion in blob.exists() is not fully supported in a Batch context.

However, note that a new raise_exception flag is added to Batch via #1043 (pending release in v2.10.0). Setting raise_exception=False allows all exceptions to be included in list of return responses. Although not recommended, it is now possible to get a list of 404 responses calling blob.exists() with storage_client.batch(raise_exception=False)

In addition, we've also added clarifications on the limited Batch support in the python client, see details in #1045