listing of large directories in gcs

[googleclouddataproc]

listing of large directories in gcs

[dennishuo]

Thanks for the suggestion! At the moment, since GCS doesn't promise returning the full maxResults number of items per page, it does appear currently to still clip to 1000 (less than the requested 1024) items per round trip; you can play around with the real GCS behavior using the apis-explorer to see if your experiences match this observation.

That said, you're right that we should at least expose it as a Hadoop configuration for ease of use when/if GCS supports larger pages on the backend in the future. That was the original intent with wiring it at least partially into GoogleCloudStorageOptions. I've added tracking for this request on our internal issue tracker as well.

[jefimm]

By the way, AWS S3A code in hadoop requests 5000 entries for directory listing per round trip

[medb]

GCS connector has fs.gs.list.max.items.per.call property to configure number of items to return per GCS list* request.

But, currently, GCS has maximum value of 1,000 for maxResults parameter in list requests.