DynamicFileListRecordReader error

[xardazz]

DynamicFileListRecordReader error

[przemekpastuszka]

Hi,
I've just ran into exactly same problem (my job was hanging on the export). I found this issue and as a workaround I tried to explicitly disable sharded export by doing:
hadoopConf.setBoolean(BigQueryConfiguration.ENABLE_SHARDED_EXPORT_KEY, false);
and to my surprise I got an error like this:

java.io.IOException: Cannot read and write in different locations: source: EU, destination: US
    at com.google.cloud.hadoop.io.bigquery.BigQueryUtils.waitForJobCompletion(BigQueryUtils.java:97)
    at com.google.cloud.hadoop.io.bigquery.UnshardedExportToCloudStorage.waitForUsableMapReduceInput(UnshardedExportToCloudStorage.java:87)

so what I did is that I fixed my input and outputs to be in the same location and then everything worked fine with sharded export enabled, no workaround needed.

This leads me to conclusion that this behaviour only occurs with cross-location exports. This is obviously bug in the connector, as it should throw an exception instead of hanging.

[AngusDavis]

@rtshadow - thank you for bringing up that case. I've opened an internal bug to track that case (have the DynamicRecordReader monitor export status).

[jeffreyolchovy]

Any update on this issue? I am experiencing similar behavior resulting in the DynamicFileListRecordReader instance hanging indefinitely.

If I try to export an unpartitioned table with a few records and BigQueryConfiguration.ENABLE_SHARDED_EXPORT_KEY is set to true, the 0 record file is never created and the record reader continues to wait for another resource.

Moreover, the DynamicFileListRecordReader exhibits the same behavior when exporting data from partitioned tables, even when using wildcards in the uri. So, AFAICT, this is a bug and it is does not behave as advertised.

10:26:47.403 [LocalJobRunner Map Task Executor #0] INFO  c.g.c.h.i.b.DynamicFileListRecordReader - Initializing DynamicFileListRecordReader with split 'InputSplit:: length:5 locations: [] toString(): gs://<elided>/shard-0/data-*.json[5 estimated records]', task context 'TaskAttemptContext:: TaskAttemptID:attempt_local949659226_0001_m_000000_0 Status:'

The only way I can successfully use the BigQuery input format is if BigQueryConfiguration.ENABLE_SHARDED_EXPORT_KEY is set to false, which is undesirable.

[juliankeppel]

Hi, I use the spotify spark-bigquery connector (https://github.com/spotify/spark-bigquery) in version 0.2.2-s_2.11 which uses bigdata-interop in version 0.10.2-haddop2. First I load meta data from dataset.__TABLES__ and then the actual data. Since some days I face the same problem mentioned here. The job hangs at DynamicFileListRecordReader when I want to load the (pretty small) meta table from BigQuery. Loading the actual "data" tables works as before.

Since I haven't change any lib versions or code in my application, it seems that there has something changed in BigQuery or Cloud Storage itself?

[pmkc]

I think the issue is that BigQuery does not create the 0 record file when using a single wildcard, which is why there is MIN_SHARDS_FOR_SHARDED_EXPORT = 2.

However mapreduce.job.maps=1 (like in local mode) used to overrule that before ea2bb01.

Closing since this should the minimum should now be enforced, and BigQuery should always create the 0 byte files.