LeaseExpiredException when copying many files from gcs to HDFS

[jburos]

LeaseExpiredException when copying many files from gcs to HDFS

[jburos]

Think we figured this out, -- don't think it involves the gc_connector -- but posting here in case someone else encounters this.

Current theory is that two nodes are trying to write to the same file, and when one completes it deletes the ._COPYING_ file, causing an error on the other (or some variation of this is occurring during copying before completion).

The file listing by which we are copying includes directories, so if I run in one process (-j1) to download files to an empty directory i see the following:

Copying: gs://bucket-name/for-msk-bms-lung/downloads/0017a332-c7fd-41ca-8cb9-cd8d81f1cb24 to /datasets/bucket-name/for-msk-bms-lung/downloads/0017a332-c7fd-41ca-8cb9-cd8d81f1cb24 done
Found: /datasets/bucket-name/for-msk-bms-lung/downloads/0017a332-c7fd-41ca-8cb9-cd8d81f1cb24/C508.TCGA-60-2721-11A-01D-1522-08.5_gdc_realn.bai
Found: /datasets/bucket-name/for-msk-bms-lung/downloads/0017a332-c7fd-41ca-8cb9-cd8d81f1cb24/C508.TCGA-60-2721-11A-01D-1522-08.5_gdc_realn.bam
# ...

So it’s copying the directory, then each file or directory within the directory. When this gets parallelized, the files within the directories are then trying to be “copied” by more than one process simultaneously. This gets really messy, particularly when we have several nested directories. The same file might be being copied by several processes at once, and so it's only a matter of time until there's a collision.

After modifying the script to exclude directories, and instead do a -mkdir -p on the dirname for each file listed, we haven't seen any problems yet. Will come back to close this once it's error-free for some time.