Don't have access to seed_rl bucket

[google-research]

Don't have access to seed_rl bucket

[lespeholt]

No this is perfectly fine. This is from actors which don't use GPU.

[mcaleste]

When I commented out the line

Create bucket if doesn't exist.

gsutil ls gs://seed_rl || gsutil mb gs://seed_rl

in setup.sh I get the following error:
ERROR: (gcloud.beta.ai-platform.jobs.submit.training) Non-custom jobs must have packages.

[lespeholt]

Can you try https://cloud.google.com/storage/docs/gsutil_install#creds-gsutil

[mcaleste]

I tried that and got a slightly different error:
AccessDeniedException: 403 AccessDenied

AccessDeniedAccess denied.

[email protected] does not have storage.objects.list access to seed_rl.

Creating gs://seed_rl/...
ServiceException: 409 Bucket already exists.

BucketNameUnavailableThe requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

[mcaleste]

When I commented out the line

gsutil ls gs://seed_rl || gsutil mb gs://seed_rl

in setup.sh I get the following error:
ERROR: (gcloud.beta.ai-platform.jobs.submit.training) Non-custom jobs must have packages.

Why isn't there a package specified when we call
gcloud beta ai-platform jobs submit training ${JOB_NAME}
--project=${PROJECT_ID}
--job-dir gs://seed_rl/${JOB_NAME}
--region us-central1
--config /tmp/config.yaml
--stream-logs -- --environment=${ENVIRONMENT} --agent=${AGENT}
--actors_per_worker=${ACTORS_PER_WORKER} --workers=${WORKERS} --

in setup.sh?

[lespeholt]

/tmp/config.yaml has information on the image being used, e.g. see https://github.com/google-research/seed_rl/blob/master/gcp/train_atari.sh

...
workerConfig:
imageUri: ${IMAGE_URI}:${CONFIG}
...

[mcaleste]

Thanks but I still don't understand why I'm getting the error:
ERROR: (gcloud.beta.ai-platform.jobs.submit.training) Non-custom jobs must have packages.
when I run train_atari.sh.

build.sh and push.sh are running fine even when I change the image to be a new bucket I have access to, but in setup.sh when it calls
gcloud beta ai-platform jobs submit training ${JOB_NAME}
--project=${PROJECT_ID}
--job-dir gs://seed_rl/${JOB_NAME}
--region us-central1
--config /tmp/config.yaml
--stream-logs -- --environment=${ENVIRONMENT} --agent=${AGENT}
--actors_per_worker=${ACTORS_PER_WORKER} --workers=${WORKERS} --

it gets that error
ERROR: (gcloud.beta.ai-platform.jobs.submit.training) Non-custom jobs must have packages.

But in the yaml the imageUri is the same one that the docker sh files made.

[mcaleste]

When I try running the docker image locally, in run.py it isn't able to load tf_config from the environment variable TF_CONFIG. I'm guessing this is just because the docker image isn't meant to be run locally and TF_CONFIG is created during gcloud beta ai-platform jobs submit training. But it might be a reason why I'm getting the error "non-custom jobs must have packages" if the docker image is failing to load or something.

[lespeholt]

It's not quite clear to me what's going wrong. Just tested it and seems to work for me. Maybe try remove "beta" which also works for me.

It appears that it fails for you even without starting to run the docker image so probably won't help running it locally.

[mcaleste]

I have tried removing beta and that didn't work either. When I run
gcloud beta ai-platform jobs submit training ${JOB_NAME}
--project=${PROJECT_ID}
--job-dir gs://seed_rl/${JOB_NAME}
--region us-central1
--config /tmp/config.yaml
--stream-logs -- --environment=${ENVIRONMENT} --agent=${AGENT}
--actors_per_worker=${ACTORS_PER_WORKER} --workers=${WORKERS} --
in the console instead of in the .sh, I get a more informative error message. First it says that I don't have access to gs://seed_rl for the --job-dir (like before). But then when I try to use a new job-dir with my own bucket, it gives the following error message:

Job [SEED_20200430153030] submitted successfully.
INFO 2020-04-30 15:30:33 -0700 service Validating job requirements...
INFO 2020-04-30 15:30:33 -0700 service Error creating the job. Field: job_dir Error: The provided GCS path gs://seed_rl/SEED_20200430153030 cannot be written by current user. Please make sure that the bucket exists and you have write access to it.
INFO 2020-04-30 15:32:59 -0700 service Validating job requirements...
INFO 2020-04-30 15:33:00 -0700 service Job creation request has been successfully validated.
INFO 2020-04-30 15:33:00 -0700 service Job SEED_20200430153030 is queued.
INFO 2020-04-30 15:33:07 -0700 service 1 Waiting for job to be provisioned.
INFO 2020-04-30 15:35:53 -0700 service 1 Waiting for training program to start.
INFO 2020-04-30 15:35:55 -0700 service 1 Job is preparing.
ERROR 2020-04-30 15:37:46 -0700 worker-replica-14 1 FATAL Flags parsing error: flag --actors_per_worker=: invalid literal for int() with base 10: ''

So I think what is going on is elsewhere it is trying to use gs://seed_rl but I don't have access to it. I think once I can get access to it everything will work, but it isn't my bucket, so it doesn't show up in the cloud storage browser or when I do gsutil ls.

[mcaleste]

Ok now I am able to get it to run when I change the job_dir to my own bucket but get this error:

ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.566483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.568575: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (-1)
ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.568725: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (gke-cml-0501-154905-bc2-n1-standard-4-143b9948-d9g1): /proc/driver/nvidia/version does not exist
ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.569201: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.569633: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.571217: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5276ef0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
ERROR 2020-05-01 08:54:11 -0700 worker-replica-5 1 2020-05-01 15:54:11.573891: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
ERROR 2020-05-01 08:54:11 -0700 worker-replica-0 1 2020-05-01 15:54:11.579866: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
ERROR 2020-05-01 08:54:11 -0700 worker-replica-0 1 2020-05-01 15:54:11.580554: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3dfd080 initialized for platform Host (this does not guarantee t

I think I need to set up a virtual machine first with the exact driver I need before running.

[mcaleste]

Ok great, thanks for all the help!