About Dataset

[jannerm]

About Dataset

[chaojiewang94]

error when run bash files

[ utils/filesystem ] Found bucket in savepath: my-bucket | path: logs/rwr-compressed-animals-s10k/samples/0_reference
[ sizes ] gs://my-bucket/logs/rwr-compressed-animals-s10k/samples/0_reference
Traceback (most recent call last):
File "/home/Desktop/Logical_Diffusion/Diffusion_RL/pipeline/save_sizes.py", line 37, in
fnames = utils.fs.ls(args.loadpath)
File "/home/Desktop/Logical_Diffusion/Diffusion_RL/ddpo/utils/filesystem.py", line 31, in ls
paths = sorted(fs.ls(path))
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/fsspec/asyn.py", line 113, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/fsspec/asyn.py", line 98, in sync
raise return_result
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
result[0] = await coro
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/core.py", line 825, in _ls
out = await self._list_objects(path, prefix=prefix)
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/core.py", line 537, in _list_objects
items, prefixes = await self._do_list_objects(
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/core.py", line 572, in _do_list_objects
page = await self._call(
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/core.py", line 422, in _call
status, headers, info, contents = await self._request(
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/decorator.py", line 221, in fun
return await caller(func, *(extras + args), **kw)
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/retry.py", line 115, in retry_request
return await func(*args, **kwargs)
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/core.py", line 415, in _request
validate_response(status, contents, path, args)
File "/opt/conda/envs/ddpo/lib/python3.9/site-packages/gcsfs/retry.py", line 96, in validate_response
raise IOError("Forbidden: %s\n%s" % (path, msg))
OSError: Forbidden: b/my-bucket/o
[email protected] does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
WARNING:absl:Tensorflow library not found, tensorflow.io.gfile operations will use native shim calls. GCS paths (i.e. 'gs://...') cannot be accessed.

[chaojiewang94]

should I modify the bucket = "gs://my_bucket" in use.py ?

[chaojiewang94]

the dataset is generated by python sample.py

[jannerm]

Yes, there is no static dataset; new data is sampled from the model on-the-fly during finetuning (in sample.py for RWR variants and policy_gradient.py for DDPO variants).

As for the bucket path, all of the samples and checkpoints are saved to the GCP bucket specified on this line.