Downloading files/folders from synapse with additional jamboree credentials fails

[sage-bionetworks]

Downloading files/folders from synapse with additional jamboree credentials fails

[s-hoyt]

@jkiang13

Thank you! This worked and I was able to download all the files.

[jkiang13]

Hello @s-hoyt

Could you provide the synapse id of the folder you are downloading ("synID" in the script above).

Are you able to download these files individually outside of a sync, e.g.

syn.get(<syn_file_id)

If so that would help determine whether the issue is with sync or with the client being able to access these files at all.

Also, could you run the following against one of the files and let me know what the result is?

entity = syn.get(<syn_file_id>, downloadFile=False)
print(entity['_file_handle']['concreteType'])

Thank you.

[s-hoyt]

Hi @jkiang13

The synID of the folder = syn22349509

When I try to download an individual file:
syn22350959 = syn.get(entity='syn22350959')

I get the error:
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

I am not sure if this is an error in how I set up the client, or an error with this file.

When I run:
entity = syn.get(entity='syn22350959', downloadFile = False)
print(entity['_file_handle']['concreteType'])

I get:
org.sagebionetworks.repo.model.file.ExternalObjectStoreFileHandle

Thank you

[jkiang13]

Hi @s-hoyt ,

Okay this confirms that these files are accessed through external AWS credentials (I thought that's what you were referring to with the jamboree credentials but I wasn't sure). Unlike typical files stored in Synapse, the Synapse backend does not have access to files stored in this way and only serves as a directory, the clients themselves only can access these files and they need your AWS credentials to do so.

The Synapse Python client is not able to find any default AWS profile with your credentials (i.e. the credentials that you have been supplying on the web for individual downloads).

Can you create a file at path ~/.aws/credentials with the following content (filling in your values on the right side). You can also do this using the aws configure command if you have the awscli installed.

[default]
aws_access_key_id = <aws_access_key_id>
aws_secret_access_key = <aws_secret_access_key>

After creating this file then try again to retrieve an individual file and the sync as you have above.