Sign blob URL using workload identity instead of common service account credentials

[devnicolas1]

Sign blob URL using workload identity instead of common service account credentials

[devnicolas1]

Hey @frankyn! First of all, thank you so much for your help!

When trying to execute response = client.generate_access_token(name=name, scope=scope), I got the error:

  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/google/cloud/iam_credentials_v1/services/iam_credentials/client.py", line 825, in generate_access_token
    response = rpc(
               ^^^^
  File "/usr/local/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
domain: "iam.googleapis.com"
metadata {
  key: "permission"
  value: "iam.serviceAccounts.getAccessToken"
}
]

Interesting enough, I had confirmation earlier from devops guy that all permissions were correctly configured, but this does get me thinking. I'll make sure to confirm everything is configured the way it should be, but I'll at least leave the traceback to keep it all registered.

Once again, thanks for the help! Will comeback with updates as soon as I can.

[devnicolas1]

Following this example for manually signing URLs, I still didn't quite comprehend what exactly should be my google_credentials. Using both the Credentials object suggested in the example and the impersonated credentials auth.impersonated_credentials still resulted me in the error "no private key". Still trying

[devnicolas1]

So, after a lot of trying, this is the furthest I think I got:

from google.auth import default
from googleapiclient import discovery
import base64
  
bucket_name = "bucket_name"
object_name = "object_name"
expiration_time = 3600
credentials, _ = default(scopes=['https://www.googleapis.com/auth/cloud-platform'])
service_account_email = '[email protected]'
iam_service = discovery.build('iam', 'v1', credentials=credentials)
string_to_sign = f"GET\n\n\n\n{expiration_time}\n/storage/v1/b/{bucket_name}/o/{object_name}"
body = base64.b64encode(string_to_sign.encode("utf-8")).decode("utf-8")
namePath = 'projects/-/serviceAccounts/[email protected]'


response = iam_service.projects().serviceAccounts().signBlob(
      name=namePath,
      body={"bytesToSign": body}
).execute()

Still gives me the error:

googleapiclient.errors.HttpError: <HttpError 403 when requesting https://iam.googleapis.com/v1/'projects/-/serviceAccounts/[email protected]:signBlob?alt=json returned "Permission 'iam.serviceAccounts.signBlob' denied on resource (or it may not exist).". Details: "[{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'IAM_PERMISSION_DENIED', 'domain': 'iam.googleapis.com', 'metadata': {'permission': 'iam.serviceAccounts.signBlob'}}]">

I have a feeling that using default() for getting my credentials isn't exactly what I should be doing, but I don't really know what other way I have of getting it.

[frankyn]

Hi @devnicolas1,

You need to grant roles/iam.serviceAccountTokenCreator role to [email protected].

The client will do this on your behalf, iff you supply access token and service account email; can you try the following?

    import datetime

    from google.cloud import iam_credentials_v1
    from google.cloud import storage

    client = iam_credentials_v1.IAMCredentialsClient()
    service_account_email = "[email protected]"
    name = path_template.expand(
        "projects/{project}/serviceAccounts/{service_account}",
        project="-",
        service_account=service_account_email,
    )
    scope = [
        "https://www.googleapis.com/auth/devstorage.read_write",
        "https://www.googleapis.com/auth/iam",
    ]
    response = client.generate_access_token(name=name, scope=scope)
    
    storage_client = storage.Client()
    bucket = storage_client.bucket("bucket-name")
    blob = bucket.blob("object-name")
    # https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob#google_cloud_storage_blob_Blob_generate_signed_url

    url = blob.generate_signed_url(
        version="v4",
        # This URL is valid for 15 minutes
        expiration=datetime.timedelta(minutes=15),
        method="GET",
        service_account_email=service_account_email,
        access_token=response.access_token
    )

[devnicolas1]

After a lot of testing and trying different things, we opted for a bit of different way.

We generated a pair of HMAC keys and used Boto3 for signing URLs. It worked out perfectly and was easier to make it work for both development/devops teams.

Still, thank you so much @frankyn for your help! Your comment surely contributed to our success in the end, even if we opted for a different solution.

If anyone finds this issue in the future: the link I added is an easy way of achieving the desired result with HMAC keys. Can't talk about using simply workload identity since we opted for another way.

Cheers!

[frankyn]

@devnicolas1 are you using boto3 with GCS for signed URLs?

[devnicolas1]

Correct. I really don't know in details how does it work, but we got the idea from this example we found while searching. We did a quick test and everything actually worked out flawlessly, and still is in production right now.

Here is how the code look like:

from boto3 import client as boto3Client

client = boto3Client(
                "s3",
                region_name="southamerica-east1",
                endpoint_url="https://storage.googleapis.com",
                aws_access_key_id=environ['ACCESS_KEY_ID'],
                aws_secret_access_key=environ['ACCESS_KEY_SECRET'],
            )

return client.generate_presigned_url(
                'get_object',
                Params={
                    'Bucket': bucket.name,
                    'Key': name
                },
                ExpiresIn=convertTimeStringToSeconds(availabilityTime)
            )

Edit: just to make extra clear @frankyn, bucket is still the google.cloud.storage.Bucket object, and was retrieved using the GCP library as usual. Literally the only thing in the whole code that uses Boto3, is the URL signing