federation e2e gce automated tests on Jenkins fail consistently with token auth attempt failed with status: 403 Forbidden

[kubernetes]

federation e2e gce automated tests on Jenkins fail consistently with token auth attempt failed with status: 403 Forbidden

[ixdy]

Most likely ACLs for the [email protected] service account are missing somewhere.

[nikhiljindal]

I had created the project using the standard create project script which should have setup ACLs correctly.

The job is running in k8s-jkns-pr-bldr-e2e-gce-fdrtn project and we are running gcloud docker push <image-name> to the same project, so no extra credentials should be required.

@madhusudancs had suggested running gsutil acl to ensure the right ACL on the GCS bucket, but running:

gsutil acl ch -u [email protected]:R gs://artifacts.k8s-jkns-pr-bldr-e2e-gce-fdrtn.appspot.com

obviously fails with

BucketNotFoundException: BucketNotFoundException: 404 gs://artifacts.k8s-jkns-pr-bldr-e2e-gce-fdrtn.appspot.com bucket does not exist.

Trying to create that bucket using the UI asks me to verify that I own appspot.com :)

cc @colhom How did you set up artifacts.k8s-jkns-e2e-gce-federation.appspot.com GCS?

[madhusudancs]

@madhusudancs had suggested running gsutil acl to ensure the right ACL on the GCS bucket, but running:

If you are running the build jobs and tests jobs in the same project, you don't need any of these ACL changes I suggested. You only need them when the build artifacts and test VMs are in different projects.

Trying to create that bucket using the UI asks me to verify that I own appspot.com :)

I don't think you can create appspot.com buckets. And also, you don't have to :) GCR automatically creates this bucket for you when you run gcloud docker push the first time.

I think @ixdy is right. This is a credentials issue. And I think it is likely because (pure guess, zero evidence) the job that's running gcloud docker push doesn't have push access to gcr.io/k8s-jkns-pr-bldr-e2e-gce-fdrtn/hyperkube.

I know this feels like a frustrating chicken & egg problem. I think the solution here is to add the service account of the project where you run gcloud docker push (kubernetes-jenkins-pull I think?) as an editor or some other role which has write access to your k8s-jkns-pr-bldr-e2e-gce-fdrtn project. Ping me on chat, I will point you to the right links/pages.

[nikhiljindal]

Ran the following command which should have been run anyway by the create_jenkins_project script if i had set the flags correctly while calling it (which I may or may not have :)):

gcloud projects add-iam-policy-binding "k8s-jkns-pr-bldr-e2e-gce-fdrtn" --member "serviceAccount:[email protected]" --role roles/editor

Will see if that fixes the problem.
Thanks!

[nikhiljindal]

That didnt fix it.
Still the same error.

[ixdy]

@nikhiljindal link to the log?

[nikhiljindal]

@ixdy https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30458/kubernetes-pull-build-test-federation-e2e-gce/11/

[ixdy]

Argh! https://github.com/kubernetes/kubernetes/blob/master/hack/jenkins/build.sh#L56

I'm pretty sure we don't need this git clean. I'm going to test just removing it.

[ixdy]

Removing git clean appears to have worked: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/32158/kubernetes-pull-build-test-federation-e2e-gce/16/

Next issue is a lack of resources. :)

[nikhiljindal]

The image push issue is fixed now.
Thanks @ixdy !

The error now is:

ERROR: (gcloud.compute.networks.create) Some requests did not succeed:
 - Quota 'NETWORKS' exceeded. Limit: 5.0

I have submitted a request for increasing the network quota to 15.