jetbrains/teamcity-google-agent

TeamCity support for Google cloud build agents

TeamCity Google Cloud Agents

TeamCity integration with Google Compute Engine which allows using cloud instances to scale the pool of build agents.

Compatibility

The plugin is compatible with TeamCity 10.0.x and greater.

Installation

You can download the plugin and install it as an additional TeamCity plugin.

Configuration

For configuration details please take a look at the TeamCity Google Cloud plugins blog post.

The plugin supports Google Compute images to start new instances. You also need to create a new JSON private key and assign the Compute Engine Instance Admin (v1) and Project Viewer roles or create your own with a following permissions:

• compute.images.list
• compute.instances.create
• compute.instances.list
• compute.instances.setMetadata
• compute.machineTypes.list
• compute.diskTypes.list
• compute.networks.list
• compute.subnetworks.list
• compute.zones.list

Note: If you're using "Instance template" image type also assign a Service Account User role.

To verify whether your service account has all required permissions please enable Google Cloud Resource Manager API in your project.

Shared VPC permissions configuration

There is 3 ways to configure permissions for shared VPC:

1. In order to see networks and subnetworks in dropdowns in the "Add Image" dialog, you have to add the Compute Network User role to the ServiceAccount under which VMs are created from TeamCity in the Host GCP project with Shared VPC.

2. You can select a limited number of subnetworks that your ServiceAccount should have access to. In Google Cloud Console go to the Host Project -> Shared VPC tab, select the subnet you want to grant access to and add the Compute Network User role to the account. In this case, you won't see networks and subnetworks (except for those that don't require any permissions), but you can specify subnet manually. The subnet must be specified in the following format: projects/[project_id]/regions/[region]/subnetworks/[subnet_id]

   

   

3. In the case when it is necessary to show some specific subnets, first you need to do p.2, and then add the Compute Network Viewer role to the ServiceAccount in IAM to the Host Project with Shared VPC.

Image Creation

Before you can start using the integration, you need to create a new cloud image. To do that, create a new cloud instance, install the TeamCity Build Agent on it and set it to start automatically. You also need to manually point the agent to the existing TeamCity server with the Google Cloud plugin installed to let the build agent download the plugins.

Then you need to remove temporary files and create a new image from the instance disk.

Startup and shutdown scripts

To specify instance metadata you could the "Custom metadata" property in cloud image settings. It could be useful while defining startup and shutdown scripts.

Preemptible instance

If you are using preemptible instances you have to specify shutdown script to gracefully reschedule build from preempted VM on another build agent like that.

For Linix instances:

{
  "shutdown-script": "#! /bin/bash\n/opt/buildagent/bin/agent.sh stop force"
}

For Windows instances:

{
  "windows-shutdown-script-cmd": "C:\\BuildAgent\\bin\\agent.bat stop force"
}

License

Apache 2.0

Feedback

Please feel free to post feedback in the repository issues.

teamcity-google-agent's Issues

Unable to create cloud profile - Invalid disk type URL

Hi,

I am unable to add new cloud images, or add new cloud profiles for Preemptible instances.
On the cloud profile page, I see the error https://www.googleapis.com/compute/v1/projects/*****/regions/us-central1/diskTypes/local-ssd is not a valid disk type URL on the page (screenshot below), and in the logs (teamcity-clouds.log), I see the below stacktrace.

Version info
Teamcity - 2017.2.3
Google compute plugin - 0.5.4

This has just popped up recently, I have been able to setup new images and cloud profiles successfully in the past.

Any Idea how to fix this issue or debug this further is appreciated

[2018-05-25 01:37:25,821]   INFO [io-8111-exec-17] - .google.web.SettingsController - https://www.googleapis.com/compute/v1/projects/*****/regions/us-central1/diskTypes/local-ssd is not a valid disk type URL 
java.lang.IllegalArgumentException: https://www.googleapis.com/compute/v1/projects/*****/regions/us-central1/diskTypes/local-ssd is not a valid disk type URL
        at com.google.cloud.compute.DiskTypeId.fromUrl(DiskTypeId.java:149)
        at com.google.cloud.compute.DiskType.fromPb(DiskType.java:234)
        at com.google.cloud.compute.ComputeImpl$5.apply(ComputeImpl.java:542)
        at com.google.cloud.compute.ComputeImpl$5.apply(ComputeImpl.java:539)
        at com.google.common.collect.Iterators$7.transform(Iterators.java:750)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
        at com.google.cloud.PageImpl$PageIterator.computeNext(PageImpl.java:72)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
        at jetbrains.buildServer.clouds.google.connector.GoogleApiConnectorImpl$getDiskTypesAsync$1.doResume(GoogleApiConnectorImpl.kt:222)
        at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resume(CoroutineImpl.kt:54)
        at kotlinx.coroutines.experimental.DispatchTask.run(CoroutineDispatcher.kt:123)
        at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

Not possible to create profile because "No networks found in the project"

What were you trying to do?

Set up a Google Cloud Engine Windows agent

What happened? How can we reproduce?

0. Install TeamCity (running via docker) 2017.1.3
1. Install cloud-google 0.5.0
2. Create a cloud profile - give it a name, pick GCE
3. Give it a service account key (where that service account has Compute Instance Admin v1 role, only.

• (I also tried with Compute Instance Admin v1 and Compute Network Viewer (in that order), and in that case, the plugin tells me that it cannot list zones or images or one other operation)

0. Result:

But I have these networks defined:

What did you expect to have happen?

Either of:

• I would be allowed to choose one of the networks within my project
• I am told why the plugin could not find any networks

Support additional gcloud create instance arguments

For example we need to enable the display device on our windows VMs, necessary for automated UI tests. This seems like a common requirement for TeamCity.

See --enable-display-device here:
https://cloud.google.com/sdk/gcloud/reference/compute/instances/create#--enable-display-device

We can't solve this using instance templates; Google do not yet support this configuration in instance templates. No sign of that happening:
https://issuetracker.google.com/u/1/issues/133047176

(There are other options we might want, like local SSD, or E2 machine types - but these can be solved with instance templates although it's not so convenient).

Possible to define instance metadata for startup scripts?

I have a requirement to apply licenses to software within an instance before that can be used. I cannot bake that into the image; it must happen at instance-run-time, not image-build-time.

A really convenient way to do this would be via a google startup script. I need to use both Linux and Windows machines, here.

Is this functionality planned?

Provide hints while image creation on GCE

We could automatically select zone and network for TeamCity running on GCE.

Support preemptible instances

It is the similar to hot spot instances in the Amazon EC2, so we should properly handle cases when instance is going down.

Add support for minimum num of instances as hot standbys

Feature Request

Use case
Add ability to define a minimum number of instances for pools that are busy, that are known to have intermittent quiet periods that exceed the Terminate instance idle time configuration in a Cloud Profile. In our scenario, we're ok with a set number of instances that are kept alive and idle to avoid the time a build needs to wait for at least one agent to spin up and become available.

Functionality
For a first version, it'd be useful to add another configuration to the Instances limit under an image configuration to allow for configuring a minimum number of instances for the image group:

Start an agent on custom network gcp

Hi,
The agent is able to start on gcp default network, but when change the google cloud "Network" to our own internal subnet, in this case called "staging" the agent fails and not even get created on google cloud console.
Here is the error on teamcity:
Error:

com.google.cloud.compute.ComputeException: Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode networkcom.google.cloud.compute.ComputeException: Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network
at com.google.cloud.compute.spi.v1.HttpComputeRpc.translate(HttpComputeRpc.java:96)
at com.google.cloud.compute.spi.v1.HttpComputeRpc.createInstance(HttpComputeRpc.java:889)
at com.google.cloud.compute.ComputeImpl$61.call(ComputeImpl.java:1664)
at com.google.cloud.compute.ComputeImpl$61.call(ComputeImpl.java:1661)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:91)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.compute.ComputeImpl.create(ComputeImpl.java:1661)
at jetbrains.buildServer.clouds.google.connector.GoogleApiConnectorImpl$createVmAsync$1.doResume(GoogleApiConnectorImpl.kt:104)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resume(CoroutineImpl.kt:54)
at kotlinx.coroutines.experimental.DispatchTask.run(CoroutineDispatcher.kt:123)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network",
"reason" : "invalid"
} ],
"message" : "Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.compute.spi.v1.HttpComputeRpc.createInstance(HttpComputeRpc.java:887)
... 14 more

Error starting Gcloud instances

Hello,
I'm running into this issue when I'm using this plugin:

[2019-12-11 11:50:45,351]   WARN [atcher-worker-1] - clouds.google.GoogleCloudImage - Bad Request: com.google.api.gax.rpc.InvalidArgumentException: Bad Request (enable debug to see stacktrace)

here is the full log:

[2019-12-11 11:50:45,209]   INFO [5 Flush Queue 1] - uds.base.AbstractCloudInstance - Changing tc_agent1490(13b56b26) status from Unknown to Scheduled to start 
[2019-12-11 11:50:45,209]   INFO [5 Flush Queue 1] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to start' state, profile 'Gcloud profile'{id=google-2, projectId=project1}, GoogleCloudInstance{myName='tc_agent1490'}
[2019-12-11 11:50:45,209]   INFO [5 Flush Queue 1] - .server.impl.CloudEventsLogger - Cloud instance start succeeded: profile 'Gcloud profile'{id=google-2, projectId=project1}, GoogleCloudInstance{myName='tc_agent1490'}
[2019-12-11 11:50:45,210]   INFO [atcher-worker-1] - clouds.google.GoogleCloudImage - Creating new virtual machine tc_agent1490
[2019-12-11 11:50:45,351]   WARN [atcher-worker-1] - clouds.google.GoogleCloudImage - Bad Request: com.google.api.gax.rpc.InvalidArgumentException: Bad Request (enable debug to see stacktrace)
[2019-12-11 11:50:45,351]   INFO [atcher-worker-1] - uds.base.AbstractCloudInstance - Changing tc_agent1490(13b56b26) status from Scheduled to start to Error 
[2019-12-11 11:50:45,351]   INFO [atcher-worker-1] - clouds.google.GoogleCloudImage - Removing allocated resources for virtual machine tc_agent1490
[2019-12-11 11:50:45,446]   WARN [atcher-worker-2] - clouds.google.GoogleCloudImage - Failed to delete allocated resources for virtual machine tc_agent1490: Bad Request: com.google.api.gax.rpc.InvalidArgumentException: Bad Request (enable debug to see stacktrace)

as a result my gcloud instances get fired up and immediately shut down. Any idea how I can fix this? Thanks

400 error when starting an instance in a custom subnet: URL is malformed

We are trying to create instances in a custom subnetwork and are getting a 400 error from GCP (Stacktrace below).

According to https://cloud.google.com/compute/docs/reference/rest/v1/instances/updateNetworkInterface the subnetwork value provided should be an URL and from the the stacktrace it looks like the plugin is using the subnetwork name property

at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:90)
at com.google.api.gax.httpjson.HttpJsonExceptionCallable$ExceptionTransformingFuture.onFailure(HttpJsonExceptionCallable.java:106)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52)
at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "Invalid value for field 'resource.networkInterfaces[0].subnetwork': 'tools-network'. The URL is malformed."
}
],
"code": 400,
"message": "Invalid value for field 'resource.networkInterfaces[0].subnetwork': 'tools-network'. The URL is malformed."
}
}

at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1072)
at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:130)
... 7 more```

Properly handle change of zone in the image settings

Currently after zone changes integration fails to control already started instances or instances with a different zone settings.

Add ability to define "active hours" for image groups

Feature Request

Relates to #30

Use Case
Based on the same busy pool scenario, it'd be great to be able to shutdown all agents outside "active" or "busy" hours, which usually aligns with business hours. This would be a good money saver for those busy agent pools.

Functionality
A date picker for the Instances limit configuration on an image would describe when are the active/busy hours for the corresponding agent pool.

Update to Google Compute SDK v1

Current Compute SDK is deprecated, so we need to switch to new v1 SDK.

Missing Metadata with TeamCity 2020.1.2

When running Google Cloud Agent 0.7.3, we are getting the following output in the build. It looks like :/META-INF/build-agent-per-build-plugin-*.xml is missing.

[15:10:15]E: Snapshot dependency "... GetMetadata" failed
[15:10:15] : Starting the build on the agent "gce-1"
[15:10:16]i: Agent time zone: Etc/UTC
[15:10:16] : Failed to start build id=85175 on agent. java.lang.RuntimeException: Failed to create context for build: Could not resolve bean definition resource pattern [classpath*:/META-INF/build-agent-per-build-plugin-*.xml]; nested exception is java.io.FileNotFoundException: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar.dpkg-new (No such file or directory)
[15:10:16]W: java.lang.RuntimeException: Failed to create context for build: Could not resolve bean definition resource pattern [classpath*:/META-INF/build-agent-per-build-plugin-*.xml]; nested exception is java.io.FileNotFoundException: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar.dpkg-new (No such file or directory)
	at jetbrains.buildServer.agent.impl.runContext.RunningBuildContextFactory.createContext(RunningBuildContextFactory.java:52)
	at jetbrains.buildServer.agent.impl.runContext.RunningBuildContextFactory.createContext(RunningBuildContextFactory.java:40)
	at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:304)
	at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:56)
	at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:276)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Could not resolve bean definition resource pattern [classpath*:/META-INF/build-agent-per-build-plugin-*.xml]; nested exception is java.io.FileNotFoundException: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar.dpkg-new (No such file or directory)
	at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:229)
	at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
	at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
	at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
	at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
	at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
	at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:614)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:514)
	at jetbrains.buildServer.spring.SpringSubContainerFactoryImpl.createSubContext(SpringSubContainerFactoryImpl.java:46)
	at jetbrains.buildServer.agent.impl.runContext.RunningBuildContextFactory.createContext(RunningBuildContextFactory.java:48)
	... 5 more
Caused by: java.io.FileNotFoundException: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar.dpkg-new (No such file or directory)
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindPathMatchingJarResources(PathMatchingResourcePatternResolver.java:602)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(PathMatchingResourcePatternResolver.java:504)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(PathMatchingResourcePatternResolver.java:280)
	at org.springframework.context.support.AbstractApplicationContext.getResources(AbstractApplicationContext.java:1297)
	at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:216)
	... 14 more

No networks found in the project when adding image

Hey. I can't add an image because of "No networks found in the project" error.

I have two networks in my GCP account (including default one).
TC server is hosted on GCE instance.

There is also one suspicious thing in teamcity-clouds.log

[2017-07-07 20:57:22,310]   WARN [nio-8111-exec-4] - .beans.CloudTabFormFactoryImpl - Can't find projectId by projectExtId '' 
[2017-07-07 22:34:40,464]   WARN [io-8111-exec-11] - .beans.CloudTabFormFactoryImpl - Can't find projectId by projectExtId '' 

Could you suggest a workaround?
Thanks.

Add support for using images from a different GCP project

We have an existing CI image build pipeline whereby GCP images are built in one project, but are launched in a different project.

Currently, this plugin doesn't support this workflow.

unauthorized on registration

Hello,
I'm getting unauthorized on registration when trying to start new agents. It seems like CloudInstanceUserData.myAuthToken is always empty ( https://github.com/JetBrains/teamcity-google-agent/blob/master/google-cloud-server/src/main/kotlin/jetbrains/buildServer/clouds/google/GoogleCloudImage.kt#L90 ) I'v tried using both, SNAPSHOT-20200127160240 and 0.7.5 from https://plugins.jetbrains.com/plugin/9704-google-cloud-agents/versions and a locally built version.

using Teamcity 2018.1.2 (build 58537)

any ideas, Im guessing I've configured it wrong?

Service account setting ignored in agent image definition when using auth From machine environment

We are currently investigating Google Cloud authentication with service accounts, but without service JSON credentials (essentially Application Default Credentials where applicable).

In our Cloud Profile definitions we use the "From machine environment" feature and have it set to service account A. In the individual cloud agent image definitions we would like to use service account B.

While running a TeamCity build we see service account B is being attached to a Compute Engine VM, but TeamCity always authenticates with service account A in the build itself.

Is there a way to force the use of a per-image service account B?

Request: new plugin release for additional fixes

I would like to request for a new release for this plugin in order to use some of the fixes that were addressed after 0.7.4, namely 633e61e and https://youtrack.jetbrains.com/issue/TW-61898

Programatically rotate the Google SA key in Cloud Profile

Our company policy restricts us to issue long-lived Google Service Account Keys.

Is there a way to programatically inject the GCP SA-key into the Team City config that we can rotate it as needed?

Also, any chance for OAuth token support?

Agent numbering reuse causes confusing build history

Unlike the Azure plugin, where agent names are never reused, the GCE plugin seems to reuse names when possible. For example, if an idle agent, say "gce-agent-7", is terminated, that name becomes available for reuse when new agents are started.

Because TeamCity seems to use the agent name as its identity, the "Build History" tab will show builds from every "gce-agent-7" ever created. This is quite confusing.

Is this behavior intentional or is it a bug?

Unable to edit/delete agent images after deleting the associated cloud image

To reproduce:

• Add an agent image associated with an existing Cloud Image
• From Google Console, delete the Cloud Image used in the step before
• Try to edit the agent

Seems like this scenario is not taken into account in some js logic:
Uncaught TypeError: self.sourceImages(...).every is not a function

Error loading plugin '0.8.0'

Hey @dimyriy

So I just tried to load the 0.8.0 plugin release in a local TC 2020.2 docker image, and it fails to start.

The error stacktrace is:

[2020-12-03 14:27:00,775]   WARN [{id=-42}; http-nio-8111-exec-3] -   jetbrains.buildServer.SERVER - Error during loading plugin
jetbrains.buildServer.plugins.exceptions.LoadPluginException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jetbrains.buildServer.clouds.google.GoogleCloudClientFactory#0' defined in Byte array resource [plugin: cloud-google#google-cloud-server.jar!/META-INF/build-server-plugin-teamcity-cloud-google.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [jetbrains.buildServer.clouds.google.GoogleCloudClientFactory]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
        at jetbrains.buildServer.plugins.PluginManagerImpl.loadPlugin(PluginManagerImpl.java:124)
        at jetbrains.buildServer.web.plugins.web.ModifiedPluginsImpl.setEnabledAndLoad(ModifiedPluginsImpl.java:77)
        at jetbrains.buildServer.web.plugins.web.PluginsActionsController$LoadAllPlugins.process(PluginsActionsController.java:2)
        at jetbrains.buildServer.controllers.BaseActionController.doAction(BaseActionController.java:59)
        at jetbrains.buildServer.controllers.BaseAjaxActionController$1.handleRequest(BaseAjaxActionController.java:47)
        at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:48)
        at jetbrains.buildServer.controllers.BaseAjaxActionController.doHandle(BaseAjaxActionController.java:44)
        at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:114)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
        at jetbrains.buildServer.controllers.BaseController.handleRequest(BaseController.java:93)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
        at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:27)
        at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:42)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:5)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:1)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at jetbrains.buildServer.web.UserIdProviderFilter.doFilter(UserIdProviderFilter.java:6)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at jetbrains.buildServer.web.NodeInfoHeaderFilter.doFilter(NodeInfoHeaderFilter.java:6)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
        at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:4)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:8)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at jetbrains.buildServer.diagnostic.web.HttpRequestsDurationMetricsReporter.doFilter(HttpRequestsDurationMetricsReporter.java:8)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at jetbrains.buildServer.web.HttpSecurityHeadersFilter.doFilter(HttpSecurityHeadersFilter.java:61)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at jetbrains.buildServer.controllers.filters.ClearSecurityContextFilter.doFilter(ClearSecurityContextFilter.java:8)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
        at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:61)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:41)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jetbrains.buildServer.clouds.google.GoogleCloudClientFactory#0' defined in Byte array resource [plugin: cloud-google#google-cloud-server.jar!/META-INF/build-server-plugin-teamcity-cloud-google.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [jetbrains.buildServer.clouds.google.GoogleCloudClientFactory]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:279)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1197)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
        at jetbrains.buildServer.plugins.spring.SpringPluginLoader.pluginClassesLoaded(SpringPluginLoader.java:118)
        at jdk.internal.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:138)
        at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
        at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:132)
        at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:82)
        at com.sun.proxy.$Proxy27.pluginClassesLoaded(Unknown Source)
        at jetbrains.buildServer.plugins.PluginManagerImpl$5.visitPlugin(PluginManagerImpl.java:422)
        at jetbrains.buildServer.plugins.PluginsCollection$1.run(PluginsCollection.java:98)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        ... 1 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [jetbrains.buildServer.clouds.google.GoogleCloudClientFactory]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:154)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
        at jetbrains.buildServer.spring.InstantiationStrategySelector$1.instantiate(InstantiationStrategySelector.java:80)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
        ... 27 more
Caused by: java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
        at jetbrains.buildServer.clouds.google.GoogleCloudClientFactory.<init>(GoogleCloudClientFactory.kt)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
        ... 30 more
Caused by: java.lang.ClassNotFoundException: Class 'kotlin.jvm.internal.Intrinsics' was not found
        at jetbrains.buildServer.plugins.classLoaders.TeamCityClassLoader.doLoadClass(TeamCityClassLoader.java:85)
        at jetbrains.buildServer.plugins.classLoaders.TeamCityClassLoader.loadClass(TeamCityClassLoader.java:40)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 36 more

Any ideas?

preemptible instances are leaked when preempted

Preemptible instances that are preempted by GCE are "leaked" in that they are stuck forever in the "Stopped" state as show below:

Preemption happens pretty frequently with some of the larger instance types, so would be great to have them automatically cleaned up. Even though the instance isn't billed when in the "Stopped" state, the persistent disk is.

In case someone else has the same issue, as a workaround for now, we're running the following script every hour:

gcloud compute instances list --format json \
  | ./jq -r '.[] | select (.scheduling.preemptible == true and .status == "TERMINATED") | .name  + " --zone " + .zone' \
  | xargs -tL1 gcloud compute instances delete --delete-disks all

Option to not delete stopped agents

Setting the idle time after which instances are stopped seems to result in them getting not only stopped, but also deleted. This is unfortunate when you'd like to cache things on the instance's disk, such as large git repositories or intermediate build artifacts.

It would be great to have an option to avoid this deletion, and keep the instances alive but stopped.

Would removing the following code be a safe way of achieving this, preventing the plugin from ever terminating instances? Or is it unable to e.g. restart the stopped instances?

teamcity-google-agent/google-cloud-server/src/main/kotlin/jetbrains/buildServer/clouds/google/connector/GoogleApiConnectorImpl.kt

Lines 380 to 389 in ac170a0

	if ("TERMINATED" == it.status) {
	GlobalScope.launch(image.coroutineContext) {
	try {
	LOG.info("Removing terminated instance $name")
	deleteVm(GoogleCloudInstance(image, it.name, zone))
	} catch (e: Exception) {
	LOG.infoAndDebugDetails("Failed to remove instance $name", e)
	}
	}
	}

How to use subnetworks that are shared to the project

Getting below error when I choose - Json private key as credential type.

Project doesn't have any subnets, but we can use shared network. But I am unable to view any subnet that are shared.

No networks found in us-central1-a zone.

Also please let us know which default credentials will be used when we choose - "From Machine environment" and where can we configure these

Add option to choose N1 vs N2 for custom machine types

N2 custom machine types allow different configurations vs N1. For some workloads, being able to choose N2 vs N1 for custom machine types will allow for better flexibility and cost savings.

https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type#n2_custom_machine_types

UI improvements

• Rename Name prefix into Agent name prefix
• Define that agent name prefix should be unique in profile
• Preemptible move under the Machine type
• Add image should not work when credentials is invalid

Add validation of "Service Account User" role for Instance template users

Note: https://youtrack.jetbrains.com/issue/TW-57520#focus=streamItem-27-3132455-0-0

Better explanation for scopes

It took a lot of head-scratching to realise why the scope "cloud-platform" didn't work. which works when using gcloud command.

Instead here you need to use the URL: "https://www.googleapis.com/auth/cloud-platform" for example. Some popup help would be useful here.

Custom CPU and memory instances

Any chance you could add support for custom machine types? None of the stock machine sizes are quite right for our workload, so it would be great to be able to specify a custom number of CPUs/amount of memory for the created agents.

Thanks for an awesome plugin!

Support the disk type configurable

Looks like at the moment all VM instances, that are created from custom images, are created with the standard persistent disk as a boot disk by default.

Is it possible to implement possibility of configuring the disk type [pd-ssd/pd-standard]?

Our application is very demanding for build performance and using SSD disks for VM instances will definitely increase the performance.

Documentation incomplete or out-of-date

I noticed that the configuration of Cloud Profiles is far from straightforward and some feature labels a little vague. The blog post hasn't been updated since 2017 and does not showcase the full setup (setting up credentials in the Cloud Profile, creating an image definition, etc.).

Would you have some time to make the documentation more complete?

Support instance templates

By providing support of instance templates we could provide solution for fully customizable build agent machines.

Service accounts for instances

By @mayacdoit in #12 (comment):

We need to work with gsutil. From the main image there is no issue to run gsutil. From the agents started with teamcity I get: "ServiceException: 401 Anonymous users does not have storage.objects.list access to bucket mybucket"
When launching an image manually gsutil works fine. Digging a bit on the internet didn't help finding a solution.

Can't configure profile

In logs:

java.lang.NullPointerException
at com.google.cloud.compute.StorageImageConfiguration$Builder.(StorageImageConfiguration.java:69)
at com.google.cloud.compute.StorageImageConfiguration$Builder.(StorageImageConfiguration.java:49)
at com.google.cloud.compute.StorageImageConfiguration.fromPb(StorageImageConfiguration.java:202)
at com.google.cloud.compute.ImageConfiguration.fromPb(ImageConfiguration.java:188)
at com.google.cloud.compute.ImageInfo$BuilderImpl.(ImageInfo.java:172)
at com.google.cloud.compute.Image.fromPb(Image.java:212)
at com.google.cloud.compute.ComputeImpl$40.apply(ComputeImpl.java:1259)
at com.google.cloud.compute.ComputeImpl$40.apply(ComputeImpl.java:1256)

I'm also unable to switch between "use machine config" and "json private key" for an already configured profile.

Preempted agents permacancel builds

When an agent is preempted, any build that it was running is permanently canceled:

I've noticed that TeamCity can automatically retry a build when someone runs agent.sh stop force on the agent, so I tried specifying the following shutdown script in the instance metadata:

{
  "shutdown-script": "sudo -u agent /home/agent/bin/agent.sh stop force"
}

Unfortunately, this doesn't actually work. I'm not sure whether it's GCE failing to run the startup script, or the cloud plugin noticing that the instance has been shutdown and somehow marking the build as permanently canceled. Either way, it would be really nice if TC could automatically restart a build that failed because the agent was preempted/removed.

Perhaps there's some way to do this already that I'm not aware of?

Add ability to use environment authentication

We could reuse existing authentication when starting in Compute/App Engine.

Automatic Authorization

It's not clear to me what needs to be done to ensure that the agent images that have been made will automatically authorize with the teamcity server.

Can you please either point me to the docs that detail what causes automatic authorization, or help me to update the documentation?

NVMe scratch disks

I don't see a way with this plugin that I can add an extra disk to the agents. It would be nice to be able to have an NVMe scratch disk added, which could be mounted via a startup script into places requiring high IOPS. NVMe scratch is significantly faster than even the SSD drives.

SRE-2823 Teamcity GCP plugin requires org level permission to spawn agents in Shared VPC

Add support for Image Families

GCP has the concept of Image Families, which is effectively a pointer to the 'latest' version of a given image.

This is useful in CI environments where the underlying cloud image is being rebuilt on a frequent or even semi-frequent basis.

It would be extremely useful for this plugin to support launching a build agent using an instance family.

Ref's:

• https://cloud.google.com/compute/docs/images#image_families
• https://cloud.google.com/compute/docs/images/image-families-best-practices

Add ability to start agent from docker image

Google has announced that now it's possible to start docker images on GCE instances:
https://cloudplatform.googleblog.com/2017/11/introducing-an-easy-way-to-deploy-containers-on-Google-Compute-Engine-virtual-machines.html