akkornel/gcs_gcp

Infrastructure as Code to run Globus Connect Server in Google Cloud

gcs_gcp's Issues

Send Cloud Build notifications to Slack

When Cloud Build does anything, we should send those notifications to Slack. That is important to…

• Know that Cloud Build jobs are continuing to run as normal.
• Be alerted to build problems that prevent the creation of updated images.

Terraform code for Core & Packer

Terraform is going to be used to manage all of the GCP infrastructure. First up is the core code (defining the provider, securing Default Service Accounts, etc.), followed by the code needed to create Packer images.

Switch Packer to a NAT?

Right now the Packer temporary instances run in a VPC that allows outbound traffic, and with instances that have an ephemeral public IP address.

We need the outbound connectivity, because the temporary instance is downloading software & updating packages. It's a core need that Packer be able to connect out to the outside world.

But, the traffic is all outbound, not inbound. So we could use a NAT. Instead of using a public IP, all outbound traffic would go through the NAT, making things a little more secure: It would require the use of IAP for SSH, it would insulate us from messed-up firewall rules, and it would mean any bad code that gets onto the instance wouldn't be able to allow inbound connections.

On the other hand, there is a cost to running everything through a NAT, both a per-instance-per-minute cost and a per-GB cost. And Packer downloads a lot of data each time it runs, so that would be a notable cost.

So, should we switch Packer to use a NAT?

Implement Packer in Cloud Build

We will be using Packer to build images for GCS. We should use Cloud Build to manage those build jobs.

Manage the cloudbuild bucket

The first time someone (with appropriate permissions) does a manual run of Cloud Build from the gcloud CLI, the CLI creates a new bucket. This happens because CLI-based builds work by uploading a .tgz file with the source.

The bucket name is [PROJECT_ID]_cloudbuild, the prefix for uploads is source, and files are named with something that looks like a timestamp plus a hash, with a .tgz extension. It doesn't look like there are any special permissions on the bucket; Cloud Build has access to the bucket through the "Cloud Build Service Account" role.

This bucket should be managed in Terraform. That'll let us do a few things:

• Auto-delete files after 18 months (365.25 * 1.5 days, which we'll round to 548 days).

• Move to Coldline after 32 days.

• Set write permissions to store some build artifacts (like package lists).

And it also lets me keep track of the bucket, in case we want to do anything else.

Manage the Terraform state bucket

At the very beginning of deployment, a Cloud Storage bucket is created to hold Terraform state.

It would be nice to do a certain amount of work on this bucket. For example, ensuring that versioning is enabled, and deleting old versions after some time has passed.

That could all be specified in the documentations, but it would be better if we could do it automatically.

Lifecycle compute images

We need a way to automatically mark old images as deprecated or obsolete, and to eventually delete them. The packer Cloud Build job only creates new ones, it doesn't do anything to images that already exist.

Compute Engine is smart enough to choose the newest image out of a family, so this is really just about minimizing costs.

How to SSH from the management node to the DTN

Found this while working on Terraform code for the GCS nodes.

Logged in to gcs-management-1 with IAP with OSLogin. Trying to SSH to gcs-dtn-2.

akkornel_stanford_edu@gcs-management-1:~$ gcloud compute ssh gcs-dtn-2 --internal-ip --zone us-west1-a
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

It's trying to log in to the DTN using the Compute Engine service account on the management node. That's a weird exit code for SSH, though…

Implement Packer container for Cloud Build

In order to use Cloud Build for Packer, we need to have Packer in a container image. Google has this documented, but they don't provide a ready-made container image. So, we need to make one ourselves.

This blocks #2.

Terraform code for GCS core

The core part of GCS needs to be defined in Terraform. This includes a new VPC and subnet, plus separate firewall rules for HTTPS+GridFTP and SSH (applied based on instance network tags). We also need some service accounts and permissions, but that may be better to cover in the issues for GCS management and GCS DTN Terraform.

Terraform code for GCS management node

There are several operations that are better done on some sort of management node.

The management node should use the Globus image, but be on a low-end instance type. It should be tagged for management (not gcs), so that it allows in SSH instead of HTTPS and GridFTP. It should use OSLogin for authentication.

We also need a Service Account for instances to run. In addition to whatever basic functions the instance needs, it also needs write access to appropriate secrets.

The Terraform code should define an instance template only, and not actually start any instances. This depends on #6 for the core configuration.