mansagroup/gcs-cache-action

A GitHub Action to cache your workload to a Google Cloud Storage bucket.

Google Cloud Storage Cache Action

GitHub already provides an awesome action to cache your workload to Azure's servers hosted in United States. However, if you are using self-hosted runners hosted far away from the cache location, or if you pay external network way more than internal network, you may want to host your cache elsewhere for better performance and lower costs.

This action does have the same set of inputs as the @actions/cache action from GitHub, in addition to a new bucket input which should contain your target Google Cloud Storage bucket. As simple as that.

Usage

workflow.yml

- name: Authenticate to Google Cloud
  uses: google-github-actions/auth@v0
  with:
    workload_identity_provider: projects/your-project-id/locations/global/workloadIdentityPools/your-identity-pool/providers/your-provider
    service_account: [email protected]

- name: Cache the node_modules
  id: node-modules-cache
  uses: mansagroup/gcs-cache-action@v2
  with:
    bucket: my-ci-cache
    path: node_modules
    key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
    restore-keys: |
      node-modules-${{ runner.os }}-

- name: Install dependencies
  if: steps.node-modules-cache.outputs.cache-hit == 'false'
  run: npm ci

Inputs

This GitHub action can take several inputs to configure its behaviors:

Name	Type	Default	Example	Description
bucket	String	ø	my-ci-cache	The name of the Google Cloud Storage bucket to use
path	String[]	ø	node_modules	One or more path to store
key	String	ø	node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}	Key to use as cache name
restore-keys	String[]	ø	node-modules-${{ runner.os }}-	Alternative keys to use when looking for the best cache available

Note: the path and restore-keys inputs can contains multiple value separated by a new line.

Outputs

This GitHub action will output the following values:

Name	Type	Description
cache-hit	String	A boolean string representing if the cache was successfully restored

Examples

With multiple paths

workflow.yml

- name: Authenticate to Google Cloud
  uses: google-github-actions/auth@v0
  with:
    workload_identity_provider: projects/your-project-id/locations/global/workloadIdentityPools/your-identity-pool/providers/your-provider
    service_account: [email protected]

- name: Cache the node_modules and npm cache
  id: node-modules-cache
  uses: mansagroup/gcs-cache-action@v2
  with:
    bucket: my-ci-cache
    path: |
      node_modules
      ~/.npm
    key: npm-and-node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
    restore-keys: |
      npm-and-node-modules-${{ runner.os }}-

Compression algorithm

When compressing or decompressing the cache archive, the action will lookup for the best compression algorithm to use. If zstd is available, it will be used instead of gzip by default. The compression method will be added to the object's metadata on the Bucket. Thanks to this, when decompressing, the correct algorithm will be used.

Installing zstd on Ubuntu is simple as doing a apt install zstd.

Note that if a cache archive was compressed using one algorithm, this same algorithm should be installed to decompress it after.

Terraform

Here is a little snippet allowing you to create your cache bucket with Terraform (which you should probably use):

resource "google_storage_bucket" "ci_cache" {
  name                        = "your-ci-cache"
  location                    = "your-location" # "EUROPE-WEST1"
  uniform_bucket_level_access = true

  lifecycle_rule {
    action {
      type = "Delete"
    }

    condition {
      age = 7
    }
  }
}

resource "google_storage_bucket_iam_member" "ci_cache_write_github_ci" {
  bucket = google_storage_bucket.ci_cache.name
  role   = "roles/storage.objectCreator"
  member = "serviceAccount:[email protected]"
}

resource "google_storage_bucket_iam_member" "ci_cache_read_github_ci" {
  bucket = google_storage_bucket.ci_cache.name
  role   = "roles/storage.objectViewer"
  member = "serviceAccount:[email protected]"
}

resource "google_storage_bucket_iam_member" "ci_cache_legacy_write_github_ci" {
  bucket = google_storage_bucket.ci_cache.name
  role   = "roles/storage.legacyBucketWriter"
  member = "serviceAccount:[email protected]"
}

Q&A

Could I use this action on multiple repositories with the same bucket?

Yes you can. When storing to the bucket, this action will use the following the following path:

[repository owner]/[repository name]/[cache key].tar.gz

License

This project is MIT licensed.

Contributors

Thanks goes to these wonderful people (emoji key):

Jérémy Levilain 💻 📖 🤔

This project follows the all-contributors specification. Contributions of any kind welcome!

gcs-cache-action's Issues

Zstd v1.5.5 isn't detected correctly

Github runners on Ubuntu 20.04 are prepackaged with zstd v1.5.5 at the time of this writing. The command output of zstd --version has changed and it doesn't quite match the string that this action relies on (see here). As a result, zstd is assumed to be absent and slower gzip is used instead.

Please update the check to detect the presence of zstd.

Zstd fails with Frame requires too much memory for decoding

Bug Report

Current behavior

/usr/bin/tar -x --use-compress-program zstd -d -P -f /tmp/tmp-195888-ch6Z2OTlm0qa -C /home/github/actions-runner/_work/foo/foo
  /*stdin*\ : Decoding error (36) : Frame requires too much memory for decoding 
  /*stdin*\ : Window size larger than maximum : 1073741824 > 1342[17](https://github.com/gajus/foo/runs/5548683535?check_suite_focus=true#step:9:17)728 
  /*stdin*\ : Use --long=30 or --memory=1024MB 
  /usr/bin/tar: Child returned status 1
  /usr/bin/tar: Error is not recoverable: exiting now

Expected behavior

Compress file.

Possible Solution

Use --long=30 or --memory=1024MB

Make post cache step optional

At the moment, if post action fails, it makes the entire workflow fail too.

While I am unsure what is causing these failures, they are happening often enough for this to be a problem.

Add timing information to logs

Feature Request

Is your feature request related to a problem? Please describe.

Current logs do not describe how long individual steps are taking.

Describe the solution you'd like

It would be nice if the logs said how long each step took, e.g.

🔍 Searching the best cache archive available (25 seconds)
🌐 Downloading cache archive from bucket (40 seconds)
🗜️ Extracting cache archive (20 seconds)

Teachability, Documentation, Adoption, Migration Strategy

What is the motivation/use case for changing the behavior?

With this information I could seek for opportunities to optimize the process.

Fails by simply dumping the script

Here is how we have it setup:

- uses: actions/setup-node@v2
  with:
    node-version: '16'
- name: Cache node_modules
  id: node-modules-cache
  uses: mansagroup/[email protected]
  with:
    bucket: github-action-runner-cache
    path: node_modules
    key: node_modules-${{ hashFiles('**/yarn.lock') }}
    restore-keys: node_modules-${{ hashFiles('**/yarn.lock') }}

Here is how it fails:

cache-hit output should match actions/cache output

Bug Report

The gcs-cache-action currently behaves slightly differently than the regular github actions/cache action. According to https://github.com/marketplace/actions/cache#outputs, the cache-hit output only returns true on an exact match hit. I would expect gcs-cache-action to behave the same, but it is currently returning true on exact and partial matches.

Current behavior

gcs-cache-action is returning cache-hit=true on exact and partial matches.

Expected behavior

gcs-cache-action should only return cache-hit=true on exact matches like actions/cache does. Partial matches should return cache-hit=false

Possible Solution

https://github.com/MansaGroup/gcs-cache-action/blob/main/src/main.ts#L169

Should be:

core.setOutput('cache-hit', bestMatchKind=='exact');

instead of

core.setOutput('cache-hit', 'true');

---

Others:

Wrong version in README

Bug Report

Current behavior

In the README example show a v2 version this action, but there is no v2 release

Expected behavior

Should be v2 version or just stick with v1