link-/actions-at-scale-ghoh

GitHub Office Hours - Adopting GitHub Actions at scale in the Enterprise

📺 Latest Youtube Video
Debugging and troubleshooting complex software failures!	✨ 🧙🏼‍♂️ ✨ ✨ glich.stream ✨

actions-at-scale-ghoh's Issues

auto scaling is not getting triggered

thanks for your work on this topic. YouTube videos were awesome! documentation is neat.
followed the instructions and implemented in a sandbox environment. Looks like it is unable to auto scale runners when the workflow is triggered.

Here is the autoscale_webhook.yaml file:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: azure-runners
  namespace: default
spec:
  replicas: 0
  template:
    spec:
      organization: myOrg
      labels:
        - ephemeral-aks-gharunner-vanilla
      env:
      - name: RUNNER_FEATURE_FLAG_EPHEMERAL
        value: "true"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: azure-runners-deployment-autoscaler
  namespace: default
spec:
  scaleTargetRef:
    name: azure-runners
  minReplicas: 0
  maxReplicas: 5
  scaleUpTriggers:
  - githubEvent: {}
    duration: "5m"

Here is the workflow file:

name: Matrix jobs

# Controls when the workflow will run
on:
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  matrix:
    strategy:
      matrix:
        node: [1, 2, 3, 4, 5, 6, 7]
    # The type of runner that the job will run on
    runs-on: ['ephemeral-aks-gharunner-vanilla']

    steps:

      # Runs a single command using the runners shell
      - name: Run a one-line script
        run: echo ${{ matrix.node }}

      # Runs a set of commands using the runners shell
      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          sleep 20
          echo test, and deploy your project.

If I configure minReplicas:7 (or any number other than 0, but less than or equal to maxReplicas), jobs are triggered in parallel and finishes eventually. However, if I keep minReplicas:0, runners are not created and workflow jobs are in queue indefinitely. Any idea where I am going wrong?

Pre-requisites section in README?

I noticed there's no mention of what kind of access is needed for Azure.

I was attempting to follow from an Enterprise sandbox subscription with Contributor rights. I ran into quite a few issues with role assignments.

Would it be useful to call out a pre-requisites section in the README with the recommended Azure roles?

Maybe it was mentioned in the first video of the series and I missed it.

Failed to get new registration token and no pods, only runners shows up

I'm not getting corresponding pods for azure-runners-* and keep getting failed to create registration token:, not sure why the controller is fetching https://api.github.com/app/installations/95639346/access_tokens" which (95639346) is not my github_app_installation_id, sample error below for your review
ERROR actions-runner-controller.runner Failed to get new registration token {"runner": "example-runnerdeploy-zzwdj-4xgfz", "error": "failed to create registration token: Post "https://api.github.com/orgs/POC-CICD-DSO/actions/runners/registration-token\": could not refresh installation id 95639346's token: received non 2xx response status "404 Not Found" when fetching https://api.github.com/app/installations/95639346/access_tokens"}
DEBUG events Warning {"object": {"kind":"Runner","namespace":"default","name":"example-runnerdeploy-lzm59-qlw6b","uid":"000f526c-af02-4a88-81d8-3985900bc5fb","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"62074"}, "reason": "FailedUpdateRegistrationToken", "message": "Updating registration token failed"},
My appid is not on

Updating the DNS Label Manually

In the instructions, there's a set of comments like the following.

# !!! IMPORTANT !!!
#
# Add a DNS alias for the public ip manually before proceeding
#
# !!! IMPORTANT !!!

From the Azure CLI, when creating a Public IP, you can specify the --dns-label so it probably doesn't have to be done manually.

# First create a public IP resource
az network public-ip create \
  --resource-group GitHubActionsRunners \
  --name APGWPublicIp \
  --allocation-method Static \
  --sku Standard \
  --dns-label mydnslabel

Happy to open a PR if it helps.

GitHub Packages Registry?

Thanks for the work you've put into this.

I'm curious, any particular reason why using Azure Container Registry over GitHub Packages?