qetza/replacetokens-action

An action to replace tokens in text files with variables and/or secrets.

ReplaceTokens

This GitHub Action replaces tokens in text files with variables and/or secrets.

What's new

Please refer to the release page for the latest release notes.

Usage

Inputs

- uses: qetza/replacetokens-action@v1
  with:
    # A multiline list of files to replace tokens in.
    # Each line supports:
    #   - multiple glob patterns separated by a semi-colon ';' using fast-glob syntax 
    #     (you must always use forward slash '/' as a directory separator, on win32 will 
    #     automatically replace backslash with forward slash)
    #   - outputing the result in another file adding the output path after an arrow '=>' 
    #     (if the output path is a relative path, it will be relative to the input file)
    #   - wildcard replacement in the output file name using an asterix '*' in the input 
    #     and output file names
    #
    # Example: '**/*.json; !local/ => out/*.json' will match all files ending with '.json' 
    # in all directories and sub directories except in `local` directory and the output 
    # will be in a sub directory `out` relative to the input file keeping the file name.
    #
    # Required.
    sources: ''

    # A JSON serialized object containing the variables values.
    # The object can be:
    #   - an object: properties will be parsed as key/value pairs
    #   - a string starting with '@': value is parsed as multiple glob patterns separated 
    #     by a semi-colon ';' using fast-glob syntax to JSON or YAML files
    #   - a string starting with '$': value is parsed as an environment variable name 
    #     containing JSON encoded key/value pairs
    #   - an array: each item must be an object or a string and will be parsed as 
    #     specified previously
    #  
    # Multiple entries are merge into a single list of key/value pairs and all JSON 
    # supports comments.
    #
    # Example: '[${{ toJSON(vars) }}, ${{ toJSON(secrets) }}]' will pass all defined 
    # variables and secrets.
    #
    # Required.
    variables: ''

    # Add BOM when writing files.
    #
    # Optional. Default: false
    add-bom: ''

    # Enable case-insensitive file path matching in glob patterns (sources and variables).
    #
    # Optional. Default: false
    case-insensitive-paths: ''

    # The characters to escape when using 'custom' escape.
    #
    # Optional.
    chars-to-escape: ''

    # Encoding to read and write all files.
    #
    # Accepted values:
    #   - auto: detect encoding using js-chardet
    #   - any value supported by iconv-lite
    #
    # Optional. Default: auto
    encoding: ''

    # Character escape type to apply on each value.
    #
    # Accepted values:
    #  - auto: automatically apply JSON or XML escape based on file extension
    #  - off: don't escape values
    #  - json: JSON escape
    #  - xml: XML escape
    #  - custom: apply custom escape using escape-char and chars-to-escape
    #
    # Optional. Default: auto
    escape: ''

    # The escape character to use when using custom escape.
    #
    # Optional.
    escape-char: ''

    # The behavior if no files are found.
    #
    # Accepted values:
    #   - ignore: do not output any message, the action do not fail
    #   - warn: output a warning but do not fail the action
    #   - error: fail the action with an error message
    #
    # Optional. Default: ignore
    if-no-files-found: ''

    # Include directories and files starting with a dot '.' in glob matching results for sources 
    # and additionalVariables.
    #
    # Optional. Default: false
    include-dot-paths: ''

    # The log level.
    #
    # Accepted values:
    #   - debug
    #   - info
    #   - warn
    #   - error
    #
    # Debug messages will always be sent to the internal debug system.
    # Error messages will always fail the action.
    #
    # Optional. Default: info
    log-level: ''

    # The behavior if variable is not found.
    #
    # Accepted values:
    #   - none: replace the token with an empty string and log a message
    #   - keep: leave the token and log a message
    #   - replace: replace with the value from missing-var-default and do not 
    #     log a message
    #
    # Optional. Default: none
    missing-var-action: ''

    # The default value to use when a key is not found.
    #
    # Optional. Default: empty string
    missing-var-default: ''

    # The level to log key not found messages.
    #
    # Accepted values:
    #   - off
    #   - warn
    #   - error
    #
    # Optional. Default: warn
    missing-var-log: ''

    # Opt out of the anonymous telemetry feature.
    # You can also set the 'REPLACETOKENS_TELEMETRY_OPTOUT' environment variable to '1' 
    # or 'true'.
    #
    # Optional. Default: false
    no-telemetry: ''

    # Enable token replacements in values recusively.
    #
    # Example: '#{message}#' with variables '{"message":"hello #{name}#!","name":"world"}' 
    # will result in 'hello world!'
    #
    # Optional. Default: false
    recursive: ''

    # The root path to use when reading files with a relative path.
    #
    # Optional. Default: ${{ github.workspace }}
    root: ''

    # The separtor to use when flattening keys in variables.
    #
    # Example: '{ "key": { "array": ["a1", "a2"], "sub": "s1" } }' will be flatten as 
    # '{ "key.array.0": "a1", "key.array.1": "a2", "key.sub": "s1" }'
    #
    # Optional. Default: .
    separator: ''

    # The token pattern to use.
    #
    # Accepted values:
    #   - default: #{ ... }#
    #   - azurepipelines: $( ... )
    #   - custom: token-prefix ... token-suffix
    #   - doublebraces: {{ ... }}
    #   - doubleunderscores: __ ... __
    #   - githubactions: #{{ ... }}
    #   - octopus: #{ ... }
    #
    # Optional. Default: default
    token-pattern: ''

    # The token prefix when using 'custom' token pattern.
    #
    # Optional.
    token-prefix: ''

    # The token suffix when using 'custom' token pattern.
    #
    # Optional.
    token-suffix: ''

    # Enable transforms on values.
    # The syntax to apply transform on a value is '#{<transform>(<name>[,<parameters>])}#'.
    #
    # Supported transforms:
    #   - base64(name): base64 encode the value
    #   - indent(name[, size, firstline]): indent lines in the value where size is the 
    #     indent size (default is '2') and firstline specifies if the first line must be 
    #     indented also (default is 'false')
    #   - lower(name): lowercase the value
    #   - raw(name): raw value (disable escaping)
    #   - upper(name): uppercase the value
    #
    # Example: 'key=#{upper(KEY1)}#' with '{ "KEY1": "value1" }' will result in 
    # 'key=VALUE1'
    #
    # Optional. Default: false
    transforms: ''

    # The tranforms prefix when using transforms.
    #
    # Optional. Default: (
    transforms-prefix: ''

    # The tranforms prefix when using transforms.
    #
    # Optional. Default: )
    transforms-suffix: ''

Outputs

Name	Description	Example
defaults	The number of tokens replaced with the default value if one was specified.	1
files	The number of source files parsed.	2
replaced	The number of values replaced by a value different than the default value.	7
tokens	The number of tokens found in all files.	8
transforms	The number of transforms applied.	2

Examples

Multiple sources

- uses: qetza/replacetokens-action@v1
  with:
    sources: |
      **/*.json;!**/*.dev.json;!**/vars.json => _tmp/*.json
      **/*.yml
    variables: '[${{ toJSON(vars) }},${{ toJSON(secrets) }}]' # use variables & secrets

Multiple variables

- uses: qetza/replacetokens-action@v1
  with:
    sources: '**/*.yml'
    variables: >
      [
        ${{ toJSON(vars) }},                                           # variables
        ${{ toJSON(secrets) }},                                        # secrets
        ${{ toJSON(format('@{0}/settings.json', github.workspace)) }}, # read from file
        "@**/vars.(json|jsonc);!**/local/*"                            # read from JSON files
        "@**/settings.(yml|yaml);!**/local/*"                          # read from YAML files
        "$ENV_VARS",                                                   # read from env
        { "VAR2": "${{ github.event.inputs.var2 }}" }                  # inline values
      ]
  env:
    ENV_VARS: '{ "VAR4": "env_value4" }'

Access outputs

steps:
- uses: qetza/replacetokens-action@v1
  id: replace-tokens
  with:
    sources: '**/*.yml'
    variables: '[${{ toJSON(vars) }},${{ toJSON(secrets) }}]'
- run: |
    echo "defaults  : ${{ steps.replace-tokens.outputs.defaults }}"
    echo "files     : ${{ steps.replace-tokens.outputs.files }}"
    echo "replaced  : ${{ steps.replace-tokens.outputs.replaced }}"
    echo "tokens    : ${{ steps.replace-tokens.outputs.tokens }}"
    echo "transforms: ${{ steps.replace-tokens.outputs.transforms }}"

Data/Telemetry

The ReplaceTokens GitHub Action collects anonymous usage data and sends them by default to its author to help improve the product. If you don't wish to send usage data, you can change your telemetry settings through the no-telemetry input or by setting the REPLACETOKENS_TELEMETRY_OPTOUT environment variable to 1 or true.

The following anonymous data is send:

• the hash of the owner and repository name (GITHUB_REPOSITORY)
• the hash of the workflow name (GITHUB_WORKFLOW)
• the hosting (server or cloud)
• the runner operating system (RUNNER_OS)
• the inputs values for
  • add-bom
  • chars-to-escape
  • encoding
  • escape
  • escape-char
  • if-no-files-found
  • log-level
  • missing-varvaction
  • missing-var-default
  • missing-varvlog
  • recursive
  • separator
  • token-pattern
  • token-prefix
  • token-suffix
  • transforms
  • transforms-prefix
  • transforms-suffix
• the number of sources entries
• the number of variables entries referencing file
• the number of variables entries referencing environment variables
• the number of variables inline entries
• the task result (success or failed)
• the task execution duration
• the outputs (defaults, files, replaced, tokens and transforms)

You can see the JSON serialized telemetry data sent in debug logs.

replacetokens-action's Issues

ensure versioning is always correct

Did you know I wrote this? It can help you ensure tags and branches in your repo are setup correctly to ensure proper versioning

https://github.com/jessehouwing/actions-semver-checker

Extension not working with x64 agent on vsts on premises

Facing this:
A supported task execution handler was not found. The task does not carry an implementation that is compatible with your current operating system 'Windows(X64)'. Contact the task author for more details.

Make variable names are case-insensitive

As GitHub variables and secrets names are automatically put in uppercase the action should be case insensitive with all variable names to avoid any issue.

secrets logged in workflow logs if ${{ toJSON(secrets) }} is used

When I use this step with the example:

- name: Replace tokens
  uses: qetza/replacetokens-action@v1
  id: replacetokens
  with:
    sources: ~/somefile.yml
    variables: '[${{ toJSON(vars) }},${{ toJSON(secrets) }}]'
    if-no-files-found: 'error'
    recursive: 'true'

The logs of the action include all of the secrets in an unredacted form in my GitHub Environment:

There should be a way to pass in secrets without being logged into the output of the workflow.

Can this action be used to save the transform to a new file or overwrite an existing file?

Is the only option to save to the same file that was loaded as a source?

I have a config.json file in my NodeJs project that has default configurations that all work for local development. I have a config.template.json file that has all the tokens in it. When I build via GitHub Actions, I want to load up the config.template.json file, transform it with your action to replace those tokens with the env vars and secrets from my repo, then save the resulting file over config.json (not back into config.template.json).

I enjoy the mechanism of having a config.json that is pre-populated with working, default values, that doesn't require any work/build/action/deployment to get it working right after cloning the repo. I'd also like to be able to overwrite that file with env vars and secrets via a GitHub Action. The above is a straightforward way to do it, assuming I can save a transformed token file to a different filename than the source.