tmccombs/hcl2json

Convert hcl2 to json

hcl2json

This is a tool to convert from HCL to json, to make it easier for non-go applications and scripts to process HCL inputs (such as terraform config).

If passed the -pack option, it converts to the JSON pack format used by hclpack, which contains the original structure of the document. However, all it has for expressions is the source text, which isn't always very useful.

If no options are passed, it converts the native HCL file to an (almost) equivalent HCL JSON file. Note, however, that there are some corner cases where it may not be exactly equivalent, especially if the target application makes use of static analysis.

Alternatives

At the time this project created, there weren't really any other object. However, the HCL project itself now includes an hcldec command that can convert hcl to json if you give it a specification. One advantage of hcl2json is it doesn't require a specification, so for use cases where the heuristics are correct it may be easier to use. However, if you have cases hit the limitations of conversion listed above, hcldec is probably a better fit.

Usage

# convert a file from hcl to json
$ hcl2json some-file.hcl > out.json
# reading from stdin also works
$ hcl2json < infile.hcl > out.json
# simplify any expressions that don't use variables or unknown functions
$ hcl2json -simplify infile.hcl > out.json

If you use the docker image, you can invoke with

$ docker run -it --rm -v $PWD:/tmp tmccombs/hcl2json /tmp/infile.hcl

Installation

Mac OS

Install with Homebrew:

brew install hcl2json

hcl2json can also be installed via MacPorts:

sudo port install hcl2json

Others

Prebuilt binaries are available on the releases page. There is also a docker image on dockerhub.

Alternatively you can build from source (see next section).

Building

You can build and install hcl2json using go get. Since hcl2json uses Go modules, you will need to run this as GO11MODULE=on go get github.com/tmccombs/hcl2json.

Alternatively, you can clone and build the repository:

$ git clone https://github.com/tmccombs/hcl2json
$ cd hcl2json
$ go build

This will build an hcl2json executable in the directory.

hcl2json's Issues

Download and build instructions not working

Go n00b here. Followed instructions verbatim; no worky. Request/recommend adding some robustness to the instructions to ease adoption/usage by plebes like me. Thanks!

username@machine:~/Downloads$ git clone https://github.com/tmccombs/hcl2json
Cloning into 'hcl2json'...
remote: Enumerating objects: 245, done.
remote: Counting objects: 100% (123/123), done.
remote: Compressing objects: 100% (83/83), done.
remote: Total 245 (delta 48), reused 90 (delta 24), pack-reused 122
Receiving objects: 100% (245/245), 63.10 KiB | 923.00 KiB/s, done.
Resolving deltas: 100% (112/112), done.
username@machine:/Downloads$ cd hcl2json/
username@machine:/Downloads/hcl2json$ go build
main.go:11:2: cannot find package "github.com/tmccombs/hcl2json/convert" in any of:
/usr/lib/go-1.10/src/github.com/tmccombs/hcl2json/convert (from $GOROOT)
/home/username/go/src/github.com/tmccombs/hcl2json/convert (from $GOPATH)

Add version output

It would be nice to add a version output.

The version seems to be passed in

hcl2json/.github/workflows/deploy.yml

Lines 47 to 58 in 75e097e

	- name: Update environment
	run: |
	echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_ENV
	echo "TARGET_NAME=hcl2json_${{ matrix.os.name }}_${{ matrix.arch}}${{ matrix.os.ext }}" >> $GITHUB_ENV
	- name: Build
	run: 'go build -ldflags="-X main.Version=${{ env.VERSION }}"'
	- name: Upload
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: |
	mv hcl2json${{ matrix.os.ext }} $TARGET_NAME
	gh release upload v$VERSION $TARGET_NAME

But not exposed as a flag

hcl2json/main.go

Lines 17 to 20 in 75e097e

	var options convert.Options
	flag.BoolVar(&options.Simplify, "simplify", false, "If true attempt to simply expressions which don't contain any variables or unknown functions")
	flag.Parse()

---

It could be as simple as this

import (
	"bytes"
	"encoding/json"
	"flag"
+	"fmt"
	"io"
	"log"
	"os"

	"github.com/tmccombs/hcl2json/convert"
)

+var (
+    version = "dev"
+)

func main() {
	logger := log.New(os.Stderr, "", 0)

	var options convert.Options

	flag.BoolVar(&options.Simplify, "simplify", false, "If true attempt to simply expressions which don't contain any variables or unknown functions")
+	flag.BoolVar(&options.Version, "version", false, "return version")
	flag.Parse()

+	if options.Version {
+		fmt.Println(options.Version)
+		os.Exit(0)
+	}

Including range (start and end) value in the generated JSON

Hi, thanks for building such a great utility for HCL
We are currently using this library to perform static analysis checks on Terraform HCL files and we wanted to use the range (start and end position of tokens) info in order to point to the location of the resource in terraform template. I was wondering if you could help me find a solution for this issue

Parsing error occurs

Thank you for this very handy tool!
I experienced hcl2json fail under the special condition of not ending with a blank line. Not sure whether this is a works as designed, but terragrunt (hclfmt, plan, etc.) does not stumble over the same hcl syntax. So I thought I share this with you anyways, in case you'd like to look into it.

Find attached a working and a failing version of a terragrunt.hcl.

terragrunt_working.hcl.txt
terragrunt.hcl.txt

Here is the stack trace:

panic: runtime error: index out of range [1212] with length 1212

goroutine 1 [running]:
github.com/tmccombs/hcl2json/convert.(*converter).rangeSource(...)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:100
github.com/tmccombs/hcl2json/convert.(*converter).wrapExpr(0xc00019fd68, 0x12cae40, 0xc0001621e0, 0x1218468, 0x123f7a0)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:311 +0x172
github.com/tmccombs/hcl2json/convert.(*converter).convertExpression(0xc00019fd68, 0x12cae40, 0xc0001621e0, 0x0, 0x0, 0x38, 0x2900108)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:198 +0xfa
github.com/tmccombs/hcl2json/convert.(*converter).convertBody(0xc00019fd68, 0xc000074840, 0x0, 0x0, 0xc00006a180)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:87 +0x1f8
github.com/tmccombs/hcl2json/convert.convertFile(0xc00006d8c0, 0x0, 0x0, 0xc000018320, 0x1)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:68 +0x8d
github.com/tmccombs/hcl2json/convert.File(0xc00006d8c0, 0x0, 0x0, 0x7ffeefbff600, 0xe, 0x1, 0x1)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:37 +0x38
github.com/tmccombs/hcl2json/convert.Bytes(0xc000132000, 0x4bc, 0x6bc, 0x7ffeefbff6b9, 0xe, 0x0, 0x0, 0x0, 0xc000063db0, 0xc000105f18, ...)
	/home/runner/work/hcl2json/hcl2json/convert/convert.go:27 +0xef
main.main()
	/home/runner/work/hcl2json/hcl2json/main.go:34 +0x29b

Add v prefix to tags

Now that the convert package is exposed, could we add a v to the latest tag (and the tags going forward?).

In order to add the dependency to other projects go.mod files, the tag must be in the format vX.Y.Z

Use of the archived and experimental hcl2 library

It seems that we are pulling a library that was experimental and is no longer supported by the vendor see Hashicorp's HCL Readme.

According to their docs, they advise everyone to move to the newest versions, hosted in the hcl2 branch of the HCL Project.

See for more information: Version Selection Guide.

I'll open a PR to upgrade to this version of the HCLv2 library.

Add usage instructions

Thank you firstly for putting this together!

I'm not a Go programmer so it took me a little while to figure out how to use this.

My first guess did not yield fruit:

$ go get github.com/tmccombs/hcl2json
package github.com/hashicorp/hcl/v2: cannot find package "github.com/hashicorp/hcl/v2" in any of:
        /usr/local/go/src/github.com/hashicorp/hcl/v2 (from $GOROOT)
        /Users/tim/go/src/github.com/hashicorp/hcl/v2 (from $GOPATH)
package github.com/hashicorp/hcl/v2/hclsyntax: cannot find package "github.com/hashicorp/hcl/v2/hclsyntax" in any of:
        /usr/local/go/src/github.com/hashicorp/hcl/v2/hclsyntax (from $GOROOT)
        /Users/tim/go/src/github.com/hashicorp/hcl/v2/hclsyntax (from $GOPATH)

(might be similar to #8)

But I eventually settled on:

git clone [email protected]:tmccombs/hcl2json.git
cd hcl2json
docker build -t hcl2json .
docker run -i hcl2json < /path/to/*.tf

Just wondering if you could add your recommend mode of usage to the README to help others coming across this? (I would PR, but I'm sure you will know what I did wrong with go get!)

linux_arm64 binary

Is it possible to add additional binary to the latest release?
would be super cool to have linux_arm64 next to the other binaries.

I hope you will consider! thanks :)

Add Docker image with ARM64 architecture

0.4.0 version from homebrew always returns an empty body

Regardless of input, hcl2json is always returning {} for me in the latest 0.4.0 version from homebrew:

$ brew install hcl2json
==> Fetching hcl2json
==> Downloading https://ghcr.io/v2/homebrew/core/hcl2json/manifests/0.4.0
Already downloaded: /Users/tflint/Library/Caches/Homebrew/downloads/645754b91374f38924e46fceedce7070c5f7041c0979adcb20f69426cfcf5fcb--hcl2json-0.4.0.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/hcl2json/blobs/sha256:aa5c2bec588ec15249d88fa73969e51f854ea3601521eb3491162c081a04efc6
Already downloaded: /Users/tflint/Library/Caches/Homebrew/downloads/771670a987eb164154c4e173c182609d976ee3908e6bcfc3b47d1b5fd95d00a4--hcl2json--0.4.0.ventura.bottle.tar.gz
==> Pouring hcl2json--0.4.0.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/hcl2json/0.4.0: 6 files, 3.8MB

$ echo 'variable "foo" { default = "bar" }' | hcl2json
{}

This works in the latest release from this repository:

$ wget https://github.com/tmccombs/hcl2json/releases/download/v0.3.6/hcl2json_darwin_amd64
$ chmod +x hcl2json_darwin_amd64
$ echo 'variable "foo" { default = "bar" }' | hcl2json_darwin_amd64
{
    "variable": {
        "foo": [
            {
                "default": "bar"
            }
        ]
    }
}

$ in type paramater of a variable

type = map(string)
is getting converted to
"type": "${map(string)}"
Is there a way , it can convert to json without introducing $

Returning list instead of object in latest master

Hi,

The output of our file with hcl2json being built from master is the following:

{
  "locals": [
    {
      "example": []
    }
  ]
}

but 0.3.1 produces:

{
    "locals": {
        "example": [],
      }
  ]
}

This is with the same input:

locals {
  example = [
    module.example,
    module.example2,
  ]
}

We would expect locals to be an object per 0.3.1.

Expose HCL to JSON as package

We would like to use this functionality in another Go app, specifically taking an HCL file and getting its JSON representation. However, all of the methods are private.

Any objections to packaging 🥁 up this logic into a package? Happy to do a PR!

Add to homebrew when the repo is popular enough

Hi.

So I wrote a homebrew formula for this tool after discovering that everything else that does CLI-based hcl->json conversion is either old or bad (or both).

Homebrew, however, is a little obnoxious, and doesn't think this tool is notable enough:

18:25 $ brew audit --new-formula hcl2json
hcl2json:
  * GitHub repository not notable enough (<30 forks, <30 watchers and <75 stars)
Error: 1 problem in 1 formula detected

So... I'm basically just dropping this note here. You can poke me when this threshold is reached, and/or submit it yourself.

In the meantime, anyone who is interested can install it manually using:

brew install https://raw.githubusercontent.com/RulerOf/homebrew-core/ff9109f212601f57be60aedb7dfdf91c6347b8ba/Formula/hcl2json.rb

hcl2son directory

Is there a way to convert entire directory?

Escape sequences in string values

Hello

I've been using hcl2json to verify some terraform files - it's working great, but I've found a small snag in escape sequence handling that results in ambiguous string value output.

It's probably easiest to just show an example:

resource "my_type" "my_resource" {
  my_text_value = "$${data.vault_generic_secret.my_secret.data[\"xyz\"]}"
  my_expr_value = data.vault_generic_secret.my_secret.data["xyz"]
}

The above generates this json:

"my_type": {
  "my_resource": [
    {
      "my_expr_value": "${data.vault_generic_secret.my_secret.data[\"xyz\"]}",
      "my_text_value": "${data.vault_generic_secret.my_secret.data[\"xyz\"]}"
    }
  ]
}

where both my_text_value and my_expr_value get output as the same literal string because $${ is the escape sequence for a literal ${. (Other escape sequences also get converted to their literal values - \r, \n, etc).

I was trying to do some downstream processing on the attribute values with some very basic expression evaluation - it would be good if there was some sort of --no-unescape switch to preserve the literal string values as they appear in the terraform files:

"my_type": {
  "my_resource": [
    {
      "my_expr_value": "$${data.vault_generic_secret.my_secret.data[\"xyz\"]}",
      "my_text_value": "${data.vault_generic_secret.my_secret.data[\"xyz\"]}"
    }
  ]
}

That way it'd be possible to differentiate an expression string from a literal string with $${ escapes...

(For now my code is just going to assume no-one actually uses the $${ escape sequence so if a string attribute contains ${ it'll trigger my evaluation logic - I don't think that'll be a problem for my use-case, but it would still be nice to remove the ambiguity).

Please provide pre-built binaries

Hi,

This is great and exactly what we needed since after the 0.12 upgrade one of our less important workflows which was using json2hcl broke.
As a small addition, could you please provide pre-built binaries somewhere, ideally in the github releases?

regards,
Jörg

Terraform fails for depends_on and list("string") cases

Hello.
hcl2json works nice, but there are some cases that after conversion don't work.

Here you can see a list of fixtures used for different converter.
Passing of those tests almost garanties that json file will be executable and valid for terraform.

Terraform init/plan fails for at least two cases. depens_on(depends_on.hcl) case and type case.
Depends_on case has to be without interpolation.
In case of type - interpolation don't has to present as well. type is reserved key for variables.
Example :

variable "availability_zones" {
  type    = list(string)
  default = ["az1", "az2", "az3"]
}

Thank you.

Single and Multiple provider definitions

Thanks a ton for exposing this functionality, it'll allow us to stop maintaining a local version of this library. Now onto the issue at hand!

@sarcasticadmin explained it best in this comment: open-policy-agent/conftest#266 (comment)

The overall issue is that when an HCL file has a single provider definition, the resulting JSON is just a single object.

"provider": {
    "aws": { "version": "=2.46.0", "alias": "one" }
}

But when an HCL file has multiple provider definitons of the same type, aliased, the JSON is turned into a set.

"provider" : {
    "aws": [
        { "version": "=2.46.0", "alias": "one" },
        { "version": "=2.46.0", "alias": "two" }
    ]
}

We would like to always return a set for the provider block for consistency.

Question being, do you agree with this approach as well and is something that hcl2json should do? If so, again would be more than happy to do a PR! Otherwise, we'll handle this specific conversion ourselves.

Context: open-policy-agent/conftest#266

Add usage section to the Readme

Hi

I wonder if you could add a Usage section to your readme. A line like below to show how you can use the docker image of this tool will be helpful

docker run -it --rm -v $PWD:/tmp tmccombs/hcl2json /tmp/common.hcl

GO version critical CVEs

Based on a scan we did on the image, it looks like the GO version used in the binaries is vulnerable.

GHSA-rxv8-v965-v333 and GHSA-679v-hh23-h5jh
Path: /0.5.0/bin/0.6.0/hcl2json_linux_arm64

Recommended course of action: Upgrade GO to 1.21.2 or higher

Making convert Body/ Blocks/ Attributes etc public?

I would like to thank you @tmccombs for writing this library!
You think we can make convertBody, convertBlock convertExpression public? I have a requirement where I need to update several terraform files and hclwrite package makes is possible to perform that operation. However hclwrite doesn't provide easy access to all the attributes/ expression. e.g. if I want to read existing attribute and update them programatically.

I think after I convert hclwrite.Expression to hclsyntax.Expression, I should be able to read attributes from hcl to json using your library. I wish this was organically available in HCLWrite but unfortunately it's not.

hclsyntaxExpr, _ := hclsyntax.ParseExpression(tagsAttr.Expr().BuildTokens(nil).Bytes(), "", hcl.InitialPos)

Fails to parse tags with periods in name

This configuration works in terraform and some other HCL->Json converters but results in an error for hcl2json:

resource "aws_s3_bucket" "design_bucket" {
  # non-relevant config

  tags {
    something.more.even_more = "True"
  }
}

This is an example of the error output:

Failed to convert file: main.tf:30,5-20: Argument or block definition required; An argument or block definition is required here. To set an argument, use the equals sign "=" to introduce the argument value.

Convert from JSON to HCL2

Having a result.json file with the following content:

{
    "var1": "foo",
    "var2": "bar",
    "var3": {
        "key": "value"
    }
}

Is it possible to retrieve HCL2 content back?

var1 = "foo"
var2 = "bar"
var3 = {
  key = "value"
}

Parsing error occurs when trying to convert null key as string

We experienced hcl2json fail under the special condition of having null as a key in a hc2 file.
This is happening on lines https://github.com/tmccombs/hcl2json/blob/main/convert/convert.go#L238-L243

where go-cty panics when trying to return a null value asString: https://github.com/tmccombs/hcl2json/convert/convert.go#27

Here is an example of the offending block:

  providers = {
    aws     = aws
    archive = archive
    null    = null
  }

And the logs:

> hcl2json my-tf.tf
panic: value is null

goroutine 1 [running]:
github.com/zclconf/go-cty/cty.Value.AsString(0x12b3be0, 0xc00001a19a, 0x0, 0x0, 0x12b3be0, 0xc00001a19a)
	github.com/zclconf/[email protected]/cty/value_ops.go:1254 +0x185
github.com/tmccombs/hcl2json/convert.(*converter).convertStringPart(0xc00018fd68, 0x12b42b8, 0xc000117ce0, 0xc00001a3a6, 0x7, 0x59, 0x8)
	github.com/tmccombs/hcl2json/convert/convert.go:243 +0x2f3
github.com/tmccombs/hcl2json/convert.(*converter).convertKey(0xc00018fd68, 0x12b4338, 0xc00000cba0, 0x7, 0xc00011cab8, 0x0, 0x0)
	github.com/tmccombs/hcl2json/convert/convert.go:266 +0x1ad
github.com/tmccombs/hcl2json/convert.(*converter).convertExpression(0xc00018fd68, 0x12b42f8, 0xc000174000, 0x0, 0x0, 0x1482390, 0x38)
	github.com/tmccombs/hcl2json/convert/convert.go:187 +0x2d1
github.com/tmccombs/hcl2json/convert.(*converter).convertBody(0xc00018fd68, 0xc0001284d0, 0x0, 0x0, 0x0)
	github.com/tmccombs/hcl2json/convert/convert.go:87 +0x218
github.com/tmccombs/hcl2json/convert.ConvertFile(0xc000119940, 0x0, 0x0, 0xc00001a358, 0x1)
	github.com/tmccombs/hcl2json/convert/convert.go:68 +0x8d
github.com/tmccombs/hcl2json/convert.File(0xc000119940, 0x0, 0x0, 0x7ffeefbff700, 0x8, 0x1, 0x1)
	github.com/tmccombs/hcl2json/convert/convert.go:37 +0x38
github.com/tmccombs/hcl2json/convert.Bytes(0xc00015a000, 0x82, 0x200, 0x7ffeefbff751, 0x8, 0x0, 0x0, 0xc00007c000, 0x14828e0, 0x1, ...)
	github.com/tmccombs/hcl2json/convert/convert.go:27 +0xef
main.main()
	github.com/tmccombs/hcl2json/main.go:34 +0x296

Even if the above hcl block might be invalid, we could still handle the panic from go-cty here and not panic.

--slurp option

If I have two JSON files:

{
    "k":{
        "a":"foo"
    }
}

{
    "k":{
        "b":"bar"
    }
}

I can do:

$ jq -s '.' *.json
[
  {
    "k": {
      "a": "foo"
    }
  },
  {
    "k": {
      "b": "bar"
    }
  }
]

it would be nice to have the same functionality for hcl2json

Fail to grab github.com/hashicorp/hcl/v2

Looks like the latest commit is using a bad url for the hcl dependency.

[2020-03-20T06:23:29.632Z] package github.com/hashicorp/hcl/v2: cannot find package "github.com/hashicorp/hcl/v2" in any of: [2020-03-20T06:23:29.632Z] /usr/local/go/src/github.com/hashicorp/hcl/v2 (from $GOROOT) [2020-03-20T06:23:29.632Z] /go/src/github.com/hashicorp/hcl/v2 (from $GOPATH) [2020-03-20T06:23:29.632Z] package github.com/hashicorp/hcl/v2/hclsyntax: cannot find package "github.com/hashicorp/hcl/v2/hclsyntax" in any of: [2020-03-20T06:23:29.632Z] /usr/local/go/src/github.com/hashicorp/hcl/v2/hclsyntax (from $GOROOT) [2020-03-20T06:23:29.632Z] /go/src/github.com/hashicorp/hcl/v2/hclsyntax (from $GOPATH)

Publish new Docker image for 0.3.4

Is it possible for a new image to be published to Docker Hub for the latest release? Last published image is a couple years old now.

test strategy

I want to create a test folder and verify whether it is normally converted to hcl -> json or json -> hcl through the go test command. Can you tell me if you have any good strategies?

Imports of github.com/hashicorp/hcl/v2 are broken

Hi! Thanks for hcl2json. go get github.com/tmccombs/hcl2json is failing right now because you depend on github.com/hashicorp/hcl/v2 and it looks like that no longer exists in github.com/hashicorp/hcl.

depends_on attribute is converted incorrectly.

hcl:
resource "aws_sns_topic" "test_topic" {
name = "test_topic"
display_name = "Test Topic"
depends_on = [
aws_iam_role_policy.example,
]
}
With the above terraform init and plan works fine.

json generated by hcl2json:
"aws_sns_topic": {
"test_topic": [
{
"depends_on": [
"${aws_iam_role_policy.example}"
],
"display_name": "Test Topic",
"name": "test_topic"
}
]
}
When I try to do terraform init it gives the following error:
Error: Invalid expression

on test.tf.json line 27, in resource.aws_sns_topic.test_topic[0].depends_on:
56: "${aws_iam_role_policy.example}"

A single static variable reference is required: only attribute access and
indexing with constant keys. No calculations, function calls, template
expressions, etc are allowed here.