Cannot archive directories with symbolic links (symlinks)

[hashicorp]

Cannot archive directories with symbolic links (symlinks)

Really sad this isn't fixed in 2019 :[

Honestly, the archive file provider is so woefully broken, as a result of this bug, that it really shouldn't even be included in terraform, if you ask me. Why provide functionality that is all but guaranteed not to work correctly? This provider should be undocumented and/or not included in distributions until such time as it can actually create a zip file correctly. Especially in the case of anyone trying to deploy a lambda zip file out of a monorepo, node_modules is basically guaranteed to have symlinks within it. Why cause people to waste their time trying out this provider when it is known that it is totally dysfunctional in a very normal use case?

All it is doing is wasting people's time and forcing us all to independently figure out that it is broken, figure out a workaround to prove it is broken, come here to report it, only to find that it has been a known issue for a YEAR AND A HALF!!! Why waste all of our time like that?

It ain't that hard to zip up a file or directory. If you can't provide a working shortcut, don't provide a shortcut at all.

[danieljarrett74]

Ditto for 2022.

Really sad this isn't fixed in 2019 :[

[dmrzzz]

FWIW, this is only a problem with symlinked directories. Symlinking (and archiving) individual files works fine.

For me, symlinks to individual files don't work either, but in a much more insidious fashion. While Terraform does create an archive file in this case, it is malformed; extracting the archive yields a symlink whose symbolic target is the original file contents.

Using same Terraform config as OP,

$ tree
.
├── dir_with_symlink
│   └── textfile -> ../textfile
├── main.tf
└── textfile

1 directory, 3 files

$ cat textfile 
text file contents

$ terraform apply
data.archive_file.dir: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

$ unzip -d tmp dir.zip 
Archive:  dir.zip
  inflating: tmp/textfile            -> text file contents^J 
finishing deferred symbolic links:
  tmp/textfile           -> text file contents^J

$ ls -la tmp/
total 8
drwxrwxr-x 2 dmrz dmrz 4096 Mar 16 18:59 .
drwxrwxr-x 5 dmrz dmrz 4096 Mar 16 18:59 ..
lrwxrwxrwx 1 dmrz dmrz   19 Mar 16 18:59 textfile -> text file contents?

$ cat tmp/textfile 
cat: tmp/textfile: No such file or directory

$ terraform --version
Terraform v0.11.3
+ provider.archive v1.0.2

[sorjef]

@combinatorist, indeed, with that version of the script, terraform will always redeploy a zip file, even if the zip file is not changed. Below, you can find an updated version of the script I use now:

ZIP_PATH=$(pwd)/$1
zip -r -X $ZIP_PATH . %1>/dev/null %2>/dev/null
echo "{ \"hash\": \"$(cat "$ZIP_PATH" | shasum -a 256 | cut -d " " -f 1 | xxd -r -p | base64)\", \"md5\": \"$(cat "$ZIP_PATH" | md5)\" }"

In terraform, you can then use result.md5 or result.hash outputs:

data "external" "worker_zip" {
  program = ["./zip.sh", "../dist/${var.name}-worker.zip"]
}

resource "aws_s3_bucket_object" "worker" {
  bucket = "${var.package_bucket}"
  key = "${var.name}-worker.zip"
  source = "../dist/${var.name}-worker.zip"
  etag = "${data.external.worker_zip.result.md5}"
}

UPD: The script may only work on OSX. I did not bother checking it on linux

[mildwonkey]

Hi all! I don't want to shut down the conversation, so please don't take this as a request not to comment, but please do not post "+1" comments here, since it creates noise for others watching the issue and ultimately doesn't influence our prioritization because we can't actually report on these. Instead, react to the original issue comment with 👍, which we can and do report on during prioritization.

[jmrah]

I have successfully been able to use the archive_file data source with a symlinked directory by appending a trailing / to the source_dir filepath. Here's a minimum reproducable example:

Create a symlink to ./real_dir using the linux command ln -s ./real_dir ./dir_with_symlink

Create this archive_file resource without the trailing slash:

data "archive_file" "dir" {
  type = "zip"
  source_dir = "./dir_with_symlink"
  output_path = "./dir.zip"
}

terraform apply and see error Error: error archiving directory: error reading file for archival: read ./symlink_dir: is a directory

Add trailing slash from the source_dir property, so the `archive_file resource becomes:

data "archive_file" "dir" {
  type = "zip"
  source_dir = "./dir_with_symlink/"
  output_path = "./dir.zip"
}

terraform apply successfully.

[jsonmaur]

Have the same issue. Here is the workaround I've been using with JQ (a command line JSON parser) and query data for the external Terraform provider:

data "external" "main" {
  program = ["sh", "${path.module}/archive.sh"]
  query = {
    input_path = "${path.module}/my_function_code/"
    output_path = "${path.module}/my_function_code.zip"
  }
}

resource "aws_lambda_function" "main" {
  filename = "${data.external.main.result.output_path}"
  source_code_hash = "${data.external.main.result.output_hash}"
  # ...
}

#!/bin/bash

ARGS=$(cat -)
INPUT_PATH=$(echo $ARGS | jq -r ".input_path")
OUTPUT_PATH=$(echo $ARGS | jq -r ".output_path")

rm -f $OUTPUT_PATH

cd $INPUT_PATH
zip -rqX $OUTPUT_PATH ./*

OUTPUT_HASH=$(cat $OUTPUT_PATH | openssl sha -binary -sha256 | base64)

jq -ncM \
  '{ "output_path": $output_path, "output_hash": $output_hash }' \
  --arg output_path "$OUTPUT_PATH" \
  --arg output_hash "$OUTPUT_HASH"

[jonathanstockton]

+1

[byustephen]

It's really, really sad this isn't fixed in December 2022. It is almost 2023!

[shaiguitar]

+1

[OJFord]

I think what's most sad is all the (to varying degrees of passiveness) aggression in this bug report in free open source software.

If it's so much more important to any of us than it is to Hashicorp, we can put the work in and open a PR as invited.

If it's not the most pressing thing to us either, then maybe we should just move on - since it's easily workaround-able - and be happy when someone does eventually find the time to work on it so we can do easy symlink zipping for free.

[bookshelfdave]

I've added this issue to our triage meeting notes. We'll prioritize and take a look within the next few weeks.

[phinze]

I very much understand your frustration here, @sgendler-stem. I myself have definitely run into old bugs that seem like they should have been fixed by the time I got around to tripping over them, and it's always a test to my patience.

I know that it's extra annoying to hit such an old known issue on a repo being overseen by HashiCorp employees. You'd expect for us to be able to keep up with issues like this one, and--believe me--we really want to! But the reality is that users are many and we are few (but growing!). We are still at the stage where we have to do our best to prioritize our attention while admitting that we can't get to everything in a timely manner. I'm sorry this isn't fixed yet. I'll make sure this issue has visibility as we continue to prioritize the work our employees have lined up, but there are thousands of issues across dozens of providers so it's hard for me to make specific promises here.

I think you make a good point about how common symlinks are in the wild, and how that makes users of this provider likely to hit this bug. I think rather than un-publishing the provider we could instead try to prevent users from tripping over this issue by tossing a notice in the docs linking back here until this bug is fixed. That will also help promote the bug to anyone in the community who may want to offer some help in fixing it.

We prioritize PRs higher than issues since they're likely close to a solution, so if anyone from the community wants to take a stab at this we'd love to see that! And we're always happy to help provide guidance to point you in the right direction. We don't want to just say "contributions welcome," we do our best to prioritize and solve our own issues, but if an issue isn't being resolved fast enough for you, then this is the best way to enact change in the projects you use.

Once again, I'm sorry this isn't fixed, and we do appreciate everyone our community reporting these issues and voting for issues they care about. I'll follow up with a PR to add a notice about this issue to the docs to help inform our users.

[sorjef]

For anybody looking for a workaround.

Create zip.sh file with the following content:

#!/bin/sh
zip -r -X $1 $2 %1>/dev/null %2>/dev/null
echo "{}"

then add this data source to your terraform file:

data "external" "foursquare_worker_zip" {
  program = ["./zip.sh", "archive.zip", "../folder/to/archive/"]
}

[byustephen]

@OJFord I don't want to get into a comment battle with you.

I disagree about locking a thread just because someone commented on it. If there is a known bug, that was reported over 5 years ago, I think it is great that people comment on it and voice their concerns. It adds weight to an issue, and makes it visible than just a thumbs up on an original bug report.

As serverless apps become more and more common, and Terraform becomes more of an industry standard, this issue becomes more concerning. And this is a bug, and this bug should be fixed. I use symlinks all the time in my projects, and it is arguable that if you have a multi-environment terraform project without symlinks then it is arguable that you are doing it wrong.

I hope that Hashicorp can ressurect this bug, and put some smarter minds on this issue and get it fixed. I'd offer to do a pull request, but I'm 1) wise enough not to get into a fight, and 2) wise enough to know that there are people smarter than I am who can craft a more elegant solution to this problem.

[sorjef]

As an option, an additional parameter in archive_file data source could be added, named something like follow_symlinks with a default value set to true.

If there is any deep recursion, the code may throw an error after getting N levels deep.

Unfortunately, I do not have enough hands-on Go experience to create a pull request myself, but here are my thoughts on how this could be implemented:

The check below could be added before this line

fi.Mode() & os.ModeSymlink != 0

And Readlink should be executed along with a recursive walk.

Hope these thoughts will be helpful.

[combinatorist]

@sorjef, will your workaround generate appropriate dependencies in terraform?

For example, if I were using AWS lambda functions, I need to deploy code in a zip, but terraform needs a SHA (or some other trigger) to know that zip has changed (e.g. 8344). But, if I'm using your external data "program" instead of an archive, then my lambda function would statically depend on the zipped file, so it might check the SHA before this is rerun (i.e. before the file actually changes). Is that right?

Or, does terraform always compute data sources before computing resource state / changes? If that's the case, then this should work fine.

[combinatorist]

FWIW, this is only a problem with symlinked directories. Symlinking (and archiving) individual files works fine.

[haidaraM]

Any news on this issue ?

[jantman]

This is really problematic for me. I'm having the same problem with files too, like @dmrzzz ... except if the length of the file contents is more than trivial, unzipping the resulting archive completely fails with "File name too long" because it attempts to create a symlink pointing to the full content of the file.

IMO this makes the archive_file provider pretty much broken...

[kristof9851]

+1

[artkay]

+1

[karthikns16yahoo]

I have problem with python packages when built using

pip install -r requirements.txt --target python

and zipped using data_archive

data "archive_file" "python" {
  type = "zip"
  output_path = "${path.module}/python.zip"
  source_dir = "${path.module}/python"
  depends_on = [null_resource.python_with_packages]
}

and subsequently uploaded as lambda layer in AWS results in lamdba failure with error

Unable to import module 'lambda_layer': No module named requests

If I used local-exec to zip the file such as below

resource "null_resource" "zip_the_python_folder" {
  triggers = {
    build_number = timestamp()
  }

  provisioner "local-exec" {
    command = "zip -r9 python.zip python"
    working_dir = "${path.module}"
  }

  depends_on = [null_resource.python_with_packages]
}

and create a lambda layer version it works. I checked and found that were no symlinks in hte python directory

[Wambosa]

Have the same issue. Here is the workaround I've been using with JQ (a command line JSON parser) and query data for the external Terraform provider:

data "external" "main" {
  program = ["sh", "${path.module}/archive.sh"]
  query = {
    input_path = "${path.module}/my_function_code/"
    output_path = "${path.module}/my_function_code.zip"
  }
}

resource "aws_lambda_function" "main" {
  filename = "${data.external.main.result.output_path}"
  source_code_hash = "${data.external.main.result.output_hash}"
  # ...
}

#!/bin/bash

ARGS=$(cat -)
INPUT_PATH=$(echo $ARGS | jq -r ".input_path")
OUTPUT_PATH=$(echo $ARGS | jq -r ".output_path")

rm -f $OUTPUT_PATH

cd $INPUT_PATH
zip -rqX $OUTPUT_PATH ./*

OUTPUT_HASH=$(cat $OUTPUT_PATH | openssl sha -binary -sha256 | base64)

jq -ncM \
  '{ "output_path": $output_path, "output_hash": $output_hash }' \
  --arg output_path "$OUTPUT_PATH" \
  --arg output_hash "$OUTPUT_HASH"

Thank y'all very much for this discussion. I took the above and made it work for my usecase which was slightly different. Combining posts from @jsonmaur and @sorjef

lambda.tf

data "external" "my_lambda" {
  program = ["sh", "${path.module}/../../util/hack-zip.sh"]
  query = {
    input_path = "${path.module}/.build/my_lambda"
    output_path = "../auth_basic.zip"
  }
}

hack-zip.sh

#!/bin/bash
ARGS=$(cat -)
INPUT_PATH=$(echo $ARGS | jq -r ".input_path")
OUTPUT_PATH=$(echo $ARGS | jq -r ".output_path")

cd $INPUT_PATH
zip -rXD --symlinks $OUTPUT_PATH ./* %1>/dev/null %2>/dev/null

OUTPUT_HASH=$(cat $OUTPUT_PATH | sha256sum | awk '{print $1}' | base64)

jq -ncM \
  '{ "output_path": $a, "hash": $b }' \
  --arg a $OUTPUT_PATH \
  --arg b $OUTPUT_HASH

This ended up working very well. I tried what feels like a hundred different approaches. Even trying to use working_dir param as provided by hashicorp. That route in particular produced a slew of strange path errors.

edit: I should mention that this approach fails to strip out the metadata during hash calculation, so that consecutive runs produce a different hash; this triggers a redeploy of the lambda/resource every time.

[OJFord]

Then put in the work and propose a PR as invited:
#6 (comment)

All this 'so sad me too want this fixed' is pointless and boring.

@phinze Maybe time to lock this one?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.