Comments (24)
Here's something that I've been doing for some time now, and is working reliably cross-platform (local runs on darwin, CI runs on linux):
data "archive_file" "this" {
type = "zip"
output_path = "${path.root}/.terraform/gke-cluster-notification-${random_id.this.hex}.zip"
dynamic "source" {
for_each = toset([
"files/function/function.go",
"files/function/go.mod",
"files/function/go.sum",
])
content {
content = file("${path.module}/${source.value}")
filename = basename(source.value)
}
}
}
I.e. instead of using source_dir
I'm enumerating the files the resulting zip should contain, and since terraform is reading their contents in via the fil()
function, there is no opportunity for umask, permissions, etc to cause a difference in the resulting zip on different platforms.
from terraform-provider-archive.
@appilon @radeksimko Any chance we could get this merged into the module? This issue has been around for awhile and has the most upvotes/comments
from terraform-provider-archive.
I can confirm this behavior still exists as the patch from @saveriomiroddi wasn't merged (sadly). With out these changes the archive_file production isn't idempotent. At a minimum this should be reflected in the documentation so that appropriate steps can be taken to avoid the problem
from terraform-provider-archive.
Hello all, apologies for the long time without any word. Unfortunately solving this problem is inherently difficult, we recently setup multi-os testing in #71 to try and have a better chance at catching problems such as this, but this issue specifically isn't something that we can practically catch in CI.
We are evaluating the solutions proposed in #41 and #47, but in this circumstance trying to code against operating system or library defaults just so a common pattern for a different provider is resolved, might not be something we want to set precedent for in an individual provider (our apologies if that is a frustrating stance to take).
With that said, we are still looking at the two proposed PRs, and will be attempting to either come up with a solution, or provide a final word/guidance.
UPDATE: #53 is likely the enhancement we want to make to try and remedy the problem.
from terraform-provider-archive.
I noticed that the produced zip file differs depending on the permissions of the source file. Why would the source file permissions be different? One common case is having a different umask between systems. For example, my laptop's default umask is 0022
thus when I clone a repository, the permissions of non-executable files are -rw-r--r--
. However, the Amazon Linux 2 AMI has a default umask of 0002
thus newly created (or cloned) files have permissions of -rw-rw-r--
.
Prior to realizing that archive_file worked for directories, I wrote a little tool to produce deterministic zip files. To handle permissions it either sets all files in the zip to be -r--r--r--
or -r-xr-xr-x
depending on if the file should be executable or not. I could see a use case for also handling the write permission, but I have yet to need that.
The reason for these permissions is, as according to the lambda documentation all files should have global read permissions. While it's not explicitly stated, I interpret that to mean any executables thus need to have global execute permissions. Hence why I think zip files created by this tool should apply consistent permissions to all users (i.e., user, group, other sections).
Edit: I'm going to see I can quickly make a PR for this. I've never worked with go, but it seems straightforward enough.
from terraform-provider-archive.
While this is a 2 year old comment (hashicorp/terraform#18422 (comment)), it suggests that zip files should be created by a build process.
https://www.reddit.com/r/Terraform/comments/aupudn/building_deterministic_zips_to_minimize_lambda/ also handles it by creating zip files in the build process instead of using archive_file
It would be great if Hashicorp would give an official status on if archive_file
is the "wrong" way to be doing this. There are a bunch of different ways to produce deterministic zip files in your build process, including strip-nondeterminism
from Debian (https://salsa.debian.org/reproducible-builds/strip-nondeterminism) and deterministic_zip https://github.com/bboe/deterministic_zip
from terraform-provider-archive.
There is a proposed workaround for this issue in #90, which adds an optional output_file_mode
attribute to force consistency in file modes across OSs.
I would be very grateful if some of you experiencing this could test this fix by running @virgofx's provider locally.
@virgofx has kindly provided binaries from his branch for testing: #90 (comment)
You can find instructions for using a locally built provider here: https://www.terraform.io/docs/cli/config/config-file.html#development-overrides-for-provider-developers
from terraform-provider-archive.
A workaround we applied was to have:
- exact same
umask
on both platforms source_dir
fordata "archive_file"
-- see https://www.terraform.io/docs/providers/archive/d/archive_file.html- dockerized terraform, ie.
docker run --rm -it -v ${HOME}/.aws:/root/.aws -v ${PWD}:/code -w /code -e AWS_PROFILE=${AWS_PROFILE} hashicorp/terraform:0.12.24 $@
The zipinfo
command will give you a hint for what is the possible difference between one archive and another
Note (in our case) the first column: -rw-rw-r--
vs -rw-r--r--
Linux - zipinfo
output:
Archive: tmp/archive.zip
Zip file size: 14604 bytes, number of entries: 15
-rw-rw-r-- 2.0 unx 412 bl defN 49-Jan-01 00:00 index.js
-rw-rw-r-- 2.0 unx 1707 bl defN 49-Jan-01 00:00 node_modules/...
[...]
15 files, 26615 bytes uncompressed, 12122 bytes compressed: 54.5%
MacOS - zipinfo
output:
Archive: tmp/archive.zip
Zip file size: 14604 bytes, number of entries: 15
-rw-r--r-- 2.0 unx 412 bl defN 49-Jan-01 00:00 index.js
-rw-r--r-- 2.0 unx 1707 bl defN 49-Jan-01 00:00 node_modules/...
[...]
15 files, 26615 bytes uncompressed, 12122 bytes compressed: 54.5%
TF code Example
data "archive_file" "lambda" {
type = "zip"
source_dir = "${path.module}/lambda_source/"
output_path = "${path.module}/tmp/archive.zip"
}
from terraform-provider-archive.
I had encountered this issue too.
We used Lubuntu 16.04 and macOS 10.13.6.
from terraform-provider-archive.
I came across this issue where my centos 7 VM has different permissions for mounted volumes and was removing the execute bit. Annoyingly, trying to change the permissions doesn't work (see https://superuser.com/questions/1083393/virtualbox-guest-shared-folder-ignoring-umask)
some kind of parameter on the archive_file data source block to set the execute bit would make sense as its generating the zip, but wouldn't work for my scenario.
from terraform-provider-archive.
perhaps a better solution would be to provide a source hash i.e.
source_code_hash = "${data.archive_file.zip.input_base64sha256}"
After all, the source_code_hash element is used to trigger an update on change so inspecting the source rather than the exported binary zip should fulfill the same requirement.
from terraform-provider-archive.
I think you misunderstand. AWS does indeed confirm the hash of the uploaded binary(https://docs.aws.amazon.com/lambda/latest/dg/API_CreateFunction.html) but I expect this is intended to confirm that the binary has uploaded correctly. However, according to the terraform docs the source_code_hash element in aws_lambda_function is "Used to trigger updates". so it would be good to have a trigger that can identify that the code has changed separate from the hash result of the binary.
Of course this doesn't answer the question, why is the binary output different across OS's. And if terraform is verifying the hash of the binary in the background anyway this still needs to be resolved.
from terraform-provider-archive.
That only works when you're only archiving a single file. What if you're archiving a directory?
from terraform-provider-archive.
I submitted this PR some time ago but haven't heard anything from maintainers of the project. 🤷♂
from terraform-provider-archive.
This is not an easy or straight forward task (even if it looks like it) and I seriously doubt that Terraform is going to fix it. A couple of years ago a new project called TorrentZip was created by the MAME community to tackle this exact problem (reproducible zip files across multiple operating systems). More information here: http://rescene.wikidot.com/torrentzip
It seems that someone implemented a Go version of it: https://github.com/uwedeportivo/torrentzip
from terraform-provider-archive.
Still having the issue today
from terraform-provider-archive.
@appilon Thanks so much for the update. Looking forward to a hopeful resolution soon.
from terraform-provider-archive.
Nicely done, that's great @karolkania 👏👏👏
from terraform-provider-archive.
Yeah, I'd love to see some traction on this issue. I don't think this is out of scope of Terraform though. As this thread has mentioned, there are tons of workarounds to get it working. Just need Hashicorp to actually put some resources on this and get it working.
from terraform-provider-archive.
Hi there! I've deleted all of my comments are closed the PR; please don't (manually) CC me.
from terraform-provider-archive.
The workaround in #90 has been released in terraform-provider-archive v2.2.0. If output_file_mode
does not solve your problem, please comment on this issue or open a new one.
from terraform-provider-archive.
This didn't fix it for me:
data "archive_file" "lambda_zip" {
type = "zip"
output_file_mode = "0777"
source_dir = "${path.module}/files/${var.name}"
output_path = "${path.module}/files/${var.name}.zip"
}
However, the zips I were having the problem with were zips created inside a module that's used multiple times... If anyone reading this is also having a problem in this scenario, check out my comment @ #31 (comment)
from terraform-provider-archive.
worked for me as well. Thanks @davidalger
from terraform-provider-archive.
@davidalger cam across this one. we are uploading our compiled go code into the zip, but the hashes are unstable (like the whole internet has this problem). sadly file(binary_file)
wont work. how you deal with that?
from terraform-provider-archive.
Related Issues (20)
- Migrate to terraform-plugin-framework HOT 1
- Run archive_file on each apply HOT 2
- data.archive_file does not support resource tainting HOT 2
- Bump Development/Build Minimum Go Version to 1.17 HOT 2
- Issue archiving base64 encoded content w/ source block HOT 5
- Bump Expected Minimum Go Version to 1.18 HOT 1
- archive_file doesn't re-create the archive upon content change
- Source_dir conflicts with source HOT 1
- Zip file created by terraform archive_file cannot be properly read by python
- Generated archive contents include an extra (empty) file when `output_path` is configured within same directory as `source_dir`. HOT 2
- Migrate acceptance testing to terraform-plugin-testing HOT 1
- Bump Expected Minimum Go Version to 1.19 HOT 1
- GitHub Actions - deprecated warnings found - action required! HOT 2
- archive_file data source gets created during "terraform plan" vs "terraform apply" and also is not deleted during destroy HOT 10
- Error generated during the execution of acceptance test on archive_file resource
- Documentation and changelog require updating HOT 1
- Support Additional Compression Types(Ex: tar.gz format) HOT 4
- Update Go Module to Go 1.20 Minimum HOT 1
- archive_file produces a corrupted zip file HOT 5
- Error generating archive with archive_file when symlink is present and exclude_symlink_directories is set to true HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from terraform-provider-archive.