aws-quickstart/quickstart-git2s3

AWS Quick Start Team

quickstart-git2s3

This repository has been deprecated in favor of https://github.com/aws-ia/cfn-ps-git2s3.

We will archive this repository and keep it publicly available until May 1, 2024.

quickstart-git2s3's Issues

Symlinked files to nowhere in repo causes failure in zipping

By mistake, my git repo had symlinked files in it. (A poorly named virtualenv directory had been checked in, it seems.) This causes zip to fail and thus the function to fail. There seems to be a --symlinks option to zip which would make this not fail.

(This isn't particularly important to me since I cleaned up the repo to remove the bad files, but this issue might be useful to someone else who stumbles upon this, and it'd be a convenient thing to not fail on.)

ZipDL method creates zip with an additional folder

when the repo contents are in a sub-folder integrations with things like CodeDeploy do not work.

Unable to find template file

Trying to follow the steps provided to deploy the required stack, but I get an error right away. It seems that the template files do not exist in the given S3 path anymore? please advise.

Thanks

Failed to authenticate SSH session: Waiting for USERAUTH response: GitError

I created the stack with oauth key/secret for my bitbucket account. I webhook that calls the GitPullLambda runs when I push the code, however, I get this error:

Traceback (most recent call last):
File "/var/task/lambda_function.py", line 191, in lambda_handler
repo = create_repo(repo_path, remote_url, creds)
File "/var/task/lambda_function.py", line 69, in create_repo
repo = clone_repository(remote_url, repo_path, callbacks=creds)
File "/var/task/pygit2/__init__.py", line 263, in clone_repository
check_error(err)
File "/var/task/pygit2/errors.py", line 64, in check_error
raise GitError(message)
GitError: Failed to authenticate SSH session: Waiting for USERAUTH response

The key/secret for the Oauth consumer in my bitbucket account has read access to projects and repositories. Does it need more access?

unable to connect to Gitlab

I see the lambda was invoked in the cloud watch logs, however it is failed with the following error. I need some help to resolve this issue.

failed to resolve address for git.xxxx.xxxxx.xxxxxxx.com: Name or service not known: GitError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 191, in lambda_handler
repo = create_repo(repo_path, remote_url, creds)
File "/var/task/lambda_function.py", line 69, in create_repo
repo = clone_repository(remote_url, repo_path, callbacks=creds)
File "/var/task/pygit2/init.py", line 263, in clone_repository
check_error(err)
File "/var/task/pygit2/errors.py", line 64, in check_error
raise GitError(message)
GitError: failed to resolve address for git.xxx.xxxx.xxxxxxx.com: Name or service not known

Configuring AWS services

So i followed the steps and deployed the template and created a github hook successfully. I am stuck at next step which is "Configuring AWS services" i am not sure what to do here ? Can someone please help what needs to be done in this step.

https://aws.amazon.com/quickstart/architecture/git-to-s3-using-webhooks/

Issue regarding git2s3 connectivity with GIT in prd env

git2s3issuelogs.docx

Same git2s3 CFT i used for dev stg and prd environment, everything is working correctly in dev and stg but failing in prd env I am getting below error in codebuild log:

--
Error while executing
command: git clone —single-branch —depth=1 —branch $Branch $GitUrl .. Reason: exit status 128

ssh: connect to host git.autodesk.com port 22: Connection timed out
fatal: Could not read from remote repository

Please make sure you have the correct access rights and the respository exists.

Can you please help me with this, as it is blocking my production deployment.

GitLab: The project you were looking for could not be found.

I am unable to fetch the code its shows me the below error

Please suggest what is wrong?

[INFO] Keys not found on Lambda container, fetching from S3...
.
.
.

GitLab: The project you were looking for could not be found.
: GitError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 160, in lambda_handler
repo = create_repo(repo_path, remote_url, creds)
File "/var/task/lambda_function.py", line 64, in create_repo
repo = clone_repository(remote_url, repo_path, callbacks=creds )
File "/var/task/pygit2/init.py", line 255, in clone_repository
check_error(err)
File "/var/task/pygit2/errors.py", line 64, in check_error
raise GitError(message)
GitError: > GitLab: The project you were looking for could not be found.

JSON error when using ZipDl

When I use the ZipDl endpoint with a Bitbucket webhook I get the following error:

{
"message": "Could not parse request body into json: Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n at [Source: [B@2ef71fad; line: 60, column: 6]"
}

It appears to be an error from API Gateway, but the format of the message is valid JSON, and resubmitting the same payload to the GitPull endpoint works as expected.

I can send via email an example payload, configured endpoints and my tweaked template.

CI/CD pipeline

When I am creating CI/CD from CodePipeline it gives error in the build section.

S3 error: The specified key does not exist.

Upon clicking launch quickstart from the docs, we get this error:

S3 error: The specified key does not exist. For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

EDIT: this was only from the link in the PDF. Launching from the AWS web quickstart main page worked.

CustomDomainName certificate is created in bucket/lambda region, not 'us-east-1'

The CustomeDomainName certificate is created in the wrong region

Invalid certificate ARN: arn:aws:acm:ap-southeast-2:7########:certificate/??????????. Certificate must be in 'us-east-1'.

GitError: failed to resolve address for git.xxxxxx.com: No address associated with hostname

I have generated the SSH key and registered it in the enterprise GitHub. I have added the web hook api. When I make a commit to my git repo, the lambda function is triggered. I get the following error (partial stack from cloud watch logs below):

failed to resolve address for git.autodesk.com: No address associated with hostname: GitError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 226, in lambda_handler
repo = create_repo(repo_path, remote_url, creds)
File "/var/task/lambda_function.py", line 70, in create_repo
repo = clone_repository(remote_url, repo_path, callbacks=creds)
File "/var/task/pygit2/init.py", line 263, in clone_repository
check_error(err)
File "/var/task/pygit2/errors.py", line 64, in check_error
raise GitError(message)
GitError: failed to resolve address for git.autodesk.com: No address associated with hostname

Any help in resolving this is appreciated

GitLab : 'fullName': KeyError full_name = event['body-json']['repository']['fullName']

After deploying using Quick start guide. Repo never sync and Lambda function fails with following error.

'fullName': KeyError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 155, in lambda_handler
full_name = event['body-json']['repository']['fullName']
KeyError: 'fullName'

Following is the payload for GitLab events.
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "[email protected]",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"[email protected]:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"[email protected]:mike/diaspora.git",
"ssh_url":"[email protected]:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "[email protected]:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"[email protected]:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "[email protected]"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev@dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
Ref : https://docs.gitlab.com/ee/user/project/integrations/webhooks.html

Python 3.7 reached end of Life

Due to the Python releases page, python 3.7 reached the end of life and is now no longer supported. Therefore the "CopyZipsFunction" need to be updated to preferred 3.10.
When performing this, all other Lambdas should be upgraded to 3.10 as well.

Gitlab Project Id

Hi - I'm going through the web hook POST requests from Gitlab and I noticed we need a slight tweak.

The key event['body-json']['project_id'] exists in a push request, but not a merge request, so the lambda caught a KeyError with a merge. Below is my propose solution, let me know if you want a PR.

Thanks!

#project_id = event['body-json']['project_id']
# project_id key works in push events, but not merge requests
project_id = event['body-json']['project']['id']

How to setup it for non master branches

how we can use this quick start for branches other than master ?

Zip destination different to documentation

As far as I can see the destination that the lambda functions upload the zip file to is different to what the documentation claims it is.

The ZipDlLambda function for example in line 123

s3_archive_file = "%s/%s/%s/%s.zip" % (owner, name, branch, name)

The documentation https://aws-quickstart.s3.amazonaws.com/quickstart-git2s3/doc/git-to-amazon-s3-using-webhooks.pdf on page 14 states

S3://output-bucket-name/git-user/git-repository/git-user_git-repository.zip

Misunderstanding here or documentation issue?

git pull with HTTPS

This is a great tool for the integration git and AWS codepipeline. It seesm that the git has to support SSH.

Can it support git pull with HTTPS?

Facing issue when I'm using SecurityGroupIds and SubnetIds.

I was modifying template according to my use case
that is I want to launch this template so it can assess a particulate given subnet and security group
and lambda is giving that property https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html
so I took 2 extra parameter that is SecurityGroupIds and SubnetIds.

I have refer same where lambda is going to perform creation of GitPullLambda

having error like "lambda not able to find keys in bucket"
can you tell me what I'm doing wrong here I tried to solve but couldn't.

Fix 'branch' in zipdl Lambda code

In the zipdl lambda source code, the branch is incorrectly set to a variable named branch_name which is different from other sources.

That means the archive_url is incorrectly constructed for github.

It should read as follows:

branch = event['body-json']['ref'].replace('refs/heads/', '')
archive_url = archive_url.replace('{archive_format}', 'zipball').replace('{/ref}', '/' + branch)

Upgrade CopyZips from Python 2.7 to 3.x

Python 2.7 is EOL in January 2020. We need to upgrade the CopyZips function prior to then.

BadZipFile error

Hi I'm getting a badzipfile error. When i wget the file in the lambda logs that shows the zip file, it's definitely a zip. Also, is there a way to include the .git dir from the provider?

File is not a zip file: BadZipfile
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 121, in lambda_handler
zip = ZipFile(f)
File "/usr/lib64/python2.7/zipfile.py", line 770, in __init__
self._RealGetContents()
File "/usr/lib64/python2.7/zipfile.py", line 811, in _RealGetContents
raise BadZipfile, "File is not a zip file"
BadZipfile: File is not a zip file

Deployment guide doesn't show how to replicate a github repo into an s3 location with this?

I have read the deployment guide and got the deployment up. But I'm not clear on how to actually get a git repo to be replicated in the bucket from the documentation. what would the next steps be?

I came across this when I encountered the limitation of a cloudformation module referencing modules in an s3 bucket, and since I need to edit those, this seems to be the only way to manage that.

There isn't any instructions on whether the cmmunication uses json or http in the github webhook config, or what to put in the secret form entry.

when I attempt to establish the connection using the url provided by GitPullWebHookApi

I get:

Headers
Content-Length: 37
Content-Type: application/json
Date: Sat, 17 Oct 2020 09:29:51 GMT
Via: 1.1 60e5707b1bd322c0a9qerh35h5h5esh51b57.cloudfront.net (CloudFront)
X-Amz-Apigw-Id: UjHuZEyeCYcFtVg=
X-Amz-Cf-Id: ewZpl_LU2bDTWXutf_yRuDUq0X3w54gw3w5hgw5h-fFUg==
X-Amz-Cf-Pop: SFO20-C1
X-Amzn-Errortype: BadRequestException
X-Amzn-Requestid: 0b1a73b1-8d01-4273-9212-5ed7e33fa448
X-Cache: Error from cloudfront
Body
{"message": "Unsupported Media Type"}

I have added the public ssh key to my github user's accepted ssh keys, so I'm not sure what else I might be missing.

Thanks for any help!

make exclude_git configurable

It would be useful to have the option to include the .git folder in the source archive. Currently it's hardcoded to exclude_git = True:
https://github.com/aws-quickstart/quickstart-git2s3/blob/master/lambdas/GitPullS3/lambda_function.py#L20

Adding additional IPs

After completing the quickstart I had an error in the GitPullLambda logs when code was committed to the specified repo stating that the source IP was not allowed.

The error was fixed and the repo successfully deployed to the S3 bucket when the GitHub Hooks IPs were added the the CloudFormation stacks allowed IPs.

The quickstart does point you to change the API Secret parameter settings in Step 2 but, doesn't explicitly say to add to the allowed IPs or the anything about aforementioned error that could occur.

Add support for in-VPC source control instances

This helps support other Quick Starts like Bitbucket Datacenter and GitHub Enterprise which are installed on instances in a VPC.

The changes should mostly involve a parameters to tell it to configure in VPC like list of subnets and create a security group based on a security group ID used for incoming traffic from the source control instances.

Github GitPullS3 issue - Failed to start SSH session: Unable to exchange encryption keys

I believe that recent changes to github have broken the GitPull method.

The error received is:

Failed to start SSH session: Unable to exchange encryption keys: GitError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 158, in lambda_handler
repo = create_repo(repo_path, remote_url, creds)
File "/var/task/lambda_function.py", line 64, in create_repo
repo = clone_repository(remote_url, repo_path, callbacks=creds )
File "/var/task/pygit2/__init__.py", line 255, in clone_repository
check_error(err)
File "/var/task/pygit2/errors.py", line 64, in check_error
raise GitError(message)
GitError: Failed to start SSH session: Unable to exchange encryption keys

Here are the changes announced by github:
https://github.com/blog/2507-weak-cryptographic-standards-removed

After reading through a similar issue in another project, it seems that the lambda runtime provides version 1.4 of the libssh2 dependency, yet version 1.8 is required.

“'full_name': KeyError” in GitPullLambda when processing BitBucket WebHook request

Full error message:

'full_name': KeyError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 142, in lambda_handler
repo_name = event['body-json']['repository']['full_name']
KeyError: 'full_name'

BitBucket POSTs a body like the one shown below. To me it seems the repo name would have to be read from event['body-json']['repository']['fullName'] (no underscore, upper-case N)

{
  "actor": {
    "username": "Joe.Average",
    "displayName": "Average, Joe"
  },
  "repository": {
    "scmId": "git",
    "project": {
      "key": "~JOE.AVERAGE",
      "name": "Average, Joe"
    },
    "slug": "demo",
    "links": {
      "self": [
        {
          "href": "https:\/\/bitbucket.exmaple.com\/users\/joe.average\/repos\/demo\/browse"
        }
      ]
    },
    "public": false,
    "fullName": "~JOE.AVERAGE\/demo",
    "ownerName": "~JOE.AVERAGE",
    "owner": {
      "username": "~JOE.AVERAGE",
      "displayName": "~JOE.AVERAGE"
    }
  },
  "push": {
    "changes": [
      {
        "created": false,
        "closed": false,
        "old": {
          "type": "branch",
          "name": "master",
          "target": {
            "type": "commit",
            "hash": "cd193abca3fc51336de56e75c0d3237a01d4e58d"
          }
        },
        "new": {
          "type": "branch",
          "name": "master",
          "target": {
            "type": "commit",
            "hash": "24dc6ec44645411b5b6ee7e925c66e740ada3ba9"
          }
        }
      }
    ]
  }
}

full_name': KeyError” in GitPullLambda when processing BitBucket WebHook request

Hi All,

I am facing the same problem as #14. We have an in-house bitbucket and trying to integrate the webhook with AWS service. While troubleshooting with AWS we found that the payload from the bitbucket is completely different from the one which the lambda function expects.

Lambda error : -

'project': KeyError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 158, in lambda_handler
full_name = event['body-json']['project']['path_with_namespace']
KeyError: 'project'

Bitbucket body sent : -

{
"eventKey":"repo:refs_changed",
"date":"2018-10-30T12:06:21+0000",
"actor":{
"name":"sathyanarayanan.ramamurthy",
"emailAddress":"[email protected]",
"id":269,
"displayName":"Sathyanarayanan Ramamurthy",
"active":true,
"slug":"sathyanarayanan.ramamurthy",
"type":"NORMAL"
},
"repository":{
"slug":"demo-aws-codedeploy",
"id":361,
"name":"demo-aws-codedeploy",
"scmId":"git",
"state":"AVAILABLE",
"statusMessage":"Available",
"forkable":true,
"project":{
"key":"IT",
"id":154,
"name":"Internal Testing",
"public":false,
"type":"NORMAL"
},
"public":false
},
"changes":[
{
"ref":{
"id":"refs/heads/master",
"displayId":"master",
"type":"BRANCH"
},
"refId":"refs/heads/master",
"fromHash":"ebb01dc966228ea93cc6036d9efe94f56480b89d",
"toHash":"3f8371baa548e516eee3d68b09fb4f608ff42920",
"type":"UPDATE"
}
]
}

local variable 'params' referenced before assignment

Deployed this stack for use with Bitbucket cloud.

2 things:

• the bitbucket cloud IPs have changed, which wasn't a big deal
• the ZipDlLambda throws this error when it runs:

local variable 'params' referenced before assignment: UnboundLocalError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 126, in lambda_handler
r = requests.get(archive_url, verify=verify, headers=headers, params=params)
UnboundLocalError: local variable 'params' referenced before assignment

Which makes this quickstart unusable as it is.

Can this please be fixed.
Thanks.

Resolve DNS for On prem Github

All,
I have the webhook setup, but when lambda tries to call the github address which is an internal DNS entry on prem it fails.

I'd like to understand how to get the DNS to resolve on prem. I attempted to setup something in route53, but it doesn't appear to work.

Update other lambdas to 3.x

This is a spin-off of #46

GitPullLambda, ZipDlLambda & DeleteBucketContents also need an update to 3.x

GitPullLambda "No space left on device"

Hello,

I'm having a hard time getting git2s3 to properly work for my GitHub repository. I've configured the Webhook through my repository settings on github.com, but GitHub is showing a 200 empty body response when I test a push.
Checking the lambda funciton logs, I see this error:

[Errno 28] No space left on device: IOErrorTraceback (most recent call last): File "/var/task/lambda_function.py", line 232, in lambda_handler zipfile = zip_repo(repo_path, repo_name) File "/var/task/lambda_function.py", line 109, in zip_repo zf.write(os.path.join(dirname, filename), os.path.join(zdirname, filename)) File "/usr/lib64/python2.7/zipfile.py", line 1225, in write self.fp.seek(zinfo.header_offset, 0)IOError: [Errno 28] No space left on device

The error sounds straightforward, but I'm not understanding what device it's referring to here, and how I can go about resolving it. I understand that this is more of a question on the usage rather than an issue, but I wasn't able to find a solution in the guide or anywhere online. Any insight is greatly appreciated!

Lambda functions handler is invalid - No module named lambda_function

This is a critical error. ZipDl lambda function add a folowing log entry when executed:

Unable to import module 'lambda_function': No module named lambda_function

This is happening, because source\lambda.zip contains the folder ZipDl which includes file lambda_function.py. But CloudFormation when creates Lambdas sets the Handler to be lambda_function.lambda_handler.

There are two options to fix this:

1. Edit each of the Lambdas via AWS Console and set Handler to be ZipDl/lambda_function.lambda_handler (use proper lambda fucntion name as a folder) - This is a workaround I'm currently using
2. Zip packages without including parent folder - I think this way it was planned.

@santiagocardenas might be related to your last commit

Not able to fetch the commit id inside my lambda_fuction

unable to fetch the commit id from gitlab inside my lambda function and tag the zip file along with commit into s3 bucket. please provide solution to this.

Can't pull master using Git Pull Endpoint and BitBucket

This exception is always thrown when using the Git Pull Endpoint with BitBucket

https://github.com/aws-quickstart/quickstart-git2s3/blob/master/functions/source/GitPullS3/lambda_function.py#L193

End effect is that master is always pulled regardless of which branch triggered the BitBucket webhook.

No IP range gives error

The error below is given when a blank entry is used for Allowed IPs, forcing execution to stop. When I put in a valid CIDR address, things seemed to process further.

u'' does not appear to be an IPv4 or IPv6 network: ValueError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 128, in lambda_handler
ipranges.append(ip_network(u'%s' % i))
File "/var/task/ipaddress.py", line 202, in ip_network
address)
ValueError: u'' does not appear to be an IPv4 or IPv6 network

Building libgit2 and pygit2

Hey, this isn't really an issue but would someone please be able to supply documentation on how libgit2 and pygit2 were built/installed. I can't quite get the linking working whilst trying to do something similar in my own deployment package so any info on this would be great.

Thanks!