LFS: Authorization error from Cloudfront template

[alanedwardes]

LFS: Authorization error from Cloudfront template

[alanedwardes]

Hi,

I will need a little more information to help debug this. Could you check the following:

1. Could you provide the console output from the LFS push?
2. Can you verify that your .lfsconfig file was committed into the root of the repository with the correct URL?
3. Is the Lambda function being invoked at all? Can you see any log streams in CloudWatch logs?

Thanks!

[debuggins]

Thanks for the quick response!

Below is the console output:

$ git push
Uploading LFS objects:   0% (0/1), 0 B | 18 KB/s, done                                                      
LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/eu-west-1/s3/aws4_request&X-Amz-Date=20180921T214543Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
error: failed to push some refs to '[email protected]:***.git'

My .lfsconfig and .gitattributes files are both committed into the root of my repository with a copy/paste of the output from CloudFront (LfsEndpoint).

Here's my .gitattributes file:

*.uasset filter=lfs diff=lfs merge=lfs -text
*.umap filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.tga filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.icns filter=lfs diff=lfs merge=lfs -text
*.ico filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.pdb filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text

I do see some CloudWatch Logs:

START RequestId: 94005ad2-bded-11e8-92fe-432357142bf8 Version: $LATEST
Incoming POST requests to /locks/verify
[Information] Microsoft.AspNetCore.Hosting.Internal.WebHost: Request starting POST https://***.execute-api.eu-west-1.amazonaws.com/locks/verify application/vnd.git-lfs+json; charset=utf-8 
[Debug] Microsoft.AspNetCore.Routing.Tree.TreeRouter: Request successfully matched the route with name '(null)' and template 'locks/verify'. 
[Debug] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executing action Estranged.Lfs.Api.Controllers.LocksController.Verify (Estranged.Lfs.Api) 
[Information] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executing action method Estranged.Lfs.Api.Controllers.LocksController.Verify (Estranged.Lfs.Api) with arguments ((null)) - ModelState is Valid 
[Debug] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executed action method Estranged.Lfs.Api.Controllers.LocksController.Verify (Estranged.Lfs.Api), returned result Microsoft.AspNetCore.Mvc.NotFoundResult. 
[Information] Microsoft.AspNetCore.Mvc.StatusCodeResult: Executing HttpStatusCodeResult, setting HTTP status code 404 
[Information] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executed action Estranged.Lfs.Api.Controllers.LocksController.Verify (Estranged.Lfs.Api) in 841.0929ms 
[Information] Microsoft.AspNetCore.Hosting.Internal.WebHost: Request finished in 5197.0132ms 404 
Response Base 64 Encoded: False
END RequestId: 94005ad2-bded-11e8-92fe-432357142bf8
REPORT RequestId: 94005ad2-bded-11e8-92fe-432357142bf8	Duration: 9572.50 ms	Billed Duration: 9600 ms Memory Size: 128 MB	Max Memory Used: 67 MB	
START RequestId: 9ac7e337-bded-11e8-a5a1-b55eaae88d22 Version: $LATEST
Incoming POST requests to /objects/batch
[Information] Microsoft.AspNetCore.Hosting.Internal.WebHost: Request starting POST https://***.execute-api.eu-west-1.amazonaws.com/objects/batch application/vnd.git-lfs+json; charset=utf-8 
[Debug] Microsoft.AspNetCore.Routing.Tree.TreeRouter: Request successfully matched the route with name '(null)' and template 'objects/batch'. 
[Debug] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executing action Estranged.Lfs.Api.Controllers.ObjectsController.BatchAsync (Estranged.Lfs.Api) 
[Debug] Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder: Selected input formatter 'Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter' for content type 'application/vnd.git-lfs+json; charset=utf-8'. 
[Information] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executing action method Estranged.Lfs.Api.Controllers.ObjectsController.BatchAsync (Estranged.Lfs.Api) with arguments (Estranged.Lfs.Api.Entities.BatchRequest) - ModelState is Valid 
[Debug] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executed action method Estranged.Lfs.Api.Controllers.ObjectsController.BatchAsync (Estranged.Lfs.Api), returned result Microsoft.AspNetCore.Mvc.ObjectResult. 
[Debug] Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor: Could not find an output formatter based on content negotiation. Accepted types were (application/vnd.git-lfs+json; charset=utf-8) 
[Debug] Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor: Selected output formatter 'Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter' and content type 'application/vnd.git-lfs+json' to write the response. 
[Information] Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor: Executing ObjectResult, writing value Microsoft.AspNetCore.Mvc.ControllerContext. 
[Information] Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker: Executed action Estranged.Lfs.Api.Controllers.ObjectsController.BatchAsync (Estranged.Lfs.Api) in 9702.5437ms 
[Information] Microsoft.AspNetCore.Hosting.Internal.WebHost: Request finished in 9881.2062ms 0 application/vnd.git-lfs+json; charset=utf-8 
Response Base 64 Encoded: False
END RequestId: 9ac7e337-bded-11e8-a5a1-b55eaae88d22
REPORT RequestId: 9ac7e337-bded-11e8-a5a1-b55eaae88d22	Duration: 9923.01 ms	Billed Duration: 10000 ms Memory Size: 128 MB	Max Memory Used: 81 MB	

Please let me know if there is any other information you need! Thanks!

[alanedwardes]

Thanks. I will investigate this tomorrow, that information is helpful.

[debuggins]

Here's the LFS env:

$ git lfs env
git-lfs/2.5.2 (GitHub; darwin amd64; go 1.11)
git version 2.15.1 (Apple Git-101)

Endpoint=https://***.execute-api.eu-west-1.amazonaws.com/lfs (auth=none)
LocalWorkingDir=/Users/***/Documents/Git/test-lfs-s3
LocalGitDir=/Users/***/Documents/Git/test-lfs-s3/.git
LocalGitStorageDir=/Users/***/Documents/Git/test-lfs-s3/.git
LocalMediaDir=/Users/***/Documents/Git/test-lfs-s3/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/***/Documents/Git/test-lfs-s3/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/***/Documents/Git/test-lfs-s3/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
git config filter.lfs.process = "git-lfs filter-process --skip"
git config filter.lfs.smudge = "git-lfs smudge --skip %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Here's the trace output for git push:

$ GIT_TRACE=1 git push
18:26:46.323680 git.c:344               trace: built-in: git 'push'
18:26:46.325059 run-command.c:626       trace: run_command: 'ssh' '[email protected]' 'git-receive-pack '\''***.git'\'''
18:26:47.920410 run-command.c:626       trace: run_command: '.git/hooks/pre-push' 'origin' '[email protected]:***.git'
18:26:47.930265 git.c:576               trace: exec: 'git-lfs' 'pre-push' 'origin' '[email protected]:***.git'
18:26:47.930892 run-command.c:626       trace: run_command: 'git-lfs' 'pre-push' 'origin' '[email protected]:***.git'
18:26:47.948538 trace git-lfs: exec: git 'version'
18:26:47.956773 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
18:26:47.963095 trace git-lfs: exec: git 'config' '-l'
18:26:47.966107 trace git-lfs: exec: git 'config' '-l' '-f' '/***/.lfsconfig'
18:26:47.969462 trace git-lfs: pre-push: refs/heads/master a5b13ff312c5d3a62202b2cbf9143dfad14c09b9 refs/heads/master 821fdf2b2cd79ab23bab133b6f84e2419c91b408
18:26:49.808451 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/locks/verify
18:26:50.181089 trace git-lfs: HTTP: 404
18:26:50.181363 trace git-lfs: commands: disabling lock verification for "https://***.execute-api.eu-west-1.amazonaws.com/lfs"
18:26:50.181395 trace git-lfs: exec: git 'config' '--replace-all' 'lfs.https://***.execute-api.eu-west-1.amazonaws.com/lfs.locksverify' 'false'
18:26:50.189166 trace git-lfs: tq: running as batched queue, batch size of 100
18:26:50.189450 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
18:26:50.193258 trace git-lfs: run_command: git cat-file --batch
18:26:50.197259 trace git-lfs: tq: sending batch of size 1                                                                                                 
18:26:50.197617 trace git-lfs: api: batch 1 files
18:26:50.197662 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:26:50.687432 trace git-lfs: HTTP: 200
18:26:50.687507 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:26:50.687577 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012650Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:26:50.687818 trace git-lfs: tq: starting transfer adapter "basic"
18:26:50.729083 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:26:52.562772 trace git-lfs: HTTP: 403B | 147 KB/s                                                                                                       
18:26:52.562957 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012650Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:26:52.563102 trace git-lfs: tq: enqueue retry #1 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:26:52.563166 trace git-lfs: tq: sending batch of size 1
18:26:52.563289 trace git-lfs: api: batch 1 files
18:26:52.563347 trace git-lfs: HTTP: POST https:/***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:26:52.752515 trace git-lfs: HTTP: 200
18:26:52.752568 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:26:52.752642 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012652Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:26:52.752950 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:26:54.573063 trace git-lfs: HTTP: 403B | 45 KB/s                                                                                                        
18:26:54.573251 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012652Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:26:54.573409 trace git-lfs: tq: enqueue retry #2 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:26:54.573476 trace git-lfs: tq: sending batch of size 1
18:26:54.573636 trace git-lfs: api: batch 1 files
18:26:54.573743 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:26:54.818008 trace git-lfs: HTTP: 200
18:26:54.818050 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:26:54.818110 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012654Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:26:54.818451 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:26:57.432551 trace git-lfs: HTTP: 403B | 59 KB/s                                                                                                        
18:26:57.432664 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012654Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:26:57.432730 trace git-lfs: tq: enqueue retry #3 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:26:57.432764 trace git-lfs: tq: sending batch of size 1
18:26:57.432850 trace git-lfs: api: batch 1 files
18:26:57.432898 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:26:57.603600 trace git-lfs: HTTP: 200
18:26:57.603651 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:26:57.603720 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012657Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:26:57.604147 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:26:59.422569 trace git-lfs: HTTP: 403B | 51 KB/s                                                                                                        
18:26:59.422980 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012657Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:26:59.423220 trace git-lfs: tq: enqueue retry #4 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:26:59.423259 trace git-lfs: tq: sending batch of size 1
18:26:59.423318 trace git-lfs: api: batch 1 files
18:26:59.423370 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:26:59.592283 trace git-lfs: HTTP: 200
18:26:59.592312 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:26:59.592353 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012659Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:26:59.592566 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:27:01.430804 trace git-lfs: HTTP: 403B | 28 KB/s                                                                                                        
18:27:01.430957 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012659Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:27:01.431141 trace git-lfs: tq: enqueue retry #5 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:27:01.431207 trace git-lfs: tq: sending batch of size 1
18:27:01.431293 trace git-lfs: api: batch 1 files
18:27:01.431357 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:27:01.881318 trace git-lfs: HTTP: 200
18:27:01.881351 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:27:01.881428 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012701Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:27:01.881664 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:27:03.724781 trace git-lfs: HTTP: 403B | 35 KB/s                                                                                                        
18:27:03.724882 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012701Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:27:03.725975 trace git-lfs: tq: enqueue retry #6 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:27:03.726021 trace git-lfs: tq: sending batch of size 1
18:27:03.726085 trace git-lfs: api: batch 1 files
18:27:03.726132 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:27:04.215796 trace git-lfs: HTTP: 200
18:27:04.215834 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:27:04.215888 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012704Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:27:04.216234 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:27:06.018708 trace git-lfs: HTTP: 403B | 34 KB/s                                                                                                        
18:27:06.018816 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012704Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:27:06.018895 trace git-lfs: tq: enqueue retry #7 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:27:06.018939 trace git-lfs: tq: sending batch of size 1
18:27:06.019006 trace git-lfs: api: batch 1 files
18:27:06.019063 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:27:06.667792 trace git-lfs: HTTP: 200
18:27:06.667837 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:27:06.667916 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012706Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:27:06.668376 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:27:08.529526 trace git-lfs: HTTP: 403B | 34 KB/s                                                                                                        
18:27:08.529675 trace git-lfs: tq: retrying object 632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8: LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012706Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
18:27:08.529763 trace git-lfs: tq: enqueue retry #8 for "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8" (size: 512163)
18:27:08.529792 trace git-lfs: tq: sending batch of size 1
18:27:08.529854 trace git-lfs: api: batch 1 files
18:27:08.529913 trace git-lfs: HTTP: POST https://***.execute-api.eu-west-1.amazonaws.com/lfs/objects/batch
18:27:08.710923 trace git-lfs: HTTP: 200
18:27:08.710972 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8","size":512163,"authenticated":true,"actions":{"upload":{"href":"https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***
18:27:08.711039 trace git-lfs: HTTP: ***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012708Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***","header":{},"expires_in":3600}}}]}
18:27:08.711476 trace git-lfs: HTTP: PUT https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8
18:27:10.547226 trace git-lfs: HTTP: 403B | 41 KB/s                                                                                                        
18:27:10.547335 trace git-lfs: tq: refusing to retry "632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8", too many retries (8)              
LFS: Authorization error: https://***-storagebucket-1pyzwinoc77u4.s3-eu-west-1.amazonaws.com/632e2a277cc122dd4793909783ebac9283eefe6104c8344d2c3c20d3125ec6b8?X-Amz-Expires=3600&x-amz-security-token=***&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***/20180922/eu-west-1/s3/aws4_request&X-Amz-Date=20180922T012708Z&X-Amz-SignedHeaders=content-type;host;x-amz-security-token&X-Amz-Signature=***
Check that you have proper access to the repository
Uploading LFS objects:   0% (0/1), 0 B | 41 KB/s, done
error: failed to push some refs to '[email protected]:***.git'

[alanedwardes]

Still investigating but a cursory look at this shows that it's the bucket rejecting the signed URL, which hints at an IAM permissions issue. A bit weird because the Lambda's role has full access to S3 - if you manually go to that bucket and try to upload/download a file to/from it via the console, does that work fine?

[debuggins]

Yep! A manual upload/download from the console works as expected.

[jdannenbring]

It seems I'm having the same issue with S3 not accepting the pre-signed urls. Upload to S3 bucket manually works fine. I'm getting the same behavior as debuggins when I run a GIT_TRACE.

My IAM permission policy looks like this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] }

[alanedwardes]

I tried pushing up the stack and cannot reproduce this, so I'm a little confused as to what is going on. I am wondering if it's an environmental difference with the git lfs version, or there's some AWS account difference meaning the signed URL won't work... I wonder if updating the S3 client will help at all. I will look to do that and push a new version.

What is your output from git lfs --version?

git lfs --version
git-lfs/2.4.2 (GitHub; windows amd64; go 1.8.3; git 6f4b2e98)

[jdannenbring]

git lfs --version
git-lfs/2.5.1 (GitHub; windows amd64; go 1.10.3; git e1628484)

[jdannenbring]

Its definitely related to the git lfs version. I went ahead and reverted my git lfs binary to your 2.4.2 version and the setup works now!

[alanedwardes]

Gotcha, will try updating mine to see if I can fix!

[debuggins]

Can confirm. By downgrading to git lfs 2.4.2, it now works.

[alanedwardes]

I just updated git + lfs and could not reproduce this which is super strange.

git lfs --version
git-lfs/2.5.1 (GitHub; windows amd64; go 1.10.3; git e1628484)

I will look into it more tomorrow -- seems slightly illusive!

[alanedwardes]

I have reproduced this - will get back to you with the fix.

[alanedwardes]

This is now fixed! In addition I updated the Lambda to the latest package versions, and the latest runtime. the problem was that in later versions of git-lfs it stopped implicitly passing a Content-Type header to S3, which meant the request did not match the expected request signature from the API.

To deploy a new stack, use the new magic link: https://console.aws.amazon.com/cloudformation/home#/stacks/new?region=eu-west-1&templateURL=https://ae-infrastructure-eu-west-1.s3-eu-west-1.amazonaws.com/git-lfs/2.1.0/git-lfs.yaml

Before doing any of the below I would advise backing up your S3 content locally or into a new bucket - just in case.

To update an existing Lambda in eu-west-1:

1. Head over the CloudFormation console, click on your stack's name, and click "Update Stack".
2. Pick "Specify an Amazon S3 template URL" and use https://ae-infrastructure-eu-west-1.s3-eu-west-1.amazonaws.com/git-lfs/2.1.0/git-lfs.yaml
3. Go through the wizard, hitting next and verifying your environment variables are retained
4. Deploy - the Lambda should be updated and work with git-lfs 2.5.1.

To update an existing Lambda in a different region:

1. Upload the Lambda deployment package to your own S3 bucket in the region you deploy to: https://github.com/alanedwardes/Estranged.Lfs/releases/tag/2.1.0
2. Download the CloudFormation template, and alter S3Bucket and S3Key to match your uploaded artifacts
3. Pick your stack in the CloudFormation console, click "Update Stack"
4. Pick "Upload a template to Amazon S3"
5. Go through the wizard, hitting next and verifying your environment variables are retained
6. Deploy - the Lambda should be updated and work with git-lfs 2.5.1.

Sorry if you're way ahead of those instructions, I don't know how technical you guys are! I will update my blog post with the new version in the near future. I'll also add the upgrade notes.

Any issues, let me know.

Alan

[debuggins]

When trying to create a new CloudFormation stack, If I try to change the region, the stack fails with:
Error occurred while GetObject. S3 Error Code: NoSuchBucket. S3 Error Message: The specified bucket does not exist (Service: AWSLambda; Status Code: 400; Error Code: InvalidParameterValueException;)

I tried adding a DependsOn: StorageBucket but that doesn't do anything. If I don't update the region, the stack is created successfully.

[jdannenbring]

That was the error I was getting, see the issue I closed. You'll need to host the lamda.zip file in a bucket on your region and point the yaml stack script to it.

[alanedwardes]

@debuggins did you manage to update the stack in the end?

As @jdannenbring said, to allow the stack to work you'll have to re-host the binaries. The steps for a brand new stack in a region other than eu-west-1 (I missed this path above):

1. Upload the Lambda deployment package to your own S3 bucket in the region you deploy to: https://github.com/alanedwardes/Estranged.Lfs/releases/tag/2.1.0
2. Download the CloudFormation template, and alter S3Bucket and S3Key to match your uploaded artifacts
3. In the CloudFormation console, click "Create Stack"
4. Pick "Upload a template to Amazon S3" and pick the altered template from above
5. Go through the wizard
6. Deploy - the Lambda should be updated and work with git-lfs 2.5.1.

[debuggins]

I was able to update the stack to point to another region. However, I've been getting other errors:

$ git push
Uploading LFS objects: 100% (1/1), 62 KB | 0 B/s, done                                                                                                                                    
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 406 bytes | 406.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".
To gitlab.com:***/***.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:***/***.git'

I get these errors using the magic link to set up an environment in EU as well as in the region of my choice. However, when I check the S3 bucket, it looks that the file has been successfully uploaded to S3. The file is not referenced in git's remote location.

Pushing LFS manually results in no error message:

$ git lfs push --all origin master
Uploading LFS objects: 100% (1/1), 62 KB | 0 B/s, done

But another git push after results in the first error message.

It might be useful to note that I've been creating new repositories from scratch for each test.

My environment:

$ git lfs env
git-lfs/2.5.1 (GitHub; darwin amd64; go devel +bd98a81dc2 Thu Jul 26 17:13:12 2018 +0000; git e1628484)
git version 2.15.1 (Apple Git-101)

Endpoint=https://***:***@***.execute-api.eu-west-1.amazonaws.com/lfs (auth=none)
LocalWorkingDir=/***
LocalGitDir=/***/.git
LocalGitStorageDir=/***/.git
LocalMediaDir=/***/.git/lfs/objects
LocalReferenceDirs=
TempDir=/***/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/***/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

[alanedwardes]

This is a GitLab specific issue when using a custom LFS backend, see https://stackoverflow.com/questions/47379344/gitlab-push-error-lfs-objects-are-missing

[debuggins]

I saw that when I was searching for a solution, however, previously, I was testing both GitHub and GitLab and had it working on both platforms when I had downgraded to git-lfs v2.4.2.

[debuggins]

@alanedwardes You're right. It is a GitLab-specific issue. Tried pushing it to GitHub and it appears to be working fine. Thanks for your help!

[alanedwardes]

I don’t quite understand why downgrading helped, but this rejection is coming from remote, so in this case the GitLab backend. You can see above that the LFS portion was successful, and as you found the object was pushed into S3.

Can you reproduce this with GitHub, BitBucket or another vendor?

[alanedwardes]

Ha, sorry, on mobile and was too slow! Would you be happy to close this issue?