Bug: Unable to run sam local invoke on Apple Silicon

[nataizya]

Bug: Unable to run sam local invoke on Apple Silicon

[aquanow-jeffen]

Local invocation works for me after change to Architectures: - arm64. I had to remove all the containers before run the invoke command.

[mndeveci]

I think we found the root cause (thanks to @lucashuy).

There was a change in cargo-lambda package which is used for building Rust functions under the hood. This PR changed the default target cpu (microarch) to be haswell. It looks like this is causing issues when executable built with that flag. If it is changed to x86_64 then it works fine.

However the reason for the change above was to make it more optimized for the target lambda runtime. So this code works when it is deployed (as intended) however when it is tried to be emulated on Apple Silicon machines, it doesn't work (possible) missing support from Docker Desktop.

We don't want to change default target since this was selected due to optimizations, for now we would ask customers to use workaround to build with SAM_BUILD_MODE=debug sam build when testing their lambda functions locally.

[mndeveci]

Thanks for reporting this issue.

Can you try changing your runtime to provided.al2023 to see if it works? I think there was a change in the rust builder we used under the hood, they changed default runtime settings to be aligned with newer runtimes, which broke the provided.al2 ones.

We've merged a fix to align with the change above, however it is not been released yet. If you confirm that changing to newer runtime works, we can assume that new SAM CLI release should fix the issue.

[nataizya]

Thanks for responding @mndeveci.

I did try it with provided.al2023 as well and faced the same issue.

Logs:

2024-05-13 23:44:07,411 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml        
2024-05-13 23:44:07,415 | Loading configuration values from [default.['local', 'invoke'].parameters]                          
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml'...      
2024-05-13 23:44:07,416 | Configuration values successfully loaded.                                                           
2024-05-13 23:44:07,416 | Configuration values are: {'stack_name': 'sam-app-2'}                                               
2024-05-13 23:44:07,420 | Using SAM Template at                                                                               
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml                                          
2024-05-13 23:44:07,480 | Using config file: samconfig.toml, config environment: default                                      
2024-05-13 23:44:07,481 | Expand command line arguments to:                                                                   
2024-05-13 23:44:07,481 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost                               
--container_host_interface=127.0.0.1                                                                                          
2024-05-13 23:44:07,482 | local invoke command is called                                                                      
2024-05-13 23:44:07,486 | No Parameters detected in the template                                                              
2024-05-13 23:44:07,505 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:44:07,506 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:44:07,507 | 0 stacks found in the template                                                                      
2024-05-13 23:44:07,508 | No Parameters detected in the template                                                              
2024-05-13 23:44:07,524 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:44:07,524 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:44:07,525 | 2 resources found in the stack                                                                      
2024-05-13 23:44:07,526 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'           
2024-05-13 23:44:07,526 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                           
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml                                          
2024-05-13 23:44:07,539 | Found one Lambda function with name 'HelloWorldFunction'                                            
2024-05-13 23:44:07,540 | Invoking bootstrap (provided.al2023)                                                                
2024-05-13 23:44:07,540 | No environment variables found for function 'HelloWorldFunction'                                    
2024-05-13 23:44:07,541 | Loading AWS credentials from session with profile 'None'                                            
2024-05-13 23:44:07,548 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,    
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                             
2024-05-13 23:44:07,549 | Resolved absolute path to code is                                                                   
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                                     
2024-05-13 23:44:07,550 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,    
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                             
2024-05-13 23:44:07,550 | Resolved real code path to                                                                          
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                                     
2024-05-13 23:44:07,551 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction is   
not a zip/jar file                                                                                                            
2024-05-13 23:44:13,273 | Local image is up-to-date                                                                           
2024-05-13 23:44:13,292 | Checking free port on 127.0.0.1:8347                                                                
2024-05-13 23:44:13,299 | Using local image: public.ecr.aws/lambda/provided:al2023-rapid-x86_64.                              
                                                                                                                              
2024-05-13 23:44:13,301 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction  
as /var/task:ro,delegated, inside runtime container                                                                           
2024-05-13 23:44:13,698 | Starting a timer for 3 seconds for function 'HelloWorldFunction'                                    
2024-05-13 23:44:13,701 | Getting lock for the key localhost-8347                                                             
2024-05-13 23:44:13,702 | Waiting to retrieve the lock (localhost-8347) to start invocation                                   
START RequestId: 55fe8a55-ca1d-436c-a3ed-eec2c3fda782 Version: $LATEST
13 May 2024 21:44:13,759 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:44:13,762 [ERROR] (rapid) Invoke failed InvokeID=261e53eb-7616-45bf-96fc-b24359daf3d3 error=Runtime exited with error: signal: illegal instruction
13 May 2024 21:44:13,763 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:44:13,767 | Failed to deserialize response from RIE, returning the raw response as is                           

2024-05-13 23:44:14,790 | [Container state] OOMKilled False                                                                   
2024-05-13 23:44:14,925 | Cleaning all decompressed code dirs                                                                 
2024-05-13 23:44:14,927 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:44:15,027 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:44:15,028 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'dc3cb305-75d5-4d40-b3b7-d281511cda09',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,   
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '9e7669abfc305db5598caf057dd1af7044b0710b8c9f99f979d86cc7bb8b69ca', 'initialCommit': None}, 
'duration': 7446, 'exitReason': 'success', 'exitCode': 0}}]}                                                                  
2024-05-13 23:44:15,028 | Unable to find Click Context for getting session_id.                                                
2024-05-13 23:44:15,032 | Sending Telemetry: {'metrics': [{'events': {'requestId': '48b8995e-d5d3-43ca-9aba-def06169a914',    
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local      
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'4a072847aba84d9eae8720975249e7d7', 'time_stamp': '2024-05-13 21:44:07.411', 'exception_name': None}, {'event_name':          
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'd0e7eb37fe8247958066a2e7d2574cd1', 'time_stamp': '2024-05-13  
21:44:07.480', 'exception_name': None}]}}}]}                                                                                  
2024-05-13 23:44:15,977 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:44:15,978 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:44:15,986 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:44:15,988 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':                                     
'd840b3ea-0bfe-4bde-8fe8-33eaee4876d5', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':                
'699a40ef-6b4d-474d-8610-c8a5640b79f5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':   
'1.116.0', 'runtimes': ['provided.al2023']}}]}                                                                                
2024-05-13 23:44:16,792 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
nataizya@Nataizyas-MacBook-Air sam-app-2 % cd ../sam-app
nataizya@Nataizyas-MacBook-Air sam-app % sam local invoke --debug 
2024-05-13 23:46:14,648 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml          
2024-05-13 23:46:14,651 | Loading configuration values from [default.['local', 'invoke'].parameters]                          
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml'...        
2024-05-13 23:46:14,652 | Configuration values successfully loaded.                                                           
2024-05-13 23:46:14,652 | Configuration values are: {'stack_name': 'sam-app'}                                                 
2024-05-13 23:46:14,656 | Using SAM Template at                                                                               
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml                                            
2024-05-13 23:46:14,702 | Using config file: samconfig.toml, config environment: default                                      
2024-05-13 23:46:14,703 | Expand command line arguments to:                                                                   
2024-05-13 23:46:14,704 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml  
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost                               
--container_host_interface=127.0.0.1                                                                                          
2024-05-13 23:46:14,705 | local invoke command is called                                                                      
2024-05-13 23:46:14,708 | No Parameters detected in the template                                                              
2024-05-13 23:46:14,727 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:46:14,728 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:46:14,729 | 0 stacks found in the template                                                                      
2024-05-13 23:46:14,730 | No Parameters detected in the template                                                              
2024-05-13 23:46:14,746 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:46:14,746 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:46:14,747 | 2 resources found in the stack                                                                      
2024-05-13 23:46:14,748 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'           
2024-05-13 23:46:14,748 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                           
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml                                            
2024-05-13 23:46:14,779 | Found one Lambda function with name 'HelloWorldFunction'                                            
2024-05-13 23:46:14,779 | Invoking bootstrap (provided.al2)                                                                   
2024-05-13 23:46:14,780 | No environment variables found for function 'HelloWorldFunction'                                    
2024-05-13 23:46:14,780 | Loading AWS credentials from session with profile 'None'                                            
2024-05-13 23:46:14,788 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,      
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                               
2024-05-13 23:46:14,789 | Resolved absolute path to code is                                                                   
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                                       
2024-05-13 23:46:14,790 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,      
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                               
2024-05-13 23:46:14,790 | Resolved real code path to                                                                          
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                                       
2024-05-13 23:46:14,791 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction is not 
a zip/jar file                                                                                                                
2024-05-13 23:46:20,664 | Local image is up-to-date                                                                           
2024-05-13 23:46:20,685 | Checking free port on 127.0.0.1:5726                                                                
2024-05-13 23:46:20,695 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.                                 
                                                                                                                              
2024-05-13 23:46:20,697 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction as 
/var/task:ro,delegated, inside runtime container                                                                              
2024-05-13 23:46:20,949 | Starting a timer for 3 seconds for function 'HelloWorldFunction'                                    
2024-05-13 23:46:20,953 | Getting lock for the key localhost-5726                                                             
2024-05-13 23:46:20,955 | Waiting to retrieve the lock (localhost-5726) to start invocation                                   
2024-05-13 23:46:21,068 | Getting lock for the key localhost-5726                                                             
2024-05-13 23:46:21,072 | Waiting to retrieve the lock (localhost-5726) to start invocation                                   
START RequestId: 2d02aba6-328d-4a67-b94f-91b4058b407e Version: $LATEST
13 May 2024 21:46:21,114 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:46:21,118 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=3bde647e-a338-4ff7-b860-7e24e6d8ff10
13 May 2024 21:46:21,120 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:46:21,122 | Failed to deserialize response from RIE, returning the raw response as is                           

2024-05-13 23:46:22,157 | [Container state] OOMKilled False                                                                   
2024-05-13 23:46:22,288 | Cleaning all decompressed code dirs                                                                 
2024-05-13 23:46:22,291 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:22,404 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:22,406 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '88d70caf-438d-43c5-bdb9-cd8b910a8459',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,   
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None}, 
'duration': 7588, 'exitReason': 'success', 'exitCode': 0}}]}                                                                  
2024-05-13 23:46:22,407 | Unable to find Click Context for getting session_id.                                                
2024-05-13 23:46:22,411 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a5e10d43-7cbc-43b6-be11-7dabe9d0a741',    
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local      
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'ba1472d1be8a4258bce10fcb3747afbc', 'time_stamp': '2024-05-13 21:46:14.647', 'exception_name': None}, {'event_name':          
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '4d7d2b65326940d7aebc4265646dfc81', 'time_stamp': '2024-05-13  
21:46:14.702', 'exception_name': None}]}}}]}                                                                                  
2024-05-13 23:46:23,369 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:46:23,375 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:46:23,379 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:23,381 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':                                     
'a53adc53-679e-478a-9032-a01c08bee7d2', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':                
'eab61e43-2617-4a4b-8b64-239884235d30', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':   
'1.116.0', 'runtimes': ['provided.al2']}}]}                                                                                   
2024-05-13 23:46:24,182 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)

[mndeveci]

Thanks for confirmation!

Can you also confirm what happens when you deploy this function? As far as we can see, when this function is deployed it works as expected, however running locally fails with the error that you mentioned above.

From our side we validated following scenarios;

• A: Building on Apple Silicon Macbook works (sam build), locally invoking fails (sam local invoke)
• B: Building on Intel Macbook works (sam build), locally invoking succeeds (sam local invoke)
• C: Deploying the artifact from A works as expected, deployed lambda function can be invoked
• D: Using the built artifact from A on an Intel Macbook works for sam local invoke

[mndeveci]

@nataizya we've tried with a Docker alternative, and it worked without issues. I believe this is coming from Docker Desktop's Apple Silicon distribution since it also works on Intel machines.

[nataizya]

I suspect that the deployment will work as well though I havent tested that.

Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?

[mndeveci]

Before going further down the list, you may try to uninstall/remove Docker Desktop completely and install it from their official installers, to see if it is going to fix your problem (I've seen that sometimes a clean installation fixes their issues).

I can't open an issue on their repository since they are asking more details which I can't provide since I am using an Intel Macbook. If you can, I will recommend creating an issue in this repository here; https://github.com/docker/for-mac/issues

Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?

There are alternative tools which can replace Docker Desktop, and they provide similar experience to use containers on your machine. Some of those are podman or rancher.

[nataizya]

Setting the architecture to arm64 allows it to work. Thanks!!!

[mndeveci]

Please keep in mind that, if you change Architecture value in the template.yaml it will also change the architecture of the deployed function. Architecture of the developer's local machine can be different from what they are planning to deploy in their account. In this case, I believe Docker Desktop for Apple Silicon has some issues with executing cross architecture code.

[nataizya]

Noted. I will try and make use of podman or rancher. But for now I can at least test the function locally by temporarily switching to arm64 in the template.yaml.

[mndeveci]

As another workaround (which might be better since this doesn't require architecture change), you can also build Rust x86 function with SAM_BUILD_MODE=debug sam build which will build in debug mode, and it looks like it works with sam local invoke.

!!! Please don't use SAM_BUILD_MODE=debug when doing a deployment as it removes the --release flag for Rust compiler. !!!

[nataizya]

Thanks @mndeveci .

I can confirm that using SAM_BUILD_MODE=debug sam build does work for local testing.