Boto3 and botocore need upgrading

[mdavis-xyz]

Boto3 and botocore need upgrading

[thomas-icomplai]

is there a way to upgrade this yourself?

[mdavis-xyz]

There's probably a way.
Note that the normal image doesn't include pip. The build one does though.

Note also that the version of botocore and boto3 in lambda tends to lag behind the latest version of the libraries. So you can't simply upgrade to the latest, you must pin the versions. I was hoping that's something that this project would handle for me, by just copy-pasting the files from the Lambda runtime regularly.

[thomas-icomplai]

I worked around this now with the wonderful blog post from https://www.mandsconsulting.com/lambda-functions-with-newer-version-of-boto3-than-available-by-default/

effectively just boils down to prioritize the pip installed packages before importing them:

if "LAMBDA_TASK_ROOT" in os.environ:
    envLambdaTaskRoot = os.environ["LAMBDA_TASK_ROOT"]
    sys.path.insert(0, "/var/lang/lib/python3.9/site-packages")

import boto3
import botocore

It's a real shame that the runtime does not do this itself before executing the script.

[Homas]

You can add a layer with updated packages
On Linux:

mkdir -p lambda_layer/python
cd lambda_layer/python
pip install boto3 -t ./
pip install botocore -t ./
cd ..
zip -r ~/lambda_layer.zip .

after that upload lambda_layer.zip as a layer to your function (it is relatively small so no need to store on S3)

[mdavis-xyz]

From memory boto3 and botocore and not relatively small. They're like 60MB, right? So that's 1/8th of the size quota gone just to the AWS SDK. If you're trying to also squeeze in some other library like matplotlib or awswrangler, that's going to pose a challenge for real deployments.

Although this project is for unit testing. so size isn't an issue. the issue is just that the docker container doesn't accurately reflect the environment its emulating. Installing the latest version of boto3 and botocore won't either, since real lambda tends to lag the latest SDK versions.

Also, you should probably install boto3 and botocore in one single pip command, to ensure the two versions are compatible with each other.