pg module not compatible with the latest version of python (python 3.8)

[aws-samples]

pg module not compatible with the latest version of python (python 3.8)

[0xSeb]

Fixed it with psycopg2, if people are still interested : https://github.com/0xSeb/aws_secrets_manager_psql_rotation_lambda

[frugecn]

The code in this repo is ok as far as python 3.8 compatibility; the problem lies in the pg driver pygresql and how it is compiled. If you download it from the Lambda site, it is compiled in python 3.7. If you try to compile it yourself in python 3.8 and try to use the postgreSQL 12 or 13 client files, it gets really complicated as the dependencies required and what's available on the Lambda are not the same.

[adampblack]

Got it working using psycopg2.

I will leave this open so that someone at AWS can update the sample.

[JoeJesse]

Thank you for opening this issue. Currently the rotation AWS Lambda functions we vend only officially support Python 3.7, as noted in our public documentation. This is in part due to the fact that we build and package all the necessary dependencies, along with the function itself, in the Lambda deployment package. As it pertains to PostgreSQL, the specific client library we use is PyGreSQL. We have a similar feature request to upgrade the officially supported Python runtime in our vended rotation Lambdas, and by association the versions of client libraries used as well.

[dalvarezquiroga]

+1 Yes, it looks like that all code was created in 2018, you can see in header of scripts. I don't know if it will be updated.

[smcoll]

Yeah... that pg package is for OpenGL and installs pillow?

[joebaro]

Thank you for opening this issue - we are looking into it.

[ctrawick]

Pillow happened to me when I tried to repackage lambda code from here into an independent layer using requirements.txt. DO NOT REFERENCE "PG" THERE. The proper name there is "PyGreSQL" which presents a "pg" module from PostgreSQL, not PIL/OpenGL. The default "pg" will break your lambdas.

I ran into this because I was forced to reimplement the public template using terraform because the serverlessrepo package for this does not allow the attachment of required permissions boundaries. If you want this fixed, please upvote #27

[dcalde]

Please get rid of pgdb and pg and just use psycopg2 or pg8000 as used by AWS Datawrangler.
Why are random libraries used in aws samples?

[ReyanshKharga]

Got it working using psycopg2.

I will leave this open so that someone at AWS can update the sample.

Could you please tell me the changes you made?

[kangsheng89]

U can do it with psycopg2, where u can follow as below:
1st, build the docker image, where this will prepare for the linux environment, using pip to download the pre-compile lib.

FROM python:3.8 as py

ARG PSYCOPG2_VER

RUN apt-get update && \
        apt-get install bison flex build-essential libpq-dev -y 

WORKDIR /package/psycopg2
RUN python -m venv venv
RUN . venv/bin/activate && \
        pip install psycopg2-binary==$PSYCOPG2_VER --target /out_lib
WORKDIR /

then use docker cp to get the binary from container, repackage it with lambda "handler.py"

.PHONY: help


help: ## This help.
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

.DEFAULT_GOAL := package
APP_NAME = postgres_env
PSYCOPG2_VER = 2.9.3

# DOCKER TASKS

# Build the container
build: clean
	docker build -t $(APP_NAME) .  --build-arg PSYCOPG2_VER=$(PSYCOPG2_VER)

## Build the container without caching
build-nc: clean
	docker build --no-cache -t $(APP_NAME) . --build-arg PSYCOPG2_VER=$(PSYCOPG2_VER)

clean: 
	rm -rf out
	rm -rf deployment.zip

clean-image: 
	docker rmi -f $(APP_NAME)

package: build
	mkdir -p out
	$(eval CONTAINER_ID=$(shell docker create $(APP_NAME) --name $(APP_NAME)))
	docker cp $(CONTAINER_ID):/out_lib/. ./out
	docker rm $(CONTAINER_ID)
	cd out && zip -r ../deployment.zip .
	zip -g deployment.zip handler.py

run make, and it will generate the "deployment.zip" and u can upload this to lambda

[biddster]

We got stung by this earlier this week.

We updated our cloudformation stack (with no changes to the PG password rotation lambdas) and it appears that our lambda functions were auto-updated to 3.9?

I've no idea why the lambda functions were updated yet as there was no change to them. Still looking into that. However, there's also no way in the serverless template to be able to insist on 3.7.

I guess I'll need a CustomResource with a bit of python/javascript to keep those lambda functions at 3.7?

[jbct]

In Q3/Q4 2022, Secrets Manager updated the python runtimes to v3.9. In certain circumstances, customers may have had a ZIP bundle that still included the 3.7 runtimes even though the Lambda runtime was updated to 3.9, causing some incompatibilities. We have exposed the Runtime attribute in the SAM template and have a blog available for impacted customers here: https://repost.aws/knowledge-center/secrets-manager-lambda-rotation. This particular issue has been resolved through documentation, so closing issue.