This repository is part of the Find Case Law project at The National Archives. For more information on the project, check the documentation.
This is the repository for the lambda function used to parse Transformation Engine judgments and insert them to Marklogic
We're using localstack, along with the awslocal-cli to enable local development of the lambda function.
An installation of make is required to use the bundled Makefile for local development. Most operating systems come with this preinstalled, including Ubuntu Linux and MacOS. On Windows, Make can be installed via the Chocolatey package manager, or using the Windows Subsystem for Linux (WSL).
You will also need both awscli and awslocal-cli installed. awslocal-cli is a Localstack-specific wrapper around awscli.
Install both from the requirements file using:
python3 -m pip install -r requirements/local.txtFirst, copy .env.example to .env and fill in the missing variables. If you are using Localstack via Docker, leave MARKLOGIC_HOST as host.docker.internal.
Then, start Localstack using:
docker-compose up -dThis will start Localstack in detached mode; logs are accessible via Docker Desktop.
Once the docker container is running, use the following make command to build a distribution of the lambda function, and setup the localstack AWS services
make setupThis will create a folder, dist, on your local machine that contains a zip file called lambda.zip - this is our compiled lambda. You can also upload this directly to the AWS console.
To send the example message bundled, use the send-message-v2 make target:
make send-message-v2This will publish a message to the SNS topic, triggering the handle function in our lambda.
(send-message-v1 exists, and sends a v1 message.)
The lambda output will be logged in the Localstack logs. Look for the lines following:
localstack.services.awslambda.lambda_executors: Lambda arn:aws:lambda:us-east-1:000000000000:function:te-lambda result / log output:
The logs will show the response from the lambda directly below this line. Any values sent to stdout (e.g. print statements), will be output beneath.
To run the tests
- [First time] create a virtualenv (
virtualenv venv -p \which python`` ) - Activate it with
. venv/bin/activate scripts/test- When you're done, you might want to
deactivate
Note that you might get a spurious errors about django config and environment variables if you're running in the wrong environment.
If you make a change to the code and need to update the lambda function, use the update make command:
make updateAnd then send a message:
make send-message-v2To test a tarfile locally:
- Add your test tarfile to
aws_examples/s3/te-editorial-out-int. - Edit
aws_examples/sns/parsed-judgment.jsonto contain your tarfile name ins3-folder-urland consignment reference inconsignment-reference. - Run
make setup aws_examples/s3/te-editorial-out-int/<your tarfile>, for examplemake setup aws_examples/s3/te-editorial-out-int/XYZ-123.tar.gz. If you runmake setupwithout an argument, the original test tarfileTDR-2022-DNWR.tar.gzwill be used - Run
make send-message-v2to ingest your tarfile.
Every change to the main branch is automatically deployed to the staging environment via GitHub actions.
Only releases are deployed to production. To trigger a deploy, create a new release named and tagged vX.Y.Z following semantic versioning. Autogenerate release notes, and publish; the release will then be tagged latest automatically and deployed to production.
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent