andrba/eltemplito

El Templito - serverless template rendering and pdf generation

El Templito

El Templito is an AWS Lambda driven templating service. It consists of RenderTemplate and GeneratePdf document processing components that can work either independently or in a combination, depending on the request parameters. See API documentation

The Document Processing Unit demonstrates how by replacing AWS Step Functions with SNS-Lambda and sacrificing the reliability when it's acceptable, it is possible to reduce the cost of state transitioning by 62 times (as of Sep 2019).

RenderTemplate

Given a docx template file with optional MailMerge fields in it, it replaces the fields with values from a merge_fields parameter and saves the rendered docx document.

Template rendering is based on Sablon

GeneratePdf

Given an office document, e.g. docx or xlsx and output_format=pdf, it converts the document into a pdf format.

PDF generation is based on LibreOffice and libreoffice-lambda-layer

API documentation

https://app.swaggerhub.com/apis-docs/eltemplito/eltemplito/1

Once the POST /documents API request is accepted, the service starts a background process of document generation. The result of the process is then sent in a message to an SNS topic defined in /${Product}/${Environment}/DOCUMENT_CREATED_SNS_TOPIC SSM parameter.

The link to a generated document in the SNS message is short-lived and expires as soon as Lambda IAM session token expires. If a subscriber can not process the message in real-time, it is recommended to have a fallback strategy that will obtain the document by making a GET /documents/{$id} request to the same API endpoint.

Building

# Build Functions
rake build

# Build Layers
rake build_layers

# Build a specific layer
rake build_layers[shared]

Deploying

# Deploy
rake deploy[stack-name,environment,s3_bucket_name]

Development

Check out events that are passed to lambda functions.

# Run API locally on http://127.0.0.1:3000
sam local start-api --env-vars .env.json

# ListenDocumentStreamFunction - handling new requests (DynamoDB Insert operation)
sam local invoke --event spec/fixtures/events/listen_document_stream_insert.json --env-vars .env.json ListenDocumentStreamFunction

# ListenDocumentStreamFunction - handling finished or failed requests (DynamoDB Modify operation)
sam local invoke --event spec/fixtures/events/listen_document_stream_modify.json --env-vars .env.json ListenDocumentStreamFunction

# DispatchrFunction - handling StateChanged event with a non-empty pipeline
sam local invoke --event spec/fixtures/events/dispatchr_not_empty_pipeline.json --env-vars .env.json DispatchrFunction

# DispatchrFunction - handling StateChanged event with an empty pipeline
sam local invoke --event spec/fixtures/events/dispatchr_empty_pipeline.json --env-vars .env.json DispatchrFunction

# RenderTemplateFunction
sam local invoke --event spec/fixtures/events/render_template.json --env-vars .env.json RenderTemplateFunction

# GeneratePdfFunction
sam local invoke --event spec/fixtures/events/generate_pdf.json --env-vars .env.json GeneratePdfFunction

Testing

Integration

BUNDLE_IGNORE_CONFIG=1 bundle install --with test shared create_document listen_document_stream dispatchr render_template generate_pdf
bundle exec rspec

Manual end to end testing

A simple end to end test can be performed by subscribing an email address to the DocumentCreated topic.

aws sns subscribe --topic-arn <DocumentCreatedSNSTopicArn> --protocol email --notification-endpoint <[email protected]>

AWS will send an email to confirm your subscription. Once the subscription is confirmed, make a POST request to the service endpoint:

curl -X POST \
  <ENDPOINT_URL>/<ENVIRONMENT>/documents \
  -H 'Content-Type: application/json' \
  -d '{
    "file_url": "https://github.com/andrba/eltemplito/raw/master/spec/fixtures/template.docx",
	  "output_format": "pdf",
	  "merge_fields": {
		  "content": "The best things in life are not things"
	  }
  }'

eltemplito's Issues

Add FailDocument lambda function

Background:

Currently, if any of the Document Processing functions fails, the failure is not captured and the event gets lost.

Required:

Implement failure handling in accordance with the service diagram.

Add acceptance specs

Add acceptance specs. The specs should be executed against the deployed stack.

Add integration specs

Add integration specs to all functions. The specs should test the integration of events, the business logic and AWS services comprising each individual function.

Add GetDocument lambda function

Background:

Pre-signed links to the generated documents have a 1 hour TTL. If a consumer of the service can not process the event within that time the event will contain an expired link.

Setting TTL to a maximum value of 7 days does not solve the problem, as the link signature is coupled with Lambda session tokens that by definition are short-lived.

Required:

Allow the consumer to obtain the generated document by making a GET request to /documents/{id}, where id is the identifier of the request that was returned in the outgoing SNS message. The schema of the response should be the same as the schema of the original SNS message.