Giter Site home page Giter Site logo

eltemplito's Introduction

El Templito

Screen Shot 2019-09-09 at 1 53 18 pm

El Templito is an AWS Lambda driven templating service. It consists of RenderTemplate and GeneratePdf document processing components that can work either independently or in a combination, depending on the request parameters. See API documentation

The Document Processing Unit demonstrates how by replacing AWS Step Functions with SNS-Lambda and sacrificing the reliability when it's acceptable, it is possible to reduce the cost of state transitioning by 62 times (as of Sep 2019).

RenderTemplate

Given a docx template file with optional MailMerge fields in it, it replaces the fields with values from a merge_fields parameter and saves the rendered docx document.

Template rendering is based on Sablon

GeneratePdf

Given an office document, e.g. docx or xlsx and output_format=pdf, it converts the document into a pdf format.

PDF generation is based on LibreOffice and libreoffice-lambda-layer

API documentation

https://app.swaggerhub.com/apis-docs/eltemplito/eltemplito/1

Once the POST /documents API request is accepted, the service starts a background process of document generation. The result of the process is then sent in a message to an SNS topic defined in /${Product}/${Environment}/DOCUMENT_CREATED_SNS_TOPIC SSM parameter.

The link to a generated document in the SNS message is short-lived and expires as soon as Lambda IAM session token expires. If a subscriber can not process the message in real-time, it is recommended to have a fallback strategy that will obtain the document by making a GET /documents/{$id} request to the same API endpoint.

Building

# Build Functions
rake build

# Build Layers
rake build_layers

# Build a specific layer
rake build_layers[shared]

Deploying

# Deploy
rake deploy[stack-name,environment,s3_bucket_name]

Development

Check out events that are passed to lambda functions.

# Run API locally on http://127.0.0.1:3000
sam local start-api --env-vars .env.json

# ListenDocumentStreamFunction - handling new requests (DynamoDB Insert operation)
sam local invoke --event spec/fixtures/events/listen_document_stream_insert.json --env-vars .env.json ListenDocumentStreamFunction

# ListenDocumentStreamFunction - handling finished or failed requests (DynamoDB Modify operation)
sam local invoke --event spec/fixtures/events/listen_document_stream_modify.json --env-vars .env.json ListenDocumentStreamFunction

# DispatchrFunction - handling StateChanged event with a non-empty pipeline
sam local invoke --event spec/fixtures/events/dispatchr_not_empty_pipeline.json --env-vars .env.json DispatchrFunction

# DispatchrFunction - handling StateChanged event with an empty pipeline
sam local invoke --event spec/fixtures/events/dispatchr_empty_pipeline.json --env-vars .env.json DispatchrFunction

# RenderTemplateFunction
sam local invoke --event spec/fixtures/events/render_template.json --env-vars .env.json RenderTemplateFunction

# GeneratePdfFunction
sam local invoke --event spec/fixtures/events/generate_pdf.json --env-vars .env.json GeneratePdfFunction

Testing

Integration

BUNDLE_IGNORE_CONFIG=1 bundle install --with test shared create_document listen_document_stream dispatchr render_template generate_pdf
bundle exec rspec

Manual end to end testing

A simple end to end test can be performed by subscribing an email address to the DocumentCreated topic.

aws sns subscribe --topic-arn <DocumentCreatedSNSTopicArn> --protocol email --notification-endpoint <[email protected]>

AWS will send an email to confirm your subscription. Once the subscription is confirmed, make a POST request to the service endpoint:

curl -X POST \
  <ENDPOINT_URL>/<ENVIRONMENT>/documents \
  -H 'Content-Type: application/json' \
  -d '{
    "file_url": "https://github.com/andrba/eltemplito/raw/master/spec/fixtures/template.docx",
	  "output_format": "pdf",
	  "merge_fields": {
		  "content": "The best things in life are not things"
	  }
  }'

eltemplito's People

Contributors

andrba avatar

Watchers

 avatar  avatar

eltemplito's Issues

Add FailDocument lambda function

Background:

Currently, if any of the Document Processing functions fails, the failure is not captured and the event gets lost.

Required:

Implement failure handling in accordance with the service diagram.

Add acceptance specs

Add acceptance specs. The specs should be executed against the deployed stack.

Add integration specs

Add integration specs to all functions. The specs should test the integration of events, the business logic and AWS services comprising each individual function.

Add GetDocument lambda function

Background:

Pre-signed links to the generated documents have a 1 hour TTL. If a consumer of the service can not process the event within that time the event will contain an expired link.

Setting TTL to a maximum value of 7 days does not solve the problem, as the link signature is coupled with Lambda session tokens that by definition are short-lived.

Required:

Allow the consumer to obtain the generated document by making a GET request to /documents/{id}, where id is the identifier of the request that was returned in the outgoing SNS message. The schema of the response should be the same as the schema of the original SNS message.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.