iot-ingest-large-size's Introduction

IoT Ingest Large-Sized Payloads

Securely ingest large sized payloads from IoT devices using AWS.

Description

This project is mainly based on this AWS Blog which leverages the power of S3 pre-signed URLs, MQTT and IoT Core to upload large files. Once the object is uploaded, the S3 Notification triggers a Lambda which sends an email to view the file.

Getting Started

Dependencies

An AWS Account
- Generate the certificate needed to authenticate the IoT device(s). The following are needed:
  - Amazon Root CA certificate file (Amazon-root-CA-1.pem)
  - Private key file (private.pem.key)
  - A certificate for this thing (device.pem.crt)
- Obtain the device-data-endpoint. Looks similar to account-specific-prefix.jobs.iot.aws-region.amazonaws.com.
- Register your email address for the SES to be able to send emails.
An IoT device (eg. a Raspberry Pi).
Terraform

Terraform setup

Clone this repository on the laptop/desktop/server

git clone https://github.com/vikramaditya91/iot-ingest-large-size.git

Get the ARN of the certificate and set it in the terraform/variables.tf
Adjust other variables in terraform/variables.tf as required
Apply the terraform configuration:

cd terraform
terraform apply

Set-up on IoT device

Copy the certificates onto the IoT device
Clone this repository on the IoT device

git clone https://github.com/vikramaditya91/iot-ingest-large-size.git

Create a virtual environment of Python3+
Install the dependencies listed in pip_requirements.txt
Set the correct values in send_large_file.py
- paths to the certificates
- bucket-name as set in variables.tf
- end-point url as obtained above
- request url topic as set in variables.tf
Run the script python send_large_file.py <full_path_to_file_to_be_uploaded>

Additional information

This project was originally meant for a motion detection system for when my cat Doudou wanted to be let in
Motion was detected using the Motion Project