dzimine/greengo

Simple YAML-based deployment framework for AWS Greengrass.

Infra as Code for AWS IoT Greengrass

Greengo: a tool, and a starter boilerplate project to bring up (and clean-up!) AWS Greengrass setup for play and profit. If you followed the GreenGrass Getting Started Guide, here you find it automated, as code.

Describe your Greengrass group in group.yaml, write Lambda functions and device clients, provision Greengrass Core in Vagrant VM, deploy, and clean up.

Inspired by aws-iot-elf (Extremely Low Friction) and aws-greengrass-group-setup.

Pre-requisits

• A computer with Linux/MacOS, Python, git (dah!)
• Vagrant with VirtualBox
• AWS CLI installed and configured. Consider using named profiles.

Set it Up

Install greengo from PyPI:

$ pip install greengo

Manually [*] download GreenGrassCore binary and place it in the ./downloads directory. Sign in to the AWS Management Console, navigate to the AWS IoT console, and download the AWS Greengrass Core Software from [Software section](https://us-west-2.console.aws.amazon.com/iotv2/home?region=us-west-2#/ software/greengrass). Yeah, manual sucks... I will automate it later. Or, submit your PR!

Play

1. Create GreenGrass Group definition in AWS

   Fancy yourself with the group definitions in group.yaml, and run greengo:

   $ greengo create
   

   When runs with no errors, it creates all greengrass group artefacts on AWS and places certificates and config.json for GreenGrass Core in ./certs and ./config for Vagrant to use in provisioning on next step.

2. Provision VM with GreenGrass Core with Vagrant

   $ vagrant up
   

3. Deploy Greengrass Group to the Core on the VM.

   $ greengo deploy
   

4. Check that everything works - see the "Check" section below.

5. Profit !

   Work on it: create, change or remove Lambda functions, subscriptions, resources, and then update Greengrass.

   $ greengo update
   

   Apply your changes by deploying it again:

   $ greengo deploy
   

6. Clean-up when done playing.

   Remove the group definitions on AWS:

   $ greengo remove
   

   Ditch the Vagrant VM:

   $ vagrant destroy
   

For any of the above commands you may specify a different yaml file using

$ greengo --config_file <name>.yaml <command>

where <name> is the name of your yaml and <command> is whatever you wish to run

NOTE: If you want to create a new group but keep the Greengrass Core in the same Vagrant VM, you must update it with newly generated certificates and config.json file before deploying the group, and also reset deployment by getting the deployments/group/group.json back to virgin.

To do it: login to the Greengrass Vagrant VM and run /vagrant/scripts/update_ggc.sh on the Vagrant VM.

Details

Check the deployment

How to be sure everything something works? Follow this:

1. Create greengrass group in AWS IoT: greengo create.
2. Prepare GGC on the VM: update certificates, reset group.json, restart the greengrassd.
3. Deploy with greengo deploy. Check:
   • Check the deployment status, should be 'Success'
4. Explore Greengrass Core on your vagrant VM.
   • Login to Vagrant VM. You should nkow Vagrant but for the off case: vagrant ssh.
   • Check the GGC logs runtime.log and python_runtime.log under /greengrass/ggc/var/log/system. Runtime log should have a line about starting your lambda, or an error why the funtion is not started. In many cases (like not enough memory for Lambda), the deployment is 'Success' but the function fails to start. The errors can only be seen in the runtime.log. If the function starts successfully, runtime.log will contain a message like

   [2018-03-31T08:48:40.57Z][INFO]-Starting worker arn:aws:lambda:us-west-2:0000000000:function:GreengrassHelloWorld:12
   

   • Find and check your own Lambda log under /greengrass/ggc/var/log/system.
   • Check the greengrassd process: ps aux | grep greengrassd. Depending on deployment you might have several processes.
5. In AWS console, check the MQTT topic with IoT MQTT Test page:

   REGION=`aws configure get region`; open https://$REGION.console.aws.amazon.com/iot/home?region=$REGION#/test
   

   Subscribe to the topic (e.g., hello/world), see the messages sent by the Greengrass Lambda function.

When something goes wrong

At this time greengo is just a prototype, a work-in-progress. Therefore it's not if but when somethings throws out, leaving the setup in half-deployed, and you gotta pick up the pieces. Remember:

• You are still not worse off doing this manually: you at least have all the ARN and Id of all resources to clean-up.
• DON'T DELETE .gg/gg_state.json file: it contains references to everything you need to delete. Copy it somewhere and use the Id and Arn of created resources to clean up the pieces.
• Do what it takes to roll forward - if you're close to successful deployment, or roll-back - to clean things up and start from scratch.

Please pay forward: PR a patch to whatever broke for you to prevent it from happening again.

Development

Clone the project, set up your environment, install dependencies and setup greengo CLI in dev mode:

$ git clone https://github.com/dzimine/greengo.git
$ cd greengo
$ virtualenv venv

$ . venv/bin/activate
$ pip install -r requirements.txt
$ pip install -e .

Run the unit tests:

pytest -s

greengo's Issues

Support other lambda runtimes (Python 3, nodejs, etc.)

Now that Greengrass supports python3 lambda functions, and python 2 is on the path of deprecation, would be good to have greengo supports deploying python3 lambdas

rootCA

any chance of adding in

if not os.path.isfile(self.group['certs']['keypath']+"/root-CA.crt"):
            urllib.urlretrieve("https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem",self.group['certs']['keypath']+"/root-CA.crt")

save getting root certificate

Write to Dynamodb

This is a bit odd and may require me forking but worth raising anyho, what about the option to write to a dynamodb, as I keep a list of my device in a device-table for access via appsync and user mapping

something like

DynamoTable: gen-devices

       # in __init__
        self._dynamodb = s.resource('dynamodb')

        # in create
        if 'DynamoTable' in self.group:
            self._create_table_entry()

    #create table function
    def _create_table_entry(self):
        thingTable = self._dynamodb.Table(self.group['tables']['thingTable'])
        self.group['thing']['id'] = self.id
        thingTable.put_item(
            Item=self.thing
        )

Also while on odd user cases, the name for your group, anychance this could be auto generated, as we will deploy to potential thousands of devices and save adjusting the yaml everytime

something like

# in __init__
        self.state = _load_state()
        if self.state:
            self.uuid = self.state['id']
        else:
            self.uuid = str(uuid.uuid4())
        self.name = "G"+self.uuid
# in create 
        self.state['id'] = self.uuid

Just some ideas, I can fork when you finish v2 but it would be nice to keep using the original repo

Creating Multi Core/Devices

Hi Dimitri,
First off thanks for the time taken to write greengo, I have used it and will be happy to contribute back if I can solve my issue. You mentioned in a aws forum creating devices and then having just one core.

This means I will then still need to pull down the pythonSDK for the device and then run a script to communicate with the lambda running on the core.

My hopes was to have a lambda that has all the SDKs in it and then just deploy these lambdas to all cores in the group.

You mentioned the one group/core with multiply devices in the post of the forum, I have picked apart the greengo code and have used a brief retooling to get a device created and the keys down.
However a overview of how this works still eludes me.

Are you suggesting
cloud->system->core->device->sensor
and duplicating this. Would the device just talk with core locally and core would communicate back to the cloud. If so I would need to write code for the devices and each device would communicate to the same core locally. That would mean all the deployed product would have exactly the same keys running exactly the same core, correct? I would need to set up subscriptions for every new device back to the core?

Sorry if this is confusing i'm still coming to grips with ggc and core iot for that matter.

I guess i'm dreaming of one set of lambdas with one subscription deploying to all cores

Deploy fails in Python 3 Environment

Please see this PR: #24

Basically, there is a floating point calculation inside a range() in the implementation of deploy.

Existing Lambdas

Hi Dmitri,
Loving you work and will be using greengo in our production devices.
What do you feel about using existing lambdas, as I deploy and manage all my lambdas with serverless, being able to use existing lambdas would be helpful something like;

ExistingLambdas:
  - name: d1Site
    aliasarn: arn:aws:lambda:<your lambda arn>
    greengrassConfig:
      MemorySize: 128000 # Kb, ask AWS why
      Timeout: 10 # Sec
      Pinned: True # Set True for long-lived functions
      Environment:
        AccessSysfs: False
        # ResourceAccessPolicies:
          # - ResourceId: 1_path_to_input
          #   Permission: 'rw'
        Variables:
           name: value

    def create_fn_defn(self):
        for l in self.group['ExistingLambdas']:
            functions = []
            functions.append({
                'Id': l['name'],
                'FunctionArn': l['aliasarn'],
                'FunctionConfiguration': l['greengrassConfig']
            })
            fd = self._gg.create_function_definition(
                Name=l['name'],
                InitialVersion={'Functions': functions}
            )
            self.state['FunctionDefinition'] = rinse(fd)
            _update_state(self.state)

            fd_ver = self._gg.get_function_definition_version(
                FunctionDefinitionId=self.state['FunctionDefinition']['Id'],
                FunctionDefinitionVersionId=self.state['FunctionDefinition']['LatestVersion'])

            self.state['FunctionDefinition']['LatestVersionDetails'] = rinse(fd_ver)
            _update_state(self.state)

This way I could run greengo create_fn_defn and I could deploy my existing lambda

Auto create greengrass service role

What are your thoughts on having greengo auto creating the greengrass service role if it doesn't exist ?
REF
https://docs.aws.amazon.com/greengrass/latest/developerguide/service-role.html