kalleeh/gurum-api

Gurum Management API

This reference architecture provides a set of YAML templates for deploying the Gurum Management API with AWS CloudFormation.

Overview

The architecture consists of two parts, the supporting platform and the management API. This repository and templates deploy the management API. To modify or deploy the platform, please see the gurum-platform repository.

Architecture Overview

API Overview

Template details

The templates below are included in this repository and reference architecture:

Template	Description
lambda_layers/dependencies/	Shared Python libraries deployed as a Lambda Layer.
src/	Source code for Lambda functions corresponding to API actions.
template.yaml	This is the master template - deploy it to CloudFormation and it includes all of the others automatically.

Deployment Instructions

Prerequisites

AWS X-Ray SDK

This repository does not include a shared lambda layer with the AWS X-Ray SDK. You can either disable the use of AWS X-Ray in the source code or follow below instructions to create a Lambda Deployment Package that is referenced in the template.yaml and automatically attached as a AWS Lambda Layer to the required lambda functions.

[project_root] pip3 install aws-xray-sdk --target lambda_layers/aws-xray-sdk/python

Manual Deployment

You can use the included bash script to quickly deploy the API in your account. Modify the properties in the deploy.sh and then run the following commands.

./deploy.sh

User Account Setup

Once you have the API up and running you will need to configure your developer accounts so that they can interact with the platform. Follow the below steps to create groups and user accounts for your development teams.

1. Configure Cognito

1.1 Create Teams (Cognito Groups)

1.1.1 Use the ./helpers/cognito_quick_group.sh script to create a new Gurum Team that your users can collaborate on. This effectively functions as a tenant in Gurum and ownership of apps and services are linked to the Team.

1.2 Create Users

1.2.1 Use the ./helpers/cognito_quick_user.sh script to create a new cognito user.

gurum-api's Issues

Resolve remaining lint issues.

src/services/name/delete_service.py:17:0: W0611: Unused transform_utils imported as tu (unused-import)
************* Module dependencies.python.transform_utils
lambda_layers/dependencies/python/transform_utils.py:13:0: W0611: Unused import json (unused-import)
************* Module dependencies.python.stackmanager
lambda_layers/dependencies/python/stackmanager.py:335:-1: W0105: String statement has no effect (pointless-string-statement)
lambda_layers/dependencies/python/stackmanager.py:347:4: R0201: Method could be a function (no-self-use)
lambda_layers/dependencies/python/stackmanager.py:418:-1: W0105: String statement has no effect (pointless-string-statement)
lambda_layers/dependencies/python/stackmanager.py:429:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
lambda_layers/dependencies/python/stackmanager.py:429:4: R0201: Method could be a function (no-self-use)
lambda_layers/dependencies/python/stackmanager.py:439:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
lambda_layers/dependencies/python/stackmanager.py:452:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
************* Module dependencies.python.eventmanager
lambda_layers/dependencies/python/eventmanager.py:30:0: W0223: Method '_generate_params' is abstract in class 'StackManager' but is not overridden (abstract-method)

------------------------------------------------------------------
Your code has been rated at 9.90/10 (previous run: 9.90/10, +0.00)```

Setup CI pipeline

Running tests on PR, block if failing.
Automatic versioning.
Create GitHub release when merged to master

Fix validate python3 version of x-ray

Documentation refers to "pip install x-ray dependencies" which can install the x-ray dependencies for python 2 version. This breaks api. Change documentation to pip3 install and see if we can verify the python x-ray package version to be 3 or raise an error otherwise.

Add tag based authorization for CW Logs

Right now CloudWatch Log Groups doesn't get tags inherited from their CloudFormation stags.
CW Logs doesn't support tag based authorization either.

Currently the Cognito IAM Roles inherited by users through Cognito Groups can read from any log group but when this is added we could easily tag the log groups and then modify the IAM Role that cognito users assume to add tag based authorization to their respective log group similar to CFN templates etc.

https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html

{
    "Effect": "Allow",
    "Action": [
        "logs:FilterLogEvents"
    ],
    "Resource": "arn:aws:logs:[region]:[account-id]:log-group:*",
    "Condition": {
        "StringEquals": {"ec2:ResourceTag/gureume-groups": "team1"}
    }
}

Todo: Outline steps to enable Cognito login and role mapping

We need to clarify the steps around creating the initial users and groups and IAM role mappings through cognito federated identities for first setup users.
We don't want to build a user management system in the API since this should use the default Cognito API's but we should provide configuration steps.

Something like;

1. Create Cognito user.
2. Create Cognito group for each Platform Tenant.
3. Create IAM Role with correct CloudWatch Logs Read-permissions (right log groups). Add the right trust permissions on the IAM role.

{
      "Effect": "Allow",
      "Principal": {
        "Federated": "cognito-identity.amazonaws.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "cognito-identity.amazonaws.com:aud": "eu-west-1:b3df4e00-5aea-4e69-8b60-85ec38731d17"
        },
        "ForAnyValue:StringLike": {
          "cognito-identity.amazonaws.com:amr": "authenticated"
        }
      }
    }

4. Map Cognito group login to the IAM role created in step 3.
5. Go into Cognito Federated Identities and go to Authentication providers, Cognito. Under "authenticated role selection" select "choose role from token" and "use Authenticated role".

Add API for orchestrating app pipelines and services

Add API endpoint for initialize an app. Creating a pipeline to parse service.yaml and orchestrate the other API calls accordingly.

Backend for this
kalleeh/gurum-cli#10

Add CDK generation of Gurum Products to simplify parameter, mapping and linking of templates

Consider updating the template_generator function to generate the templates through CDK rather than picking static templates from an S3-bucket.

This could enable easier mapping of applications to services, dynamically generating least privilege IAM policies etc.

Add support for multiple types and "dynamic parameters"

Add more dynamic support in the API for dynamic parameters and product types. Today they are somewhat hardcoded which makes it hard to extend applications, pipelines and services to new types with totally different parameters.

CI/CD Pipeline Overwrites existing parameters due to ParameterOverrides in CodePipeline

We need to implement a way to fetch already existing parameters and configure a parameter file as part of the pipeline rather than using CodePipelines Parameter Override function since it just sets the parameters specifically configured, and doesn't reuse existing ones.

App pipeline names miss project prefix, failing deploy.

Enhance API setup with new Cognito CloudFormation support

See what missing automation steps we can add with the recent extended CloudFormation support for Cognito.

https://aws.amazon.com/about-aws/whats-new/2019/10/amazon-cognito-increases-cloudformation-support/

Optimise first deployment

Run the below script from the ./deploy script.
pip install aws-xray-sdk --target lambda_layers/aws-xray-sdk/python

Update the readme once complete.

Package application in Serverless Application Repository

https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverlessrepo-publishing-applications.html

• Evaluate steps needed to package to sar
• Decide between 1 or 2 packages

Setup linting on pre-commit hook

We need to look at setting up a linting git hook when a commit is made.
This will help us catch errors earlier.

Maybe something along the lines of this:
https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/

Lambda IAM Roles doesn't automatically get Template Bucket permissions

Since the original S3 Bucket is created manually outside the lifecycle of either the Gureume Platform or API stack currently there is no way to properly update the S3 Bucket policy.
This is a manual step that requires the copy pasting of IAM Role ARN for Create and Update Lambda functions to get GetObject-access to the bucket holding the app/pipeline/service templates.