2ndwatch/cloudendure-python

A Python 3.7+ client and utilities for CloudEndure

cloudendure-python

Python wrapper and CLI for CloudEndure

Requirements

Python 3.7+

Installation & Usage

Basic Installation / pip

pip install cloudendure
cloudendure version

Local Development with Poetry

brew install poetry # if not installed
poetry install

Local Development with Docker

docker run --rm -it cloudendurepy/cloudendure bash

Usage

Then import the package:

import cloudendure

Getting Started

CloudEndure Pipeline Flow

CloudEndure Data Flow

Pipeline Flow (as seen in AWS Console)

Logging in via CLI using environment variables

Please note: cloudendure and ce can be used interchangeably

export CLOUDENDURE_USERNAME=<your_ce_user>
export CLOUDENDURE_PASSWORD=<your_ce_password>
export CLOUDENDURE_DESTINATION_ACCOUNT=<destination_aws_account_id>

cloudendure api login

or

export CLOUDENDURE_USER_API_TOKEN=<your_ce_user_api_token>
export CLOUDENDURE_DESTINATION_ACCOUNT=<destination_aws_account_id>

ce api login

Logging in via CLI inline

Please note: cloudendure and ce can be used interchangeably

cloudendure api login --user=<your_ce_user> --password=<your_ce_password>

or

ce api login --token=<your_ce_user_api_token>

Logging in for the first time will generate the ~/.cloudendure.yml file.

Coming Soon

This project is currently a work in progress and will actively change. This client has not yet been finalized and is entirely subject to change.

Changelog

Check out the CHANGELOG

cloudendure-python's Issues

Update CloudEndureConfig to use destination_account vs accounts

In #48, multiple destination account support was removed, but the config.py@L50 destination_accounts was not updated. We need to update all references for configuration of the CE module/cli to use destination_account/CLOUDENDURE_DESTINATION_ACCOUNT now that the .get() has changed.

cc @twarnock2w

Add security policy

As a user, I want to see a new Github recommended SECURITY.md in this project.

Upgrade docker images to buster

Upgrade both docker files to use buster versus stretch

Generate generic output infrastructure-as-code projects for migration waves

User Story

As a CE CLI user, I want the ability to output a base Terraform/CFTs project for all relevant resources (instances, volumes, amis, etc) for migrated servers.

Acceptance Criteria

• Add base Terraform cookiecutter template to generate instance, volumes, attachments, and relevant data stubs.
• Add mechanism for outputting CE/AWS derived data into the base Terraform cookiecutter.
• Setup plugin module to encapsulate all Terraform related outputs.
• Setup a stub for output CFTs.

Update API explorer to use Bravado

As a user, I want the ability to utilize the cloudendure API directly with a cursor.

We should implement an API client/cursor using: https://github.com/Yelp/bravado and simply point it at the swagger manifest and handling authentication.

Update main CLI handling to employ defined exceptions/feedback loops

Currently, we have exceptions defined for the project, but they aren't being used appropriately throughout the main CLI for providing feedback loop to the user

Missing classifiers in pyproject.toml

The pyproject.toml is missing the classifiers list field

not_synced missing hosts doing initial sync

not_synced is currently not evaluating if replicated_storage_bytes is not equal to total_storage_bytes so those machines that are added to a project are missed in the not_synced output.

Make lambdas more generic

Right now they're focused towards servicenow.

Add python-fire to package dependencies

As a user, I shouldn't be required to install fire manually.

Image creation failure

Need to fix AMI names for AWS - ec2client

Migrate all config items to config module

User Story

As a CE user, I want to know all of my configuration items stem from the Config class and maintain local state throughout the span of my CLI interactions.

Acceptance Criteria

• Modify all configuration handling throughout the project to stem from the Config class versus direct environment variables.
• Remove any overlapping configuration variants
• Multiple project configurations in yaml?

Migrate environment management from pipenv to poetry

This conversion from pipenv to poetry is at the request of the project maintainer. The conversion has been completed and will be offered as a PR for review at the maintainer's convenience.

Running gen-terraform against tagless AMIs results in error

pipenv run ce gen-terraform --image-id ami-00d277459299cdf7b
Found project_id: <project_id>
Loading EC2 resource for region:  <aws_region>
Traceback (most recent call last):
  File "/.../virtualenvs/cloudendure-gXgidy3v/bin/ce", line 11, in <module>
    load_entry_point('cloudendure', 'console_scripts', 'ce')()
  File "/.../cloudendure/cloudendure.py", line 796, in main
    fire.Fire(CloudEndure)
  File "/.../virtualenvs/cloudendure-gXgidy3v/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/.../virtualenvs/cloudendure-gXgidy3v/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/.../virtualenvs/cloudendure-gXgidy3v/lib/python3.7/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/.../cloudendure/cloudendure.py", line 764, in gen_terraform
    for tag in image.tags:
TypeError: 'NoneType' object is not iterable

New CE projects need non-admin accounts associated with them.

When we create a project we need to be able to relate non-admin/owner accounts that should have access to them.

POST /projects/assignUsers

{ "items": [ { "projectId": "string", "userIDs": [ "string" ] } ] }

Bug in last launch checks in main cli

There is currently a bug in lastTestLaunchDateTime/lastCutoverDateTime checks

Should make use of replica

Add version cli entry

Currently, we don't provide cloudendure version

Remove 3.6 support and prepare 0.0.7

Remove support for Python 3.6 and prepare the release/tag: 0.0.7.
Relevant to: https://github.com/mbeacom/cloudendure-python/pull/35

CE worker lambda diagram

Get some diagrams that explain the Lambda/Step Function flow.

Handle MFA/SAML between AWS and CloudEndure

Bug in update blueprint flow

Configurable PublicIPAction for blueprints

As a user, I want the ability to configure update_blueprint to modify the publicIPAction CE API attribute.

Acceptance criteria

• CE users can update all blueprints to either ALLOCATE, DONT_ALLOCATE, or AS_SUBNET (CE default values) to control public access to migrated instances.
• CE users should be able to adjust this from either env vars or yaml configuration.

Missing depedency

Environment

On a up to date Mac, using python3 installed wit Brew, cookiecutter is missing.

uname -a

Darwin rotten.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64

python --version

Python 2.7.16

python3 --version

Python 3.7.6

Symptom

└> pip3 install cloudendure
Collecting cloudendure
  Downloading https://files.pythonhosted.org/packages/e7/3e/526bfddb6c354ab154a8b1d54a1030bf09ae39ca898d4fc4ebf3436ae052/cloudendure-0.1.9-py3-none-any.whl (811kB)
     |████████████████████████████████| 819kB 23.8MB/s
Collecting fire
  Downloading https://files.pythonhosted.org/packages/d9/69/faeaae8687f4de0f5973694d02e9d6c3eb827636a009157352d98de1129e/fire-0.2.1.tar.gz (76kB)
     |████████████████████████████████| 81kB 18.2MB/s
Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from cloudendure) (2.22.0)
Requirement already satisfied: boto3 in /usr/local/lib/python3.7/site-packages (from cloudendure) (1.11.3)
Requirement already satisfied: six in /usr/local/lib/python3.7/site-packages (from fire->cloudendure) (1.12.0)
Collecting termcolor
  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->cloudendure) (1.25.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests->cloudendure) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->cloudendure) (2019.6.16)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->cloudendure) (2.8)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.7/site-packages (from boto3->cloudendure) (0.3.0)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3->cloudendure) (0.9.4)
Requirement already satisfied: botocore<1.15.0,>=1.14.3 in /usr/local/lib/python3.7/site-packages (from boto3->cloudendure) (1.14.3)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/site-packages (from botocore<1.15.0,>=1.14.3->boto3->cloudendure) (2.8.0)
Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.7/site-packages (from botocore<1.15.0,>=1.14.3->boto3->cloudendure) (0.15.2)
Building wheels for collected packages: fire, termcolor
  Building wheel for fire (setup.py) ... done
  Created wheel for fire: filename=fire-0.2.1-py2.py3-none-any.whl size=103527 sha256=5e7bd3deda8ab2ca086561f3cf9e1e140e6b881e0d51c3263b6e4200e002798f
  Stored in directory: /Users/ohmer/Library/Caches/pip/wheels/31/9c/c0/07b6dc7faf1844bb4688f46b569efe6cafaa2179c95db821da
  Building wheel for termcolor (setup.py) ... done
  Created wheel for termcolor: filename=termcolor-1.1.0-cp37-none-any.whl size=4832 sha256=e74556863bb03d40b32f2255deebd5ba844ace866d460c66f9b60f90a7c1c2fc
  Stored in directory: /Users/ohmer/Library/Caches/pip/wheels/7c/06/54/bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6
Successfully built fire termcolor
Installing collected packages: termcolor, fire, cloudendure
Successfully installed cloudendure-0.1.9 fire-0.2.1 termcolor-1.1.0
┌ohmer@rotten ~/w/c/w/g/cloudendure
└> ce
Traceback (most recent call last):
  File "/usr/local/bin/ce", line 5, in <module>
    from cloudendure.cloudendure import main
  File "/usr/local/lib/python3.7/site-packages/cloudendure/cloudendure.py", line 23, in <module>
    from .templates import TerraformTemplate
  File "/usr/local/lib/python3.7/site-packages/cloudendure/templates.py", line 7, in <module>
    from cookiecutter.main import cookiecutter
ModuleNotFoundError: No module named 'cookiecutter'

Workaround

Install package manually

> pip3 install cookiecutter
Collecting cookiecutter
  Downloading https://files.pythonhosted.org/packages/86/c9/7184edfb0e89abedc37211743d1420810f6b49ae4fa695dfc443c273470d/cookiecutter-1.7.0-py2.py3-none-any.whl (40kB)
     |████████████████████████████████| 40kB 7.9MB/s
Collecting whichcraft>=0.4.0
  Downloading https://files.pythonhosted.org/packages/b5/a2/81887a0dae2e4d2adc70d9a3557fdda969f863ced51cd3c47b587d25bce5/whichcraft-0.6.1-py2.py3-none-any.whl
Collecting jinja2>=2.7
  Downloading https://files.pythonhosted.org/packages/27/24/4f35961e5c669e96f6559760042a55b9bcfcdb82b9bdb3c8753dbe042e35/Jinja2-2.11.1-py2.py3-none-any.whl (126kB)
     |████████████████████████████████| 133kB 35.1MB/s
Collecting future>=0.15.2
  Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
     |████████████████████████████████| 829kB 48.7MB/s
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.7/site-packages (from cookiecutter) (7.0)
Collecting binaryornot>=0.2.0
  Downloading https://files.pythonhosted.org/packages/24/7e/f7b6f453e6481d1e233540262ccbfcf89adcd43606f44a028d7f5fae5eb2/binaryornot-0.4.4-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.18.0 in /usr/local/lib/python3.7/site-packages (from cookiecutter) (2.22.0)
Collecting poyo>=0.1.0
  Downloading https://files.pythonhosted.org/packages/42/50/0b0820601bde2eda403f47b9a4a1f270098ed0dd4c00c443d883164bdccc/poyo-0.5.0-py2.py3-none-any.whl
Collecting jinja2-time>=0.1.0
  Downloading https://files.pythonhosted.org/packages/6a/a1/d44fa38306ffa34a7e1af09632b158e13ec89670ce491f8a15af3ebcb4e4/jinja2_time-0.2.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23
  Using cached https://files.pythonhosted.org/packages/ce/c6/f000f1af136ef74e4a95e33785921c73595c5390403f102e9b231b065b7a/MarkupSafe-1.1.1-cp37-cp37m-macosx_10_6_intel.whl
Requirement already satisfied: chardet>=3.0.2 in /usr/local/lib/python3.7/site-packages (from binaryornot>=0.2.0->cookiecutter) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests>=2.18.0->cookiecutter) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests>=2.18.0->cookiecutter) (1.25.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests>=2.18.0->cookiecutter) (2019.6.16)
Collecting arrow
  Downloading https://files.pythonhosted.org/packages/92/fa/f84896dede5decf284e6922134bf03fd26c90870bbf8015f4e8ee2a07bcc/arrow-0.15.5-py2.py3-none-any.whl (46kB)
     |████████████████████████████████| 51kB 20.8MB/s
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/site-packages (from arrow->jinja2-time>=0.1.0->cookiecutter) (2.8.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil->arrow->jinja2-time>=0.1.0->cookiecutter) (1.12.0)
Building wheels for collected packages: future
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-cp37-none-any.whl size=491057 sha256=487edebec6e82f91b8a59cd5460b74ac75f288bb9a7c013f595cbd1338cf7445
  Stored in directory: /Users/ohmer/Library/Caches/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
Successfully built future
Installing collected packages: whichcraft, MarkupSafe, jinja2, future, binaryornot, poyo, arrow, jinja2-time, cookiecutter
Successfully installed MarkupSafe-1.1.1 arrow-0.15.5 binaryornot-0.4.4 cookiecutter-1.7.0 future-0.18.2 jinja2-2.11.1 jinja2-time-0.2.0 poyo-0.5.0 whichcraft-0.6.1
┌ohmer@rotten ~/w/c/w/g/cloudendure
└> ce
No CloudEndure YAML configuration found! Creating it at: (%s) /Users/ohmer/.cloudendure.yaml
You must configure your authentication credentials!
Failed to authenticate with CloudEndure! Please check your credentials and try again!

Fix

Not a Python expert, but I guess there is something wrong with the setup.py. The dependency is well declared in the Pipfile and cookiecutter was not missing when I installed via pipenv.

Migrate environment management from pipenv to poetry

This conversion from pipenv to poetry is at the request of the project maintainer. The conversion has been completed and will be offered as a PR for review at the maintainer's convenience.

Configurable disk type for blueprints

As a user, I want the ability to configure update_blueprint to modify the disk_type CE API attribute.

Acceptance criteria

• CE users can update all blueprints to either SSD, PROVISIONED_SSD, etc.
• CE users should be able to adjust this from either env vars or yaml configuration.

Add `has_lag` to inspect_ce_project

Add a check that will return any machines in a CE project that has lag.

Lag is determined by the time delta between lastConsistencyDateTime and lastSeenDateTime in replicationInfo for returned items

Event handling - track wave status

Add event handling for CE interactions (launch success, failure, etc) to maintain some logging and state client side.

CLI authentication

As a user, I want to be able to configure authentication credentials directly from the CLI.

Extension of #63

Deprecate launch_type throughout project

As a CE user, I don't want to have the complexity of launch types anywhere in the module or CLI. A user of this module should opt for test launch types at all times over cutover and manually mark the test as completed.

Update Blueprint Removes Disks

Disks disappear from blueprint due to a bug in the update blueprint code. Interesting that this has not shown up before. CE API might have changed behavior.

Bad print str sub

config.CloudEndureConfig has a bad print sub

Update replication settings to conform to desired KMS usage

As a user, I want the ability to set a destination KMS ID and be certain CloudEndure will use it for the current wave.

Acceptance criteria

• Add check for whether or not encryption is enabled on destination AMIs
• Local setting options (yaml/env) to establish destination KMS IDs for copying AMIs
• Add method to update destination KMS ID to desired setting, if encryption is enabled.
• Ensure the library always performs the check for both settings and update accordingly if they don't match local.
• Handle mismatch errors if the destination AWS account doesn't contain the locally defined KMS ID.

Adjust machine replication

As a user, I want the ability to start, stop, and pause replication for specific CloudEndure machines.

Docker build errors

Currently, docker build fails since changes in #97

Automate lambda deployment via CLI command or equivalent

Documentation parity

As a CE CLI/lambda user, I want to know how to properly use the CLI and module properly with official documentation that reflects the current codebase.

Split up cloudendure.py logic

User Story

As a CE contributor, I want to more easily find appropriate methods throughout the codebase.

Acceptance Criteria

• Breakout functionality defined in cloudendure/cloudendure.py into relevantly named modules.
• Remove overlap/duplicate code and DRY up the codebase.

Simplify project method args and init flow

Currently, we pass around project_name and dry_run to all methods relevant to CE projects...

We should offload this to init and perform the check once per session.

CloudEndure token usage not working

As a user, I want to provide a CloudEndure token. If provided, this token should take precedence and bypass the username/password authentication method.

Share AMI should pull image id from env/config

Adjust python module import init

Currently, we don't define any module initialization instructions in init.py - we should make programmatic import simpler.

YAML module missing after pipenv install

Using pipenv to install the packages for cloudendure-python and trying to run the CLI commands throws an error for a missing module.

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master
$ pipenv install
Creating a virtualenv for this project…
Pipfile: /Users/nselpa/gitrepos/cloudendure-python/Pipfile
Using /usr/local/bin/python3 (3.7.6) to create virtualenv…
⠇ Creating virtual environment...Already using interpreter /usr/local/opt/python/bin/python3.7
Using base prefix '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/bin/python3.7
Also creating executable in /Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/local/bin/python3

✔ Successfully created virtual environment! 
Virtualenv location: /Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7
Installing dependencies from Pipfile.lock (bb2ed3)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 25/25 — 00:01:02

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master
$ pipenv run ce
Traceback (most recent call last):
  File "/Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/bin/ce", line 11, in <module>
    load_entry_point('cloudendure', 'console_scripts', 'ce')()
  File "/Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
    return ep.load()
  File "/Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2445, in load
    return self.resolve()
  File "/Users/nselpa/.local/share/virtualenvs/cloudendure-python-4DQesPY7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2451, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/nselpa/gitrepos/cloudendure-python/cloudendure/cloudendure.py", line 18, in <module>
    from .api import CloudEndureAPI
  File "/Users/nselpa/gitrepos/cloudendure-python/cloudendure/api.py", line 24, in <module>
    from cloudendure.config import CloudEndureConfig
  File "/Users/nselpa/gitrepos/cloudendure-python/cloudendure/config.py", line 10, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master
$ 

Adding pyyaml to the Pipfile's [packages] section results in a successful run:

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master
$ vi Pipfile

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master [!]
$ pipenv install
Pipfile.lock (f49ee6) out of date, updating to (bb2ed3)…
Locking [dev-packages] dependencies…
✔ Success! 
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (f49ee6)!
Installing dependencies from Pipfile.lock (f49ee6)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 26/26 — 00:00:41

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master [!]
$ cloudendure
bash: cloudendure: command not found

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master [!]
$ pipenv run cloudendure
You must configure your authentication credentials!
Failed to authenticate with CloudEndure! Please check your credentials and try again!

nselpa at pa-ps-mac-0508 in ~/gitrepos/cloudendure-python on master [!]
$ 

A PR with the updated Pipfile and Pipfile.lock files to address the issue will be provided.