kabirbaidhya / boss Goto Github PK

View Code? Open in Web Editor NEW

37.0 5.0 10.0 877 KB

Deploy like a boss.

License: MIT License

Python 97.58% Shell 1.13% CSS 0.85% HTML 0.45%

boss cli fabric deployment automation remote ssh python

boss's People

Contributors

Stargazers

Watchers

Forkers

squgeim sanjeevkpandit enzosk8 sshikhrakar nhuzaa safalpandey samirpdl behara psyphernix bhupesh7shakya

boss's Issues

Support for zero-downtime frontend deployment

Frontend deployment where the app is built locally, zipped and uploaded to the remote server.

The target is to at least support the deployment of frontend applications that could be built in the local system and served as static files via a regular web server like nginx, apache etc.

Introduce "inferex" for easier configuration and better UX

inferex

The ability to "infer" configurations, defaults or know-hows based on the project type, platform, language, project configs or tools detected.

The aim of this is to make configuration easier, hassle free, reduce the config boilerplate and most importantly the cognitive overhead of the need of being familiar with configuration options for the first time users.

That is to make boss really easy to use and ideally without doing any configurations.

Examples

If boss detects package.json it should try to use default config for node preset w/o the user having to set preset to node explicitly, and will figure out to run npm install while installing dependencies in addition if it detects yarn.lock it might want to run yarn instead and so forth.
If boss detects pom.xml it knows it's a java project, it further looks up for packaging config to know about whether it's deployed as a war file or a jar
Et cetera

Use Cases

This will be used in two different features:

Boss Init: Generating initial config file boss.yml and other boilerplates like fabfile.py, pm2.config.js and all sort of stuff on boss init.
Auto-inference: On the fly inference of configuration options like scripts, preset or even other defaults when tasks are executed based on the inferred.

Inspiration and motivation

Dan Abramov's talk on building tools and JS Ecosystem - Dan Abramov - The Melting Pot of JavaScript
Now and zeit/deployment-type
Too Many Tools. Need to learn using each of them differently. - JavaScript Fatigue
Your app makes me fat.
Heroku

Initial Support

Initially, we'll try to support this thing for few languages/frameworks:

Node Project
Web - React Project

Ability to override user for each stage

Unable to override the user for a specific stage.

Feature Discussion: A public "reflection" or "introspection" API

I'm opening this issue with the aim of it being more of a discussion than a Yes/No answer. Maybe you already have certain plans in place for this 😄

What do you think about Boss having a public facing "reflection" or "introspection" API - a way for external tools to communicate with Boss to introspect deployment targets. It would answer questions like:

What's the latest git commit SHA?
What's the last deployment timestamp for branch master? etc.

For instance, a service could create reports based on this information (perhaps visualize it in GUI). A service could reconcile branches deployed with project management board tickets (e.g JIRA) to know what ticket's been deployed where. In general it would really useful for building things on top of what Boss already provides, possibly in other languages.

The technicalities of communication method, protocol etc. would be another point of discussion if you are interested in this!

Zero-downtime rollback

Ability to rollback to previous builds

Fabric Independence - rewrite core SSH and remote functionality directly over paramiko

Rewrite the core ssh and remote execution using paramiko.
- Add basic SFTP functions like get, put, read, write directly over paramiko #98
- Rewrite logic with basic ssh read and write sftp operations using the new ssh module w/o relying on fabric #101
- Ability to upload local directory to the remote host #105
- Better file/directory upload API with progress #110
- Better file/directory download API with progress
- Error handling for run function.
Get rid of Fabric
- Get rid of fabric's context managers
- Get rid of fabric's run operation to run remote commands
- Get rid of get, put and upload apis to transfer files.
Refactor all remote execution code.
Integration/Regression Tests:
- Write more integration tests for the api/core functions #87
- Write end to end tests for whole deployment process.
- Containerized SSHD for running E2E/integration tests for deployments

Send notifications on failure

If the deployment fails due to some reason after it has been initiated, boss doesn't notify.

Ability to use the Boss CLI interface

Ability to use boss cli tool instead of just using the fab command

Write tests for boss api and utils

Write tests for:

Modules and functions under boss.api
Code under boss.util
Any other code that requires tests

Integrate dotenv for heroku-cli like config setting

Use dotenv and add heroku-cli like features config:set, config:get, config:list and so on.

Add support for deployment presets

Add support for deployment preset to support various kinds of application deployment like like frontend, nodejs, java-war, java-jar, php etc.

Need a new cli command `boss init`

It's nice to have a CLI command boss init which generates bare minimal configuration and fabfile.

Just similar to how npm init or yarn init works. It should generate files:

boss.yml
fabfile.py

Add React app project deployment example

Add react project deployment example. Should be based upon app structure bootstrapped using create-react-app.

Unable to deploy due to git checkout error

Getting this error on deploying my-branch

[example.com] Executing task 'sync'
[example.com] run: git fetch --prune

Checking out to my-branch branch
[example.com] run: git checkout my-branch
[example.com] out: public/example.html: needs merge
[example.com] out: public/sample.html: needs merge
[example.com] out: error: you need to resolve your current index first
[example.com] out: 


Fatal error: run() received nonzero return code 1 while executing!

Requested: git checkout my-branch
Executed: /bin/bash -l -c "cd /var/www/html/example.com >/dev/null && git checkout my-branch"

Aborting.
Disconnecting from example.com:22... done.

Add configuration documentation

Please provide complete details for configurations.

Add Node/TypeScript project deployment example

Add Node, TypeScript, PM2 project deployment example.

Generic task notifications about configured scripts

Generic Notifications for scripts

Support generic notifications for scripts.
Ability to get notified for the when certain task or script is run.
Need a new configuration block such as notified_scripts (or a better name)
For instance:
Configuration could be as simple as:

project_name: my-app
scripts: 
  db_migrate: 'yarn db:migrate'
  db_rollback: 'yarn db:rollback'
  reload: 'pm2 reload'

notifications:
  slack:
    ...
  hipchat:
    ...

notified_hooks: 
  scripts: ['reload', 'db_migrate', 'db_rollback']

Now, if the user executes the command:

$ fab staging run:db_migration

In hipchat or slack, the notifications would look like:

- kabir is running db_migrate for my-app on staging server.
- kabir finished running db_migrate for my-app on staging server.

Support docker based deployment with `docker` preset

Have a preset docker that can deploy any docker based project.

Pretty much similar to how now-cli does this https://zeit.co/now#demo.

Fabfile independence for trivial scenarios

Ability to avoid writing fabfile.py for using boss-cli for most trivial cases as most of the functionality would already be available out of the box with just boss.yml config.

For advance scripts you'll need fabfile.py anyway as everything is just plain old fabric.

Symlink should be the last operation

Current situation: Symlink happens before installation of the modules (backend)

If something is wrong when installing the modules, the deployment breaks because node_modules is obsolete.

Reverting back the symlink to previous deployment might be the solution.

Remove deprecated features

Remove deprecated features that print the warning right now like the service config option and etc.

Deployment fails if public_url is not provided for each stage

Config option public_url should be an optional config and remote execution should not fail it it's not provided.

Independence from the tools like npm, systemctl

Currently boss-cli is tightly coupled with tools like npm, systemctl etc.

Does not support OS without systemctl

It needs a service for deployment and log tasks.

> fab dev logs                                                                                                         
[dev.abc.com] Executing task 'logs'
No service has been configured yet.

Deployment rollback notification

Send deployment rollback notification on hipchat and slack

Display CI build link on notification messages if running on CI

Display CI build link if running on CI (Travis only support).

Slack message would look like this:

Payload:

{
    "attachments": [
        {
      "color": "good",
            "text": "<https://link|CI> · travis finished deploying <https://link|app> (<https://link|39bf6a1>) to <https://link|prod> server."
        }
    ]
}

Hipchat message will have similar link too.
The CI link should point to the corresponding build page on Travis.

When vault server is not reachable - boss throws `Error loading config file "boss.yml"` error

When vault server is not reachable - boss throws Error loading config file "boss.yml" error.

This should show specific error instead.

Cleaner UX - Get rid of all those fabric logs unless users explicitly says so

Have cleaner UX and CLI output. Get rid of all those fabric's logs and output unless users explicitly says to display them.

By default hide everything on deployment or while running tasks and display a clean, compact output. In case the user wants to see the original fabric logs it should be configurable with a config option like show_fabric_output: true

Inspiration: https://zeit.co/now

Ability to use vault for credentials for deployment

Pull creds from vault and inject as env vars while running boss
Env vars interpolation in boss.yml should work the same regardless of whether they're injected from host's environment or vault.

Reference: https://github.com/hvac/hvac

Support for zero-downtime java war application deployment

Add basic support for building of java (war) based applications and deploy built war applications to remote server.

Firstly, we'll need to create a new preset java-war just the way we have presets for web, node and remote-source.

Error installing boss-cli==0.2.1 due to README.md error

Got this error while trying to install boss-cli==0.2.1

suresh@suresh-lf /var/www/html/abc/xyz (SL-237) $ sudo pip install boss-cli==0.2.1
The directory '/home/suresh/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/suresh/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting boss-cli==0.2.1
  Downloading boss-cli-0.2.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-qGYz3c/boss-cli/setup.py", line 14, in <module>
        with open(join(this_dir, 'README.md'), encoding='utf-8') as file:
      File "/usr/lib/python2.7/codecs.py", line 896, in open
        file = __builtin__.open(filename, mode, buffering)
    IOError: [Errno 2] No such file or directory: '/tmp/pip-build-qGYz3c/boss-cli/README.md'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-qGYz3c/boss-cli/

Remote environment variables injection

Remote Environment Variables Injection

Load environment variables from a remote host file.
Inject the remote environment variables into the local build during deployment or while running any tasks.
Have a remove_env_path stage-specfic config option which determines from which file on the remote host to load the env variables (default None).
Have a configuration option remote_env_injection to enable this (default False).

Use cases

Web Apps (eg: React, Angular, Vue etc) that requires builds to be injected with different environment variables or configuration depending upon different environment ie. DEV, STAGING, PROD etc.
Any other applications that requires environment specific variables on build/deployment process.

Unable to deploy with UnicodeDecodeError

Command:
DEBUG_BUILD_CREDS=true fab dev deploy

Output:

Traceback (most recent call last):
  File "~/myEnvironment/lib/python2.7/site-packages/fabric/main.py", line 757, in main
    *args, **kwargs
  File "~/myEnvironment/lib/python2.7/site-packages/fabric/tasks.py", line 386, in execute
    multiprocessing
  File "~/myEnvironment/lib/python2.7/site-packages/fabric/tasks.py", line 276, in _execute
    return task.run(*args, **kwargs)
  File "~/myEnvironment/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "~/myEnvironment/lib/python2.7/site-packages/boss/api/deployment/preset/web.py", line 87, in deploy
    buildman.build(stage, config)
  File "~/myEnvironment/lib/python2.7/site-packages/boss/api/deployment/buildman.py", line 411, in build
    runner.run_script(known_scripts.BUILD, remote=False)
  File "~/myEnvironment/lib/python2.7/site-packages/boss/api/runner.py", line 74, in run_script
    host_info(info_text, remote=remote)
  File "~/myEnvironment/lib/python2.7/site-packages/boss/util.py", line 25, in host_info
    host_print(green(msg), remote=remote)
  File "~/myEnvironment/lib/python2.7/site-packages/boss/util.py", line 20, in host_print
    _local(cmd)
  File "~/myEnvironment/lib/python2.7/site-packages/fabric/operations.py", line 1202, in local
    with_env = _prefix_env_vars(command, local=True)
  File "~/myEnvironment/lib/python2.7/site-packages/fabric/operations.py", line 712, in _prefix_env_vars
    for k, v in env_vars.iteritems()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 123: ordinal not in range(128)
Disconnecting from IP_ADDRESS... done.

System details:

boss: 1.0.0-alpha.19
Fabric: 1.13.2
Paramiko: 2.4.2
pip: 18.1
Python: 2.7.15
OS: Arch Linux

Write unit tests and configure Travis CI

Write few unit tests.

Fix SSH user not being overridden for each stage for frontend deployment

Upgrade fabric to 2.x

Upgrade fabric to 2.4.x (http://www.fabfile.org/upgrading.html)
There are a lot of breaking changes, need to fix all those and do a regression test to ensure things still work.
There might be few breaking changes in boss too, but should be minimized as much as possible.
This enables python3 support too.

Support deploying to AWS S3 bucket

Support deploying static files, web app project to S3 bucket

Add support for custom scripts

Support custom scripts like the npm scripts defined in package.json. These custom scripts could be one liner scripts/commands that could be defined directly in the boss.yml file without having to write any python code.

Such as this:

scripts:
  build: npm run build
  logs: pm2 logs

These scripts could be run on the server using the run task which might have similar signature:

def run(script):
    ''' Run a custom script. '''
    # Code that would run the defined script

And from our local system we would run something like this:

$ fab dev run:build
$ fab dev run:logs

One major benefit of having this is that, we no longer have to depend tightly upon other tools like npm and systemctl like we do now. For instance right now we call npm run build internally to build the application. After this we can expect the user to define a build custom script instead.

Boss Command Line Interface

CLI Commands

init: Initialize minimal configuration using default options to get started. Done #62.
init --interactive, -i: Initialize minimal configuration after interactively prompting and asking users for required config values. Done #75.
deploy <stage>: Deploy to <stage>, same as fab <stage> deploy
rollback <stage>: Rollback deployment on <stage>, same as fab <stage> rollback
list or ls: List available commands, fabric tasks or custom scripts.
run <task>: Run as task on <stage>
run <script>: Run a custom script (defined on boss.yml) on <stage>
setup <stage>: Run setup command on the <stage>, same as fab <stage> setup

Nice to haves

deploy <stage> --dry-run: Dry run deployment, prints out all the necessary UNIX commands to be run on the stage on deployment.
deploy --stage=<stage>: Another syntax of using deploy <stage> command.
rollback --stage=<stage>: Another syntax of using deploy <stage> command.
list --type=[all|commands|scripts|tasks]: List available commands, fabric tasks or custom scripts.

Priority Support

@nhuzaa is using Mintycode to fund this issue.
If you would like to accept bounty for solving this issue join Mintycode.
Thank you in advance for helping.

Ability to set port from boss.yml file

For now, I have to mention port in the fab file. It would be easier to add an option to set the port in the boss.yml file.

Use the local branch name by default on remote-source deployment too

Have a config option for remote-source deployment deployment.use_local_git_ref
When deployment.use_local_git_ref is true it should use the local git branch with fab <stage> deploy instead of having to do fab <stage> deploy:<local-branch>
Use the local branch name by default on remote-source deployment too

Ability to inject environment variables in boss.yml

Ability to inject environment variables in boss.yml just like what docker-compose.yml allows
Load environment vars from .env file if it exists in the same directory as boss.yml.

Inability to run fabric' tasks or even fab --list command when vault is enabled

 fab --list
Traceback (most recent call last):
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/fabric/main.py", line 667, in main
    docstring, callables, default = load_fabfile(fabfile)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/fabric/main.py", line 168, in load_fabfile
    imported = importer(os.path.splitext(fabfile)[0])
  File "/home/kabir/lf/test/db/fabfile.py", line 15, in <module>
    init(__name__)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/boss/init.py", line 20, in init
    config = load_config(stage=stage)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/boss/config.py", line 103, in load
    use_vault_if_enabled(config_str, stage)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/boss/config.py", line 167, in use_vault_if_enabled
    vault.env_inject_secrets(path, silent=raw_config['vault']['silent'])
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/boss/core/vault.py", line 44, in env_inject_secrets
    secrets = read_secrets(path)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/boss/core/vault.py", line 31, in read_secrets
    result = client.read(path)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/hvac/v1/__init__.py", line 157, in read
    return self._adapter.get('/v1/{0}'.format(path), wrap_ttl=wrap_ttl).json()
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/hvac/adapters.py", line 90, in get
    return self.request('get', url, **kwargs)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/hvac/adapters.py", line 233, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/home/kabir/envs/test/local/lib/python2.7/site-packages/hvac/utils.py", line 33, in raise_for_error
    raise exceptions.Forbidden(message, errors=errors)
hvac.exceptions.Forbidden: 1 error occurred:

* permission denied

Add Node/JavaScript project deployment example

Add an example nodejs + pm2 (javascript) project deployment sample.

Write documentation for Node project deployment

Write documentation for Node project deployment on the README.