Giter Site home page Giter Site logo

fabricio's Introduction

Fabricio

Fabricio is a Docker deploy automation tool used along with the Fabric.

image

image

Features

  • build Docker images
  • create containers and services from images with provided tags
  • unlimited infrastructures
  • Fabric's parallel execution mode compatibility
  • rollback containers or services to previous version
  • public and private Docker registries support
  • deploy over SSH tunnel (e.g. access to image registry, proxy, etc.)
  • migrations apply and rollback
  • data backup and restore
  • Docker services (Swarm mode)
  • Docker stacks (Docker Compose 3.0+)
  • Kubernetes configurations

See changelog for detailed info.

Basic example

The most basic fabfile.py you can use with the Fabricio may look something like this:

from fabricio import docker, tasks

app = tasks.DockerTasks(
    service=docker.Container(
        name='app',
        image='nginx:stable-alpine',
        options={
            'publish': '80:80',
        },
    ),
    hosts=['[email protected]'],
)

Type fab --list in your terminal to see available Fabric commands:

Available commands:

    app.deploy  deploy service (prepare -> push -> backup -> pull -> migrate -> update)

Finally, to deploy such configuration you simply have to execute following bash command:

fab app.deploy

See also Fabricio examples and recipes.

Requirements

Local

  • Python 2.7, 3.4*, 3.5*, 3.6*, 3.7*, 3.8*
  • (optional) Docker 1.9+ for building Docker images

* Fabric3 is used for compatibility with Python 3.x

Remote

  • sshd
  • Docker 1.9+
  • Docker 1.12+ for using Docker services

Install

pip install fabricio

Note for macOS users

Just use latest version of Python instead of one installed by default. The easiest way to install fresh version of Python is using Homebrew:

brew install python

Contribute

All proposals and improvements are welcomed through a pull request or issue. Just make sure all tests are running fine.

Install test dependencies

pip install ".[test]"

Running tests

python -m unittest2 discover tests --verbose

Roles and infrastructures

You can define as many roles and infrastructures as you need. The following example shows 'production' and 'test' configurations for two-roles deploy configuration:

from fabric import colors, api as fab
from fabricio import docker, tasks, infrastructure

@infrastructure
def testing():
    fab.env.roledefs.update(
        api=['[email protected]'],
        web=['[email protected]'],
    )

@infrastructure(color=colors.red)
def production():
    fab.env.roledefs.update(
        api=['[email protected]', '[email protected]'],
        web=['[email protected]'],
    )

web = tasks.DockerTasks(
    service=docker.Container(
        name='web',
        image='registry.example.com/web:latest',
        options={
            'publish': ['80:80', '443:443'],
            'volume': '/media:/media',
        },
    ),
    roles=['web'],
)

api = tasks.DockerTasks(
    service=docker.Container(
        name='api',
        image='registry.example.com/api:latest',
        options={
            'publish': '80:80',
        },
    ),
    roles=['api'],
)

Here is the list of available commands:

Available commands:

    production  select production infrastructure, 'production.confirm' skips confirmation dialog
    testing     select testing infrastructure, 'testing.confirm' skips confirmation dialog
    api.deploy  deploy service (prepare -> push -> backup -> pull -> migrate -> update)
    web.deploy  deploy service (prepare -> push -> backup -> pull -> migrate -> update)

'production' and 'testing' are available infrastructures here. To deploy to a particular infrastructure just provide it before any other Fabric command(s). For example:

fab testing api.deploy web.deploy

See Infrastructures and roles example for more details.

Tags

Almost every Fabricio command takes optional argument 'tag' which means Docker image tag to use when deploying container or service. For instance, if you want to deploy specific version of your application you can do it as following:

fab app.deploy:release-42

By default, value for tag is taken from Container/Service Image.

Also it is possible to completely (and partially) replace registry/account/name/tag/digest of image to deploy:

fab app.deploy:registry.example.com/registry-account/app-image:release-42
fab app.deploy:nginx@sha256:36b0181554913b471ae33546a9c19cc80e97f44ce5e7234995e307f14da57268

Rollback

To return container or service to a previous state execute this command:

fab app.rollback

Idempotency

Fabricio always tries to skip unnecessary container/service update. However, update can be forced by adding force=yes parameter:

fab app.deploy:force=yes

Private Docker registry

It is often when production infrastructure has limited access to the Internet or your security policy does not allow using of public Docker image registries. In such case Fabricio offers ability to use private Docker registry which can be used also as an intermediate registry for the selected infrastructure. To use this option you have to have local Docker registry running within your LAN and also Docker client on your PC. If you have Docker installed you can run up Docker registry locally by executing following command:

docker run --name registry --publish 5000:5000 --detach registry:2

When your local Docker registry is up and run you can provide custom registry which will be used as an intermediate Docker registry accessed via reverse SSH tunnel:

from fabricio import docker, tasks

app = tasks.DockerTasks(
    service=docker.Container(
        name='app',
        image='nginx:stable-alpine',
        options={
            'publish': '80:80',
        },
    ),
    registry='localhost:5000',
    ssh_tunnel='5000:5000',
    hosts=['[email protected]'],
)

See Hello World example for more details.

Building Docker images

Using Fabricio you can also build Docker images from local sources and deploy them to your servers. This example shows how this can be set up:

from fabricio import docker, tasks

app = tasks.ImageBuildDockerTasks(
    service=docker.Container(
        name='app',
        image='registry.example.com/registry-account/app-image:latest-release',
    ),
    hosts=['[email protected]'],
    build_path='.',
)

By executing command app.deploy Fabricio will try to build image using Dockerfile from the folder provided by build_path parameter. After that image will be pushed to the registry (registry.example.com in the example above). And deploy itself will start on the last step.

See Building Docker images example for more details.

Docker services

Fabricio can deploy Docker services:

from fabricio import docker, tasks

service = tasks.DockerTasks(
    service=docker.Service(
        name='my-service',
        image='nginx:stable',
        options={
            'publish': '8080:80',
            'replicas': 3,
        },
    ),
    hosts=['user@manager'],
)

See Docker services example for more details.

Docker stacks

Docker stacks are also supported (available since Docker 1.13):

from fabricio import docker, tasks

stack = tasks.DockerTasks(
    service=docker.Stack(
        name='my-docker-stack',
        options={
            'compose-file': 'my-docker-compose.yml',
        },
    ),
    hosts=['user@manager'],
)

See Docker stacks example for more details.

Kubernetes configuration

Kubernetes configuration can be deployed using following settings:

from fabricio import kubernetes, tasks

k8s = tasks.DockerTasks(
    service=kubernetes.Configuration(
        name='my-k8s-configuration',
        options={
            'filename': 'configuration.yml',
        },
    ),
    hosts=['user@manager'],
)

See Kubernetes configuration example for more details.

fabricio's People

Contributors

renskiy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fabricio's Issues

RuntimeError during attempt to delete dangling images

Any attempt to execute deploy configuration fails with following message:

RuntimeError: local() encountered an error (return code 1) while executing 'for img in $(docker images --filter "dangling=true" --quiet); do docker rmi "$img"; done'

This happens when Docker can't delete some images (due to existing containers from such images, for example).

Make Fabricio work on Windows

There is at least one place where used bash-specific command which can not be executed by default on Windows: DockerTasks.delete_dangling_images().

TypeError when trying to get detailed info about command

fab -d test
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/fabric/main.py", line 694, in main
    display_command(options.display)
  File "/Library/Python/2.7/site-packages/fabric/main.py", line 479, in display_command
    task_details = command.__details__()
  File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 183, in __details__
    return get_task_details(orig)
  File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 42, in get_task_details
    argspec = inspect.getargspec(task)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 816, in getargspec
    raise TypeError('{!r} is not a Python function'.format(func))
TypeError: <functools.partial object at 0x10c9a2260> is not a Python function

Kubernetes support

Same way as #127 (Docker stacks / Docker Compose) was resolved there is opportunity to add Kubernetes support.

Upload compose-file for docker stack

I cannot upload a compose-file from a folder other than the fabfile.py folder since the "upload_configuration_file" script will try to upload it to the path of the mentioned local folder.
Example:
stack = tasks.DockerTasks(
service=docker.Stack(
name='stack',
options={
'compose-file':'/folder0/../foldern/docker-compose.yml',
},
),
hosts=fab.env.hosts,
destroy_command=True,
)

fab will try to upload the docker-compose file to "/folder0/../foldern/docker-compose.yml" and not to temp_dir folder which will cause error if the path does not exist in the distant machine

start multiple dockers

can I start 4 running versions of same docker image?

I see a replicas option, can It be used to spin 3 dockers

ProvisionTasks

I think it would be nice to have ProvisionTasks inheriting base Tasks and providing some basic provisioning commands. Such as:

  • init (default)
  • remove
  • update

Such class would be useful for management of Docker networks, volumes and other entities.

Transparent proxy over SSH tunnel

The idea is to forward outgoing registry requests (80 and 443 ports) to SSH tunnel with proxy in transparent mode (e.g. squid) on the other side of tunnel.

Forwarding can be set up with iptables:

iptables --table nat --append OUTPUT --protocol tcp --dport 80 --dport 443 --jump DNAT --to-destination 127.0.0.1:3128

Transparent proxy may be started on the admin side as a Docker container.

PS: don't forget to reset iptables rules to original state after all. This can be reached by following steps: reset => set => reset, where first step need to be sure there is no any rules from previous (possibly failed) deploy.

'Tasks.name' property implementation

This property can be used to assign custom commands name for groups of tasks, including infrastructures. For example:

name_with_underscores = Tasks(name='name-with-hyphens')

or

@infrastructure(name='UPPER')
def lower()
    pass

Additional service options

There is a bunch of new removable options added to the Docker services recently:

  • config
  • dns
  • dns-option
  • placement-pref
  • group
  • host
  • secret

Warning: local() encountered an error (return code 1)

Hi, what could the source of the problem?
Beating the head against a wall

Fabric==1.14.0
fabricio==0.3.1

local: docker build --tag front:latest --pull --build-arg nginx_config_filename=local.conf .
Sending build context to Docker daemon 436.6MB
Step 1/15 : FROM centos:centos7
Get https://registry-1.docker.io/v2/: EOF

Warning: local() encountered an error (return code 1) while executing 'docker build --tag front:latest --pull --build-arg nginx_config_filename=local.conf .'

Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/fabric/main.py", line 757, in main
*args, **kwargs
File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
results[''] = task.run(*args, **new_kwargs)
File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
return self.wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/fabricio/tasks.py", line 368, in deploy
fab.execute(self.prepare, tag=tag, no_cache=no_cache)
File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
results[''] = task.run(*args, **new_kwargs)
File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
return self.wrapped(*args, **kwargs)
File "/Users/myusername/Develop/chat-bot/frontend/fabfile.py", line 206, in prepare
use_cache=True,
File "/usr/local/lib/python2.7/site-packages/fabricio/init.py", line 74, in local
**kwargs
File "/usr/local/lib/python2.7/site-packages/fabricio/init.py", line 27, in _command
raise RuntimeError(result)
RuntimeError

Delete service

Hi
How can I remove containers and services related to a task.

DockerTasks.update(): fabric.api.remote_tunnel fails with exception when no actual host found

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/fabric/main.py", line 745, in main
    *args, **kwargs
  File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 427, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/fabricio/tasks.py", line 105, in update
    local_host=self.local_registry_host,
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Library/Python/2.7/site-packages/fabric/context_managers.py", line 554, in remote_tunnel
    transport = connections[env.host_string].get_transport()
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 158, in __getitem__
    if key not in self:
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 173, in __contains__
    return dict.__contains__(self, normalize_to_string(key))
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 397, in normalize_to_string
    return join_host_strings(*normalize(host_string))
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 387, in join_host_strings
    template = "%s@[%s]:%s" if host.count(':') > 1 else "%s@%s:%s"
AttributeError: 'NoneType' object has no attribute 'count'

Docker stack. Links in docker-compose are ignored.

Hi.
I tried to use fabricio to deploy my django project to prod.
I used docker and docker-compose before on local so I created docker.Stack service with link to my docker-compose file.
But the problem is that fabricio copies docker-compose.yml to /tmp and looks for all files, linked in docker-compose.yml, in this directory. For example, my docker-compose.yml has line:

env_file:
          - docker/dev.env

And at building container, docker looks for env file in /tmp/docker/dev.env and I get error.
What is worse, dockerfiles are not found too.
I tried:

  1. Create docker.Stack with temp_dir parameter
  2. Subclass Stack and point temp_dir to my project directory (and re-define upload_configuration_file)
  3. Create symlink in tmp to my project directory.
    Without any success.
    Did I miss something important?

Skip tasks without any host provided

There are two alternatives if Fabric can't determine host for task:

  1. Fabric can ask host using prompt
  2. use --abort-on-prompts option

Both cases lead to inability to complete automatic scripts based on Fabricio. It's necessary to have graceful skipping of tasks which's host can't be determined at runtime.

localhost infrastructure

Is it any way how launch containers locally?

I have tried to create

# Infrastructure
@fabricio.tasks.infrastructure
def localhost():
    fab.env.roledefs.update(
        amqp=['anton@localhost'],
        reverse_proxy=['anton@localhost'],
        postgres=['anton@localhost'],
        tarantool=['anton@localhost'],
        backend=['anton@localhost'],
    )

but got error

Fatal error: Low level socket error connecting to host localhost on port 22: Unable to connect to port 22 on 127.0.0.1 (tried 1 time)

Underlying exception:
    Unable to connect to port 22 on 127.0.0.1

Use remote_tunnel only for local Docker registries

In situations when using custom registry with PullDockerTasks, remote_tunnel can take port which already used by some of containers. Therefore, PullDockerTasks should detect if remote_tunnel is really necessary.

In most cases remote_tunnel is necessary only for registries which point to the local host.

digital-ocean deploy

why is vagrant needed? can I deploy it without it i.e. to digital ocean?

can you please add documentation or example on how to deploy docker image from docker-hub to digital ocean?

denied: requested access to the resource is denied

During installation encountered the following error:

fabricio.operations.Error: local() encountered an error (return code 1) while executing 'docker push...

Environment:
Mac OS 10.x
Fabric==1.14.0
fabricio==0.5.6

denied: requested access to the resource is denied

Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/fabric/main.py", line 757, in main
*args, **kwargs
File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
results[''] = task.run(*args, **new_kwargs)
File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
return self.wrapped(*args, **kwargs)

...

'pull' не срабатывает, если не указан хост

Собираю образ, пытаюсь его запулить в локал регистри (у меня потом на базе этого образа собирается основной)
.pull не срабатывает, так как не указан хост в таске, но мне же хост и не нужен, registry то указан, я хочу просто сохранить образ, а следующий таск уже соберет основной на его базе и запулит его на бой.

'pull' execution was skipped due to no host provided (command: edcontainer.pull)

Можно ли подумать над изменением этой логики?

Not able to deploy locally without ssh

Version of fabricio==0.3.1

To force deploy locally I started describing the task as following

@tasks.infrastructure
def localhost():
    print("Trying to run fab locally")
    fabricio.run = functools.partial(fabricio.local, capture=True)
    fab.env.update(
        roledefs={
            'queue': ['localhost'],

...

Still it had no effect, it tries to deploy locally via ssh

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.