marcosnils/cmt

Container migration tool

cmt

Container migration tool for the Docker Global HackDay #3

https://www.youtube.com/watch?v=pwf0-_cs6U4

Description

Checkpoint & Restore is still a feature which is not generically available to container users. Certain understanding about how it works is needed and it’s most likely that users get errors when trying to perform CR due to some restrictions or differences between the source and the target host. The purpose of the project is to create an external command line tool that can be either used with docker or runC which helps on the task to live migrate containers between different hosts by performing pre-migration validations and allowing to auto-discover suitable target hosts.

IMPORTANT!!

This project uses custom patched versions of CRIU and runC to work. It's important to install these specific versions for CMT to work. CRIU patch has been already proposed to upstream, we hold on runC on the other hand because we needed to implement it fast and we're not sure of any possible impact on the project.

Update 09/21/15: CRIU patch as been merged to upstream here. Master version should work with CMT.

Usage

go get github.com/marcosnils/cmt

cmt --help should list all possible CMT commands and options

Authentication

CMT uses ssh-agent authentication when trying to communicate between hosts. Make sure your agent has the corresponding credentials before trying to perform any action.

Instruction about how to setup ssh-agent can be found here: http://sshkeychain.sourceforge.net/mirrors/SSH-with-Keys-HOWTO/SSH-with-Keys-HOWTO-6.html

Design / performance

CMT was thought to be as portable and lightweight as possible. As it relies on ssh heaviliy for remote communication we also took into account SSH session optimizations and concurrent executions to speed up the whole process.

It was also designed with the idea to be easily adaptable to any underlying mechanism of C/R. This means that when Docker finally implements C/R natively, CMT can take care of all the necessary heavy duty to perform container migration (image layer diffs included).

Hooks

CMT supports 3 kind of hooks. A hook is any command that you provide and that CMT will run when reaching some specific state in the migration process. The supported hooks are:

• Pre-restore: which is executed right before restoring the container on the destination host.
• Post-restore: which is executed after successfully restoring the container on the destination host.
• Failed-restore: which is executed after a failing to restore the container on the destination host.

For example:

cmt migrate --hook-pre-restore "echo pre restore" --hook-post-restore "echo post restore" --hook-failed-restore "echo failed restore"

There are some very useful scenarios for this. For example in AWS you could use the pre-restore hook to move an Elastic Network Interface so the destination has the same IP address of the source.

FAQ

What kind of validations does CMT do?

• Binary existence (runC, criu)
• Binary version matching
• Destination host free memory
• Kernel capabilities to perform c/r (criu check --ms)
• CPU capability problem (http://criu.org/Cpuinfo)

Can CMT perform TCP live migration without end-user disconnection?

Yes, although all the heavy work is done by CRIU, CMT provides some help when migrating TCP connections to avoid end-user disconnect. We've accomplished this in AWS using ENI and VPC peering connections.

(Hope to find the time to demo this soon)

Is it necessary to perform validations each time when migrating?

No, validations are performed by default as a security measure, using --force flag bypasses them.

What does pre-dump exactly do?

Please refer to the official CRIU documentation for iterative migration specifics. http://criu.org/Iterative_migration

What does downtime mean?

Refer to the CRIU documentation for downtime/freeze time. (http://criu.org/)

TODO

Redo this project as it should be done (tests please!!).

We do have some issues we though about implementing but we couldn't find the time.

Special mention to:

• Docker and the community for making us leave our comfort zone and hack on cool stuff. We've learnt a lot these past 4 days.
• Medallia Argentina for hosting the Buenos Aires Docker meetup and being excellent people.
• All OS contributors who can make this happen.
• Ross Boucher (@boucher) for dedicating his personal time to help us answering our annoying questions.

cmt's Issues

ssh: must specify HostKeyCallback in dst

RUNC

[root@node02 mycontainer]# runc --version
runc version 1.0.0-rc4+dev
commit: e6516b3d5dc780cb57a976013c242a9a93052543
spec: 1.0.0

GO
go version go1.9.2 linux/amd64

OS

[root@node02 mycontainer]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

CMT
[root@node02 mycontainer]# $GOBIN/cmt --version
cmt version 0.1

Passwordless ssh

[root@node02 mycontainer]#
[root@node02 mycontainer]# ssh node01
Last login: Thu Jan 18 17:55:53 2018 from node02
[root@node01 ~]# exit
logout
Connection to node01 closed.

SSH agent
Followed this website for ssh agent
http://sshkeychain.sourceforge.net/mirrors/SSH-with-Keys-HOWTO/SSH-with-Keys-HOWTO-6.html

Error
[root@node01 mycontainer]# $GOBIN/cmt migrate --src pwd --dst root@node02:/root/mycontainer
2018/01/18 18:09:18 Performing validations
2018/01/18 18:09:18 Connection error: ssh: must specify HostKeyCallback in dst
[root@node01 mycontainer]#

[root@node01 mycontainer]# $GOBIN/cmt validate --src pwd --dst root@node02
2018/01/18 18:10:59 Connection error: ssh: must specify HostKeyCallback in dst

Could you please help me fix this?

Leave process stopped when migrating

Leaving process stopped in the source host makes easier to recover in case restore fails in the target host. As to resume it rollback hooks (#2) need to be executed and process resumed.

Rollback if can't restore container

We said we'll leave the process stopped and resume it if we can't restore.

Version RUNC CRIU

Could you please tell me which runc and criu version you guys used for your demo? Also the docker version.

Allow to auto-discover suitable target hosts

Either a CIDR could be specified or eventually it'd be nice to integrate it with Docker Swarm

Allow to ignore validations when performing migrate

have you seen criu?

http://criu.org/Docker

cmt error: command not found

Hi
Following usage instructions from https://github.com/marcosnils/cmt didn't work as expected. Currently i have installed runc and criu but cmt commands are not accepted. Can anyone help me in this issue. Thanks

Command History:

go get github.com/marcosnils/cmt
netcom1@cloudlet1:/cmt$ ls
bin pkg src test
netcom1@cloudlet1:/cmt/src/github.com/marcosnils/cmt$ ls
cmd cmt.go iptables LICENSE magic_migrator.sh migrate README.md validate

netcom1@cloudlet1:~/cmt/src/github.com/marcosnils/cmt$ cmt --help
No command 'cmt' found, but there are 18 similar ones
cmt: command not found

Validation error with patched criu

Error in validation:
cmt validate --src pwd --dst [email protected]
2016/01/11 17:31:53 Error criu does not exist in dst

I am sure that the destination is the same and i tried it the other way too.

Add support for Pre/Post migrate hooks

The migration flow would be as follows:

• Validations (CPU / Memory / CRIU/RUNC versions, etc, IP, file transfer)

• Checkpoint

• Copy files

  (restore-before) >> Cambiar la ruta de AWS

• Restore
  (restore-fail) >> if restore fails
  (restore-success) >> cleanup

Improve README

😄

Allow to specify an identity file

As today only SSH agent authentication is supported. It'd be nice to allow to specify an identity file to perform remote executions.

Add --force parameter to ignore some validations

Such as:

• Version check (criu / runc)
• CPU check

Error in installing

go get github.com/marcosnils/cmt
# golang.org/x/crypto/ssh
/usr/local/go/src/golang.org/x/crypto/ssh/buffer.go:8: import /usr/local/go/pkg/linux_amd64/io.a: not a package file