clusterhq/flocker-docker-plugin

SUPERCEDED, functionality is now part of Flocker

flocker-docker-plugin's Issues

Use with Docker API

I have a quick question regarding the flocker docker plugin.
I should be able to use it if I'm not using Compose, right?
I intend to drive each host's Docker API (using Salt) - flocker should take care of ensuring volumes follow containers around?

restarting results in stray socket

Rebooting a machine results in:

2015-07-05 13:31:05-0400 [-]   File "/usr/local/lib/python2.7/dist-packages/twisted/application/internet.py", line 131, in _getPort
2015-07-05 13:31:05-0400 [-]     'listen%s' % (self.method,))(*self.args, **self.kwargs)
2015-07-05 13:31:05-0400 [-]   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 405, in listenUNIX
2015-07-05 13:31:05-0400 [-]     p.startListening()
2015-07-05 13:31:05-0400 [-]   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/unix.py", line 302, in startListening
2015-07-05 13:31:05-0400 [-]     raise CannotListenError, (None, self.port, le)
2015-07-05 13:31:05-0400 [-] twisted.internet.error.CannotListenError: Couldn't listen on any:/usr/share/docker/plugins/flocker.sock: [Errno 98] Address already in use.

The Flocker Docker plugin should remove stale socket files.

Anonymous volumes can cause confusingly names volumes

e.g.:
When moving volumes using swarm node constraints, I also see some extra datasets being created with machine generated names. What are these for? Can they be avoided?

ashwinp@flocker-aws:~/clusters/swarm$ flocker-volumes list
DATASET                                SIZE      METADATA                                     STATUS         SERVER                  
4edea715-ab6e-4ad9-86da-57c2e39d39d1   100.00G   name=ba39eb98876792cde7690142322b16be7ae88   attached ✅   df2c4a33 (172.31.2.206) 
                                                 dcc95ff2d049cd0dade67c084c6                                                         
942f09b3-1124-43a0-9ff2-c85f57bb3b36   100.00G   name=e5d93a2d6d4c79d910ffe00d4195b18776453   attached ✅   df2c4a33 (172.31.2.206) 
                                                 4351fd4a80dc953b328654c9ec9                                                         
7b0c8197-bc5f-4399-9287-1e69b57bf7f2   100.00G   name=6dd1e22aced8372de2e6a29e256d6edc07936   attached ✅   df2c4a33 (172.31.2.206) 
                                                 ce4d3b4bda4cd61394c54183b5a                                                         

This behaviour should be documented.

flocker dataset api doesn't always return a primary

For example, a user ran across this: https://docs.clusterhq.com/en/1.0.2/labs/installer.html#comment-2141822548

2015-07-17 06:20:13+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] while processing configured datasets
Traceback (most recent call last):
Failure: twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <type 'exceptions.keyerror'="">: 'primary'
/usr/local/lib/python2.7/dist-packages/treq/client.py:48:connectionLost
/usr/local/lib/python2.7/dist-packages/treq/content.py:35:connectionLost
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:393:callback
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:501:_startRunCallbacks
--- <exception caught="" here=""> ---
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:588:_runCallbacks
/usr/local/lib/python2.7/dist-packages/flockerdockerplugin/adapter.py:182:check_dataset_exists
]]
2015-07-17 06:20:13+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Stopping factory <twisted.web.client._http11clientfactory instance="" at="" 0x7f180e4e5a28="">

We can workaround this in flocker-docker-plugin by making check_dataset_exists more lenient about this, assuming it's a transitory condition.

Flocker plugin doesn't cope with datasets without name metadata

It expects that all datasets have been created with a name.

Missing configuration causes unhandled errors

exceptions.Exception: Not enough information to construct TLS context: user_crt: FilePath('/etc/flocker/plugin.crt'), cluster_crt: FilePath('/etc/flocker/cluster.crt'), user_key: FilePath('/etc/flocker/plugin.key'), target_hostname: None


Failed to load application: Not enough information to construct TLS context: user_crt: FilePath('/etc/flocker/plugin.crt'), cluster_crt: FilePath('/etc/flocker/cluster.crt'), user_key: FilePath('/etc/flocker/plugin.key'), target_hostname: None

This should probably be an error like "docker daemon hostname not specified".

Throw away this repository as soon as next Flocker release happens

README should point to new docs or something.

docker has changed plugin paths

we need to update the plugin to match the new docker plugin path. flocker docker plugin is now broken on experimental nightly builds as well as upcoming 1.8 stable release.

simplest fix is to change the socket to listen on /run/docker/plugins in powerstripflocker.tac.

if we care about backward compat, we could make the plugin listen on two sockets, flocker.sock in both of /usr/share/docker/plugins and /run/docker/plugins.

we should also update the docs to tell users to mkdir -p /run/docker/plugins. or we could make the flocker docker plugin FilePath(...).makedirs() itself, and update the docs to remove that instruction.

bonus points if we could at the same time fix the bug which causes the plugin to fail to start if there is a stray socket file.

when fixing this, bump the version in setup.py to 0.3 so that users can upgrade by simply running sudo pip install ... again as per the docs.

Doesn't stop user from ripping volumes out from under running containers

Currently it's possible for "user error" (start container α with volume A on host 1, start container β with volume A on host 2 without stopping container α first) to cause this plugin to cause volumes to be ripped out from underneath running containers, because it hasn't made flocker aware of when a container is running using a volume. (This is despite the Docker plugins API providing Flocker with sufficient information to deduce this.)

In my experience this makes docker grumpy and in need of a restart.

It should be possible to fix this once https://clusterhq.atlassian.net/browse/FLOC-1425 is delivered by making the flocker plugin toggle on and off the lease boolean on the flocker volumes api when >0 containers are started/stopped.

We need to be careful about what happens when docker restarts or the plugin restarts (or the server reboots!). I think that assuming that no containers are running the first time the plugin reconnects to Docker in the handshake is probably a safe bet (it will receive Mount() requests as they come into use) but these cases require careful thought and thorough (automated) testing.

schema not supported

docker run -ti -v mytestvol:/data --volume-driver=flocker busybox sh

Aug  5 11:58:48 localhost flocker-docker-plugin: 2015-08-05 11:58:48+0000 [-] path: {u'Name': u'test'}
Aug  5 11:58:48 localhost flocker-docker-plugin: 2015-08-05 11:58:48+0000 [-] "-" - - [05/Aug/2015:11:58:47 +0000] "POST /VolumeDriver.Path HTTP/1.1" 200 652 "-" "Go 1.1 package http"
Aug  5 11:58:48 localhost flocker-docker-plugin: 2015-08-05 11:58:48+0000 [-] Unhandled error in Deferred:
Aug  5 11:58:48 localhost flocker-docker-plugin: 2015-08-05 11:58:48+0000 [-] Unhandled Error
Aug  5 11:58:48 localhost flocker-docker-plugin: Traceback (most recent call last):
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/treq/client.py", line 187, in request
Aug  5 11:58:48 localhost flocker-docker-plugin: bodyProducer=bodyProducer)
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/twisted/web/client.py", line 1857, in request
Aug  5 11:58:48 localhost flocker-docker-plugin: deferred = self._agent.request(method, uri, headers, bodyProducer)
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/twisted/web/client.py", line 1925, in request
Aug  5 11:58:48 localhost flocker-docker-plugin: deferred = self._agent.request(method, uri, headers, bodyProducer)
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/twisted/web/client.py", line 1732, in request
Aug  5 11:58:48 localhost flocker-docker-plugin: d = self._agent.request(method, uri, headers, bodyProducer)
Aug  5 11:58:48 localhost flocker-docker-plugin: --- <exception caught here> ---
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/twisted/web/client.py", line 1558, in request
Aug  5 11:58:48 localhost flocker-docker-plugin: endpoint = self._getEndpoint(parsedURI)
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/twisted/web/client.py", line 1542, in _getEndpoint
Aug  5 11:58:48 localhost flocker-docker-plugin: return self._endpointFactory.endpointForURI(uri)
Aug  5 11:58:48 localhost flocker-docker-plugin: File "/usr/lib/python2.7/site-packages/twisted/web/client.py", line 1422, in endpointForURI
Aug  5 11:58:48 localhost flocker-docker-plugin: raise SchemeNotSupported("Unsupported scheme: %r" % (uri.scheme,))
Aug  5 11:58:48 localhost flocker-docker-plugin: twisted.web.error.SchemeNotSupported: Unsupported scheme: ''
Aug  5 11:58:48 localhost flocker-docker-plugin: 2015-08-05 11:58:48+0000 [-] path: {u'Name': u'myvol'}
Aug  5 11:58:48 localhost flocker-docker-plugin: 2015-08-05 11:58:48+0000 [-] "-" - - [05/Aug/2015:11:58:47 +0000] "POST /VolumeDriver.Path HTTP/1.1" 200 652 "-" "Go 1.1 package http"

Experimental version

# docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Experimental (client): true
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64
Experimental (server): true

[root@mdm2 flocker]# systemctl status -l flocker-docker-plugin
flocker-docker-plugin.service - flocker-plugin - flocker-docker-plugin job file
   Loaded: loaded (/etc/systemd/system/flocker-docker-plugin.service; disabled)
   Active: active (running) since Wed 2015-08-05 12:03:33 UTC; 5s ago
 Main PID: 14568 (twistd)
   CGroup: /system.slice/flocker-docker-plugin.service
           └─14568 /bin/python /usr/bin/twistd -noy /opt/flocker/flocker-1.0.3/flocker-tutorial/lib/python2.7/site-packages/flockerdockerplugin/flockerdockerplugin.tac

Aug 05 12:03:33 mdm2.scaleio.local systemd[1]: Starting flocker-plugin - flocker-docker-plugin job file...
Aug 05 12:03:33 mdm2.scaleio.local systemd[1]: Started flocker-plugin - flocker-docker-plugin job file.

I am running this on CentOS, with VFS, but would like to get this working.

vague error on centos 7

 sudo docker run -v name:/data --volume-driver=flocker busybox sh
Error response from daemon: no such file or directory

sudo docker --version
Docker version 1.7.0-dev-experimental, build 4caa939

cat /etc/systemd/system/flocker-plugin.service
[Unit]
Description=flocker-plugin - flocker-plugin job file

[Service]
Environment=FLOCKER_CONTROL_SERVICE_BASE_URL=https://172.31.23.166:4523/v1
Environment=MY_NETWORK_IDENTITY=172.31.23.165
ExecStart=/usr/bin/twistd -noy flockerdockerplugin.tac
WorkingDirectory=/root/fdp

[Install]
WantedBy=multi-user.target

ps ax | grep flocker
12748 ?        Ss     0:01 /bin/python /usr/bin/twistd -noy flockerdockerplugin.tac
13331 pts/1    S+     0:00 grep --color=auto flocker

In here there is an errr Bailing out because we couldn't find a path, passing docker an error:

 systemctl status -l flocker-plugin
flocker-plugin.service - flocker-plugin - flocker-plugin job file
   Loaded: loaded (/etc/systemd/system/flocker-plugin.service; enabled)
   Active: active (running) since Sun 2015-06-21 22:18:38 UTC; 2min 45s ago
 Main PID: 12748 (twistd)
   CGroup: /system.slice/flocker-plugin.service
           └─12748 /bin/python /usr/bin/twistd -noy flockerdockerplugin.tac

Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTPChannel,0,]
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTPChannel,0,] "-" - - [21/Jun/2015:22:19:09 +0000] "POST /VolumeDriver.Create HTTP/1.1" 200 13 "-" "Go 1.1 package http"
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTPChannel,0,] path: {u'Name': u'name'}
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTPChannel,0,] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x25407e8>
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x25431b8>
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x25407e8>
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Bailing out because we couldn't find a path, passing docker an error:
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client]     unable to find name
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] "-" - - [21/Jun/2015:22:19:09 +0000] "POST /VolumeDriver.Path HTTP/1.1" 200 48 "-" "Go 1.1 package http"
Jun 21 22:19:10 ip-172-31-23-165.us-west-2.compute.internal twistd[12748]: 2015-06-21 22:19:10+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x25431b8>

stray socket files stop plugin starting

I suspect that the plugin fails to start if there is a stray socket file.

Make default volume size configurable

It is desirable to be able to ask Flocker to create volumes other than 100GB in size, for example on EBS where you might not want to pay for lots more storage than you need.

See https://groups.google.com/forum/#!topic/docker-user/Ouq94UGpBwU

The constant at https://github.com/ClusterHQ/flocker-docker-plugin/blob/master/flockerdockerplugin/adapter.py#L18 could be changed to be an optional environment variable, and then the user would be able to configure sizes globally for the plugin at least in the upstart script for the flocker plugin.

not all exceptions get logged

for example, #9 was only findable by running twistd --debug -noy, it didn't show up in the flocker plugin logs