containers configuration tool
License: MIT License
This tool is deprecated you should use Concreate instead.
containers configuration tool
not supported yet, use tool directly from git
$ export PYTHONPATH="${CCT_REPO_PATH}:${PYTHONPATH}"then you can run in project dir:
$ python cct/cli/main.py -hpython cct/cli/main.py -v dummy.yamlif command completes successfully you should get 0 result code and no error logged on stdout
this variable and the CCT_MODULE_PATH_NAME should point to directory with image.yaml
jboss-cli.sh --connect command=:shutdown
$ docker run -it --rm -e CCT_CHANGES=http://raw.githubusercontent.com/containers-tools/cct/master/samples/jboss_cli.yaml cctjboss/wildfly:latest 1 ↵
2015-07-03 05:50:13,192 - cct - INFO - executing change samples.jboss_cli
2015-07-03 05:50:13,201 - cct - INFO - running jboss-cli command data-source remove --name=ExampleDS
2015-07-03 05:50:14,793 - cct - INFO - running jboss-cli command ls
2015-07-03 05:50:15,752 - cct - INFO - module jboss_cli succesfully processed all steps
Processed module: jboss_cli
setup : Passed
run_cli : Passed
run_cli : Passed
2015-07-03 05:50:15,753 - cct - INFO - executing command /opt/jboss/wildfly/bin/standalone.sh with args -b 0.0.0.0
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/jboss/wildfly
JAVA: /usr/lib/jvm/java/bin/java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
05:50:16,385 INFO [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final
05:50:16,827 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.2.Final
05:50:17,085 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: WildFly 8.2.0.Final "Tweek" starting
05:50:18,711 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
05:50:18,731 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.3.0.Final
05:50:18,744 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.3.0.Final
05:50:18,788 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 31) WFLYIO001: Worker 'default' has auto-configured to 8 core threads with 64 task threads based on your 4 available processors
05:50:18,799 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 32) JBAS010280: Activating Infinispan subsystem.
05:50:18,798 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.6.Final
05:50:18,818 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
05:50:18,825 INFO [org.jboss.as.security] (ServerService Thread Pool -- 45) JBAS013171: Activating Security Subsystem
05:50:18,831 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 38) JBAS012615: Activated the following JSF Implementations: [main]
05:50:18,838 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 40) JBAS011800: Activating Naming Subsystem
05:50:18,845 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 46) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.
05:50:18,865 INFO [org.jboss.as.security] (MSC service thread 1-6) JBAS013170: Current PicketBox version=4.0.21.Final
05:50:18,871 INFO [org.jboss.as.connector.logging] (MSC service thread 1-4) JBAS010408: Starting JCA Subsystem (IronJacamar 1.1.9.Final)
05:50:18,909 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) JBAS010417: Started Driver service with driver-name = h2
05:50:18,919 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
05:50:18,973 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) JBAS017502: Undertow 1.1.0.Final starting
05:50:18,974 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 47) JBAS017502: Undertow 1.1.0.Final starting
05:50:19,031 INFO [org.jboss.as.naming] (MSC service thread 1-5) JBAS011802: Starting Naming Service
05:50:19,032 INFO [org.jboss.as.mail.extension] (MSC service thread 1-3) JBAS015400: Bound mail session [java:jboss/mail/Default]
05:50:19,328 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 47) JBAS017527: Creating file handler for path /opt/jboss/wildfly/welcome-content
05:50:19,351 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) JBAS017525: Started server default-server.
05:50:19,378 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) JBAS017531: Host default-host starting
05:50:19,468 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.undertow.listener.default: org.jboss.msc.service.StartException in service jboss.undertow.listener.default: Could not start http listener
at org.wildfly.extension.undertow.ListenerService.start(ListenerService.java:137)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) [rt.jar:1.7.0_79]
at sun.nio.ch.Net.bind(Net.java:444) [rt.jar:1.7.0_79]
at sun.nio.ch.Net.bind(Net.java:436) [rt.jar:1.7.0_79]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) [rt.jar:1.7.0_79]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) [rt.jar:1.7.0_79]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) [rt.jar:1.7.0_79]
at org.xnio.nio.NioXnioWorker.createTcpConnectionServer(NioXnioWorker.java:182)
at org.xnio.XnioWorker.createStreamConnectionServer(XnioWorker.java:243)
at org.wildfly.extension.undertow.HttpListenerService.startListening(HttpListenerService.java:113)
at org.wildfly.extension.undertow.ListenerService.start(ListenerService.java:134)
... 5 more
05:50:19,722 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-5) JBAS015012: Started FileSystemDeploymentService for directory /opt/jboss/wildfly/standalone/deployments
05:50:19,763 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
05:50:19,767 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.serverManagement.controller.management.http: org.jboss.msc.service.StartException in service jboss.serverManagement.controller.management.http: JBAS015811: Failed to start the http-interface service
at org.jboss.as.server.mgmt.UndertowHttpManagementService.start(UndertowHttpManagementService.java:258) [wildfly-server-8.2.0.Final.jar:8.2.0.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
Caused by: java.lang.RuntimeException: java.net.BindException: Address already in use
at org.jboss.as.domain.http.server.ManagementHttpServer.start(ManagementHttpServer.java:156)
at org.jboss.as.server.mgmt.UndertowHttpManagementService.start(UndertowHttpManagementService.java:224) [wildfly-server-8.2.0.Final.jar:8.2.0.Final]
... 5 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) [rt.jar:1.7.0_79]
at sun.nio.ch.Net.bind(Net.java:444) [rt.jar:1.7.0_79]
at sun.nio.ch.Net.bind(Net.java:436) [rt.jar:1.7.0_79]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) [rt.jar:1.7.0_79]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) [rt.jar:1.7.0_79]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) [rt.jar:1.7.0_79]
at org.xnio.nio.NioXnioWorker.createTcpConnectionServer(NioXnioWorker.java:182)
at org.xnio.XnioWorker.createStreamConnectionServer(XnioWorker.java:243)
at org.jboss.as.domain.http.server.ManagementHttpServer.start(ManagementHttpServer.java:135)
... 6 more
05:50:19,777 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
("core-service" => "management"),
("management-interface" => "http-interface")
]) - failure description: {"JBAS014671: Failed services" => {"jboss.serverManagement.controller.management.http" => "org.jboss.msc.service.StartException in service jboss.serverManagement.controller.management.http: JBAS015811: Failed to start the http-interface service
Caused by: java.lang.RuntimeException: java.net.BindException: Address already in use
Caused by: java.net.BindException: Address already in use"}}
05:50:19,781 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
("subsystem" => "undertow"),
("server" => "default-server"),
("http-listener" => "default")
]) - failure description: {"JBAS014671: Failed services" => {"jboss.undertow.listener.default" => "org.jboss.msc.service.StartException in service jboss.undertow.listener.default: Could not start http listener
Caused by: java.net.BindException: Address already in use"}}
05:50:19,785 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([("subsystem" => "webservices")]) - failure description: {"JBAS014879: One or more services were unable to start due to one or more indirect dependencies not being available." => {
"Services that were unable to start:" => ["jboss.ws.config"],
"Services that may be the cause:" => ["jboss.remoting.remotingConnectorInfoService.http-remoting-connector"]
}}
05:50:19,788 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
("subsystem" => "ejb3"),
("service" => "remote")
]) - failure description: {"JBAS014771: Services with missing/unavailable dependencies" => ["jboss.ejb3.connector is missing [jboss.remoting.remotingConnectorInfoService.http-remoting-connector]"]}
05:50:19,862 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
JBAS014775: New missing/unsatisfied dependencies:
service jboss.remoting.remotingConnectorInfoService.http-remoting-connector (missing) dependents: [service jboss.ejb3.connector]
JBAS014777: Services which failed to start: service jboss.serverManagement.controller.management.http: org.jboss.msc.service.StartException in service jboss.serverManagement.controller.management.http: JBAS015811: Failed to start the http-interface service
service jboss.undertow.listener.default: org.jboss.msc.service.StartException in service jboss.undertow.listener.default: Could not start http listener
05:50:19,966 INFO [org.jboss.as] (Controller Boot Thread) JBAS015964: Http management interface is not enabled
05:50:19,967 INFO [org.jboss.as] (Controller Boot Thread) JBAS015954: Admin console is not enabled
05:50:19,967 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: WildFly 8.2.0.Final "Tweek" started (with errors) in 4108ms - Started 171 of 230 services (4 services failed or missing dependencies, 82 services are lazy, passive or on-demand)
we have some not nice wording in module.yaml.
we should change language options to: python, bash, script
we should not have name there - only description
Probably with implementation of possibility to run CCT operations as arbitrary user CCT started to report all modules/operations as NotRun even if everything was run successfully.
Logs and any other signs of launching the application server should be cleaned up after execution.
If you define CCT_CHANGES=, but then run cct as
RUN CCT_MODULES_PATH=/tmp/cct/ cct -v /tmp/cct/cct.yaml
The instructions in /tmp/cct/cct.yaml are ignored and the file named in CCT_CHANGES is executed.
I think if changes are supplied via command-line they should always be executed.
Provide a way to request/check version of a module in yaml file.
The current proposal is to use YAML, because of two reasons:
Since this tool is meant to have all login in modules (plugins) the goal for the descriptor file is to execute these plugins in a parametrized way, therefore we can focus on the readability of the descriptor file, because we don't need to put there a lot of specific things (like paths, specific values, etc). Another good feature of YAML is that we can easily (and it'll be still readable) define block of commands to be executed that provide atomic change (for example: add datasource definition to EAP).
Comments?
/cc @hhorak
If you have python on your cct module which is used at runtime but not install time, and that Python is broken (e.g. "import jinja" instead of "import jinja2"), cct will fail at build-time with very unusual errors e.g.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cct/module.py", line 188, in run
method(*args, **kwargs)
File "/tmp/cct/openshift-eap/install.py", line 17, in install
self.openshift_scripts()
File "/tmp/cct/openshift-eap/install.py", line 40, in openshift_scripts
jboss_home = os.getenv("JBOSS_HOME")
AttributeError: 'NoneType' object has no attribute 'getenv'
In the case of the above, install.py was fine, and fixing run.py (which is not referenced at all for build-time) made the problem go away. I think this is to do with the module scanner.
I will try to provide a test case!
CCT still lacks error handling, so there is a need to implement in and maybe we need some extension yaml file like saying that step should run:
or at least stop execution of other steps.
bash modules with functions doesn't seem like a good fit now - drop it.
Reading through Wiki and dummy.yaml I see there is term environment mentioned many times. From what I understood it only applies to environment defined by that yaml file, correct? Do you expect users to be able to override environment defined in yaml config by real environment variables - as they are basically the only (or at least easiest) way how to pass config options to a container.
My question relates to Nulecule integration - Nulecule spec defines params which are passed to artifact for the given provider. We could use dummy.yaml as artifact, but I expect it to live inside an image. How is then the user expected to modify it on runtime? Or is he expected to mount it in the container - that would be a bit problematic in kubernetes or OSv3 environment.
CCT should be extended to be able to execute its modules from zip file.
changes should be extended about some "module" parameter which should point to url/git repo with module
Currently we try to process module even if the variables are not resolved.
This is basically the same issue as jboss-dockerfiles/dogen#13 . All the same arguments apply. IMHO I would like to see that resolved in dogen first, but then cct be consistent with dogen.
If CCT modules would be available in the cct/MODULE_NAME directory - these copies should be used.
Currently modules are cloned every time. We should cache them locally and fetch only required changes.
Additionally - cloning module every times invalidates Docker cache.
in current state we run shell module via 'bash -x <module_path>' - if we make module runner configurable we will be able to run script in any language.
Some modules needs snippet for exact configuration and we need to provide a way to distribute them on CI without internet access.
pykwalify uses the same logging module as cct proper, so when cct's log level is set to debug (via -v, so is pykwalify, which is very verbose.
I think we should add a new flag for debug log level, and change -v so that it increases verbosity by default, but not enough to pykwalify is too noisy.
Or put another way, I think we should explicitly list out the logging levels provided by the logging module, and map the -v and (new) --debug (or whatever we call it) flags to different places, picked so we don't get the pykwalify output by default with -v.
The main goal for cct is to execute atomic updates decribed in the YAML format. We initially thought that it makes sense to make it possible to run selected part of the module from CLI, but I'm not sure if this makes sense. This would probably make the tool much more complicated.
It would be good to create a module and attach it to the release page, as a download, so other tools like Dogen could fetch it and use directly without any changes. This would take the tag and apply any changes to it, so that the resulting zip would be consumable. Additionally this zip would contain all dependencies (like fat jar).
@dbecvarik WDYT? It would simplify Dogen's installing of CCT and we would not care about deps.
architecture, use cases, contribution guide, packaging guide.,..
when Module.run method was refactored to _run , it was kept in a list of methods excluded form running - currently it is 'setup' and 'teardown' only.
in CCT modules invocation we have scheme:
name.package.operation
in script:
in python:
We should change name of python file to a dir containing image.yaml
2017-02-14 13:43:28,048 - cct - DEBUG - Cannot process module.yaml [Errno 2] No such file or directory: '/usr/lib/python2.7/site-packages/cct-0.0.1.dev0-py2.7.egg/cct/module.yaml'
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cct-0.0.1.dev0-py2.7.egg/cct/module.py", line 185, in __init__
with open(os.path.join(directory, "module.yaml")) as stream:
IOError: [Errno 2] No such file or directory: '/usr/lib/python2.7/site-packages/cct-0.0.1.dev0-py2.7.egg/cct/module.yaml'
It would be a nice feature if we would be able to add to image env variables and labels that are defined in module.yaml.
Low priority.
Dogen made artifact fetching better -> sync to same version.
https://github.com/jboss-dockerfiles/dogen/blob/develop/docs/descriptor.adoc#sources
Some recent change to entrypoint/cmd handling has broken the tutorial:
$ docker run -it -v $(pwd)/tutorial:/mnt -e CCT_CHANGES=/mnt/dummy.yaml asdasd
usage: cct [-h] [-v] [-q] [-l] [-s SHOW] [-c COMMAND] [-g GET_CHANGES]
[--version]
[changes [changes ...]]
snip
Error: unrecognized arguments: -b
-b is an argument in the CMD so should be being passed to the wildfly start script not cct.
After running the module we should create some easy to read and parse report of steps taken, and their statuses. This can be stored also in a Docker image for audit purposes.
Since this commit to the base module
containers-tools/base@cd294de
I get build-time failures such as
2016-07-28 10:28:37,870 - cct - DEBUG - inspecting /tmp/cct/base/tests/test_unit_module_shell.py
2016-07-28 10:28:37,870 - cct - DEBUG - importing module /tmp/cct/base/tests/test_unit_module_shell.py to cct.module.tests
Traceback (most recent call last):
<snip>
File "/tmp/cct/base/tests/test_unit_module_shell.py", line 2, in <module>
import mock
ImportError: No module named mock
Now that modules are being removed from cct to external repos, tests will eventually need adjusting to handle this. But how? Do we
The tests/test_unit_module_amq.py will need this when amq module goes for example.
The current PoC of this tool is written in Python. This doesn't necessarily mean that the end goal is to have it win Python. Python was selected because:
This issue is about selecting the right language to write this tool in. My personal favorite is here golang. This would let us to just add one static binary which would work everywhere. Pros/cons?
/cc @praiskup
Additionally all options in the modules should be provided to cct so it could print usage instructions. We would need to support something like cct MODULE --help where it would print all the information about expected parameters and what is possible.
Another call, like cct --list-modules should be implemented to list all available modules.
Logging should be enhanced, so module not found exception is printed - probably is a good idea to list available modules too.
This should not be hardcoded: https://github.com/containers-tools/cct/blob/master/cct/modules/jboss_cli/cct_module.py#L16
This issue talks about runtime execution (we want to customize container), in contrast to build time execution (we want to build container).
I would expect two ways of customizing the execution:
This is the easiest case where the descriptor file is baked into the image and we just act on the environment variables that are set in the container.
This is "internal" configuration - we let the cct tool react based on descriptor file we have baked in and environment variables that are provided.
In this way we inject a custom descriptor file to the container. This can be done using for example volumes or by getting it from a store like etcd. This way of configuring container heavily depends on the environment in which we're running the container.
This can be called also as "external" configuration, because we prepare the descriptor file externally and make it available to cct.
One thing to note: the cct tool does not care about how we inject the descriptor as along as it'll be available to the tool.
Any other ideas?
/cc @vpavlin
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.