test-kitchen/kitchen-habitat

A Test Kitchen Driver for Habitat

Test Kitchen

Website	https://kitchen.ci/
Source Code	https://kitchen.ci/docs/getting-started/introduction/
Slack	#test-kitchen channel on Chef Community Slack

Test Kitchen is an integration tool for developing and testing infrastructure code and software on isolated target platforms.

Getting Started Guide

To learn how to install and setup Test Kitchen for developing infrastructure code, check out the Getting Started Guide.

If you want to get going super fast, then try the Quick Start next...

Quick Start

Test Kitchen is a RubyGem and can be installed with:

gem install test-kitchen

If you use Bundler, you can add gem "test-kitchen" to your Gemfile and make sure to run bundle install.

Next add support to your library, Chef cookbook, or empty project with kitchen init:

kitchen init

A kitchen.yml file will be created in your project base directory. This file describes your testing configuration; what you want to test and on which target platforms. Each of these suite and platform combinations are called instances. By default your instances will be converged with Chef Solo and run in Vagrant virtual machines.

Get a listing of your instances with:

kitchen list

Run Chef Infra Client on an instance, in this case default-ubuntu-2004, with:

kitchen converge default-ubuntu-2004

Destroy all instances with:

kitchen destroy

You can clone a Chef Infra cookbook project that contains Test Kitchen support and run through all the instances in serial by running:

kitchen test

Usage

There is help included with the kitchen help subcommand which will list all subcommands and their usage:

kitchen help test

More verbose logging for test-kitchen can be specified when running test-kitchen from the command line using:

kitchen test -l debug

For the provisioner (e.g. chef-solo or chef-infra) add a log_level item to the provisioner section of the .kitchen.yml For more information on setting log_level see the configuration documentation.

Documentation

Documentation is being added on the Test Kitchen website. Please read and contribute to improve them!

Versioning

Test Kitchen aims to adhere to Semantic Versioning 2.0.0.

Community and Ecosystem

If you would like to see a few of the plugins or ecosystem helpers, please look at ECOSYSTEM.md.

Development

• Source hosted at GitHub
• Report issues/questions/feature requests on GitHub Issues

Pull requests are very welcome! Make sure your patches are well tested. Ideally create a topic branch for every separate change you make. For example:

1. Fork the repo
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Authors

Created by Fletcher Nichol ([email protected]).

Maintained by Chef Software and a growing community of contributors.

License

Apache License, Version 2.0 (see LICENSE)

kitchen-habitat's Issues

supervisor 0.56.0 breaks converge

It appears that the release of 0.56.0 has broken converges. Converging a fresh container results in an infinite loop on the following:

   ✗✗✗
   ✗✗✗ [Err: 1] IoError while reading or writing ctl secret, /hab/sup/default/CTL_SECRET, Permission denied (os error 13)
   ✗✗✗

Habitat license acceptance

Currently there is no way to use habitat versions greater than 0.79.1 due to license acceptance.

Multiple Package Support

In order to accurately simulate production systems we need to be able to install multiple packages in individual supervisors.

Omitting hab_sup_bind from service that requires it does not fail 'kitchen converge'

I have a kitchen instance that's using docker + habitat that was pausing/waiting forever at at this location during kitchen converge:

[... snip ...]
    75.23 KB / 75.23 KB - [==============================] 100.00 % 42.00 MB/s  
       ☛ Verifying core/zlib/1.2.11/20180608050617
       ✓ Installed core/cacerts/2018.03.07/20180608102212
       ✓ Installed core/gcc-libs/7.3.0/20180608091701
       ✓ Installed core/glibc/2.27/20180608041157
       ✓ Installed core/haproxy/1.6.11/20180609190214
       ✓ Installed core/linux-headers/4.15.9/20180608041107
       ✓ Installed core/openssl/1.0.2n/20180608102213
       ✓ Installed core/pcre/8.41/20180608092740
       ✓ Installed core/zlib/1.2.11/20180608050617
       ✓ Installed kmott/haproxy/1.8.13/20180807232806
       ★ Install of kmott/haproxy/1.8.13/20180807232806 complete with 9 new packages installed.
D      Attempting to execute command - try 1 of 1.
D      [SSH] kitchen@localhost<{:user_known_hosts_file=>"/dev/null", :port=>32791, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/kmott/IdeaProjects/ford/ford-plans/haproxy/.kitchen/docker_id_rsa"], :auth_methods=>["publickey"], :verify_host_key=>false, :logger=>#<Logger:0x00007f9b68b88188 @level=4, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f9b68b88110 @datetime_format=nil>, @formatter=nil, @logdev=#<Logger::LogDevice:0x00007f9b68b880c0 @shift_period_suffix=nil, @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<IO:<STDERR>>, @mon_owner=nil, @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007f9b68b88048>>>, :password_prompt=>#<Net::SSH::Prompt:0x00007f9b68b88020>, :user=>"kitchen"}> (sh -c '
TEST_KITCHEN="1"; export TEST_KITCHEN
        
        
                [ -f ./run.pid ] && echo "Removing previous supervisor and unloading package. "
        [ -f ./run.pid ] && sudo -E hab svc unload kmott/haproxy/1.8.13/20180807232806
        [ -f ./run.pid ] && sleep 5
        [ -f ./run.pid ] && sudo -E kill $(cat run.pid)
        [ -f ./run.pid ] && sleep 5

        echo "Running kmott/haproxy/1.8.13/20180807232806."
                  [ -f ./run.pid ] && rm -f run.pid
          [ -f ./nohup.out ] && rm -f nohup.out
          nohup sudo -E hab sup run  --peer jenkins-master --topology standalone --strategy rolling --channel stable & echo $! > run.pid

          until sudo -E hab svc status
          do
            sleep 1
          done

          sudo -E hab svc load kmott/haproxy/1.8.13/20180807232806  --topology standalone --strategy rolling --channel stable

          until sudo -E hab svc status | grep kmott/haproxy/1.8.13/20180807232806
          do
            sleep 1
          done

          [ -f ./nohup.out ] && cat nohup.out || (echo "Failed to start the supervisor." && exit 1)

')
       Running kmott/haproxy/1.8.13/20180807232806.
       nohup: appending output to ‘nohup.out’
       ✗✗✗
       ✗✗✗ Connection refused (os error 111)
       ✗✗✗
       ✗✗✗
       ✗✗✗ Connection refused (os error 111)
       ✗✗✗
       ✗✗✗
       ✗✗✗ Connection refused (os error 111)
       ✗✗✗
       ✗✗✗
       ✗✗✗ Connection refused (os error 111)
       ✗✗✗
       No services loaded.
       The kmott/haproxy/1.8.13/20180807232806 service was successfully loaded

The haproxy service I was trying to start had a pkg_binds requirement that I forgot to add. For some reason, it just stayed at that last prompt above for successfully loaded, and never returned control back to the caller (or error'd out saying you have a pkg_binds requirement that cannot be met).

It should be pretty easy to reproduce by adding a kitchen suite for any package that has a pkg_binds requirement, but leave out hab_sup_bind from the provisioner section.

ident regex misidentifies packages with `-` in the version

sensu beta versions are of the following format:
2.0.0-beta.4-1

and a resulting hart artifact looks like so:
local-sensu-backend-2.0.0-beta.4-1-20181001132122-x86_64-linux.hart

this results in kitchen-habitat throwing an ident mismatch error:

✗✗✗
✗✗✗ [Err: 0] hab-sup(ER)[components/sup/src/error.rs:464:9]: Artifact ident local/sensu-backend/2.0.0-beta.4-1/20181001132122 for `local-sensu-backend-2.0.0-beta.4-1-20181001132122-x86_64-linux.hart' does not match expected ident local/sensu-backend-2.0.0-beta.4/1/20181001132122
✗✗✗

How to build and test a local package

I would like to use kitchen to test and build a local package with docker as the provisioner and inspec as the verifier. I have a structure similar to the one in the tutorial: https://github.com/habitat-sh/habitat-example-plans. I think I would like to put a .kitchen.yml inside of the habitat folder there.

Issue with starting supervisor under nohup

When running kitchen converge I get the following error:

[kitchen@95b91cf4607d ~]$ tailf nohup.out
→ Using core/zlib/1.2.8/20170513201911
✓ Installed core/hab-launcher/8133/20180810175521
★ Install of core/hab-launcher/8133/20180810175521 complete with 1 new packages installed.
hab-sup(MR): Supervisor Member-ID a51483973986496fa19f8dbe0b20e494
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(MR): Starting ctl-gateway on 127.0.0.1:9632
hab-sup(MR): Starting http-gateway on 0.0.0.0:9631
hab-sup(AG): The skylerto/vault/0.10.5/20180817131403 service was successfully loaded
hab-launch(SV): Supervisor process killed by signal 1; shutting everything down now
ERROR 2018-08-20T14:49:14Z: hab_launch: Launcher exiting with code 86

This happens consistently.

Using the docker provider, and centos docker image.

This is a confirmed issue on v0.9.0 and v0.10.0.

Confirmed earliest break time was Friday, August 17, 2018.

Support multiple converges

A second converge results in an error from the supervisor that the service is already loaded. I think this could be resolved either through:

a) Adding --strategy at-once to the hab svc load command, and skipping it on subsequent runs, or
b) Unloading the service if its running, installing the new pkg, and then loading the service again.

Thoughts?

Repro steps:

---
driver:
  name: vagrant

provisioner:
  name: habitat
  install_latest_artifact: true
  package_origin: jvogt
  package_name: habichefdemo

platforms:
  - name: centos-7

suites:
  - name: default-habitat

1. hab pkg build
2. kitchen converge
3. hab pkg build
4. kitchen converge

...
      ✗✗✗
       ✗✗✗ [Err: 3] Service already loaded, unload 'jvogt/habichefdemo/0.1.0/20190118055505' and try again
       ✗✗✗
...

Create an integration test suite

There should be an integration test suite for the kitchen habitat plugin and it should be added to our TravisCI build jobs.

release a > 0.11.0 version

The current latest released version 0.11.0 has a pessimistic version constraint on Test Kitchen ~1.4. The version constraint was loosened on master in May to require anything between 1.14 and 3.0, but that change is currently unreleased.

Chef Workstation is shipping with Test Kitchen 2.x currently, so installing this gem from RubyGems results in the latest TK 1.x (1.25.0 at the moment) being installed and kitchen-habitat is unable to be found by TK 2 at runtime.

Error Output

 > kitchen list
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the 'habitat' provisioner from the load path. Did you mean: chef_apply, chef_base, chef_solo, chef_zero, dokken, dummy, policyfile_zero, shell ? Please ensure that your provisioner is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

> chef gem install kitchen-habitat
Fetching mixlib-shellout-2.4.4.gem
Fetching test-kitchen-1.25.0.gem
Fetching net-ssh-4.2.0.gem
Fetching kitchen-habitat-0.11.0.gem
Successfully installed mixlib-shellout-2.4.4
Successfully installed net-ssh-4.2.0
Successfully installed test-kitchen-1.25.0
Successfully installed kitchen-habitat-0.11.0
4 gems installed

> chef gem list test-kitchen

*** LOCAL GEMS ***

test-kitchen (2.3.3, 1.25.0)

> kitchen --version
Test Kitchen version 2.3.3

> kitchen list
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the 'habitat' provisioner from the load path. Did you mean: chef_apply, chef_base, chef_solo, chef_zero, dokken, dummy, policyfile_zero, shell ? Please ensure that your provisioner is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

CI build missing

We could use some sort of TravisCI style auto-build as a part of inbound pull requests to validate builds. it should hopefully include

• Style
• Unit Tests (when available)
• Some simple gem compilation and functional tests on basic habitat packages in the core origin

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

bundler

Gemfile

• rake ">= 11.0"
• rspec "~> 3.2"
• chefstyle "2.2.3"

github-actions

.github/workflows/linters.yml

• actions/checkout v4
• ruby/setup-ruby v1

.github/workflows/unit.yml

• actions/checkout v4
• ruby/setup-ruby v1

---

• Check this box to trigger a request for Renovate to run again on this repository

Use public signing key cache from host

It would be nice to not have to upload a public origin key to bldr in order to test a hart.

Right now it will fail if you hab setup a new origin on a workstation, build a package, and then try to test it. Looks a bit like this:

$ hab pkg build (which creates results/jvtest-acme_app_1-0.1.0-20190531222846-x86_64-linux.hart)
$ kitchen converge
....
       » Installing jvtest/acme_app_1/0.1.0/20190531222846
       ☛ Verifying jvtest/acme_app_1/0.1.0/20190531222846
       ↓ Downloading jvtest-20190531222734 public origin key
       ✗✗✗
       ✗✗✗ [Err: 0] hab-sup(ER)[components/sup/src/error.rs:459:55]: [404 Not Found]
       ✗✗✗

Add Unit Test Coverage for artifact_name_to_package_ident_regex function

artifact_name_to_package_ident_regex was utilized in #25 and there is no unit test coverage. Cover cases:

• .hart file with - in the package and origin names
• .hart file without - in the package and origin names

Basic Unit Tests Needed

We need some basic RSpec coverage using test-kitchen/kitchen-vagrant as an example.

Basic Windows Support

Support for a windows supervisor is needed. functional tests should include IIS site packages and windows services

Feat: allow taking from last built hart to be installed

It would be nice to just specify package_origin and package_name, and kitchen should first look inside results folder for a previously built .hart before committing to going to the depot for installation.

Currently in order to do any local testing without deploying to the depot, I need to specify the entire artifact_name down to the version and release. This makes it hard to do complete local testing before deploying artifact into "public".

depot_url is not respected

Setting a value for depot_url is ignored by the hab CLI. It's not clear if this is because the env var the provisioner sets is not respected by the CLI or the provisioner is not setting the value in the right execution context.

driver:
  name: docker

provisioner:
  name: habitat
  depot_url: https://totally.obviously.fake.com

platforms:
  - name: centos-7.4

suites:
  - name: postgresql
    provisioner:
      package_origin: core
      package_name: postgresql
    driver:
      instance_name: postgresql
      forward:
        - 5432:5432

This configuration, for example, successfully converges while pulling packages from the default depot.

Error using local artifact if origin has hyphen

When setting artifact_name in the provider, if the origin has a hyphen, the package ident gets incorrectly parsed.

e.g.

origin = "my-origin"
package name = "my-package"

the package ident should be: my-origin/my-package/[version]/build]
instead it gets parsed to: my/origin-my-package/[version]/[build]

Possible solution:

• Require "package_origin" to be set and use it to parse the pkg_ident correctly

I think I'd like to also suggest allowing a "use_lastbuild" option to parse pkg_artifact and pkg_ident directly from the lastbuild.env in the results directory.