stackstorm/packer-st2

Packer templates for building & deploying Vagrant & OVA with StackStorm installed

packer-st2's Issues

Packer: Break out linting

          Naturally it's always nicer to have the linting checks as a dedicated jobs in the CI pipeline like it was before (`inspec-lint`, `packer-lint` running in parallel as a dependency to the main "packer-build" task). Provides some better visuals and less monolytics.

But because there was already a lot of work, not a requirement. Just nice to have, if you'll have a few more cycles to spend on it.

Originally posted by @armab in #64 (comment)

Integrate with the st2cd Release Workflows

Releasing/Deploying a Vagrant box should be done automatically within st2cd release workflows.

Once a new StackStorm version is released, - run automation to deploy new Vagrant box.

Automation sketch that workflow should do:

• Version bump
• Changelog update
• New Github release
• Add instructions to https://github.com/StackStorm/discussions/wiki/Release-Process

Ship OVA via Github releases

Related to https://github.com/StackStorm/ova/issues/104

The way we'll distribute OVA is via Github Releases.

On every new git tag (aka Github release) CircleCI will build and upload the artifact to respective Github release page.

It means we should also start versioning the OVA and add CHANGELOG.md for this repo which is a good thing.

---

Resources for implementation

• https://medium.com/@jgautheron/publish-your-golang-binaries-in-github-with-circleci-e0b64cb21bf8
• https://jedkirby.com/blog/github-releases-with-circleci
• https://github.com/tcnksm/ghr

Add Vagrant as an installation method to st2docs

Include Vagrant & OVA Virtual Appliance as an installation method in st2docs.

Cover basic steps:

• Quick start: vagrant up & running
• OVA & Virtual Appliance
• Tests
• Point to the sources (this repo)

Switch from Ubuntu Xenial to Bionic

As part of the maintenance work, switch from Ubuntu Xenial which relies on py2 to Ubuntu Bionic with python3 and MongoDB 4 for the Vagrant box we produce.

Deploy box to Vagrant Cloud/Atlas

Add to the Packer pipeline steps to deploy produced Vagrant boxes to Vagrant Cloud/Atlas.

Start with Virtualbox provider, but eventually should be done for VMware boxes as well.

Auto-generate SSH/SSL secrets on first boot

There are concerns about security keys generated during the CI process and so considered insecure.
And while Vagrant is considered as a solution for demo purposes, OVA could be used in prod environments.

The following needs to be auto-generated on first VM boot:

• SSH host keys (see related https://github.com/StackStorm/ova/issues/21)
• Stanley private SSH key & take care of Authorized key
• SSL certificate used for nginx/st2web

Solution would be systemd configs which rely on ssh/nginx services (runs as "before" dependency) and creates keys/certs if they're not there already.
This will ensure that SSH/SSL secrets are unique and not the same across all the users after our Packer builds.

Apart of that #17, - need to document as insecure:

• vagrant Linux user with default vagrant password
• vagrant Linux user with default/known Hashicorp SSH private key

Install Virtualbox guest additions

To have fully operable VM running under Virtualbox, guest additions should be installed.

See https://www.packer.io/docs/builders/virtualbox-iso.html#guest-additions

Include CHANGELOG records in Github release

During the deploy CircleCI stage, parse CHANGELOG and copy-paste list of changes into Github Releases page.

Apart of that, include:

vagrant init stackstorm/st2 --box-version ${CIRCLE_TAG}
vagrant up

Remove Enterprise/EWC mentions

When someone SSH-in into a running container EWC/Enterprise is mentioned.

Remove those references as EWC is opensourced now:
https://stackstorm.com/2020/05/27/extreme-networks-donates-ewc-to-linux-foundation/

Build failure prior to running integration tests

Looks like a SIGTERM is received prior to running integration tests:

https://circleci.com/gh/StackStorm/packer-st2/79

This is an intermittent bug. Re-running CI again can "fix" the issue.

Per https://www.packer.io/docs/other/debugging.html#debugging-packer, use PACKER_LOG=1 to see if that sheds more light on what's going on...

Incorrect IP address range

Hello,

I am getting the following error, which I am unable to override.

The IP address configured for the host-only network is not within the
allowed ranges. Please update the address used to be within the allowed
ranges and run the command again.

  Address: 10.10.10.10
  Ranges: 192.168.56.0/21

Valid ranges can be modified in the /etc/vbox/networks.conf file. For
more information including valid format see:

  https://www.virtualbox.org/manual/ch06.html#network_hostonly

From what I understand the need to be changed or ST2 needs to run on a different IP address.

Release 3.7.0 Build Issue

The v3.7.0 build in the CircleCI CICD pipeline is failing with the below issue:

TASK [Create server] ***********************************************************
FAILED - RETRYING: Create server (5 retries left).
FAILED - RETRYING: Create server (4 retries left).
FAILED - RETRYING: Create server (3 retries left).
FAILED - RETRYING: Create server (2 retries left).
FAILED - RETRYING: Create server (1 retries left).
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Error: Error 422: sjc1 is not a valid facility
fatal: [localhost]: FAILED! => {"attempts": 5, "changed": false, "msg": "failed to set device state active, error: Error 422: sjc1 is not a valid facility"}
	to retry, use: --limit @/root/ova/.circleci/ansible/create.retry

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1   


Exited with code exit status 2
CircleCI received exit code 2

https://app.circleci.com/pipelines/github/StackStorm/packer-st2/1502/workflows/e109e3e4-bdfd-4e0a-8260-a1b89830a808/jobs/4282

Because of this failure, there is no updated Vagrant box in Vagrant cloud.
https://app.vagrantup.com/stackstorm/boxes/st2

Move from CircleCI/Packet.net to GH Actions

At the moment we build ST2 vagrant box + OVA by requesting a bare-metal instance at Packet.net, making a Packer build there and deploying produced artifact to the Vagrant Cloud.

Turns out GH Actions can now support Vagrant + Virtualbox (via nested virtualization? ).

Explore/research the https://stackoverflow.com/a/60380518/4533625 and https://github.com/jonashackt/vagrant-github-actions and moving the pipeline to GH Actions which will save us some $ and complexity by not using external Packet.net bare-metal instance. Side effects could be: slow/unstable build process, needs evaluation.

Add Hashicorp/Vagrant SSH key

Add the Hashicorp private key for vagrant linux user to get the vagrant up working.

Use /30 for host network, not /31

The use of a /31 prefix for the host network does not work on all operating systems. See StackStorm/st2docs#744

This should be changed to /30.

VMware & ovftool

Optimize ovf for VMware use with ovftool & established processes from ova repo.

Not sure yet if we need it for Community version.

Make sure ~/.st2/config file is only readable by the owner, delete ~/.st2/st2admin-token file

We should make the following changes to the published image:

• Make sure ~/.st2/config file which contains credentials is only readable by the owner (and also make sure that the owner user is correct, in vagrant case that should be vagrant user)
• Delete ~/.st2/st2admin-token file which contains cached admin auth token. It seems like that file is generated during the CI process and it's owned by root which means vagrant user can't read / write to it. We should delete it before generating the final artifact.

From Slack:

@kami yes, seen that before. It's indeed created during the CI runs.

It happens because we do `sudo st2 ...` somewhere and so token is saved with `root` permissions.

I even thought about submitting a PR for st2 so the dir `/home/${USERNAME}/.st2/` is created by StackStorm client with `setgid` flag, meaning token will take the folder's owner group.

But we were pretty time limited on this OVA work.
Can you create an issue in https://github.com/stackstorm/packer-st2 ?
Another issue I noticed during that work, - `.st2/config` has read-all permissions and so everyone can read st2 username:password.
This is bug by our `curl|bash` installer:
```$ ls -la ~/.st2/config 
-rw-r--r-- 1 vagrant vagrant 54 May 23 14:09 /home/vagrant/.st2/config

Release automation: add a CHANGELOG message on new st2 version

During the st2cd release automation, we need to include a message in CHANGELOG.md when new st2 version is released.

Otherwise we'll be in a strange situation when new packer-st2 release/tag is present, - but CHANGELOG is empty.

MOTD/Welcome message

Add MOTD/Welcome message with after logging in to default shell.
It'll show useful links to StackStorm docs, forums, Slack.

Extract ST2_VERSION and BOX_VERSION from the git tag

During the CircleCI deploy step (which happens on git tag/git release) we'll need to extract the ST2_VERSION and BOX_VERSION from the git tag.

That'll allow us to pin specific version, instead of using "latest" default.

Update release instructions

          One minor issue: we're trying to remove links to the forum whenever we can. Since your already editing this, could you adjust the links?

Otherwise, LGTM. It's wonderful to finally have this fixed!

Note: We will need to update other the release instructions and probably the st2cicd workflows for release after this gets merged.

Originally posted by @cognifloyd in #64 (review)

Warn about insecure login credentials used

We should warn a user that insecure login credentials are used: vagrant:vagrant and Hashicorp SSH key.
And so it's not recommended to use the box for prod-related environments before removing insecure creds.

Try to print a warning message via:

• MOTD (related to #15)
• Vagrantfile

Add 'system' integration tests

Add Inspec tests related to how system is configured and works: Vagrant user accounts, etc.

Ubuntu 16.04.06 x64 ISO no longer exists

Need to upgrade to 16.04.07 x64 iso

Expose st2web access in Vagrant

Open access to st2web UI via Virtualbox port fowarding.
Apparently, it's easy to do, having control over the Vagrant template we ship with the .box.

Create a KVM box

Would be nice if the vagrant box could run in KVM for those of us running Linux.

I know on my host i have to stop KVM to get VirtualBox running.

Here's the snippet in my Vagrantfile that i edit after i clone st2vagrant

    st2.vm.box = "elastic/ubuntu-16.04-x86_64"
    st2.vm.provider :libvirt do |vb|
      vb.host = "#{hostname}"
      vb.memory = 2048
      vb.cpus = 2
      vb.uri = "qemu:///system"
      vb.storage_pool_name = "images"
    end

Add 'stackstorm' integration tests

Add stackstorm-related Inspec Tests

Link stackstorm tests with an executable entrypoint available in user's bin, so user can invoke StackStorm integration tests

CI

Port Packet CI from ova repo.

Update the box Ubuntu18->Ubuntu20

Currently the build is based on Ubuntu18 iso which will be dropped in st2 v3.9.0
Switch to U20 box, update https://github.com/StackStorm/packer-st2/blob/master/st2.json and other places.

We run full packer build pipeline in Github Actions.