stackstorm/st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html

StackStorm is a platform for integration and automation across services and tools, taking actions in response to events. Learn more at www.stackstorm.com.

---

TL;DR

• Install Get yourself a clean 64-bit Linux box that fits the system requirements. Run the installer script:

  curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe

• Read the docs: https://docs.stackstorm.com/index.html

• Questions? Check out forum.stackstorm.com

• Or join our Slack community

StackStorm Overview

About

StackStorm is a platform for integration and automation across services and tools. It ties together your existing infrastructure and application environment so you can more easily automate that environment -- with a particular focus on taking actions in response to events.

StackStorm helps automate common operational patterns. Some examples are:

• Facilitated Troubleshooting - triggering on system failures captured by Nagios, Sensu, New Relic and other monitoring, running a series of diagnostic checks on physical nodes, OpenStack or Amazon instances, and application components, and posting results to a shared communication context, like Slack or JIRA.
• Automated remediation - identifying and verifying hardware failure on OpenStack compute node, properly evacuating instances and emailing VM about potential downtime, but if anything goes wrong - freezing the workflow and calling PagerDuty to wake up a human.
• Continuous deployment - build and test with Jenkins, provision a new AWS cluster, turn on some traffic with the load balancer, and roll-forth or roll-back based on NewRelic app performance data.

StackStorm helps you compose these and other operational patterns as rules and workflows or actions; and these rules and workflows - the content within the StackStorm platform - are stored as code which means they support the same approach to collaboration that you use today for code development and can be shared with the broader open source community via StackStorm Exchange.

Who is using StackStorm?

See the list of known StackStorm ADOPTERS.md and Thought Leaders.

How it works

StackStorm architecture

StackStorm plugs into the environment via an extensible set of adapters: sensors and actions.

• Sensors are Python plugins for inbound integration that watch for events from external systems and fire a StackStorm trigger when an event happens.

• Triggers are StackStorm representations of external events. There are generic triggers (e.g., timers, webhooks) and integration triggers (e.g., Sensu alert, JIRA issue updated). A new trigger type can be defined by writing a sensor plugin.

• Actions are StackStorm outbound integrations. There are generic actions (SSH, HTTP request), integrations (OpenStack, Docker, Puppet), or custom actions. Actions are either Python plugins, or any scripts, consumed into StackStorm by adding a few lines of metadata. Actions can be invoked directly by user via CLI, API, or the web UI, or used and called as part of automations - rules and workflows.

• Rules map triggers to actions (or to workflows), applying matching criterias and map trigger payload data to action inputs.

• Workflows stitch actions together into "uber-actions", defining the order, transition conditions, and passing context data from one action to the next. Most automations are multi-step (eg: more than one action). Workflows, just like "atomic" actions, are available in the action library, and can be invoked manually or triggered by rules.

• Packs are the units of content deployment. They simplify the management and sharing of StackStorm pluggable content by grouping integrations (triggers and actions) and automations (rules and workflows). A growing number of packs is available on the StackStorm Exchange. Users can create their own packs, share them on GitHub, or submit them to the StackStorm Exchange organization.

• Audit trail is the historical list of action executions, manual or automated, and is recorded and stored with full details of triggering context and execution results. It is also captured in audit logs for integrating with external logging and analytical tools: LogStash, Splunk, statsd, or syslog.

StackStorm is a service with modular architecture. It is comprised of loosely coupled microservice components that communicate over a message bus, and scales horizontally to deliver automation at scale. StackStorm has a full REST API, CLI client, and web UI for admins and users to operate it locally or remotely, as well as Python client bindings for developer convenience.

StackStorm is an established project and remains actively developed by a broad community.

Documentation

Additional documentation, including installation procedures, action/rule/workflow authoring, and how to setup and use triggers/sensors can be found at https://docs.stackstorm.com.

Hacking / Contributing

To set up a development environment and run StackStorm from sources, follow these instructions.

For information on how to contribute, our style guide, coding conventions and more, please visit the Development section in our documentation.

Security

If you believe you found a security issue or a vulnerability, please send a description of it to our private mailing list at info [at] stackstorm [dot] com.

Once you've submitted an issue, you should receive an acknowledgment from one our of team members in 48 hours or less. If further action is necessary, you may receive additional follow-up emails.

For more information, please refer to https://docs.stackstorm.com/latest/security.html

Copyright, License, and Contributor Agreement

Copyright 2020 The StackStorm Authors. Copyright 2019 Extreme Networks, Inc. Copyright 2014-2018 StackStorm, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

http://www.apache.org/licenses/LICENSE-2.0

By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the project, present and future, pursuant to the license of the project.

st2's Issues

Positional arguments don't work for run-remote-script (v. 0.6.0)

Maybe same on later, I haven't checked.

MESSAGE: Additional properties are not allowed (u'cmd' was unexpected)

Failed validating 'additionalProperties' in schema:
    {'additionalProperties': False,
...

Better stdout support

Unless I've missed something in the docs, the stdout of runners is restricted to a sub-portion of the overall runner output.

While most tools that we would house under st2 are cron, batch, daemon, and other types of non-interactive actions, there are some reporting tools where the full, unaltered output is necessary.

The best I've been able to come up with is reviewing the results of the last runner through the audit history.

I noticed that the Roadmap includes better output support -- is this the same thing?

Preserve type in ActionChain publish variables

Since ActionChain publishing uses Jinja the types for published variables are not preserved and everything gets a string representation.

Fix either by switch all this to YAQL or some other post-jinja magic.

st2 run should show the output of the failed task

• st2 commands should show the error of the failed task. The task failed but I have to go searching for why. It definitely slows down the development workflow. It would be awesome if we could include the failure of the last task in the st2 run output. Something like below.

vagrant@st2express:/opt/stackstorm/packs$ st2 run deployer.environment.create domain=BLAH name=master owner=jb
...
id: 551986fd9c99384380121844
action.ref: deployer.environment.create
status: failed
start_timestamp: 2015-03-30T17:25:17.218963Z
end_timestamp: None
+--------------------------+-----------+-------------------------------+-------------------------------+-------------------------------+
| id                       | status    | task                          | action                        | start_timestamp               |
+--------------------------+-----------+-------------------------------+-------------------------------+-------------------------------+
| 551986fd9c99384380121846 | succeeded | get_current_epoch             | deployer.epoch                | Mon, 30 Mar 2015 17:25:17 UTC |
| 551986fe9c99384380121848 | succeeded | set_deployer_environment_crea | st2.kv.set                    | Mon, 30 Mar 2015 17:25:18 UTC |
|                          |           | ted_key                       |                               |                               |
| 551986fe9c9938438012184b | succeeded | set_deployer_environment_owne | st2.kv.set                    | Mon, 30 Mar 2015 17:25:18 UTC |
|                          |           | r_key                         |                               |                               |
| 551986ff9c9938438012184e | failed    | create_environment            | deployer.create_staging_envir | Mon, 30 Mar 2015 17:25:19 UTC |
|                          |           |                               | onment                        |                               |
+--------------------------+-----------+-------------------------------+-------------------------------+-------------------------------+

Failure:
   <FAILURE OUTPUT>

st2ctl should validate if mistral/component is installed. if not skip warning.

Reported by: @skarmark on freenode.

17:07:24 <hikertechy> lol. For me, whatever lets me install and start without mistral is good..since, it is not a requirement for other services :)
17:07:32 <hikertechy> warning is fine as well

Update st2_deploy script to use virtualenv

Currently st2_deploy installs all the Python dependencies in a global Python site-packages directory.

This means it will collide with other user or system installed packages. Currently, st2_deploy.sh also breaks on Fedora 21 which ships with older version of six Python package by default. register-content and StackStorm components (st2api, etc.) try to use system instead of our user installed six library by default which results in the breakage.

Issue with alias and default value

Some bugs with aliases and default values -

---
name: "google_query"
action_ref: "google.get_search_results"
formats:
  - "google {{query}} {{count=10}}"

google test count=10 - works
google test - fails

needs to be fixed.

Fix / update linux.scp action

Currently, linux.scp action is mostly useless if you want to download files from a remote server to a local one since it expects a dest_server argument.

We have a couple of ways to solve this:

1. Make scp action "desination agnostic" - instead of taking dest_server parameter, make it just take source and destination arguments. This way it works both ways (upload, download) since source and destination can either be a remote host with a path or just a path.
2. Rename existing scp action to scp_upload and add a new inverse scp_download action.

BUG: Unable to use dashes in rules for YAML values

Turns out, this rule is invalid.

---
    name: 'st2.webhook.cicd.github.events'
    description: 'Webhook listening for pushes to our CI/CD (SeatShare) repository'
    trigger:
        type: core.st2.webhook
        parameters:
            url: cicd/github/events
    criteria:
        trigger.header.X-GitHub-Event:
            pattern: push
            type: equals
    action:
        ref: jenkins.build_job
        parameters:
            project: {{trigger.body.repository.name}}
    enabled: true

Yields this:

ConstructorError: while constructing a mapping               |
|                 |   in "/opt/stackstorm/packs/cicd/rules/github-push.yaml",    |
|                 | line 15, column 22                                           |
|                 | found unacceptable key (unhashable type: 'dict')             |
|                 |   in "/opt/stackstorm/packs/cicd/rules/github-push.yaml",    |
|                 | line 15, column 23                                           |

Going to try rewriting in JSON to see if I can work around it...

Pack deploy sub-tree option not supported in install action-chain

Support for/requirement to use the subtree flag was added as part of fd27924. But the install action-chain (https://github.com/StackStorm/st2/blob/master/contrib/packs/actions/install.meta.yaml) was not updated to also support it.

[BUG] st2 reload capability leaves stale actions (maybe not only actions)

If there was an action and it was removed.

1. ls -1 /opt/stackstorm/packs/default/actions/*.yaml
/opt/stackstorm/packs/default/actions/show.indices.yaml /opt/stackstorm/packs/default/actions/show.snapshots.yaml
2. st2 run packs.load register=all
3. st2 action list -p default

ref	pack	name	description
default.show	default	show	Show indices
default.show.indices	default	show.indices	Show indices
default.show.snapshots	default	show.snapshots	Show snapshots

We've introduced stale default.show action.

Installation script bug: `st2_deploy.sh latest`

According to Installation Docs It's possible to pass latest parameter to installation script.

sudo ./st2_deploy.sh latest
sudo ./st2_deploy.sh stable

However it doesn't work.
The reason is that $LATEST (or $STABLE) from st2_deploy.sh looks like:

LATEST=`curl -Ss -q https://downloads.stackstorm.net/deb/pool/trusty_unstable/main/s/st2api/ | grep 'amd64.deb' | sed -e "s~.*>st2api_\(.*\)-.*<.*~\1~g" | sort | uniq`
echo "$LATEST"
0.10dev
0.9
0.9.0
0.9dev

which generates incorrect download URL:
https://downloads.stackstorm.net/releases/st2/0.10dev 0.9 0.9.0 0.9dev/debs/st2_deploy.sh

Also: https://downloads.stackstorm.net/releases/st2/0.9/debs/st2_deploy.sh - 404

Broken links in docs /reference/packs

I stumble on some broken links in http://docs.stackstorm.com/reference/packs.html.
See pull request #783 for fix.

global name 'verifyssl' is not defined

when running
st2 run packs.install packs=ansible repo_url=https://github.com/StackStorm/st2contrib.git

action output contains this error

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/st2actions/runners/python_action_wrapper.py", line 116, in <module>
    obj.run()
  File "/usr/lib/python2.7/dist-packages/st2actions/runners/python_action_wrapper.py", line 61, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 23, in run
    abs_local_path = self._clone_repo(repo_url, branch=branch)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 40, in _clone_repo
    if not verifyssl:
NameError: global name 'verifyssl' is not defined

vagrant install
st2 0.9.1.18
Vagrant 1.7.2
VirtualBox 4.3.28

Allow users to pass parameters which are large in size to Python actions

Currently parameters are passed to the Python action wrapper via command line arguments. This was done to provide better visibility and make Python actions easier to test and debug.

Both on Windows and Linux there is a limit on maximum size of the command line arguments (ARG_MAX, 8k on Windows, platform specific) which means users can't use really large values for parameters.

There are different ways around this limitation (e.g. store parameter value in the datastore instead), but instead of passing the arguments using command line arguments we could pass it to the action using stdin. This way users could pass parameters of (almost) arbitrary size to an action, although I don't think it makes much sense to pass arguments which are more than 4-8k or so in size.

I'm personally not convinced yet that's the best thing to do since switching to stdin decreases visibility, makes actions harder to test and if you want to pass really large value to an action as a parameter (> 8k) you should probably handle that in a different way anyway (e.g. store value in a datastore, upload it to object storage service or similar), but I'm open to feedback and I can be convinced otherwise :)

Note: This was originally reported by a user on the IRC.

REFACTOR: st2reactor/container/utils needs some refactor love.

Sensor model creation code is all in container/utils.py. These should be moved to st2common/services/sensor.py just like triggers. This way container/utils.py is smaller and focuses on high level APIs.

Ability to define a task which is always called after an action chain workflow finishes (either fails or succeeds)

It would be useful if user could define a step which is always called when a workflow finishes and / or fails. You can think of it as "finally" in Python and other programming languages.

Similar end result can be achieved by adding "on-failure" to every step, but defining this function once is nicer.

Here is an example from a workflow I'm currently working on:

0.8dev-38 breaks web

Just want to file this here...

one of these changes:

• #1030
• #1028

Has rendered the UI broken. Just hangs. Reverted back to 0.8dev-36 to get moving again.

/cc @Kami @enykeev

TriggerWatcher and SensorWatcher code should be refactored.

st2 run packs.install should register all pack components by default

It seems related to recent: Action aliases are registered by default. (improvement) change.

The problem I'm experiencing in chatops is that not everything loaded/unloaded when running install/uninstall for packs. It's just nice to have install/uninstall process complete, so user shouldn't care about low-level details.

My behavior:

• st2 run packs.install packs=hubot
• install, configure and run hubot with hubot-stackstorm
• expect that I get command output from st2 in chat !pack info hubot
• because chatops.notify_hubot rule shipped with hubot pack wasn't registered - I don't get any output in chat

I think install could do all the things by default, not only register actions. Maybe I'm missing some use cases here when only actions should be registered?

---

Related problem for uninstall. It could be better if that command unload all pack contents as well.

st2 run packs.uninstall packs=hubot
...

After uninstall there is still rule registered:

st2 rule list -p hubot
+----------------------------+-------+----------------------+------------------------------------------+
| ref                        | pack  | name                 | description                              |
+----------------------------+-------+----------------------+------------------------------------------+
| hubot.chatops.notify_hubot | hubot | chatops.notify_hubot | Notification rule to send messages to    |
|                            |       |                      | Hubot                                    |
+----------------------------+-------+----------------------+------------------------------------------+

---

Third problem,
seems action-aliases are still not registered:

st2 run packs.install packs=st2-google repo_url=jfryman/st2-google
...
st2 action-alias list | grep google
# no action aliases registered

These commits might be related: [ 1 ] [ 2 ]

---

^^ Tested on latest version.

st2 --version
st2 ('0.12dev.31',)

If that makes sense I can submit PR and verify & test everything again. My intent is to install/uninstall st2 packs in chatops way without any additional steps.

Refactor remote runners, use two classes instead of one

Currently, remote command and remote script runners both point to the same FabricRunner class and inside that class we do nasty branching depending on the value of the runner_name inside the action metadata file.

Instead of doing that, we should have two classes RemoteCommandRunner and RemoteScriptRunner and reference those classes when registering the runners.

This way, the classes itself don't need to be aware of the actual runner names which also decreases the coupling.

Also see: #1364

BUG: Rules that are renamed/removed not removed on reload

I ended up changing the namespace for my CI/CD workflow... from a url of cicd/github/events to cicd/events. On a reload however, the old endpoint is still registered.

Should this be removed, as it is no longer a defined endpoint? It is what I would expect to happen.

Refactor st2client, expose a better and more user-friendly API

Currently the API client is mostly designed around the CLI. The client API is unfriendly and needs to be refactored and improved:

• Get rid of the **kwargs abuse
• Better client construction (pass token to the constructor, etc.)
• Improve the way models work - perform attribute validation, etc.
• ...

Ideally, we should also release it as a separate package so we have st2client and st2cli. This way users don't need to install CLI if they just want to use the client and it also "forces" better coding practices which result in less coupling.

st2Auth printing traceback on wrong credentials

(26140) accepted ('172.168.50.1', 49492)
2015-03-13 05:14:01,784 DEBUG [-] Invalid password for user "test"
2015-03-13 05:14:01,785 AUDIT [-] Invalid credentials provided
2015-03-13 05:14:01,786 ERROR [-] API call failed.
Traceback (most recent call last):
  File "/vagrant/code/stanley/st2common/st2common/models/api/base.py", line 163, in callfunction
    result = f(*args, **kwargs)
  File "/vagrant/code/stanley/st2auth/st2auth/controllers/access.py", line 47, in post
    return self._handle_standalone_auth(request=request, **kwargs)
  File "/vagrant/code/stanley/st2auth/st2auth/controllers/access.py", line 95, in _handle_standalone_auth
    self._abort_request()
  File "/vagrant/code/stanley/st2auth/st2auth/controllers/access.py", line 99, in _abort_request
    pecan.abort(status_code, message)
  File "/vagrant/code/stanley/virtualenv/local/lib/python2.7/site-packages/pecan/core.py", line 119, in abort
    **kw
HTTPUnauthorized: Invalid or missing credentials
172.168.50.1 - - [13/Mar/2015 05:14:01] "POST /tokens HTTP/1.1" 401 461 0.011570
(26140) accepted ('172.168.50.1', 49493)
2015-03-13 05:14:01,801 DEBUG [-] User "wronguser9932" doesn't exist
2015-03-13 05:14:01,801 AUDIT [-] Invalid credentials provided
2015-03-13 05:14:01,802 ERROR [-] API call failed.
Traceback (most recent call last):
  File "/vagrant/code/stanley/st2common/st2common/models/api/base.py", line 163, in callfunction
    result = f(*args, **kwargs)
  File "/vagrant/code/stanley/st2auth/st2auth/controllers/access.py", line 47, in post
    return self._handle_standalone_auth(request=request, **kwargs)
  File "/vagrant/code/stanley/st2auth/st2auth/controllers/access.py", line 95, in _handle_standalone_auth
    self._abort_request()
  File "/vagrant/code/stanley/st2auth/st2auth/controllers/access.py", line 99, in _abort_request
    pecan.abort(status_code, message)
  File "/vagrant/code/stanley/virtualenv/local/lib/python2.7/site-packages/pecan/core.py", line 119, in abort
    **kw
HTTPUnauthorized: Invalid or missing credentials
172.168.50.1 - - [13/Mar/2015 05:14:01] "POST /tokens HTTP/1.1" 401 461 0.007859

That's a minor problem, but it attracts too much attention to such an ordinary event.

actionsensor race in triggertype registration.

https://gist.github.com/giampierod/2f323652ad32bc0a8c67
https://gist.github.com/giampierod/a4d59f66337a8064bfdc

Reported by: giampierod
Reported on: freenode #stackstorm

BUG: Not all named parameters have two dashes

Working on a pack for fpm, which takes named arguments. (https://github.com/jordansissel/fpm/wiki)

Not all named parameters have two dashes, as assumed in the current documentation. For example, some of the condensed attributes (like -s or -t or even -n) have only a single dash, yet there does not appear to be a place for me to adjust how parameters are being passed.

Use WinRM for Windows management

I just wanted to pitch using this: https://github.com/diyan/pywinrm instead of winexe for windows management.

Using Winexe along with windows shares is a fairly hacky solution, and Microsoft has already made this (relatively) cool API for managing windows machines remotely, and pywinrm looks like a fairly mature implementation.

If this seems like a good idea, I can get to work on implementing it straight away.

BUG: Named parameters are not being used

Working on a pack for fpm, which takes named arguments. (https://github.com/jordansissel/fpm/wiki)

Code:
https://gist.github.com/jfryman/4cfc501ca4153a2cae64

Based on the documentation, a named parameter does not have any positional parameters.

named - those parameters don’t include position attribute

However, it appears it is not doing the needful. Is this me, or something else?

/cc @manasdk

Carry triggerinstance context everywhere

1. add triggerinstance.correlation_id to supply reference to an external event in a triggerinstance
2. triggerinstance.id accessible in a rule so that it can be passed into an action
3. on-start notification to notify when an execution is started
4. ActionTrigger_start TriggerType
5. triggerinstance.id and triggerinstance.correlation_id should be available in notifier, actiontrigger etc.

Add SIGUSR1 handler for re-opening the log files

See #1336 (comment) for details.

HTML docs for action help. (Whatever st2 run blah.blah -h spits out)

Reported by: tobijb/jtimbrown on IRC.

CLI for rule creation does not support yaml input

vagrant@st2express:~$ st2 --version
st2 0.5.1

vagrant@st2express:~$ st2 rule create ./sample_rule_with_webhook.yaml
ERROR: No JSON object could be decoded

Filter executions by triggerinstance - API and CLI support.

We want the ability to figure out an execution matching a triggerinstance id.

Action chain failure should say on which action the chain failed.

See example:

https://www.irccloud.com/pastebin/6AZmoQaS

Reported by: tobijb/jtimbrown on IRC

• st2 run commands should bubble up the failures in a more human readable way. Status is failed below, but no error and the ordering of the output actually should have an entry for the execution that was never created and failed. The error was that an action did not exist for the Mistral task that follows wait_for_environment.

vagrant@st2express:/opt/stackstorm/packs$ st2 run deployer.gittobi-trunk.deploy environment=master-jb.staging-tobi.com
..
id: 551981d59c993843801217df
action.ref: deployer.gittobi-trunk.deploy
status: failed
start_timestamp: 2015-03-30T17:03:17.596028Z

Magic variables in yaml files

There are already some environment variables available to the actions.
Most probably there will be more and more useful of these during time.

The proposal is to make those variables available in yaml files too, bringing more flexibility.

---
name: example_action
runner_type: local-shell-cmd
entry_point: ""
enabled: true
parameters:
  cwd:
    default: "/opt/stackstorm/virtualenvs/{{ ST2_ACTION_PACK_NAME }}/bin"
    immutable: true
  cmd:
    ...

Btw, this approach is pretty obvious thing in Ansible, which operates with yaml files too.

BUG: "'FabricRemoteAction' object has no attribute 'id'"

Fabric based actions are failing due to an incorrectly named attribute.

root@st2stage201:/tmp# st2 execution get 5493450cce36d2111eb26c79
STATUS: failed
RESULT:
{
"message": "'FabricRemoteAction' object has no attribute 'id'",
"traceback": " File "/usr/lib/python2.7/dist-packages/st2actions/container/base.py", line 117, in _do_run
run_result = runner.run(action_params)
File "/usr/lib/python2.7/dist-packages/st2actions/runners/fabricrunner.py", line 106, in run
result = self._run(remote_action)
File "/usr/lib/python2.7/dist-packages/st2actions/runners/fabricrunner.py", line 120, in _run
remote_action.name, remote_action.id, remote_action.get_command(),
"
}

Refactor and improve st2ctl

Currently, st2ctl is a hard to maintain mess. The problem is that we are keep adding stuff without refactoring it.

We could make the whole script 10 times easier to read and maintain if we just refactored the code a bit and do things:

1. Use arrays and for loops (get rid of nasty hard-coded ifs)
2. Reduce 4 levels of if nesting (use return early approach)
3. Reduce duplicated code

EPOLL_CTL_ADD failed

I am trying to use the windows-script runner to execute a PowerShell script, but i continually get the following error:

{
    "message": "Failed to retrieve absolute path for share "C$"",
    "traceback": "  File "/usr/lib/python2.7/site-packages/st2actions/container/base.py", line 100, in _do_run
    (status, result, context) = runner.run(action_params)
  File "/usr/lib/python2.7/site-packages/st2actions/runners/windows_script_runner.py", line 91, in run
    base_path = self._get_share_absolute_path(share=self._share)
  File "/usr/lib/python2.7/site-packages/st2actions/runners/windows_script_runner.py", line 231, in _get_share_absolute_path
    raise Exception(msg + stdout + stderr)
"
}

Some additional debugging (printing stdout) showed this error:

Unknown parameter encountered: "max log size"
Ignoring unknown parameter "max log size"
Unknown parameter encountered: "passdb backend"
Ignoring unknown parameter "passdb backend"
Unknown parameter encountered: "load printers"
Ignoring unknown parameter "load printers"
Unknown parameter encountered: "cups options"
Ignoring unknown parameter "cups options"
Unknown parameter encountered: "writable"
Ignoring unknown parameter "writable"
Unknown parameter encountered: "guest ok"
Ignoring unknown parameter "guest ok"
Unknown parameter encountered: "writable"
Ignoring unknown parameter "writable"
tevent: EPOLL_CTL_ADD failed (Operation not permitted) - falling back to select()

I also made it print out the command it was trying to execute, and tried manually executing the outputted command, which works as expected. Any ideas?

Fix a bug with TriggerInstance payload not able to contain special characters such as "{{"

See #1492 for context and details.

Custom webhook fails action depending on payload.

Hi there, I've faced a weird issue.

Say it we have a webhook action like this: https://gist.github.com/dennybaa/00d1936af2f441cfff15
If I issue the following commands:

1. curl -X POST --data "{\"message\": \"6fd781b849bb\"}" http://172.17.0.8:9101/v1/webhooks/sample
2. curl -X POST --data "{\"hits.rate_15m\": 0.4765927732670934, \"hits.count\": 30, \"tags\": [\"metric\"], \"@timestamp\": \"2015-04-23T13:58:28.319Z\", \"hits.rate_5m\": 0.31123263029226506, \"@version\": \"1\", \"message\": \"6fd781b849bb\", \"hits.rate_1m\": 0.08765952620175455}"

In both cases trigger is dispatched. But the action takes place only in the #1 case.

Column width increase

Minor issue, but with the default column width set to 28, I'm seeing output like this:

# st2 execution list --action cybera.hello
+--------------------------+--------------+--------------+-----------+------------------------------+------------------------------+
| id                       | action.ref   | context.user | status    | start_timestamp              | end_timestamp                |
+--------------------------+--------------+--------------+-----------+------------------------------+------------------------------+
| 54eb49e8bda19059c109d178 | cybera.hello | stanley      | succeeded | Mon, 23 Feb 2015 15:40:24    | Mon, 23 Feb 2015 15:40:24    |
|                          |              |              |           | UTC                          | UTC                          |
+--------------------------+--------------+--------------+-----------+------------------------------+------------------------------+

Bumping it up to 29 (or higher) returns the expected format:

# st2 execution list --action cybera.hello
+--------------------------+--------------+--------------+-----------+-------------------------------+-------------------------------+
| id                       | action.ref   | context.user | status    | start_timestamp               | end_timestamp                 |
+--------------------------+--------------+--------------+-----------+-------------------------------+-------------------------------+
| 54eb49e8bda19059c109d178 | cybera.hello | stanley      | succeeded | Mon, 23 Feb 2015 15:40:24 UTC | Mon, 23 Feb 2015 15:40:24 UTC |
+--------------------------+--------------+--------------+-----------+-------------------------------+-------------------------------+

Environment variables are confusing in python actions and shell actions.

Shell actions like core.local and core.remote run as system_user whereas python actions run as user as which st2 processes run. So setting environment variables needs to happen for different users for different actions. We need a less confusing way. Perhaps some documentation around this.

Reported by: @skarmark on freenode.

st2ctl shouldn't start processes if they are already started.

https://gist.github.com/skarmark/4fe81ba2b7d353a09432

Reported by: @skarmark on freenode.

In rule criteria, handle null/undef payload

The one fix/feature we are really looking for right now is the ability to have triggers treat unset payload variables as “null/undef”, yet still run. A situation we run in to is having a pair of triggers, where we want one to match if a particular payload variable was set to true (ie. payload.whatever=true), otherwise match a different trigger criteria(ie. payload.whatever != true which also means unset or undef). Right now, if the payload does not contain a payload variable mentioned in the criteria, it will never match.

Something like 'empty' or 'not_empty', or 'exists' / 'not_exists'...
https://github.com/StackStorm/st2/blob/master/st2common/st2common/operators.py#L115-L137

st2 loads action with poorly named attributes

Carry over from I made a typo today in a pack development, and it broke the entire UI. Oh no!

I took some photos:

The error in code

The error in the UI

The same error occurs on the CLI.

ERROR: 500 Server Error: Internal Server Error
MESSAGE: Additional properties are not allowed (u'requried' was unexpected)

@enykeev tells me that the data should never get into the DB in the first place.... what's going on?

/cc StackStorm/st2web#79

Pack reinstall fails if config.yaml is missing

Related to: StackStorm/st2contrib#229

Reproduce:
Here is an empty pack which has no config.yaml file in it: https://github.com/armab/st2-pack-bug

1. Pack install (with no config.yaml)

# st2 run packs.install packs=st2-pack-bug repo_url=armab/st2-pack-bug
......................
id: 5586c6190a84b40a4a08a164
action.ref: packs.install
status: succeeded
start_timestamp: 2015-06-21T14:11:37.259169Z
end_timestamp: 2015-06-21T14:12:22.686647Z
+--------------------------+-----------+--------------------------+-------------------------+----------------------------+
| id                       | status    | task                     | action                  | start_timestamp            |
+--------------------------+-----------+--------------------------+-------------------------+----------------------------+
| 5586c61a0a84b40a402bdb88 | succeeded | download                 | packs.download          | Sun, 21 Jun 2015 14:11:38  |
|                          |           |                          |                         | UTC                        |
| 5586c62a0a84b40a402bdb8a | succeeded | virtualenv_prerun        | packs.virtualenv_prerun | Sun, 21 Jun 2015 14:11:54  |
|                          |           |                          |                         | UTC                        |
| 5586c62c0a84b40a402bdb8c | succeeded | setup_virtualenv         | packs.setup_virtualenv  | Sun, 21 Jun 2015 14:11:56  |
|                          |           |                          |                         | UTC                        |
| 5586c6370a84b40a402bdb8e | succeeded | reload                   | packs.load              | Sun, 21 Jun 2015 14:12:07  |
|                          |           |                          |                         | UTC                        |
| 5586c6410a84b40a402bdb90 | succeeded | restart-sensor-container | packs.restart_component | Sun, 21 Jun 2015 14:12:17  |
|                          |           |                          |                         | UTC                        |
+--------------------------+-----------+--------------------------+-------------------------+----------------------------+

2. Pack reinstall (if pack already exists)

# st2 run packs.install packs=st2-pack-bug repo_url=armab/st2-pack-bug
........
id: 5586c6700a84b40a4a08a166
action.ref: packs.install
status: failed
error: st2.actions.python.DownloadGitRepoAction: DEBUG    Removing existing pack st2-pack-bug in /opt/stackstorm/packs/st2-pack-bug to replace.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/st2actions/runners/python_action_wrapper.py", line 116, in <module>
    obj.run()
  File "/usr/lib/python2.7/dist-packages/st2actions/runners/python_action_wrapper.py", line 61, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 65, in run
    result = self._move_packs(abs_repo_base, packs, pack_abs_local_path, self._subtree)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 102, in _move_packs
    shutil.move(old_config_file, new_config_file)
  File "/usr/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/usr/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/opt/stackstorm/packs/st2-pack-bug/config.yaml'

traceback: None
failed_on: download
start_timestamp: 2015-06-21T14:13:04.269175Z
end_timestamp: 2015-06-21T14:13:19.928860Z
+--------------------------+--------+----------+----------------+----------------------------+
| id                       | status | task     | action         | start_timestamp            |
+--------------------------+--------+----------+----------------+----------------------------+
| 5586c6700a84b40a3f312d06 | failed | download | packs.download | Sun, 21 Jun 2015 14:13:04  |
|                          |        |          |                | UTC                        |
+--------------------------+--------+----------+----------------+----------------------------+

---

As I understand the solution is or to make config.yaml as a requirement or to fix that reinstall problem.
№2 seems more logical, since config is not needed for all packs and it's used only by python action runners (what could be improved too btw).

Document the upgrade from 0.7 to 0.8

A) re-provision an st2 box, and copy the context:

1. provision a new box
2. from the old box, copy the content of /opt/stackstorm/ to the new box (we recommend to keep all the content in SCM like git)
3. Copy your configuration from /etc/st2/st2.conf to the new box
4. install st2 0.8 on a new box
5. run st2ctl reload —register-all && st2ctl restart

B) Upgrade on the same box:

1. while on 0.7, run the cleanup script from here
2. install st2 0.8 using st2_deploy.sh

Notes

0.8 contains some changes in DB schema changes and API.

StackStorm follows "infrastructure as code": all the content and artifacts are files on disk; the database is just a content 'cache' + current state, for efficient implementation. Thus the backup, restore, and upgrade, is all about working with the file based content.

When database is dropped, the execution history won't longer be seen via st2; however it is saved in the audit files under /var/log/st2, we recommend integration with logstash / splunk / etc. for working with accumulated history data.

Use environment variables in actions

Initially posted by @jtopjian in google groups:

Is it possible for actions to inherit the environment of the user that issued the action?

I think this would be useful in many scenarios, but the one scenario that made me think of it was playing with the OpenStack pack. That pack comes with a config.yml file that asks for the OpenStack cloud information. The problem here is that it requires me to have a credentials file in a different format than the openrc file we normally deploy (which is a bit of a hassle when we have that config file under config management). Secondly it is limited to only one cloud/region. On some of our "control" servers, we have several rc files for different clouds and regions of those clouds. If we want to run a command against a certain cloud, we just source the specific rc file.

By being able to inherit the environment, we could then gain the flexibility of being able to run actions against several clouds without creating a complex config.yml as well as prefix the environment in cron and such (though StackStorm can do cron on its own ;)

Of course this would also require modifying the OpenStack pack (for this specific scenario) but that's rather easy.

Documentation correction - TTL Authentication flag

Latest 0.9.0 and 0.10dev documentation shows the following:

with TTL and password

st2 auth yourusername -p yourpassword -t 600

"usage:" however shows the following:

usage: st2 auth [-h] [-j] [-p PASSWORD] [-l TTL] [-t] username

It should also be noted that the TTL value cannot go past the default value (not sure if this is by design), only create tokens that expire with a shortened value to the TTL default.

Infinite timeout in action parameters

The problem is that parameter timeout: 0 in action is considered as 60 seconds and there is no option to make it "infinite", eg. no timeout.

Here is example action actions/test_timeout.yaml:

---
name: test_timeout
runner_type: local-shell-cmd
entry_point: ""
enabled: true
parameters:
  timeout:
    # works as default 60 seconds
    default: 0
  cmd:
    description: "Command to run"
    immutable: true
    default: "sleep 90 && echo 'done!'"

Running it:

# st2 run st2-pack-bug.test_timeout
..............................
id: 5586ca9b0a84b40a4a08a168
status: failed
result:
{
    "succeeded": false,
    "stdout": "",
    "return_code": -9,
    "failed": true,
    "stderr": "",
    "error": "Action failed to complete in 60 seconds"
}

Not sure if it's bug or not, but proposal is to consider timeout: 0 as "no timeout", instead of default 60 seconds. Lots of cases when action doesn't needs any timeouts.

Because things like this in action:

parameters:
  timeout:
    default: 99999999

definitely looks like there should be a better way.