shanejansen/touchstone

Touchstone is a testing framework for your services that focuses on component, end-to-end, and exploratory testing.

touchstone's Issues

Ability to supply Docker images instead of Dockerfiles for services

In touchstone.yml, a service can be defined with dockerfile. Support ability to specify docker_image instead

Dedicated console library?

The Touchstone develop console lacks many common console/shell features such as command history via up and down keys, jumping over words by holding control or alt/option while pressing left or right, home and end or other shortcuts to the beginning and end of a line, holding alt or option while pressing delete/backspace to delete entire words, etc.

It would be great if some Python library for console apps could supply these features for Touchstone without having to reimplement them. Bonus points if tab completion is also possible using such a system.

ANY does not match numbers?

Exactly what it says on the tin; I've got a payload json comparison where validation.ANY works for strings but for numbers (integers, specifically, not sure if that's a different type) I have to hardcode the number in order to pass.

The code doesn't have any obvious (to me) problems that would cause this…

When looking at this be aware of #23

Auto-start Docker

Touchstone will not start if Docker isn't running. It would be helpful if Touchstone automatically started Docker for you

MySql mock - misleading error message

MySql mock method does_not_exist has a misleading error message: "was found 1 time(s) but expected any time(s)"

Health check mock timeouts

Depending on the developer machine (and not the project), sometimes mocks can take longer than their health check time to finish spinning up. It would be helpful to check for a user-level or system-level configuration that can override the health check timing for the mocks, either all mocks or specific ones.

Support CI Reporters

Various CI servers such as Jenkins and Team City support test report logs in particular formats that can be parsed and read by the server in order to identify and track tests, allowing e.g. "4 out of 12 done, no failures" type indicators on the builds.

Supporting this for a variety of CI servers probably requires test result reporting to be pluggable.

See also #14 re. potentially needing to refactor output anyway.

Supply arbitrary list of supporting images

Some services need supporting images which do no include mocks. Add ability to specify a list of "supporting" Docker images to spin up when Touchstone is started

Database Not Reset

When running Touchstone Develop with snapshot_databases: true, after running an individal test, the database is no longer reset when running mocks reset or running all tests.

I'm not certain whether the reset stops happening or whether the snapshot gets overwritten (which would start getting reset to after the test state).

Mongo snapshots?

My thinking on MongoDB mocks is that, since it's a database, it should largely parallel MySQL (except where the difference between the two is relevant, if there are any).

Do any services use migrations on a MongoDB? Presumably they don't need them to create collections as collections are schema-less or not strongly typed or however you want to put it, but theoretically a service could set up initial data. (As with #46, it might make sense to allow defaults to be applied by Touchstone after the service rather than before it.)

This scenario seems relatively unlikely given the use cases for Mongo, and therefore low priority; however, I wanted to bring up matching MySQL behavior/options in general.

order of MySQL defaults and migrations

I believe the order of effects when starting up the mocks and services with mysql set to snapshot DBs is:

1. Defaults file applied on mocks start.
2. Migrations may be applied by the service on service start.
3. The DB snapshot is saved.

(The same order is used without snapshotting DBs, with similar questions/issues but it's eclipsed by the migrations getting wiped so with migrations one pretty much always wants to snapshot DBs.)

This has a couple limitations/oddities:

1. The migrations may expect a clean DB, but if you add anything to the defaults that may not be the case.
2. You cannot add default data to the service's tables without having to create the tables the migrations should create and therefore running into problem 1.

In other words, you can have migrations or defaults but effectively it's hard to use both, e.g. to populate default data in tables the service creates.

It would make sense in my opinion to defer running the defaults until after the services start, to the same point when the DB is snapshotted (presumably before the snapshot, although at that point as long as the defaults don't contain nondeterministic values/behavior there is only performance difference between "defaults once then snapshot" versus "snapshot migrations, reapply snapshot in place of wiping DB and reapply defaults after each wipe or snapshot application".)

Mongo & MySQL assertion parallels

My thinking on MongoDB mocks is that, since it's a database, it should largely parallel MySQL (except where the difference between the two is relevant, if there are any).

In MySQL assertions we have four cases:

• value, asserts equality to that value
• field not included, no filtering/asserting on that field
• None, asserts field IS NULL
• validation.ANY, asserts field IS NOT NULL

I know Mongo assertions have the first two, maybe the first three. Is there a way to implement all four in MongoDB queries?

MongoDB et al. expect 0 fixes

In 4fe636c#diff-f00ff2c353e1ab98d15ad58bb6a842e49495c52107bc800ddce3783d412b1dcbL49-R59 a bug was corrected around expecting 0 calls to a REST API. The fix was to change not n, which is also True for 0 rather than only None, to n is None, which is False for 0.

(The MySQL mock didn't seriously need this since it has row_does_not_exist; however, other mocks would be helpful to have a similar fix applied.)

All the code I could find where the logic should be corrected are:

• touchstone/touchstone/lib/mocks/networked_runnables/mongodb/docker/docker_mongodb_verify.py

  Line 24 in bc8e3ba

  if not num_expected and num_documents != 0:

• touchstone/touchstone/lib/mocks/networked_runnables/rabbitmq/docker/docker_rabbitmq_verify.py

  Line 24 in bc8e3ba

  if not num_expected and num_messages != 0:

• touchstone/touchstone/lib/mocks/networked_runnables/mysql/docker/docker_mysql_verify.py

  Line 38 in bc8e3ba

  if not num_expected and num_rows != 0:

ANY times

Several mocks have a feature where one can set the num_times to None to assert at least once but no specific number of times matched (>0, >=1).

I can't recall for certain but I think this predates the JSON APIs and validation.ANY.

It would make more sense to use ANY instead of None.

Further, if None were no longer used for this purpose it would make it easier to avoid bugs conflating 0 and None such as #44

My suggestion around backwards compatibility, to avoid confusion (old Touchstone tests use None to mean any, but a newcomer would likely expect it to mean 0 once ANY can mean any), would be to make it an error to use None. This is backwards incompatible, but fails explicitly and clearly rather than behaving unexpectedly. In my experience most tests are using either the default (implicitly 1 time) or 0 (asserting something is not there), only a few special cases are using fuzziness such as "assert 3 calls to some URL matching this wildcard pattern" or "assert this was called, but assert it was called any number of times".

While we're at it it may make sense to try to centralize more of the num times logic. As I recall we had an issue about correcting a 0 vs None bug re. equals vs. is, it had to be corrected separately in each mock.

Service logs do not cover immediate startup failures

Usually when a service fails to pass the healthcheck, --log-services provides useful info. However, sometimes the log file is empty and to see its output I have to catch the container with a Docker attach command in the fleeting window during which it is live. It seems as though Touchstone connects to the container after the initial docker run command to get the logs. I would expect it to direct the container output to a file as part of the initial container run/start command, guaranteeing all output is logged.

Multi-line default documents for mock MongoDb throws exception

Example:

---
databases:
  - name: myapp
    collections:
      - name: users
        documents:
          - { "firstName": "John", "lastName": "Smith" }
          - >
            {
                "firstName": "Jane",
                "lastName": "Brown"
            }

HTTP Mock - Default to "url" instead of "urlPattern"

When building the HTTP mock, "urlPattern" is used by default. This can be confusing because characters need to be escaped when not expected. "url" should be the default and "urlPattern" should be an option

Multiple service's urls

One thing Touchstone is great at, is setting up a test for a flow that spans multiple services. Drop a rabbit message to service-a for input, service-a talks to service-b, assert that service-b sends a rabbit message for output.

If the flow involves REST calls to both services, however, we are limited to hitting one service via self.service_url; so for instance you could not POST to service-a for input and then GET service-b for output (to assert on) in the same test.

This could be made possible with a self.services hashmap, self.services['service-name']

Add route+exchange to rabbit assertion messages

Rabbit payload assertion failures show the expected and list of actual messages, which is generally clear enough but it wouldn't hurt to show the routing key and exchange that the assertion is checking.

Rabbit message count assertion without the payload print "expected N does not match actual M," which borders on meaningless without further contextual info.

Adding routing key and exchange to both would be very handy.

Restart/refresh mocks while running in develop mode

Currently, Touchstone must be completely shutdown and restarted when mock's defaults are changed in develop mode. This significantly slows down development especially when integrating TS with an existing service. Support ability to:

• Restart/refresh mocks to use changed defaults
  • This will be difficult for some mocks. Some mocks may need to be shutdown and re-ran
• Ability to refresh a single mock

Database query logs lack values

When a database assertion fails and it prints which SQL was not successful, the values are placeholders (as they should be for the whole string escaping logic and such) so only the fields are shown; would it be possible to either substitute the values into the placeholders when printing this output, or else add the list/map of values used?

MongoDB mock only clears listed collections

The MongoDB mock will not clear the collections created by the service if they are not included in the defaults file.

It should clear any collections that the service created that are left out of the mock specification:

• if collections is not defined
• if collections is an empty array (really just a case of the below)
• if there are other collections mocked but additional ones are created by the service

Supporting images/defining custom mocks

Add support for additional Docker images to run along side mocks and services. This is important if a service has a dependency on an unsupported mock

Mongo collection defaults requirement

My thinking on MongoDB mocks is that, since it's a database, it should largely parallel MySQL (except where the difference between the two is relevant, if there are any).

MySQL does not require anything in the defaults file (beyond the existence of a particular DB for the service to connect to), as the actual tables created by the service can be used by Touchstone on the fly without Touchstone needing to set anything up before their use.

This can be contrasted with, say, in Rabbit the messages are consumed or lost unless Touchstone creates shadow queues ahead of time, hence Touchstone must know about routing in the defaults file.

In Mongo I believe MySQL could be paralleled: services can automatically create their databases and collections, it is possible to query them on the fly without needing information about them in the defaults file.

(Touchstone already parallels MySQL with wiping the entire Mongo DB, not just what's in the defaults, thereby ensuring no state carries over between tests and it must be explicitly set in defaults or per-test mocks setup calls, as is best practice for testing in general. See also #47 )

Redis support

Added Redis to supported mocks

Support HTTP Connections Between Services

Services can point to localhost/host.docker.internal to reach other services when running in Touchstone Develop. However, under Touchstone Run, the services are given randomized ports and are not reachable. This makes HTTP calls (and other direct, non-Rabbit communication) between two or more services impossible.

Cannot route multiple keys to same queue

I have an app that has a queue "all" on which it receives from multiple routing keys, "route.a", "route.b", "route.c". (This helps with ordering of different messages that affect the same object, when the app comes back up after downtime. Arguably the root problem is the app's structure but it's a legacy app we're hoping to first get Touchstone tests for then use those tests when refactoring or replacing. Then again: Rabbit plainly allows different sources to use their own routing keys and one recipient to route them into the same queue.) This gives me Touchstone failures, 'This exchange "my.exchange", routing-key "route.b" combination is not defined. Check your "rabbit.yml" defaults.' (and same for route.c) even though I have this in my.exchange:

- name: all
  routingKey: route.a
- name: all
  routingKey: route.b
- name: all
  routingKey: route.c

Use a unique reference as the value of ANY

Rather than 'TS_ANY' string, which could (though it's unlikely) be in a user data string, consider a technique like https://stackoverflow.com/q/23703863 or https://stackoverflow.com/q/1677726

This may or may not impact #22

Specifying "None" for a service's tests should pass

In touchstone.yml, services: None should pass with 0 test executions

Long-Running Touchstone Infrastructure

The slowest part of Touchstone tests is spinning up everything in Docker. While it's unlikely anything can be done about that for the services themselves, the mocks could likely be made long-running, provided some way to sandbox multiple instances of mock usage in the same container. Instead of spinning them up on demand Touchstone would have a client-server model in which an instance is running/managing mock containers and other instances connect to that.

RabbitMQ

Implementing this for Rabbit should be pretty easy. Virtual hosts! ${TS_RABBITMQ_VHOST} = randomized ID

MySQL

MySQL could probably implement this as multiple databases. Multiple databases are already supported, only, their names are hardcoded. So it looks like this…

Now

mysql.yml

databases:
- name: a
  statements:
  # any statements, or maybe migrations handle the main DB
- name: b
  statements:
  # lots of statements

properties file

spring.datasource.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/a
spring.datasource-b.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/b

Instead

mysql.yml is the same

However, instead of using those names, a randomized ID is generated and used, and a variable TS_MYSQL_DB_<capitalized_name> is set to that ID.

properties file

spring.datasource.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/${TS_MYSQL_DB_A}
spring.datasource-b.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/${TS_MYSQL_DB_B}

HTTP

If Wiremock supports using multiple ports and treating them like virtual hosts, then this is trivial, like Rabbit but without any change required for service configuration.

If not, there are a couple options:

Context Root

Instead of aiming at http://${TS_HTTP_HOST}:${TS_HTTP_PORT}/, aim at http://${TS_HTTP_HOST}:${TS_HTTP_PORT}/${TS_HTTP_ROOT} and a randomized ID (followed by /) for TS_HTTP_ROOT is used to sandbox all the mocks for a given instance. Plus, most services should be pointed at ${TS_HTTP_URL} anyway, which would get the root added to it without the service needing to be reconfigured at all!

Routing

Create a sandboxing root as above on the Wiremock side; however, for ease of use…

Stand up a simple routing server that sends each port to its respective root without needing to add the root to the URL.

MySQL actual field values

Some mocks provide an indication of the actual data, e.g. rabbit's list of published messages. MySQL one cannot tell from the Touchstone failure whether a field didn't match and if so which, vs if there are no records.

Could MySQL be updated to query against the primary keys (preferably just looking those up automatically) and compare the fields?

(Arguably related to #38 which is seeing expected field values)

Assertions Automatic Retry w/Timeout (alternative to processing_period sleep)

Right now, in order to give tests enough time for Rabbit messages to be processed, a hardcoded delay is introduced. Set too low and the tests become flakey, set higher and they are more reliable but take longer. It would be best if instead there was a timeout, either at the test level or the assertion level, for trying the assertions until they either pass or have continued failing till the timeout is hit.

Since Touchstone currently prints the test failure from within the assertion on failure, some refactoring will be needed to prevent printing assertion failures over and over.

I can think of a couple different possible solutions. They aren't mutually exclusive and could be used together to get the advantages of both.

Solution A: Throw

Most test frameworks I'm familiar with throw an exception on failed assertions, where the exception message describes the failure and the assertion exception class contains an expected value and an actual value.

This leaves it up to the test framework (or a reporting plugin for the framework; see #15) how to print the message and the comparison of expected vs actual.

Solution B: Retry Within Assertion

Cypress assertions do not actually fail until they've tried until a time limit is hit. Touchstone assertions could do likewise. There could be a default time limit for the test with a possibility to override it for individual assertions.

Wrong container name flag

At work, my local machine is able to make service-to-service REST calls as intended referencing other containers through their name. Unfortunately it isn't working in CI.

It turns out this is an oddity in our machines, not CI. As far as I have been able to figure out, Docker's DNS settings will use --hostname to resolve the host by name for that container itself, but the proper way to set it for the rest of the network appears to be --network-alias For some reason hostname works on our local, but whatever we've got on CI appears to be more strict.

Touchstone is using --hostname; I believe it should be updated to use --network-alias for maximum compatibility.

A workaround for now is to add --network-alias <your service name> to docker_options in your touchstone.yml file.

Underspecified docker errors

Sometimes when a container has catastrophically failed, Touchstone goes to stop it as part of network shutdown and spits out an error saying no such container. This isn't very detailed and isn't obvious to all developers that the real issue is the container died when it attempted to start up. (In contrast, a healthcheck never passing spits out a whole stack trace where all we need to know is which container's healthcheck never became healthy.)

Would it be possible to catch the Docker missing container error and provide a message saying which mock or service died or wasn't started, and ensure this does not block shutting down the rest of the network?

(Relatedly I would expect a similarly informative error if the Docker command to build or to start/run the container in the first place errors; it's been a while since I had that happen so I don't recall what it looks like, though I think it may be reproducible by attempting to run while the app's port is already allocated to another app or instance.)

Enable testing NOT NULL in DB via ANY

In the MySQL mock, if you omit a field it can be any value including null, and if you give it the value None then the mock checks that it IS NULL.

Sometimes I need to confirm that the field has a non-null value, but allow that value to be anything. I.e. IS NOT NULL. This seems like a use case for validation.ANY; that currently tries to see if the field = 'TS_ANY'.

See also #23