bigtestjs/server

All BigTest development has moved to https://github.com/thefrontside/bigtest

server's Issues

Query subscriptions

Right now, we support "one-shot" queries for accessing the state of the server. However, stateful user interfaces like our CLI and eventually our dashboard will need to be refreshed constantly whenever an update happens like a test starts running or is completed. This we can do by taking the concept of a one-off graphql query and turning it into a stream.

If a request is made to the command server and the web socket headers like

Upgrade: websocket
Connection: Upgrade

are detected, then instead of taking the query as a one-shot, we store the query, and every time the state changes, we run the query and push the results out of the websocket (assuming that it has changed)

Integrate an agent server

We're clear that one of the moving pieces is the BigTest agent which will be active inside the browser process that is actually running the test. It will be served from the bigtest server over http so that it can be injected as a script, for example http://localhost:4003/agent.js

That means that this code has its own entry point, and will need to be assembled independently from the server along with its own unique set of (possibly overlapping) dependencies.

In the production version of the agent that is published on NPM, we'll want to pre-compile this agent code from TypeScript and bundle all its pieces ahead of time and just have a static file server serving it. However, when developing the server itself, we'll need a mechanism to watch the files that comprise the agent, and rebuild the agent bundle any time that they change.

Serve Test suite files

The entire point of the bigtest server is to run tests, and so getting the suite bundled up and loaded into the test harness lies on the critical path. Based on the discussion of how test files will be loaded (#17) we now have a path to implementation.

1. Test files will be built by a parcel process. We'll generate an index file which is will glob all of the test suites detected and then compose them into single test suite value which it will export by default.
2. We'll watch for changes in then tests directory, and then re-write that index file, and restart the parcel process.
3. The generated javascript bundle will be made available over http (potentially via parcel)

Questions

What is in the entry point index that allows the harness to "capture the test suite": e.g.

let suite = require('__bigtest_suite__');
window['@@bigtest/mailbox'].push({type: 'suite', suite});

Show state of connected browsers from the GraphQL interface

In order to see which browsers are connected to the bigtest server, we should be able to query it directly:

type Browser {
  name: string!;
  platform: string!;
  vendor: string!;
  version: string!;
}

This means that when we connect a browser, we'll need to send a message from the agent to the orchestrator, update the state somehow, and then be able to query that state.

Unfurl Test tree

In our graphql, we made the Test tree a single JSON. Let's not do that.

Add TodoMVC app as a fixture for the test suite

In order to do development, and also have a test suite, we need to have an application of non-trivial complexity that's not to difficult to maintain. This will be critical for automated testing too.

We should be able add a todomvc script to package.json that can be used to start our server.

$ yarn todomvc -p 3000

That way, when we run in development we can start the server conveniently like:

$ yarn start --start-with 'yarn todomvc -p 3000'

In our test suite, we can also start our app and then run the components of the system against it.

Constraints

• It should be just a pre-built, vendored app that can be served in its entirety

How to test?

We started implementing the BigTest server mostly as a proof of concept for the architectural discussions and to validate or invalidate assumptions. However, it now looks like this implementation will proceed to become the product, and so we need to figure out a way to test it.

With so many moving asynchronous parts, it's agreed that we want to try and make the test as high level or "Big" as we can (it's only fitting for the BigTest server after all). For example, we may want to start a bigtest server and then send real http requests to it and make sure that we get certain responses.

A non-exhaustive list of unknowns:

• What components are tests allowed to interact with besides the command server, if any?
• How do we test error conditions?
• How do we simulate (or leverage) systems that exist outside of the bigtest server like the application under test, or a browser.

How are test files made available to the agent?

We know that test files, which are on disk somewhere will need to be executed inside the agent, and the result be communicated back to the orchestrator/server. Those test files will presumably be sent to the agent somehow. The question is how this is going to work.

Presumably there needs to be some kind of transpilation of these files so that they can be run in the browser? How do we make that work?

Make orchestrator capable of starting the application

Current process:

$ yarn start
$ cd ../my-app
$ PORT=24000 yarn start

MVP process:

$ cd my--app
$ yarn bigtest --start-with 'yarn start' --app-url http://localhost:3000

This implies that the orchestrator will be able to

1. start the app
2. wait until the app is running at the specified port before being considered ready.

For testing purposes, we'll need to use the example app.

What does communication with the command server look like?

The command server running inside the bigtest daemon will be the primary interface for interaction whether through command line interface or through a more complex GUI.

It needs to have both request/response capabilities in order to do things like run a test suite, but it also needs to have realtime capabilities so that interfaces can subscribe to test runs and report results back to the server.

key questions:

• What serialization scheme will we use and why?

  • REST
  • GRAPHQL
  • RPC
  • something else?

• What are some key use cases and how will this command flow support them? E.g. launch a browser, run a test suite comprised of multiple tests, and somehow report results back to a command line client.

Test File Server leaking event listeners

Noticed that when I was working on the test file server and it was getting reloaded multiple times, that I'd get this error:

[orchestrator] show GraphQL dashboard via: http://localhost:24002
[test files] test files updated
[test files] test files updated
[test files] test files updated
[test files] test files updated
[test files] test files updated
[test files] test files updated
[test files] test files updated
[test files] test files updated
[test files] test files updated
(node:60268) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 message listeners added to [ChildProcess]. Use emitter.setMaxListeners() to increase limit