Karma/Jest Setup

[aurelia]

Karma/Jest Setup

[eiswind]

You made my day. At least with karma It works now.

[fkleuver]

@eiswind We've built aurelia v2 in such a way that there is no need for anything special to test it, testability is built into the core. It works with anything.

For karma, you can just start a normal app the same way in a test as you would in production. In our integration tests we usually perform assertions on the dom content.

This goes against what a lot of people believe to be best practice, but it should be noted here that we have several thousands of tests like this and they run in hundreds per second. Starting a fresh Aurelia app is pretty cheap and fast.

Example:

it('works with functional api', async function () {
  const App = CustomElement.define({
    name: 'app',
    template: '${message}',
  }, class {
    message = 'Hello world';
  });

  // Just create a div on the fly as a detached host for simplicity.
  // Aurelia doesn't need to live in a connected dom to work.
  const host = document.createElement('div');

  const au = new Aurelia()
    .register(JitHtmlBrowserConfiguration)
    .app({ component: App, host });

  await au.start().wait();

  assert.equal(host.textContent, 'Hello world');

  await au.stop().wait();
});

it('works with declarative api', async function () {
  @customElement({
    name: 'app',
    template: '${message}',
  })
  class App {
    message = 'Hello world';
  }

  const host = document.createElement('div');

  const au = new Aurelia()
    .register(JitHtmlBrowserConfiguration)
    .app({ component: App, host });

  await au.start().wait();

  assert.equal(host.textContent, 'Hello world');

  await au.stop().wait();
});

We do however have a TestContext exported from @aurelia/testing that, much like the quick-start package, wraps the typical configuration to reduce the amount of boilerplate. This also makes it easier to do cross-platform testing (we use it to run our tests in the browser as well as in jsdom).

This isn't quite yet in a ready state for end users though. We will be getting that ready in the near future.

Since you mentioned Jest, I would strongly recommend against using that framework for testing in general. I've seen all kinds of weird issues with it due to the way it messes with globals. It's also the slowest testing framework out there. For running tests in nodejs, plain jsdom + mocha will give you far better mileage.

[3cp]

I will get scaffolding up for few test frameworks. Instead of using own our assertion lib, I will demo the test code using what ever native assertion provided by the test frameworks (e.g. jest, jasmine...). Because the end users unlikely have the performance issue (the reason of creating our own assertion) as the scale of Aurelia 2 monrepo tests, and we don't want to deliver wrong message that Aurelia 2 can only be tested with our own special assertion lib.

[3cp]

I almost forgot. When shadow dom is turned on, it makes really difficult to write tests.

Shadow dom open mode is possible but still tedious to test. Shadow dom closed mode is really hard for unit tests.

@EisenbergEffect do you have any idea on making shadow dom testing easier? One not-very-idea option I can think about is using conditional bundler config to always turn off shadow dom for NODE_ENV=test.

[EisenbergEffect]

I don't have too many ideas here. You can always get the shadow root in open mode though. So, you should be able to assert internals of the DOM that way without much trouble. I don't use closed mode ever. Never have. I don't even think it's recommended unless you need a strong security barrier.

[3cp]

Ok, I can conditionally avoid closed mode for shadow dom in test env. So the tests are always running in open mode.

[3cp]

Another related issue is the implicit enabling shadow dom for any custom element using slot. This will catch user off guard.

[EisenbergEffect]

I think the implicit enable should be shut off actually. However, we need to pair that with the ability to do basic default slot projection without shadow dom.

[fkleuver]

@3cp
We have an assertion helper assert.visibleTextEqual that takes au.root and it will dig all the way into shadow roots and their content recursively. It does so by traversing the controllers and getting the shadow roots from the projectors.

If you need to do different stuff besides that, just tell me the use cases and I'll add appropriate assertion helpers. Testing shadow dom is pretty trivial with that. We have a few thousand tests with shadow dom that fully verifies text + html content in complex scenarios. They use that helper.

So, I actually disagree with shadow dom being difficult to test and I also don't think special env variables to disable stuff in non-transparent ways is the way to go. There are truly trivial solutions to all of this, simply by traversing the controllers in a helper. Let's not degrade/kill features just because it "seems" difficult to test. Every other framework does that, including v1, and it's a mistake that I don't want to make in v2. When you're doing integration tests in v2 I want users to be able to rely on the fact that their app is behaving identically in the test to how it would behave in production, so that the test is maximally useful.

[3cp]

@fkleuver I like the assert.visibleTextEqual helper. But I just confirmed my worry: our assertion tool doesn't play well with other test framework other than mocha.

Here is jasmine's output, although it passed without any error (logic is successful).

Spec 'my-app renders' has no expectations.

jasmine doesn't understand our assertion. I expect same issue from jest/tape/ava.

@fkleuver @EisenbergEffect should we layer the testing lib to provider?

1. visibleText helper regardless of testing framework
2. assert.visibleTextEqual on top of visibleText to provide faster assertion in mocha.

[EisenbergEffect]

That seems reasonable to me.

[3cp]

@fkleuver @EisenbergEffect nvm.
I saw getVisibleText is exported in @aurelia/testing, I can use that directly.

This is probably the only helper we need to ease testing in shadow dom.

[3cp]

Small problem. I am able to get the jasmine test working without shadow dom. But when shadow dom is turned on (open mode), getVisibleText only gives me empty string. Anything I did wrongly?

import {getVisibleText, TestContext} from '@aurelia/testing';
import Aurelia from 'aurelia';
import {MyApp} from '../src/my-app';

describe('my-app', function() {
  it('renders', function(done) {
    const ctx = TestContext.createHTMLTestContext();
    const host = ctx.createElement('my-app');
    const au = new Aurelia(ctx.container)
      .app({ component: MyApp, host });

    au.start().wait().then(() => {
      const text = getVisibleText(au.root, host);
      expect(text.trim()).toBe('Hello World!');
      return au.stop().wait();
    }).then(done, done);
  });
});

Update: manual code works.

const text = host.shadowRoot.querySelector('.message').textContent;
expect(text.trim()).toBe('Hello World!');

[3cp]

Unit tests skeleton is up for jasmine/tape/mocha.

Jest (and ava) is not ready yet. Pending implementation of convention transformer to work in nodejs environment. (Similar to what @aurelia/webpack-loader did for browser build).

ShadowDom closed is conditionally turned off when running tests. Although Aurelia has API to enable testing closed mode, cypress (our app e2e setup) has no way to peek through closed shadowdom. So for simpler skeleton, the closed mode is off for both unit tests and e2e tests. When user chose closed mode, it's only turned on for production build.

[fkleuver]

ShadowDom closed is conditionally turned off when running tests.

Can we please not do that? If a user is using closed mode, which is a pretty rare scenario, they are probably expecting it to work like closed mode, also in tests. If they really need to peek into it, we have APIs that work just fine. There's always a way to dig into closed shadow even in cypress.

Conversely, by always turning it off in tests, you are preventing users from being able to do assertions specifically to verify the behavior of closed shadow mode, namely that they can't peek into it.

So, as I said before, and will say again, I think tampering with the app in any way at all for tests is a bad idea. We shouldn't touch stuff. Leave tampering with the environment to crappy testing frameworks like jest.

[EisenbergEffect]

I tend to agree with @fkleuver on this one. Changing the behavior in test mode could result in unexpected results.

[3cp]

There is no problem for me to turn it on for unit tests. But the issue is cypress e2e tests, we need to find a way to support that. Need some idea for that.

[fkleuver]

we need to find a way to support that.

Support what exactly? It's going to work exactly as users would expect it to.

Need some idea for that.

Wait for someone to actually complain about it?

I'd much rather see an issue titled "how can I inspect a closed shadow root" over an issue titled "why is my shadow root not closed in cypress?"

[3cp]

Support cypress e2e test which is out of aurelia. It cannot see through shadowRoot to test the content.

[3cp]

I can also turn off option of e2e after user chose closed mode. So cypess will disappear from questionnaire.

[fkleuver]

But I don't understand the problem right now. Is this preventing you from writing a test that verifies the setup or something?

[3cp]

The skeleton ships both unit tests setup and cypress e2e setup through the questionnaire presented to end user.

When closed mode is chosen, cypress setup for the final app doesn't work. I suggest to conditionally remove cypress from options after user chose closed mode. The "makes" supports this kind of dynamic conditional questionnaire.

[eiswind]

The cypress guys are currently discussing support for shadow dom. I guess it's a matter of time.
cypress-io/cypress#144

[3cp]

Cypress already got api for shadow dom, but only works for open mode which is expected.
That cypress issue started from 2016, from a glance, I did not see it discussed anything around closed mode (our problem).

[eiswind]

Seems I have to learn something about shadow dom...

[EisenbergEffect]

@3cp Why don't we remove the option for closed shadow dom all-together and just have "shadow dom" that configures the app for open mode. That should be the default way that people go anyway. I don't think you would want your entire project to be in closed mode. It's more likely that you would have one or maybe two specific components in closed mode (if any). That could be configured manually on the component.

[3cp]

That sounds like a good idea. I removed closed mode as a choice and added some comments to inform user about the complication of closed mode. I am testing all skeletons, will push up after testing.

[3cp]

All up. Tested all skeletons on my mac. I didn't test on a win32 box.