japgolly/test-state

Scala Test-State.

Scala Test-State

Test stateful stuff statelessly, and reasonably.

Contents

• What is this?
• How does this work?
• How do I use this?
  • Usage
  • Types
  • DSL
  • Composability
• Modules
• Examples
• Changelogs
• Support

What is this?

Firstly, a quick mention of what this is not:

1. This is not a test framework.
   Use it conjunction with ScalaTest, Specs2, μTest, etc.

2. This is not a property testing library.
   Use it conjunction with ScalaCheck, Nyaya, ScalaProps, etc.

Ok, so what is this? This is a library that:

1. Lets you write pure, immutable, referentially-transparent tests that verify stateful, effectful code or data.

2. Encourages composability of test concepts such as invariants/properties, pre/post conditions, dynamic actions/assertions, and more.

3. Makes test failure and inspection easy to comprehend.

Uses

• Unit-test a webapp with Scala.JS.
• Integration testing.
• UAT automation.
• Random-test (fuzz-test) like Android's monkeyrunner or ScalaCheck's Command API.
• Data migration.

Features

• Compiled for Scala & Scala.JS.
• Can run synchronously, asynchronously (Future) or in your own context-type (eg IO). Is stack-safe.
• Everything is immutable and composable.
• Everything can be transformed into (reused in) different contexts.
• Combines property and imperative testing.
• Actions and assertions can be non-deterministic and/or dependent on runtime state.
• Transparent and informative about test execution.
• Includes an abstract DomZipper which greatly simplifies the task of HTML/SVG observation.
• Comes with various DomZipper implementations and backends.
• Lots of platform-specific utilities for web testing.
• Configurable error handling. Be impure and throw exceptions or be pure and use a custom ADT to precisely maintain all forms of failure and error in your domain; it's up to you.
• Extension modules for various 3rd-party libraries. (Cats, more.)

How does this work?

The key is to take observations of anything relevant in the stateful test subject. Observations are like immutable snapshots. They capture what the state was at a particular point in time. Once an observation is captured, assertions are performed on it.

Optionally, you can specify some kind of test-only state that you modify as you test, and use to ensure the real-world observations are what you expect.
For example, if you're testing a bank account app, you could maintain your own expected balance such that when you instruct the app to make a deposit, you add the same amount to your state. You could then add an invariant that whenever the balance is shown in the app, it matches the expected state balance.

This is a (simplified) model of how tests are executed:

When retries are enabled, then test execution is like this.

How do I use this?

• Usage
• Types
• DSL
• Composability

Modules

Module	Description	JVM	JS
core	The core module.	JVM	JS
dom-zipper	Standalone utility for observing web DOM with precision with conciseness. This is the base API; concrete implementations below.	JVM	JS
dom-zipper-jsoup	DOM zipper built on Jsoup.	JVM
dom-zipper-selenium	DOM zipper built on Selenium. Also comes with a fast version with uses Jsoup for nearly all operations which is 5-50x faster. See doc/SELENIUM.md.	JVM
dom-zipper-sizzle	DOM zipper built on Sizzle.		JS
ext-cats	Extensions for Cats.	JVM	JS
ext-nyaya	Extensions for Nyaya.	JVM	JS
ext-scalajs-react	Extensions for scalajs-react.		JS
ext-selenium	Extensions for Selenium.	JVM

Examples

• Scala.Js + React - Demonstrates DomZipper, invariants, actions, basics.
• Selenium - Demonstrates Selenium testing of external web content, using retry scheduling (instead of Thread.sleep), parallelism and concurrency.
• [TODO] DB triggers. - real external state, ref.
• [TODO] Mutable sample. - fuzz, invariants.

Support

If you like what I do —my OSS libraries, my contributions to other OSS libs, my programming blog— and you'd like to support me, more content, more lib maintenance, please become a patron! I do all my OSS work unpaid so showing your support will make a big difference.

test-state's Issues

Allow post-assertion state modification

Currently state modification occurs before assertions. Add ability to tie a state update to an action, but have it run after successful post-action assertion.

(Currently using *.emptyAction.updateState(...) as a workaround)

Consider using NEV

In ((Vector[Layer[Dom]], Layer[Dom])) and in collect1n

Faster setText in Selenium

This is much faster

  private def asJsString(text: String): String = {
    val sb = new StringBuilder
    sb.append("'")
    text.foreach {
      case '\'' => sb.append("\\'")
      case '\\' => sb.append("\\\\")
      case '\n' => sb.append("\\n")
      case '\r' => sb.append("\\r")
      case '\t' => sb.append("\\t")
      case c    => sb append c
    }
    sb.append("'")
    sb.toString()
  }

  implicit class WebElementLocalExt(private val self: WebElement) extends AnyVal {
    private def currentValue(): String =
      self.getAttribute("value")

    def quickSetTextIfNeeded(text: String)(implicit d: WebDriver): Unit =
      if (text != currentValue())
        quickSetText(text)

    def quickSetText(text0: String)(implicit d: WebDriver): Unit =
      if (text0.isEmpty)
        self.clear()
      else {
        // React will only ingest the new value when it receives an event - just modifying the dom via JS will go
        // ignored by React. Therefore we set all but the last character using JS then use a real event for the last
        // char. This also has the nice effect of supporting a carriage-return at the end of the text argument to trigger
        // form submission.
        val text                  = text0.replace("\r", "")
        val n                     = text.reverseIterator.takeWhile(_ == '\n').size + 1
        val loc                   = self.getLocation
        val x                     = s"${loc.x}-window.scrollX+1"
        val y                     = s"${loc.y}-window.scrollY+1"
        val init                  = text.dropRight(n)
        val last                  = text.takeRight(n)
        val (lastText, lastOther) = if (last.endsWith("\n")) (last.dropRight(1), "\n") else (last, "")
        val cmd                   = s"document.elementFromPoint($x,$y).value=${asJsString(init)}"
        // println(s"${asJsString(init)} | ${asJsString(lastText)} | ${asJsString(lastOther)}")
        // println(cmd)

        // Step 1: Fast!
        if (currentValue() !=* init) {
          d.executeJsOrThrow(cmd)
          assert(currentValue() ==* init) // Without this, the value here can be overwritten by the steps below
        }

        // Step 2: Send 1 textual char without \n so that the app processes the expected event
        self.sendKeys(Keys.END + lastText)
        assert(currentValue() ==* (init + lastText))

        // Step 3: Send the non-text key
        // This needs to be separate from the previous step else ~2% of the time, the update is missed
        if (lastOther.nonEmpty)
          self.sendKeys(lastOther)
      }

Add Monocle support to transformers

Snippet:

  implicit val transformRT =
    RT.*.transformer
      .mapR[Ref](r => RT.Ref(r.tester.component zoomL State.rt, r.svr))
      .pmapO[Obs](_.rt)
      .mapS(TestState.project.get)((a, b) => TestState.project.set(b)(a))
      // ↑ TODO Add Monocle support

DisplayFailure isn't exported

Skipping a chooseAction should fallback on failure

It seems, when skipping a chooseAction, it tried to determine the action and use that name. Normally that's great but there are cases where it will fail because an earlier step in the test failed (thus why we're in skip-mode). In those cases we currently report the error as a second error, this is wrong. It should just fallback to the chooseAction name with None.

Add diagrams for retry logic

Fix colours on osx

getFn shouldn't be bound to Fn0

Enable .assert.empty in FocusOption

https://github.com/japgolly/test-state/blob/master/core/shared/src/main/scala/teststate/dsl/Dsl.scala#L320-L324

Error in MultiBrowser close

Exception in thread "Thread-1" org.openqa.selenium.WebDriverException: chrome not reachable
  (Session info: chrome=66.0.3359.139)
  (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Windows NT 6.1.7601 SP1 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: '3.11.0', revision: 'e59cfb3', time: '2018-03-11T20:26:55.152Z'
System info: host: 'W1H057993TZ50C', ip: '144.136.17.132', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.8.0_161'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552522 (437e6fbedfa876..., userDataDir: C:\Users\d873628\AppData\Lo...}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: XP, platformName: XP, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 66.0.3359.139, webStorageEnabled: true}
Session ID: b4d0e8b9b04952c23049fb3f5ad4d189
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
        at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
        at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
        at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
        at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
        at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
        at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:545)
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:602)
        at org.openqa.selenium.remote.RemoteWebDriver.getWindowHandles(RemoteWebDriver.java:453)
        at teststate.selenium.TabSupport$Typical.closeActive(TabSupport.scala:59)
        at teststate.selenium.TabSupport$Typical.closeActive$(TabSupport.scala:56)
        at teststate.selenium.TabSupport$Chrome$.closeActive(TabSupport.scala:65)
        at teststate.selenium.MultiBrowser$$anon$1$Browser.$anonfun$closeRoot$1(MultiBrowser.scala:54)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at teststate.selenium.Mutex$.apply$extension(Mutex.scala:12)
        at teststate.selenium.MultiBrowser$$anon$1$Browser.closeRoot(MultiBrowser.scala:51)
        at teststate.selenium.MultiBrowser$$anon$1.$anonfun$closeRoot$2(MultiBrowser.scala:174)
        at teststate.selenium.MultiBrowser$$anon$1.$anonfun$closeRoot$2$adapted(MultiBrowser.scala:174)
        at teststate.selenium.MultiBrowser$$anon$1.par(MultiBrowser.scala:190)
        at teststate.selenium.MultiBrowser$$anon$1.foreachBrowser(MultiBrowser.scala:184)
        at teststate.selenium.MultiBrowser$$anon$1.closeRoot(MultiBrowser.scala:174)
        at x.x.x.DDT.$anonfun$new$6(DDT.scala:67)
        at java.lang.Thread.run(Thread.java:748)

Remove DomCollection.{map,traverse,}{doms,extracts}?

Util for Option/Either/Disj get (need) with meaningful error msg

It's acceptable to call .get on Option etc in tests because if it fails and throws, well that just means the assertion failed, usually because what was observed wasn't in the expected state.

Add util (and probably to the main core Exports) for both

• failing with a more meaningful error message (eg start button not available instead of java.util.NoSuchElementException: None.get)
• above but going to disjunction instead of throwing
• String \/-like A → A throwing the String

Add needOne too.

Add method to disable CSS transitions & animation

https://stackoverflow.com/a/32988279/1846272

Dsl.future throws JavaScriptException: findDOMNode was called on an unmounted component

I followed the React example to set up a TestState, Dsl, and Test suite. Everything seems to work fine using the synchronous Dsl. However, when using Dsl.future I'm getting the exception: scala.scalajs.js.JavaScriptException: Invariant Violation: findDOMNode was called on an unmounted component.

When using Dsl.future, the component is initially mounted but it is no longer mounted when the observe() method gets called. Here's my runTest

  def runTest(plan: dsl.Plan): Future[Report[String]] = {
    ReactTestUtils.withRenderedIntoBody(MyComponent()) { c =>

      def observe() = {
        println(s"creating observer: ${c.isMounted()}") // c.isMounted() returns false
        new MyComponentObs(c.htmlDomZipper)
      }

      val test = plan
        .addInvariants(invariants)
        .test(Observer watch observe())

      test.runU()
    }
  }

And I call it using:

runTest(plan).map {
    m => m.assert()
}

Invariants appear twice in a row in the report on action group

When a group action runs, the invariants are run twice in a row.
Not a big deal at all but it causes some unnecessary noise in reports.

[info] ✓ Add XXXX 'XXXX'
[info]   ✓ Click 'XXXX'
[info]     ✓ Pre-conditions
[info]       ✓ XXXX modal is open should be false.
[info]     ✓ Action
[info]     ✓ Post-conditions
[info]       ✓ XXXX modal is open should be true.
[info]     ✓ Invariants
[info]       ✓ Identified pages must not identify more than one page
[info]   ✓ Set message to 'XXXX'
[info]     ✓ Action
[info]     ✓ Invariants
[info]       ✓ Identified pages must not identify more than one page
[info]   ✓ Click 'Add'
[info]     ✓ Pre-conditions
[info]       ✓ XXXX modal is open should be true.
[info]     ✓ Action
[info]     ✓ Post-conditions
[info]       ✓ XXXX modal is open should be false.
[info]     ✓ Invariants
[info]       ✓ Identified pages must not identify more than one page
[info]       ✓ Scenario messages should be XXXX
[info]   ✓ Invariants
[info]     ✓ Identified pages must not identify more than one page
[info]     ✓ Scenario messages should be XXXX

Use http://cquiroz.github.io/scala-java-time/ instead of scalajs-javatime for better java time support

Tried to use test-state for an application which is using LocalDateTime in the frontend. Getting linking errors from scalajs-javatime. Cquiroz provides a better support for the java time lib.

Or maybe make it possible to use any kind of java time lib? 🤔

Add drag-and-drop to Selenium stuff

Add .count(sel): Int to zippers

Add NamedOption#map

Obs error hidden on retry

With retry on:

1. obs pass
2. action fail
3. obs fail
4. obs fail
5. give up

The expected result should be from (4). Currently it is (2) which is very misleading.

Use sourcecode to report where assertion failures come from

[Scala 3] Use implicit fns to enforce side-effect boundaries

Side-effects are meant to appear in observations, and actions.

A potential problem is that observations then contain pure values (the actual data observations) and DOM references for use by actions. When Scala 3 comes out, implicit function types can be used to enforce side-effect capabilities such that the parts of the obs that are impure can be marked as such and be made callable only from actions and not assertions.

Collection assertions missing ∀ & ∃

Index state

It would be a huge, messy change but state should be indexed such that

compose :: Action f r o s1 s2 e
        -> Action f r o s2 s3 e 
        -> Action f r o s1 s3 e

Describe actions without running

I'd like a report equivalent to running actions.skip but without actually running anything

DomCollection failures should include outerHTML

SPA helpers

Being able to test pages in an SPA and keep everything modular requires a lot of things be done to set it all up. Hard to remember and pain in the ass to get in place. Add some direct support here.

focus.option.assert.contains is misleading

It was meant to be the same as .contains on Option but instead it just looks like substring contains.

FocusColl#map

def map[D[X] <: TraversableOnce[X], B: Display](f: C[A] => D[B]): FocusColl[D, B] =
  new FocusColl(focusName, f compose focusFn)

• I'd expect A => B
• C A => D B should be renamed
• Display no longer needed for FocusColl; remove from .map signature.

Add .domAs{Input,Textarea,Button}

[dom-zipper] Compositional query DSL

All the apply and child overloads could be merged into a single method with a composable arg.

Change dom to dom() outside of FastAndSlow

Makes it nice and consistent using DZFAS or not.

Compare obs against previous obs

Eg. "counter X should be monotonic"

Add tailrec action

For state-machines, loop allowing return of:

• next action
• exit loop
• error
• sleep cmd (or time)

Eventual assertion

When a step in an action fails, there should be an option to wait X and try again until success or time exceeds Y. Invariants are exempt; they by definition should never fail.

Also it should be configurable at various levels:

• plan
• action
• check (?)

Report default settings should colour an argument

Currently it's val coloured = ... where as it should be:

def addDefaultColours(uncoloured) =
val coloured = addDefaultColours(uncoloured)

Build for Scala 2.12.0

Waiting on:

• japgolly/univeq#6
• typelevel/cats#1450

Add getAttribute to shared DOM zipper

def getAttribute(name: String): Option[String]

Add `.{when,unless}(Boolean)`

But make sure it doesn't screw with type inference

Add support for a `(R, O, S) => S` to run with every successful obs

Use contextualised name on chooseAction(empty)

    dsl
      .chooseAction(NameFn {
        case None      => "A"
        case Some(ros) => "B"
      }) { ros =>
        dsl.emptyAction
      }
      .when(_ => true)

    dsl
      .chooseAction(NameFn {
        case None      => "A"
        case Some(ros) => "B"
      }) { ros =>
        dsl.action("C")(_ => ())
      }
      .when(_ => true)

should (on success) print "B" and "C" respectively. Instead it prints "A" and "C"

Selenium support

• Create dom-zipper that uses Selenium API
• Run test through Selenium

Add .child

.child == .child("*") i guess

Support assertMultiline

How am I supposed to read this??

      ✘ Other sources should be Some(Exclusive Tag Groups
  * Priority

Mandatory Fields
  * Business Justification(CO and FR excepted)
  * Priority(MF and FR only)
  * Released
  * Title(built-in)
  * Use Case Steps(built-in)

Req Types with Mandatory Implication
  * FR: Functional Requirement).
          Got Some(Exclusive Tag Groups
            * Priority
          
          Mandatory Fields
            * Business Justification
            * Priority(FR and MF only)
            * Released
            * Title(built-in)
            * Use Case Steps(built-in)
          
          Req Types with Mandatory Implication
            * FR: Functional Requirement), expected Some(Exclusive Tag Groups
            * Priority
          
          Mandatory Fields
            * Business Justification(CO and FR excepted)
            * Priority(MF and FR only)
            * Released
            * Title(built-in)
            * Use Case Steps(built-in)
          
          Req Types with Mandatory Implication
            * FR: Functional Requirement).

DOM zipper should expose more zippy operations

move up, left, right
expose path

[ext-scalajs-react] Add ExecutionModel for AsyncCallback

Integrate consistency checks around selenium actions