hulu/roca

A command-line tool for running brightscript tests

roca's Issues

Compile time const error

Hi all

How are you handling Conditional compilation variables?

Our project makes use of compile time costs (https://developer.roku.com/en-ca/docs/references/brightscript/language/conditional-compilation.md) and when running Hulu Roca we're seeing the following error:

Attempting to reference undefined #const with name 'SOME_COMPILE_TIME_CONST'

Reading thru the Docs there is no mention of setting up a Hulu Roca manifest or ignore error type list?

Thanks

CreateObject returns Invalid

I'm trying to test a custom component, using the following code:

• components/MyComponent.xml:

<component name="MyComponent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://devtools.web.roku.com/schema/RokuSceneGraph.xsd">
    <script type="text/brightscript" uri="pkg:/components/MyComponent.brs" />
</component>

• components/MyComponent.brs:

function init()
    print "init() called"
end function

• tests/MyComponent.test.brs:

function main(args as object) as object
    return roca(args).describe("A sample test suite", sub()
        m.it("has a task", sub()
            test = CreateObject("roSGNode", "MyComponent")
            m.assert.isValid(test, "Should be valid")
        end sub)
    end sub)
end function

Running the test always fails with the following output:

      m.assert.isValid: Should be valid
      + expected - actual

      -null
      +non-invalid
      
      at m.assert.isValid (src/tests/MyComponent.test.brs:5:12)

Whenever I try creating another type of object, for instance CreateObject("roDeviceInfo"), then the test succeeds.

Add support for parameterized tests

Parameterized tests are really helpful when attempting to remove duplication in tests. Consider a function that accepts a number and returns that number incremented by 1:

function addOne(n as integer) as integer
    ' Warning to devs: advanced, very brittle mathematics ahead

    return n + 1
end function

Testing addOne in multiple cases gets pretty repetitive though:

m.it("adds one to 1", sub()
    m.assert.equal(addOne(1), 2, "should be 2")
end sub)

m.it("adds one to 2", sub()
    m.assert.equal(addOne(2), 3, "should be 3")
end sub)

m.it("adds one to -1", sub()
    m.assert.equal(addOne(-1), 0, "should be 0")
end sub)

Parameterized testing would allow users to do something like:

m.it_each([
    [-1, 0],
    [0, 1],
    [1, 2]
], "adds one to {0}", sub(args)
    m.assert.equal(addOne(args[0]), args[1], "incorrectly added one to " + args[0].toStr())
end sub

Add documentation for deepEquals

Right now, we don't have anything in the README for m.assert.deepEquals -- we should add a blurb

Encountered an error when parsing component

Hi, I'm trying to run roca on my project but the following error is raised:

Interpreter found an error:  Error: Encountered an error when parsing component ButtonBar: TypeError [ERR_INVALID_URL]: Invalid URL: ButtonBar.brs
    at componentDefinitions.forEach (/Users/user/Projects/roca-test-roku/nod_modules/brs/lib/index.js:66:19)

This error is while it is trying to parse the ButtonBar.xml component from Roku SGDEX(https://github.com/rokudev/SceneGraphDeveloperExtensions/tree/master/extensions/SGDEX/ButtonBar).

Did I forget to do something or is there any way that I can skip that file?

Print test suite and case names before executing beforeEach

When an error occurs inside of m.beforeEach, the user is unable to tell which suite/case caused the failure, because we are printing that out after the beforeEach block. We should print it before to enable easier debugging.

Add support for beforeEach/afterEach

We should include some way to execute code in several setup/teardown phases:

• beforeEach (like in jasmine): before every test in a suite (and its subsuites)
• afterEach (like in jasmine): after every test in a suite (and its subsuites)

Note that this'll probably require making sure there's a place to put shared state between tests since BrightScript doesn't support closures like JavaScript does.

Delete `_brs_.testData` in between suites

Summary

Once brs adds _brs_.testData (sjbarag/brs#623), let's clear it in between files so that test data isn't accidentally shared/leaked. We can do this in the same place that we call _brs_.resetMocks()

Allow configurable test root directory

Not everyone wants to keep their tests in a directory called, well, tests/! Users of roca should be able to keep tests in a directory called huluhaslivesports/ if desired. Seems unlikely, but we should allow it! 💰💰💰

Update mockFunction docs

Summary

In sjbarag/brs#622 we updated the _brs_.mockFunction API. The changes are backwards-compatible so the docs are still accurate, but let's update the docs to reflect the new features.

Fail test case when `isValid` gets <UNINITIALIZED>

m.assert.isValid(...) passes when gets <UNINITIALIZED>.

m.describe("initialization", sub()
    m.it("creates a component", sub()
        ? component
        m.assert.isValid(component, "node must not be invalid")
    end sub)
end sub)

Add a subcommand to create a simple test files

Creating a test file can be annoying since you have to do the following:

• Have a main function that takes and object as arguments
• Return the top-level describe function in the file
• The top-level describe should take the arguments passed to main

It would be great to add a subcommand to roca that creates a template test file with all of the above saving us time when adding new tests.

Wrong page link

Inside the How to use page, the link for quick start got an extra .md in it

Incomplete license

I would like to reuse the assert script in a repo, but the license doesn't seem correct.

LICENSE still has the template Copyright [yyyy] [name of copyright owner] message.

__roca_deepEquals compare objects with different types as equal

Example:

    m.fit("validate associative arrays", sub()
        expected = { key: "Some value"}
        actual = {}
              
        m.assert.deepEquals(expected["key"], actual["key"], "'Some value' should not be equal invalid")
   end sub)

Result:

✓ validate associative arrays

Expected:

                error:
                    message: 'Some value' should not be equal invalid
                stack:

Spy API?

There is no nice way to spy just one function of an object.

aa = {
   someFun: function()
     ? "doing something"
   end function
}

' what about?

_brs_.spyFunction(aa, "someFun")

aa.someFun("hey") ' does nothing
aa.someFun.called ' true
aa.someFun.calledWith ' ["hey"]
aa.someFun.calledTimes ' 1

Clarification about the test's scoping

So, I wrote a few tests and put them in the source/tests/ directory. All is working fine, but after that, I tried to write tests for some util functions that are inside the components directory, and I'm getting errors.

/Users/puritanic/roku/src/components/tests/ComponentUtils.test.brs(6,61-62): 'someComponentUtilFn' is not a function and cannot be called.
Stopping execution. Interpreter encountered errors:
	Error: 'someComponentUtilFn' is not a function and cannot be called.
		at /Users/puritanic/roku/src/components/tests/ComponentUtils.test.brs(6,61-62)

Looks like Roca is not detecting the functions declared inside ComponentUtils.brs file, which somehow makes sense, but is there any way I can import that file to the test suite so that I can test the functions from it?

This is an example of how dir structure I've tried to set up tests

── components/
   ├── Utils/
       ├── ComponentUtils.brs
   ├── tests/
       ├── ComponentUtils.test.brs
── images/
── source/
   ├── Utils/
       ├── SourceUtils.brs
   ├── tests/
       ├── SourceUtils.test.brs
── manifest

Add support for beforeAll/afterAll

We should include some way to execute code in several setup/teardown phases:

• beforeAll (like in jasmine): once before all tests in a suite
• afterAll (like in jasmine): once after all tests in a suite

Note that this'll probably require making sure there's a place to put shared state between tests since BrightScript doesn't support closures like JavaScript does.

On windows roca is not finding the tests

When we run the npm test on a win machine, we are getting always a 0 passing,
we have same code in a mac and its finding and executing the tests.

is roca supporting windows? or is there some other config needed for windows

Add support for fdescribe/xdescribe

While it's currently possible to focus on a test case with fit and exclude a test case with xit, users can't yet focus on or exclude an entire suite and all the tests (recursively) included within it! Implementing fdescribe (like jasmine's version) and xdescribe (again, like jasmine's version) should make that easier for users!

Call `_brs_.resetMocks` in between test files

Summary

We converted the test runner from Brightscript to Typescript in this commit, but forgot to include the _brs_.resetMocks call in the new Typescript test runner. We should add that back in!

Add isValid assertion

We have support for m.assert.isInvalid, but that doesn't support the inverse! Let's add m.assert.isValid (or something similarly named).

Add command line option to filter tests

Summary

It'd be great to have a command line option to filter tests. This would allow filtering tests without having to edit files, and limit risk of committing m.fit or m.describe to the codebase. For reference, Mocha has a command line option -g/--grep.

Convert to yarn instead of npm

To be in sync with brs and make dev a little smoother, let's use yarn instead of npm.

Add error message when no tests are found

Change summary

Currently, if you run roca and it doesn't find any test files, we exit with code 0 and pretend everything is fine. If you do the same thing with a framework like jest, though, it exits with code 1 and displays an error. We should let the user know when no tests are found.

Incorrect exit code is returned when npm test fails

Running npm test is always returning a success exit code (0) even when unit tests failed, some wrong syntax is found or --forbid-focused is called with focused test cases.
This bug will break CI/CD system that rely on exit codes from the command because it will think that there are no failures after running the command

Support test files in nested directories

When writing test cases, it would be great to be able to organize the test files into subdirectories (often mirroring the source directory) instead of having them all flat inside of the tests directory.

Several asserts in scope of the one test case don't work properly

If we make several asserts within one test case, and one of them fails, the others that follow will not be fulfilled. So, while running the test, we do not see any information about the following asserts until we fix the error or comment it.

No support for: ?

I don't remember the name of this feature well, but I'm having problems when trying to run tests in my .brs thanks to the use of: ?

Example: myObject?.hasError

It ends up displaying several error messages and warnings about this, in addition to complaining when calling the feature flag:

Example: #if myFeatureFlag

Which is defined in the manifest only at Brightscript compile time.

Does anyone have any idea how to resolve?

Add documentation for using `m` scope in unit tests

Summary

We should add some more documentation, perhaps in the form of a guide, on the best way to use the m scope in unit tests. It can be confusing to navigate the m scope of each m.it and m.describe block, especially once you add in further customization with m.addContext, _brs_.global, _brs_.testData, etc.

Escape YAML characters in error output

Summary

We're not escaping strings in our YAML output, which is causing some weirdness:

m.assert.equal("[what?", "yeah...")

The above causes tap-mocha-parser to incorrectly parse our YAML output, so we print out the raw (normally hidden) YAML:

        ---
                error:
                    message: Unexpected result
                    name: m.assert.equal
                    stack: m.assert.equal (/sandbox/tests/source/utils.test.brs:17:14)
                    stackframes:
                        - /sandbox/tests/source/utils.test.brs:17:14
                        - /sandbox/tests/source/utils.test.brs:2:11
                        - /sandbox/tests/source/utils.test.brs:2:4
                found: [what?
                wanted: yeah...
        ...

This actually isn't awful. It gives the user more information than tap-mocha-reporter does -- but it also doesn't look very nice and is an unexpected result.