junit-team/junit5

✅ The 5th major version of the programmer-friendly testing framework for Java and the JVM

junit5's Issues

Introduce first-class support for scenario tests

Proposal

The current proposal is to introduce the following:

• @ScenarioTest: class-level annotation used to denote that a test class contains steps that make up a single scenario test.
• @Step: method-level annotation used to denote that a test method is a single step within the scenario test.

The @Step annotation will need to provide an attribute that can be used to declare the next step within the scenario test. Steps will then be ordered according to the resulting dependency graph and executed in exactly that order.

For example, a scenario test could look similar to the following:

@ScenarioTest
class WebSecurityScenarioTest {

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Related Issues

• #13
• #419
• #607
• #884

Opt-in support for parallel execution

Related to #90.

Less restrictive Spotless settings

• Try to make Spotless less strict on additional or missing newlines.
• Needs to work in Eclipse and IntelliJ IDEA.

Split TestDescriptor into two interfaces

• TestDescriptor: immutable API for users of Launcher (in junit-engine-api)
• MutableTestDescriptor extends TestDescriptor: additional methods to add children etc.

ReflectionUtils.handleException() should not wrap a target exception in an IllegalStateException

Status Quo

The handleException() method in ReflectionUtils currently extracts a target exception from an InvocationTargetException, and if the target exception is a checked Exception it will be wrapped in an IllegalStateException with the message "Unhandled exception".

This behavior, however, is inappropriate for a testing framework that reflectively invokes test methods.

Specifically, any exception thrown by a test method must be reported in TestExecutionResult. If such an exception is wrapped in an IllegalStateException, that makes it very difficult to retrieve the original exception, since it is impossible to know that some IllegalStateExceptions should be treated differently than others.

Analysis

We do not want to have to reason about the contents of an IllegalStateException. Instead, we need an explicit way to retrieve an exception thrown by a test method.

Deliverables

• Change everything that throws Throwable to throws Exception
• Change ReflectionUtlils.invokeMethod()/newInstance() to mask checked exceptions as unchecked exceptions without physically wrapping them in a RuntimeException.
• Refactor ReflectionUtils.handleException() accordingly.

Introduce method call level exception expected

Currently if we need to test some method is throwing an exception by following methods

@test(expected=BalhException.class)

OR

@test
public void meTest() {
int a = 10;
int b = 20;
Cat c = new Cat();
try{
c.talk()
fail("this is bad")
} catch(BalhException e) {
//this is good
}
}

I prefer the 2nd option since there i can really tell that the point of failure is c.talk() method.

So I suggest a annotation for a method execution level tests which would look like following

@test
public void meTest() {
int a = 10;
int b = 20;
Cat c = new Cat();
@MethodTest(expected=BalhException.class)
c.talk()

}

Investigate getting rid of external dependencies in junit-console

To be usable on the command-line, it would be nice if junit-console were a jar that includes all of its dependencies (fat-jar, uber-jar, ...).

Investigate AsciiDoc Javadoc options

e.g. https://github.com/asciidoctor/asciidoclet

Provide mechanism to order the sequence of tests

Overview

Although it is typically discouraged, having a way to order tests is useful to some people. The JUnit 4 @FixMethodOrder annotation specifically and deliberately makes it impossible to implement a simple ordering scheme, forcing users who want/need this to resort to method names like test245_ensureReasonableTestName().

Proposal

• The minimal "requirement" would be to allow users to write their own MethodSorter for @FixMethodOrder
• A reasonable implementation would be to allow @FixMethodOrder(ORDERED), and then @Order(integer) to specify individual tests. Tests could be run from smallest number to highest, with non-annotated methods having a default of 0. Methods with the same order could be run in parallel; different order tiers could not.

Related Issues

• #48
• #607
• #884
• #1707

Deliverables

• Introduce a mechanism for ordering testable methods within JUnit Jupiter.
• Validate collection of methods ordered by an extension; log warning or error if methods were added or removed.
• Introduce built-in support for alphanumeric ordering.
• Introduce built-in support for @Order based ordering.
• Introduce built-in support for random ordering.
• Introduce Configuration Parameter for setting the seed used in random ordering.
• Test ordering support in top-level classes.
• Test ordering of @Test, @TestFactory and @TestTemplate methods.
• Test ordering support in @Nested classes.
• Test ordering support in conjunction with parallel test execution.
• Extract junit.jupiter.execution.order.random.seed constant and document it.
• Write Javadoc for all new types.
• Update @API declarations for types with changed visibility, etc.
• Document new features in User Guide.
• Document new features in Release Notes.

Out of Scope

• ❌ Consider introducing a mechanism for ordering tests with the JUnit Platform -- potentially at a later date in conjunction with a separate dedicated issue.

Prepare repository for alpha phase

1. Tag current master with prototype-1.
2. Change group id to org.junit.
3. Delombok everything.
4. Delete elements we want to rewrite (e.g. primarily the implementation of junit5-engine) using JUnit Team <[email protected]> as the Git author
5. Remove all author tags
6. Create test module and move existing tests for published modules there.
7. Remove dependencies on junit:junit:4.12 from all existing modules except junit4-engine, junit4-launcher-runner, and open-test-alliance.

JUnit Team user configuration for ~/.gitconfig:

[user]
    name = JUnit Team
    email = [email protected]

Introduce support for parameterized tests

Deliverables

• Introduce @TestTemplate and accompanying extension point.
• Determine if #704 should be included in M4.
• Determine if we need an @InvocationIndex annotation or TestTemplateInfo type for injecting the current invocation index, etc. into @Test, @BeforeEach, and @AfterEach methods.
  • See d2e3a4d
• Introduce junit-jupiter-params module
  • Merge initial contribution (#728)
  • Javadoc
    • @since tags
  • API annotations
  • Add unit tests where missing (see TODOs)
  • Resolve TODOs
• Document @TestTemplate and accompanying extension point in User Guide and Release Notes
  • Provide example(s) in the documentation module and include in the User Guide. Will be addresses in #762
• Document parameterized tests in User Guide and Release Notes
  • Document the junit-jupiter-params module in the Dependency Metadata section.
  • Document the junit-jupiter-params module in the Dependency Diagram.
  • Provide example(s) in the documentation module and include in the User Guide.

---

Original Issue Description

At the time of writing, the https://github.com/junit-team/junit-lambda/wiki/Prototype-Test-Decorators page does not have all that much detail, so this might already be considered.

It would be good if the library shipped with ready-to-use simple parameter annotations that could be used inside a single test instance. It would in many ways look like JUnit 4's @Parameterized Runner, but for use inside a single class. When running the test, each @Parameterized method would appear many times, once for each data set.

class MyTest {

   // A simple test with a specified data set
   @Test @Parameterized("mod10")
   void testMod10(String value) { ... }

   @Parameterized("mod10") 
   Collection<Object[]> getMod10Values { ... }

   // Also including a method naming convention
   @Test @Parameterized("mod11", name="{0}")
   void testMod11(String value) { ... }

   @Parameterized("mod11")
   Collection<Object[]> getMod11Values { ... }

   // In the absense of a @Parameterized data provider, use reflection
   @Test @Parameterized("getMod12Values", name="{0}")
   void testMod12(String value) { ... }

   Collection<Object[]> getMod12Values{ ... }
}

---

Related Issues

• #378
• #386
• #604
• #704
• #728

Re-implement JUnit 4 engine

Discovery

• Resolve ClassSpecification elements
  • Regular JUnit 4 test class
  • Class with @Ignore annotation
  • Class with @RunWith annotation
  • Regular JUnit 3 test class
  • Class with a suite method
  • Does not discover classes it should not run
    • Class with @RunWith(JUnit5.class) annotation
    • Class without test methods
• Resolve AllTestsSpecification
• Resolve MethodSpecification
  • single method
  • multiple methods of same class
  • combination with ClassSpecification of method's class
  • Non-filterable Runner
  • Method of class filtered out by a ClassFilter
• Resolve PackageSpecification
• Resolve UniqueIdSpecification
  • single unique ID of test method
  • single unique ID of test class
  • unresolvable unique ID
  • unique ID of method of test class in suite
  • multiple unique IDs for same class
• Logging (see TODOs in code)

Execution

• Container and test events for single test class are reported correctly
• Container and test events for suite are reported correctly
• Reports InitializationErrors for malformed tests (e.g. non-public test methods) correctly
• Handle dynamic tests Not supported by JUnit 3/4
• Protect against Runners that throw exceptions
• Test class with multiple methods of same name?
• Failure in @BeforeClass or @AfterClass
• Handle (ignore?) unknown Descriptions

Complete TestDescriptors

• Consider Description#fUniqueId instead of displayName
• Report categories as tags
• Determine and set TestSources if possible

---

Idea for implementing container events

Create/Migrate Tests for project 'junit-launcher'

Allow tests to be executed in a user-defined thread

Overview

In JUnit 4, we use a specific test runner that runs the test code on a dedicated UI thread, such as the AWT/Swing thread (or the JavaFX thread). This avoids cluttering your code with EventQueue.invokeAndWait all over the place. As far as I understood, the current extensions proposal does not allow to do such thing.

The below runner simply wraps all calls to before, after, and test methods in an EventQueue.invokeAndWait call. Tests simply have to declare @RunWith(AWTThreadRunner.class) to get this behavior.

    public class AWTThreadRunner extends BlockJUnit4ClassRunner {
      public AWTThreadRunner(Class<?> klass) throws InitializationError {
        super(klass);
      }

      @Override
      protected Statement classBlock(RunNotifier notifier) {
        return onAWTThreadStatement(super.classBlock(notifier));
      }

      @Override
      protected Statement methodBlock(FrameworkMethod method) {
        return onAWTThreadStatement(super.methodBlock(method));
      }

      private static Statement onAWTThreadStatement(final Statement aStatement) {
        return new Statement() {
          @Override
          public void evaluate() throws Throwable {
            EventQueue.invokeAndWait(new Runnable() {
              public void run() {
                aStatement.evaluate();
                // left out exception handling code improve readability
              }
            });
          }
        };
      }
    }

Note: the actual implementation is a bit more complex in that it does proper exception handling and refuses JUnit's time-outs as that doesn't work together with running on a dedicated thread.

Related Issues

• #157

Simple exception testing using lambdas

Add something like it was proposed in junit-team/junit4#1055.

Introduce simple ToStringBuilder

In order to avoid code duplication and dependencies on third-party libraries such as Apache Commons, introduce a simple ToStringBuilder that generates output similar to that which is currently generated by ConditionEvaluationResult#toString().

Update CONTRIBUTING.MD

• I hereby agree…
• Section about coding standards, usage of spotless
• Section about Javadoc:
  • No author tag
  • etc.

Should JUnit support mocking as a basic feature

There are some object mocking frameworks available such as Mockito. Should JUnit care and give functionality to junit users to mock variables?

If it is yes, then I would like to see some form of automatic mocking mechanism for JPA meta-model like classes. But again the mocking should should be scoped for a single test. If im explaining the problem further, now when the hibernate meta models are generated they define variables as "SingularAttribute blah". So each of those fields will be null on Junit testing level. So the verification will not work properly unless you first mock a value and set them.

Introduce support for a custom test scheduler

Hello,
my use case is to execute the same testcase concurrently in different threads on the same test object to find race conditions. If I understand the api correctly this should be possible by using the annotation @testinstance PER_CLASS and the yet to be implemented but planned "Introduce support for registering parameterized tests #14". What is missing is an annotation for a custom scheduler which gets preferably the root EngineTestExecutionNode and is responsible for actually running the tests. Is something like this planned already?
Thank you very much
Thomas

Introduce support for type information in TestExecutionContext

TestExecutionContext is hierarchical, and currently there are "engine", "class", and "method" context types.
However, once context is created, there is no simple indication of which type it is.

I think it would be better that TestExecutionContext would have some API that can easily distinguish context type.

Implementation may be:

• keep TestDescriptor which is passed at constructor of DescriptorBasedTestExecutionContext, then user can do context.getDescriptor() instanceof ClassTestDescriptor.
• or, can have getContextType() method that returns simple enum such like:

public enum TestExecutionContextType {
  ENGINE, CLASS, METHOD
}

Split Engine API package into subpackages

Proposal

(not checked for package cycles etc., yet)

org.junit.gen5.engine.api

• TestDescriptor
• TestSource
• TestTag
• TestEngine
• ExecutionRequest
• EngineExecutionListener
• EngineAwareTestDescriptor
• discovery
  • AllTestsSpecification
  • ClassSpecification
  • ClassFilter
  • ClassFilters
  • ClassNameFilter
  • EngineFilter
  • MethodSpecification
  • PackageSpecification
  • TestPlanSpecification
  • TestPlanSpecificationElement
  • TestPlanSpecificationElementVisitor
  • UniqueIdSpecification
• support
  • descriptor
    • AbstractTestDescriptor
    • EngineDescriptor
    • FileSystemSource
    • JavaSource
  • hierarchical
    • Container
    • EngineExecutionContext
    • HierarchicalTestEngine
    • HierarchicalTestExecutor
    • Leaf

Experiment with alternative approach for Callbacks for @Before/@After

Alternative solution for pre/post callbacks.

Get InstancePostProcessor in sync with *Callbacks

It seems the InstancePostProcessor could be made in sync with all the Callbacks, and probably be named TestCallbacks, giving it both a pre and post method similar to the other Callbacks.

An example use case is measuring how long the execution of the test method took (without all before/after calls). We use this to measure the performance of our code, and fail the test if it runs significantly slower than earlier executions.

Clarify / Introduce path concept for the unique ID of the TestDescriptors

Having a tree of TestDescriptors that we use for the execution of tests, we need to guarantee that the descriptors are always resolved in the same tree structure, i.e. a MethodTestDescriptor should always have its enclosing ClassTestDescriptor as a parent, no matter how the method was found during the resolve phase (by unique id, by class reference, by method reference, etc.).

If we guarantee the tree structure to be equal we can then deduce the path of a descriptor by the tree node leading to the descriptor. As for Java test the unique id of a deeper node in the tree is always prefixed by the unique id of its parent node, we could reduce the path such that each level only adds the corresponding part.

We should investigate some time on this idea and clarify / introduce a path concept for the JUnit5TestEngine.

Enable ConsoleRunner to add additional entries to classpath

Work has started on this in #37 (37ba9a2).

Configure Spotless to add licence headers to Groovy files

Create and communicate policy for published API lifecycle

SUGGESTION:

Introduce annotation for all interfaces and classes supposed to be used from outside:

@PublicAPI(Stable)
@PublicAPI(Maintained)
@PublicAPI(Experimental)
@PublicAPI(Deprecated)
@PublicAPI(Internal)

• Stable: Will be here forever (in this major version 5.x)
• Maintained: Considered mostly stable. If considered for removal, it will be deprecated for at least one minor version.
• Experimental: Use at your own risk. Might be removed at any time.
• Internal: Only used for public members in types that have a @PublicAPI annotation.

Everything not annotated would be considered for internal use only.
Annotation would allow to produce API change reports for all releases.

Tasks

• Create annotation
• Add paragraph about API annotations to documentation
• Annotate junit-launcher
• Annotate junit-commons
• Annotate junit-console
• Annotate junit-engine-api
• Annotate junit5-api
• Annotate junit5-engine
• Annotate junit4-engine
• Annotate junit4-runner

Add mechanism for registering dynamic tests

Further Resources

• Related to #14.

Agree and document on conventions for Javadoc

Where to use Javadoc?

• Every class/method that we expose as API?
• Every class/method that is public?

How to write/format Javadoc?

• How to format paragraphs?
  • <p> on a separate line or at the beginning of the new paragraph?
• Where to put whitespace?
  • blank line between paragraphs?
  • blank line before tags?
• Define/check order of tags?

There is some good advice here:

• http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html
• http://www.oracle.com/technetwork/java/javase/documentation/index-142372.html
• https://github.com/spring-projects/spring-framework/wiki/Spring-Framework-Code-Style#javadoc

Introduce support for declaring dependencies between test methods

See https://github.com/junit-team/junit.contrib/tree/master/assumes
I hope the new JUnit add this in the core. Or, at least makes it easy to built it as an extension.

Tests for gradle plugin

Write at least one test to apply the gradle-plugin so that the most stupid mistakes don't break the build.

Re-implement JUnit 5 engine

• Resolution (in #29)
• ExtensionRegistry honors ordering
• Implement getParent() of ExtensionContext
• Encapsulate attributes for ExtensionContext, do not expose Map
• Invocation of ExtensionPoints
  • BeforeEach
  • AfterEach
  • BeforeAll
  • AfterAll
  • PostProcessTestInstance
  • MethodParameterResolver
  • Conditional test execution
• Clean up integrated tests (subclasses of AbstractJUnit5TestEngineTestCase)

Commit message template and Git hook for Indiegogo sponsors

• Only a single sponsor at a time
• Plan for 27 days with foreseeable activity (Google spreadsheet)
• Local Git hook with setup instructions

Rename junit4-launcher-runner to junit4-runner

For greater clarity:

• Rename the junit4-launcher-runner module to junit4-runner.
• Rename the org.junit.gen5.junit4runner package to org.junit.gen5.junit4.runner.
• Update artifact ID in README, Wiki, etc.

Support parameterized test

Related to #58.

Introduce uniqueId and name properties in ExtensionContext

Status Quo

ExtensionContext currently has a displayName property; however, this is limited in terms of robust identification.

Deliverables

• Introduce uniqueId property in ExtensionContext
• Introduce name property in ExtensionContext

Improve interface of TestPlanSpecification

• Change Launcher API to just use TestSpecificationElement (or better name) as marker interface, and make the collection of elements accessible to engines, as a group or by specific type
• Build DSL on top of launcher API in separate module junit-testplan-dsl
• Change filterWith() from Predicate<TestDescriptor> to Predicate<TestIdentfier>
• TestPlanSpecification must implement Serializable
• TestPlanSpecification must be immutable

Redefine and implement TestExecutionListener

• Change interface, implementors, and callers.
• Document semantics in Javadoc.

Introduce extensible reporting API

• Move TestReporter junit5-api module
• Replace Map<String, String> with a custom class, e.g. ReportingEntry
• Display entries in ConsoleRunner
• Add tests for JUnit5EngineExecutionContext#getExecutionListener()
• Add tests for TestReporterParameterResolver
• Add tests for ExecutionListenerAdapter
• Move reportingEntryPublished method to the end of TestExecutionListener, EngineExecutionListener, and their implementations.
• Move ReportEntry to junit-engine-api. TestReporter uses Map (instead of ReportEntry)
• Added TestReporter example to documentation
• Output incoming ReportEntries in junit4-runner and document that in documentation.
  • All formatted outputs of report entries should look alike (see XmlReportWriter, ColoredPrintingTestListener)

Move Open Test Alliance to new project

• separate GitHub organization
• register domain name
• create repo
• copy code into new repo
• publish artifacts in Sonatype's snapshot repo (@marcphilipp)
• use it in junit modules
• remove open-test-alliance module from JUnit Lambda repository
• invite people from #12
• incorporate feedback from #12

Launch Open Test Alliance Initiative

Status Quo

There is no standard for testing on the JVM: the only common building block we have is java.lang.AssertionError.

AssertionError is great for signaling that a test has failed, but it doesn't go far enough. Each testing framework is therefore forced to fill the gap with custom subclasses of AssertionError or RuntimeException to provide a richer feature set to end users. The downside is that each framework has its own set of custom errors and exceptions, and this makes it a challenge for frameworks to interoperate.

For example, JUnit has long supported the notion of a failed assumption via its AssumptionViolatedException, but assertion frameworks like AssertJ cannot integrate that feature without a direct dependency on JUnit. Furthermore, the status quo makes the work of IDEs and build tools more difficult than it should be.

Proposal

The only real solution to this problem is to create a foundation that we can all build on!

Based on recent discussions with IDE and build tool developers from Eclipse, Gradle, and IntelliJ, the JUnit Lambda team is working on a proposal for an open source project to provide a minimal common foundation for testing libraries on the JVM. The primary goal of the project is to enable testing frameworks like JUnit, TestNG, Spock, etc. and third-party assertion libraries like Hamcrest, AssertJ, etc. to use a common set of exceptions that IDEs and build tools can support in a consistent manner across all testing scenarios -- for example, for consistent handling of failed assertions and failed assumptions as well as visualization of test execution in IDEs and reports.

Draft Implementation

We have already begun with a small set of errors and exceptions that we consider to be common for all testing and assertion frameworks. In fact, we are already using these exceptions in the JUnit Lambda Prototype.

Please take a look at our current draft in the open-test-alliance project and let us know what you think.

Project Name and Repository

The name Open Test Alliance is simply a working title for the project. As such, the name is up for discussion, and recommendations are welcome. We will of course need a name that maps to an available Internet domain name so that the code can live in an appropriate Java package.

The draft implementation is currently a part of the JUnit Lambda repository; however, as soon as additional parties join the collaboration, we will collectively work to find a new home for the project.

Feedback is welcome!

What types of errors and exceptions should such a project support?

What types of properties should such errors and exceptions have?

What should the project be named?

Projects already contacted

We've already reached out to and asked for feedback from the maintainers of the following projects.

• Test NG
• Hamcrest
• AssertJ
• Spock
• Google Truth
• ScalaTest
• Eclipse
• IntelliJ
• Gradle
• Maven Surefire Plugin

Reach out to Maven Surefire maintainers regarding JUnit Platform provider

Agree and document team conventions for automated tests

Test Class Naming

I think we need a naming convention for our test classes. At the moment we have test classes that end with Test, Tests, and TestCase. Personally, I would prefer one of the first two variants.

Assertions

In addition, we should agree on how to write assertions. At the moment we have tests that use JUnit 4's Assert, some use our new Assertions class, others use AssertJ's Assertions. Since we plan to migrate our tests to use JUnit 5 at some point, I would propose that we do not use anything in JUnit 4, particularly not org.junit.Assert. Instead, I think, we should use our own Assertions were feasible and AssertJ, where we need its power.

Test Doubles

We had a brief discussion on Skype regarding a mocking framework. I've added Mockito to the test classpath and used it in a couple of tests. Other tests use hand-written stubs or spys. I think we can go with Mockito where easier and write doubles ourselves when we think it clearer.

Investigate more extensible solution for test resolution

Tests for junit-console

I started to make it testable in #37.

Code Conventions

It would be nice to have some code conventions defined, to make it easier for non-core-committers to provide useful pull requests.

Ideally they would come with formatters/checkstyle configurations.

Test the 'junit-engine-api' module

• Write tests for HierarchicalTestExecutor

Set up AsciiDoc-based user manual

Remove sample-extension and sample-project from test execution during build

Until we have reimplemented the junit5 engine (see #39) we should not execute the tests of the sample projects, as they are built upon the new features.

Migrate documentation from wiki to AsciiDoc

• Migrate documentation from wiki to AsciiDoc
• Extract existing Java examples from AsciiDoc into Java source files.
• Move tests from sample-project to Java test source folder in documentation module and include them in the AsciiDoc documentation.