jiial/bumbleb

A BDD framework for my master's thesis project

bumbleB

BDD framework for my master's thesis project.

bumbleB is a behavior driven testing framework for Java where tests are written in Java code, in an easy-to-read syntax. bumbleB is built on top of JUnit and leverages Java's annotation processing in order to provide plain text descriptions of the tests. The descriptions are generated based on step definitions declared in bumbleB's @Step-annotations. bumbleB supports the given-when-then syntax, but also allows the creation of custom keywords to extend the syntax. bumbleB tests are created using the ExampleBuilder in the Framework class. The builder can be provided a name, a narrative, and a set of steps to execute. The steps are passed on as method references with the help of bumbleB's keyword-methods (given/when/then/and).

A basic example written in bumbleB might look like this:

public class CalculatorTests {

    private CalculatorSteps calculatorSteps = new CalculatorSteps();

    @Example
    public void calculatorSumTest() {
        builder
                .name("The calculator can calculate the sum of two numbers correctly")
                .steps(
                        given(calculatorSteps::inputNumbers, 1, 2),
                        when(calculatorSteps::chooseFunction, "sum"),
                        then(calculatorSteps::checkResult, 3)
                )
                .build()
                .run();
    }
}

The steps used in the tests might look like this:

public class CalculatorSteps {

  private Calculator calculator = new Calculator();
  private int actualResult;

  @Step("a user input the numbers {num1} and {num2}")
  public void inputNumbers(int num1, int num2) {
    calculator.input(num1, num2);
  }

  @Step("the user chooses the {functionName} function")
  public void chooseFunction(String functionName) {
    if (functionName.equals("sum")) {
      actualResult = calculator.sum();
    } else {
      // TODO: Implement other functions
    }
  }

  @Step("the result is: {expectedResult}")
  public void checkResult(int expectedResult) {
    Assert.assertEquals(expectedResult, actualResult);
  }
}

And the resulting text file (CalculatorTests.txt) generated by the Example processor would look like this:

Example: The calculator can calculate the sum of two numbers correctly

Given a user input the numbers {num1} and {num2}
When the user chooses the {functionName} function
Then the result is: {expectedResult}

While the console output would look something like this:

Running com.jiial.myProject.tests.CalculatorTests
Example: The calculator can calculate the sum of two numbers correctly

Given a user input the numbers 1 and 2
When the user chooses the sum function
Then the result is: 3

bumbleB is designed to support BDD on different levels of testing, and so it comes with built-in support for unit testing as well. A slightly modified example of a unit test from the examplesB project looks like this:

@Example
public void addToStock() {
    builder
            .name("Adding items to stock")
            .steps(
                    given(stock::addItem, new Item("Foo", 10.0))
                            .describeAs("item {item} is added to the stock"),
                    then(stock::isEmpty).satisfies(Assertions::assertFalse)
                            .describeAs("the stock is not empty"),
                    and(stock::getAmount, "Foo").satisfies(Assertions::assertEquals, 1)
                            .describeAs("there is [amount] {item} in the stock"),
                    and(stock::getItems).satisfies(Assertions::assertEquals, expectedStock)
                            .describeAs("the stock is equal to [stock]")
                )
                .build()
                .run();
}

where the referenced methods are normal methods of the class/unit being tested (no annotations or anything else required).

The full documentation can be seen here.

Example usages are available in the examplesB-project.

Known issues

• ExampleProcessor can't handle step prefixes/keywords with more than one word
• ExampleProcessor can't handle cases where custom prefix/keyword methods are called without a static import, e.g. ExtendedFramework.after() instead of just after()
• Various issues with version compatibility between junit-platform, Maven's surefire plugin and AspectJ (unresolved)
• Unit testing: ExampleProcessor does not generate descriptions for reused steps or steps that are otherwise created outside the methods annotated with @Example if the steps are not marked with @Step (which is true for unit tests)

Features and ideas to implement

• Add support for adding steps to existing example definitions in order to support shared before- or after-steps
  • Initial attempt caused an error at runtime for unknown reasons (and the changes were reverted)
  • In addition, the text files generated by the annotation processor were missing the shared steps that were declared outside the test method annotated with @Example
• Enhance Example processor to include the actual parameters used instead of simply using the plain annotation value
  • Could be pretty complicated to implement, and unsure if this is actually a desired feature
  • It can be argued that the data used is not that relevant to the test case itself, so maybe it makes sense that the information is only available at runtime (or from the Java code of course)
  • Also, sticking with the annotation values ensures that the generated file is easy to read and understand, whereas presenting some complex data structures might make it messy