srdc/gitb

GITB TestBed PoC Implementation

gitb's Issues

Validate test artefacts upon their definition to avoid errors at test execution

Validation of test suite and test case XML files is possible in an offline manner by using the respective XML Schemas. However, there is significant room for errors that can go uncaught at the time of the test suite’s upload and that can cause unexpected problems upon execution. The test suite upload should be improved to fully validate provided input, from XML structure to content (e.g. matching transaction IDs, correct asset references, valid expressions) so that errors are caught before test case execution. This is particularly important considering that test case execution will often not be done by the test cases’ authors.

Binary file provided as configuration from testbed to SUT

Upon test initiation, the GITB testbed exposes relevant configuration properties to the tester in order to configure the SUT. This includes also the dynamically generated endpoint information for the test services (actors) that are used in the test case. Additional flexibility could be provided here since the implementation supports only text properties, whereas it could also be interesting to expose binary content (e.g. a certificate or data file for use by the SUT).

Export the detailed log of a test case’s execution

Exporting the log output generated from a test’s execution is not currently possible. Allowing this would offer significant benefits to users in the investigation of unforeseen errors or detailed validation failures.

Export the raw content of exchanged messages

Exporting the content of received messages is not possible, representing a feature that is noticeably lacking.

This is more so the case when coupled with the limitation of being unable to copy data from the message content displayed through the user interface (#5).

tests fail on artifact gitb-messaging

After a full clean install without running the tests (-DskipTests), I'm extending the SOAP messaging implementation to support MTOM in the gitb-messaging package. I'd like to make sure I don't break anything, so I'm trying to execute the tests.

They fail on AS2MessagingHandlerTest, and then keep hanging on DNSMessagingHandlerTest.

Below is the full maven tests log

Running com.gitb.messaging.AS2MessagingHandlerTest
13:23:10.264 [main] DEBUG c.g.m.s.tcp.TCPMessagingServerWorker - New server created on: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=8081]
13:23:10.269 [main] DEBUG c.g.m.s.tcp.TCPMessagingServerWorker - New server created on: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=8082]
13:23:10.274 [main] DEBUG c.g.m.s.tcp.TCPMessagingServerWorker - New server created on: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=8083]
13:23:10.274 [main] DEBUG c.g.m.s.tcp.TCPMessagingServerWorker - New server created on: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=8084]
Tests run: 5, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.395 sec <<< FAILURE!
test_1_InitSession(com.gitb.messaging.AS2MessagingHandlerTest)  Time elapsed: 0 sec  <<< ERROR!
com.gitb.exceptions.GITBEngineInternalError: [as2.name,public.key] actor configurations are required.
    at com.gitb.messaging.layer.AbstractMessagingHandler.checkRequiredParameters(AbstractMessagingHandler.java:342)
    at com.gitb.messaging.layer.AbstractMessagingHandler.validateActorConfigurations(AbstractMessagingHandler.java:350)
    at com.gitb.messaging.layer.AbstractMessagingHandler.initiate(AbstractMessagingHandler.java:43)
    at com.gitb.messaging.MessagingHandlerTest.test_1_InitSession(MessagingHandlerTest.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

test_2_SendReceiveMessage(com.gitb.messaging.AS2MessagingHandlerTest)  Time elapsed: 0.01 sec  <<< ERROR!
com.gitb.exceptions.GITBEngineInternalError: 'receive' command functionality is not available between 2 SUTs, please take a look at 'listen' command
    at com.gitb.messaging.layer.AbstractMessagingHandler.receiveMessage(AbstractMessagingHandler.java:141)
    at com.gitb.messaging.MessagingHandlerTest.test_2_SendReceiveMessage(MessagingHandlerTest.java:138)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

test_3_EndSession(com.gitb.messaging.AS2MessagingHandlerTest)  Time elapsed: 0.005 sec  <<< ERROR!
java.lang.NullPointerException
    at com.gitb.messaging.MessagingHandlerTest.test_3_EndSession(MessagingHandlerTest.java:155)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Running com.gitb.messaging.DNSMessagingHandlerTest
13:23:10.319 [main] DEBUG c.g.m.s.udp.UDPMessagingServerWorker - New server created on [53]
Exception in thread "Thread-1" java.lang.NullPointerException
    at com.gitb.messaging.MessagingHandlerTest$1.run(MessagingHandlerTest.java:105)
    at java.lang.Thread.run(Thread.java:745)
Exception in thread "Thread-2" com.gitb.exceptions.GITBEngineInternalError: 'send' command functionality is not available between 2 SUTs, please take a look at 'listen' command
    at com.gitb.messaging.layer.AbstractMessagingHandler.sendMessage(AbstractMessagingHandler.java:97)
    at com.gitb.messaging.MessagingHandlerTest$2.run(MessagingHandlerTest.java:128)
    at java.lang.Thread.run(Thread.java:745)

Provide a SQL dump

I got the version of GITB-UI in git. It has changed slightly compared to the one that was in SVN.

I've seen a slight modification on the table testCaseHasActors as a new column has been added. I cannot use my old database, so I need to start from scratch or hack my existing database.

It would help new users to get started with GITB-UI if there were an SQL dump containing the data of

• 2 default users (a system admin and a vendor admin),
• the configured sample test suites.

Allow arbitrary update of a test case execution context

Each test case execution is assigned a persistent context in which information can be recorded. This information relates to initial configuration, user input, data extracted from messages, as well as arbitrary input. Updating the context is not however possible from all test steps. The specific example that has been encountered is update in the "send" handler of a messaging module, in which case entries added to the context are not available once the module completes. To address this a current workaround is to pass the reference to a map, created in the "receive" handler that, being passed by reference, is updated and remains usable later on in the test.

Note: It should be verified whether support for this is already implemented.

Create test artefacts using online graphical editor

Test artefacts are created offline as archives of XML files and used assets. Structural validation using the GITB XSDs is of course possible but leaves significant room for errors concerning the defined content. A graphical editor to simplify test case authoring, allow for resource lookup and reuse and, very importantly, validate results, would significantly streamline test case development.

Note: This task is considered as one of significant complexity that would need to be broken down further. As, such it is marked as an EPIC.

messagingContext is null in the second transaction when the 2 transactions have the same handler

I'm writing a test case where I have 2 transactions based on the same handler SOAPMessaging.

<btxn from="DocumentSource" to="DocumentRepository" txnId="t2" handler="SoapMessaging"/>
        <receive id="soap_output1" desc="Provide and Register Set-b (one Document)" from="DocumentSource" to="DocumentRepository" txnId="t2">
            <config name="soap.version">1.2</config>
        </receive>
        <send id="soap_input1" desc="Send response" from="DocumentRepository" to="DocumentSource" txnId="t2">
            <input name="soap_message" source="$DocRepo_Template" />
            <config name="soap.version">1.2</config>
        </send>
        <etxn txnId="t2"/> 

        <btxn from="DocumentSource" to="DocumentRepository" txnId="t1" handler="SoapMessaging"/>
        <receive id="soap_output2" desc="Provide and Register Set-b (Two Documents)" from="DocumentSource" to="DocumentRepository" txnId="t1" >
            <config name="soap.version">1.2</config>
        </receive>
        <send id="soap_input2" desc="Send response" from="DocumentRepository" to="DocumentSource" txnId="t1">
            <input name="soap_message" source="$DocRepo_Template" />
            <config name="soap.version">1.2</config>
        </send>
        <etxn txnId="t1"/>

The first transaction executes without problem. However when the second transaction starts, the messagingContext is null and I got a NPE, so the second transaction fails.

7:29:05.950 [test-engine-as-akka.actor.default-dispatcher-4] DEBUG c.g.m.l.AbstractTransactionReceiver - Closing socket: Socket[addr=/127.0.0.1,port=56378,localport=8081]
17:29:05.950 [test-engine-as-blocking-processor-dispatcher-16] DEBUG com.gitb.engine.TestbedService - updateStatus ( 2d4f8b24-079a-474f-875d-a02a791a25e4 , 2 , COMPLETED , com.gitb.tr.TAR@17a378fd ) 
17:29:05.950 [TCPListenerThread:8081] DEBUG c.g.m.s.tcp.TCPMessagingServerWorker - Closing server created on: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=8081]
17:29:05.958 [test-engine-as-akka.actor.default-dispatcher-3] ERROR c.g.e.a.p.AbstractTestStepActor - akka://test-engine-as/user/session/2d4f8b24-079a-474f-875d-a02a791a25e4/tc-p/seq-p-9nVK7/begin-txn-p-YFyeL caught an exception
java.lang.NullPointerException: null
    at com.gitb.engine.actors.processors.BeginTransactionStepProcessorActor.start(BeginTransactionStepProcessorActor.java:31) ~[gitb-engine-1.0-SNAPSHOT.jar:na]
    at com.gitb.engine.actors.processors.AbstractTestStepActor.onReceive(AbstractTestStepActor.java:126) ~[gitb-engine-1.0-SNAPSHOT.jar:na]
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.3.9.jar:na]
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465) [akka-actor_2.11-2.3.9.jar:na]
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.3.9.jar:na]
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [akka-actor_2.11-2.3.9.jar:na]
    at akka.actor.ActorCell.invoke(ActorCell.scala:487) [akka-actor_2.11-2.3.9.jar:na]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) [akka-actor_2.11-2.3.9.jar:na]
    at akka.dispatch.Mailbox.run(Mailbox.scala:221) [akka-actor_2.11-2.3.9.jar:na]
    at akka.dispatch.Mailbox.exec(Mailbox.scala:231) [akka-actor_2.11-2.3.9.jar:na]
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.5.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.5.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.5.jar:na]
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.5.jar:na]
17:29:05.962 [test-engine-as-akka.actor.default-dispatcher-3] ERROR c.g.e.a.p.AbstractTestStepActor - An error status received in []

This was working with the previous release of GITB (rev 2 in the SVN repository).

I guess that having two transactions with the same handler in the same test case does not work as the messagingContext gets cleaned after the first transaction ends. Is there a workaround? I tried to wrap the test steps in the same transaction but the second receive fails.

Validate messages using external or internal assets

Content validation can take place using assets such as XML Schemas that can also be reused between different test suites. A point that could be improved nonetheless is the ability to use externally located assets for validation (e.g. an XML Schema published online).

Note: it should be checked whether it is already possible to use external validation resources based on URI.

Gracefully handle unexpected errors during test execution

Test case execution errors are handled by exposing the error messages of the corresponding test steps. Unforeseen errors do not block the execution however, resulting typically in subsequent failures that do not add value to the test outcome. The error handling can be significantly improved by allowing fatal errors to terminate a test run and automatically release used resources.

Visualise the raw content of exchanged messages

The raw content of exchanged messages is extracted and exposed in the test execution context by the underlying step’s messaging handler. All such context information is then available for full inspection through the testbed’s user interface (GITB-UI) both for the current test run but also for recorded ones.

It is not possible to copy text from the displayed message contents. This capability would be of significant benefit.

Visualise unexpected errors in the execution of a test case

Unexpected errors are represented and visualised as step failures. However, no distinction is made between unexpected errors and e.g. anticipated validation failures.

Register new components as plug-ins without impact on testbed operations

Currently test components (messaging handlers and validators) are implemented within the source code of the testbed itself. This means that adding new components requires a new build and deployment of the testbed as a whole. Test components should ideally be packaged as plugins that can be implemented and deployed in isolation, and be usable by the testbed without restarts.

Allow exporting of test artefacts

The upcoming Test Registry and Repository is foreseen to allow the discovery and export of all types of test artefacts. In the current implementation, test artefacts can be retrieved from the artefact repository in use through a simple REST API. This is however currently meant for internal purposes meaning that there is no support in discovering artefacts or in constructing the queries needed to fetch them. In addition, it is currently not possible to export a complete test suite in this manner.

It would be beneficial if through the UI it was possible to allow the export of test suite archives and possibly individual test cases.

Skip Maven test

Problem is: Test tries to connect localhost:9000 but at first run it's obvious that nothing is there.

Consequence: build fails...

Solution is to skip test:
Please modify the instruction to execute: mvn clean install -Dmaven.test.skip=true