Giter Site home page Giter Site logo

quarkus-test-framework's People

Contributors

actions-user avatar afalhambra avatar dale-fu avatar dependabot[bot] avatar fedinskiy avatar gtroitsk avatar jcarranzan avatar jedla97 avatar jirisimek98 avatar jsmrcka avatar kshpak avatar llowinge avatar michalvavrik avatar mjurc avatar mocenas avatar quarkusqe avatar rsvoboda avatar sgitario avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

sgitario pjgg mjurc jsmrcka kshpak fabiobrz fedinskiy llowinge osmman rsvoboda mmuzikar lukasinko penehyba davided dale-fu michalvavrik caolf jirisimek98 afalhambra jcarranzan gtroitsk linartova mocenas jedla97 karb101 worldsoldier

quarkus-test-framework's Issues

Dev Mode ++ OpenShift extension: Test execution fails when test module requires parent dependencies

When using DEV mode or other Quarkus OpenShift extensions to deploy the Quarkus application, the test framework is internally copying the Maven pom.xml of the example and running an inner Maven command.
The problem of this approach is when the pom.xml of the example uses a parent pom dependency that has not been previously installed.
There are different approaches to fix this issue:

  1. Document that the framework needs to install the parents beforehand: This is a really poor intuitive
  2. Obey to not have parent dependencies for examples: Again this is a poor limitation we should avoid
  3. Automatically install the parent poms before running the inner Maven command.

Refactor generation of scenario IDs

Scenario ID should be part of the ServiceContext.
Maybe, we should also add a ScenarioContext where to have the scenario ID and the test execution context (junit).

BaseService cannot return hostname without protocol

BaseService and its descendant classes, e.g. RestService, have method getHost, which returns string, which looks like "http://localhost". In most places our code either replaces "http" in this string with some other protocol(e.g. "jdbc") or glues it together with results of getPort method of the same class. Also, there are problems with API, which accepts only URLs or URIs, since we have to parse output of those methods.

I suggest to either:
a) Replace these two methods with method getURL (or getURI)
b) Change the method getHost, so it will return only a host.

Wait for logs makes the threads blocked

For example:

@Test
    public void shouldDetectNewTests() {
        // At first, there are no tests annotated with @QuarkusTest
        app.logs().assertContains("Tests paused");
        // Now, we enable continuous testing via DEV UI
        app.enableContinuousTesting();
        // We add a new test
        app.copyFile("src/test/resources/GreetingResourceTest.java.template", "src/test/java/GreetingResourceTest.java");
        // So good so far!
        app.logs().assertContains("All 1 test is passing"); // Sometimes, this makes the threads blocked and the test fails.
    }

Support of YAML configuration

At the moment, the quarkus test framework is highly tied to properties file in the form of:

a.b=value

However, when using a YAML properties file for Quarkus, it might have some side effects (mostly when the test framework triggers a build operator to create the Quarkus applications at each scenario)

Regression: copy properties files in the build deployments ++ fails on Native

At the moment, due to some instabilities, the properties files are not copied to the final service folder. This way the properties defined in the src/main/properties or src/test/properties do not interfere in the test execution. However, this is also a limitation and might cause confusion to users.

Where are we copying / filtering the files to copy? Here:

quarkus-test-framework/quarkus-test-core/src/main/java/io/quarkus/test/services/quarkus/QuarkusApplicationManagedResourceBuilder.java

Line 126 in b27c6d7

&& !path.toFile().getName().contains(PROPERTIES_SUFFIX))) {

If we don't filter the properties, the tests in https://github.com/quarkus-qe/quarkus-test-suite/tree/main/sql-db/sql-app will fail.
If we do filter the properties, the tests in https://github.com/quarkus-qe/quarkus-test-suite/blob/main/properties/src/test/java/io/quarkus/qe/properties/consul/ConsulConfigSourceIT.java will fail as the customsource.properties is missing. (I had to rename this file to customsource.prop to workaround this issue.

However, the above workaround does not work for Native, so this issue needs to be fixed.
Regression issue from 0.0.3 to 0.0.4.

Add coverage of the privileged mode feature in @Container

Related PR: #171

We need to cover this functionality in the test framework.
At the moment, the only known container that needs the privileged mode is db2, but I want to avoid using it as it needs up to 5 minutes to startup. Therefore, we should find another container that only works with privileged mode and add some basic coverage with the new property.

Improve failures in the Build for external applications

At the moment, when there are failures to build the external application in OpenShift, the framework still awaits for the application to be run.

Example:

10:17:54 08:17:54.987 INFO  oc: INFO Running 'mvn -s /configuration/settings.xml -Dquarkus.package.type=uber-jar -DskipTests=true -Dquarkus.version=999-SNAPSHOT -Dquarkus.platform.version=999-SNAPSHOT -Dquarkus.plugin.version=999-SNAPSHOT -Dquarkus-plugin.version=999-SNAPSHOT --batch-mode -Djava.net.preferIPv4Stack=true -s /tmp/artifacts/configuration/settings.xml -Dmaven.repo.local=/tmp/artifacts/m2  package'
10:17:56 08:17:56.053 INFO  oc: [INFO] Scanning for projects...
10:17:56 08:17:56.298 INFO  oc: [INFO] Downloading from redhat: https://maven.repository.redhat.com/ga/io/quarkus/quarkus-universe-bom/999-SNAPSHOT/maven-metadata.xml
10:17:56 08:17:56.985 INFO  oc: [INFO] Downloading from redhat: https://maven.repository.redhat.com/ga/io/quarkus/quarkus-universe-bom/999-SNAPSHOT/quarkus-universe-bom-999-SNAPSHOT.pom
10:17:57 08:17:57.907 INFO  oc: [ERROR] [ERROR] Some problems were encountered while processing the POMs:
10:17:57 08:17:57.907 INFO  oc: [ERROR] Non-resolvable import POM: Could not find artifact io.quarkus:quarkus-universe-bom:pom:999-SNAPSHOT in redhat (https://maven.repository.redhat.com/ga/) @ line 27, column 25
10:17:57 08:17:57.907 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-resteasy:jar is missing. @ line 38, column 21
10:17:57 08:17:57.907 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-openapi:jar is missing. @ line 42, column 21
10:17:57 08:17:57.907 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-health:jar is missing. @ line 46, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-metrics:jar is missing. @ line 50, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-graphql:jar is missing. @ line 54, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-hibernate-validator:jar is missing. @ line 58, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-jdbc-h2:jar is missing. @ line 62, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-hibernate-orm-panache:jar is missing. @ line 66, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-resteasy-jsonb:jar is missing. @ line 70, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.quarkus:quarkus-junit5:jar is missing. @ line 76, column 21
10:17:57 08:17:57.908 INFO  oc: [ERROR] 'dependencies.dependency.version' for io.rest-assured:rest-assured:jar is missing. @ line 81, column 21
10:17:57 08:17:57.908 INFO  oc:  @ 
10:17:57 08:17:57.908 INFO  oc: [ERROR] The build could not read 1 project -> [Help 1]
10:17:57 08:17:57.908 INFO  oc: [ERROR]   
10:17:57 08:17:57.908 INFO  oc: [ERROR]   The project io.quarkus.sample:todo-backend:1.0-SNAPSHOT (/tmp/src/pom.xml) has 12 errors
10:17:57 08:17:57.911 INFO  oc: [ERROR]     Non-resolvable import POM: Could not find artifact io.quarkus:quarkus-universe-bom:pom:999-SNAPSHOT in redhat (https://maven.repository.redhat.com/ga/) @ line 27, column 25 -> [Help 2]
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-resteasy:jar is missing. @ line 38, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-openapi:jar is missing. @ line 42, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-health:jar is missing. @ line 46, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-metrics:jar is missing. @ line 50, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-smallrye-graphql:jar is missing. @ line 54, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-hibernate-validator:jar is missing. @ line 58, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-jdbc-h2:jar is missing. @ line 62, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-hibernate-orm-panache:jar is missing. @ line 66, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-resteasy-jsonb:jar is missing. @ line 70, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.quarkus:quarkus-junit5:jar is missing. @ line 76, column 21
10:17:57 08:17:57.912 INFO  oc: [ERROR]     'dependencies.dependency.version' for io.rest-assured:rest-assured:jar is missing. @ line 81, column 21
10:17:57 08:17:57.913 INFO  oc: [ERROR] 
10:17:57 08:17:57.913 INFO  oc: [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
10:17:57 08:17:57.913 INFO  oc: [ERROR] Re-run Maven using the -X switch to enable full debug logging.
10:17:57 08:17:57.913 INFO  oc: [ERROR] 
10:17:57 08:17:57.913 INFO  oc: [ERROR] For more information about the errors and possible solutions, please read the following articles:
10:17:57 08:17:57.913 INFO  oc: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
10:17:57 08:17:57.913 INFO  oc: [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
10:17:58 08:17:58.204 INFO  oc: error: build error: error building at STEP "RUN /usr/local/s2i/assemble": error while running runtime: exit status 1
10:17:59 08:17:59.030 INFO  Running command: oc scale dc/app --replicas=1 -n ts-mnjvwwcwmi
10:17:59 08:17:59.212 INFO  oc: W0802 08:17:59.212383   77180 warnings.go:70] extensions/v1beta1 Scale is deprecated in v1.2+, unavailable in v1.16+
10:17:59 08:17:59.214 INFO  oc: deploymentconfig.apps.openshift.io/app scaled
10:37:59 08:37:59.465 WARNING [app] Service didn't start in PT20M minutes
10:37:59 08:37:59.473 INFO  Running command: oc delete project ts-mnjvwwcwmi
10:37:59 08:37:59.770 INFO  oc: project.project.openshift.io "ts-mnjvwwcwmi" deleted
10:37:59 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,229.708 s <<< FAILURE! - in io.quarkus.ts.external.applications.OpenShiftTodoDemoIT
10:37:59 [ERROR] io.quarkus.ts.external.applications.OpenShiftTodoDemoIT  Time elapsed: 1,229.708 s  <<< ERROR!
10:37:59 org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.bootstrap.BaseService was not fulfilled within 20 minutes.
10:37:59 	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:166)
10:37:59 	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
10:37:59 	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
10:37:59 	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:939)
10:37:59 	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:908)
10:37:59 	at io.quarkus.test.utils.AwaitilityUtils.untilIsTrue(AwaitilityUtils.java:52)
10:37:59 	at io.quarkus.test.bootstrap.BaseService.waitUntilServiceIsStarted(BaseService.java:210)
10:37:59 	at io.quarkus.test.bootstrap.BaseService.start(BaseService.java:148)
10:37:59 	at io.quarkus.test.bootstrap.RestService.start(RestService.java:20)
10:37:59 	at io.quarkus.test.bootstrap.QuarkusScenarioBootstrap.launchService(QuarkusScenarioBootstrap.java:134)
10:37:59 	at io.quarkus.test.bootstrap.QuarkusScenarioBootstrap.lambda$beforeAll$2(QuarkusScenarioBootstrap.java:70)
10:37:59 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

Expected behaviour: the framework should not wait up to 20 minutes, but quickly fail if the Build job failed.

Service for SQL server has wrong settings

  1. Database is "mydb" instead of "msdb"
  2. JDBC string is created incorrectly("databaseName=" is omitted)
  3. Properties are not used(due to the lack of onPreStart method overload)

Support computed properties in @Containers

The idea is to have the ability to get image and versions from system properties:

@KafkaContainer(vendor = KafkaVendor.STRIMZI,
            image = "${amq-streams.image}",
            version = "${amq-streams.version}",
            withRegistry = true)
static KafkaService kafka = new KafkaService();

Moreover, we can define default properties if the system property amq-streams.image is not found:

@KafkaContainer(vendor = KafkaVendor.STRIMZI,
            image = "${amq-streams.image:DEFAULT_IMAGE}",
            version = "${amq-streams.version:DEFAULT_VERSION}",
            withRegistry = true)
    static KafkaService kafka = new KafkaService();

Support Spring Boot deployments

Iteration 1: Support only Spring Boot on baremetal. Effort: Medium

1.- Create new module test-core
2.- Move existing sources from module quarkus-test-core to test-core:

  • QuarkusScenarioBootstrap: It should be renamed to ScenarioBootstrap
  • QuarkusScenario: It should be renamed to Scenario
  • All the rest of dependant interfaces and common utilities

3.- (Optional) Create new sources in quarkus-test-core for backward compatibilities purposes:

  • Create back the QuarkusScenario annotation and marked it as deprecated

4.- Create new module spring-test-core:

And https://github.com/spring-projects/spring-framework/blob/main/spring-test/src/main/java/org/springframework/test/context/junit/jupiter/SpringExtension.java

  1. Update examples

Iteration 2: Support Spring Boot on Native. Effort: Easy

According to https://docs.spring.io/spring-native/docs/current/reference/htmlsingle/, it should be easy to reuse the same artifacts from the Maven build and run the application.

Iteration 3: Support Spring Boot on OpenShift. Effort: Easy

Using the container registry strategy, it should be easy.

Iteration 4: Support Spring Boot on Kubernetes. Effort: Easy

S2i: Infer Quarkus version from the system property instead of from the model

At the moment, for S2i the Quarkus version needs to be explicitly set in the annotation:

@QuarkusApplication(gitRepositoryUri = "https://github.com/quarkusio/quarkus-quickstarts.git", contextDir = "getting-started", quarkusBuildVersion = "xxx")    static final RestService app = new RestService();

This is misleading as it should be taken from the version from the Maven configuration (system property).
Moreover, the annotation is used for others deployment strategies, so it potentially confuses the usage.

Documentation: add a section for commands on how to build the framework/run examples to the README

Hi, I noticed there's no mention in the README regarding how to build the framework and e.g. running examples.
Of course this information can be derived somehow from the POM file and by reading the holy sources but it would be great to have a small paragraph in the documentation so we can improve UX and first impression.

FYI @Sgitario
CC @rsvoboda (paying my ticket for the presentation by submitting this issue, ok? :) )

OpenShiftDeploymentStrategy.UsingOpenShiftExtension doesn't copy all maven arguments

When running a testsuite with additional maven arguments the openshift extension deployment strategy should use these arguments for the deployment. Currently it looks like some arguments get used, such as quarkus.platform.version, quarkus.platform.artifact-id and quarkus.platform.group-id. However when using properties such as -Dversion.camel.quarkus=2.2.0 the argument is not used in the subsequent mvn builds.

Nor does camel-quarkus-bom.version from https://github.com/quarkus-qe/quarkus-test-suite/blob/main/pom.xml#L33 get used

A "cherry-pick" quarkus application is failing with kafka "auto-serializers" enabled

Given a Quarkus application as the follow one:

@QuarkusApplication //(classes = { SlowTopicConsumer.class, SlowTopicResource.class })
static RestService app = new RestService()
            .withProperty("kafka.bootstrap.servers", kafka::getBootstrapUrl)
            .withProperty("quarkus.kafka-streams.bootstrap-servers", kafka::getBootstrapUrl)

Fails with message:

 Unable to create the publisher or subscriber during initialization: java.lang.IllegalArgumentException: The attribute `value.deserializer` on connector 'smallrye-kafka' (channel: login-denied) must be set

Reproducer: https://github.com/quarkus-qe/quarkus-test-suite/blob/main/messaging/kafka-streams-reactive-messaging/src/test/java/io/quarkus/ts/messaging/kafka/KafkaGratefulShutdownIT.java#L36

Note: in the above reproducer you need to remove the serializer from RestService app

Print what service / app is executed when `[app] Initialize service` is printed

Print what service / app is executed when [app] Initialize service is printed

I would at least appreciate information if the service is going to be executed in JVM / DEV / NATIVE mode

For docker based services I would like to see the image name if possible.

[app] Initialize service (Quarkus JVM mode)
[app] Initialize service (Quarkus DEV mode)
[app] Initialize service (Quarkus NATIVE mode)
[keycloak] Initialize service (jboss/keycloak:15.0.0)
etc.

Detect new tests scenario does not work on Windows

The following scenario does not work on Windows

@Test
@Order(1)
public void shouldDetectNewTests() {
    // At first, there are no tests annotated with @QuarkusTest
    app.logs().assertContains("Tests paused");
    // Now, we enable continuous testing via DEV UI
    app.enableContinuousTesting();
    // We add a new test
    app.copyFile("src/test/resources/GreetingResourceTest.java.template",
            "src/test/java/GreetingResourceTest.java");
    // So good so far!
    app.logs().assertContains("All 1 test is passing"); // fails here.
}
```

Link: https://github.com/quarkus-qe/quarkus-test-framework/blob/8e1175f1e544b188d166384ab0313d734cf98d7a/examples/greetings/src/test/java/io/quarkus/qe/DevModeGreetingResourceIT.java#L34

The output is:

```
08:01:16.541 INFO  [app] 08:01:12,863 Thread Thread[vert.x-eventloop-thread-4,5,build group] has been blocked for 2661 ms, time limit is 2000 ms
08:01:16.542 INFO  [app] Running tests for the first time
08:01:16.542 INFO  [app] Press [h] for more options>
08:01:16.542 INFO  [app] Press [r] to re-run, [h] for more options>
08:01:16.542 INFO  [app] No tests found
```

See the warning message: `has been blocked for 2661 ms, time limit is 2000 ms`

I tried to increase the time limit using:

```
quarkus.vertx.max-worker-execute-time=11s
quarkus.vertx.max-event-loop-execute-time=12s
quarkus.vertx.warning-exception-time=13s
```

But the time limit was not increased.

However, doing the same locally worked fine, that's why I didn't report this issue in `quarkusio/quarkus`, but here, for further investigation.

S2i: Native support of Quarkus Application from Git repositories

Having:

@OpenShiftScenario
public class OpenShiftQuickstartUsingS2iIT {
    @GitRepositoryQuarkusApplication(repo = "https://github.com/quarkusio/quarkus-quickstarts.git", contextDir = "getting-started")
    static final RestService app = new RestService();

It does not work for Native builds.

Configuration: Allow to setup default factor timeouts

Allow configuring timeouts for services. For example, in order to configure startup service timeout, we could use:

ts.<SERVICE_NAME>.timeout.startup=10 # 10 minutes

The default for startup is 5 minutes.

Moreover, let's allow to have some factor times to adapt our test suite to slower environments:

ts.global.timeout.factor=2.5 # Default 1.

This value will be used to multiply the timeouts (timeout x factor = total timeout). Example: if we define a timeout of 5 minutes with a factor of 2.5, the total timeout will be 5 x 2.5 = 12.5 minutes.

The factor should be suitable to be supplied from system properties.

At the moment, let's support only the startup timeout.

Logs are not printed out when using jboss logmanager

When using jboss log manager and it's properly configured in either failsafe or surefire, for example:

<plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <systemPropertyVariables>
                        <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                    </systemPropertyVariables>
                </configuration>
            </plugin>

Then, the Quarkus application logs are not printed.
The logging configuration is setup here, so something should be fixed around.

Support for Quarkus deployment with odo

Workflow:

  • deploy the test application using odo push

Stuff to figure out:

  • How to push and use custom Maven settings and repository with the odo image?

What needs to be done?

  • implement new simple Java wrapper for odo
  • use the wrapper to manage new implementation of ManagedResource for odo
  • update @*scenario annotations

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.