gradle/exemplar

Discover and verify code samples and services

exemplar's Issues

Version 0.8.0 broken when using multi-step config with `execution-subdirectory`

Expected Behavior

No broken.

Current Behavior

The worker directory keeps getting appended throughout each steps: https://github.com/gradle/exemplar/blob/master/sample-check/src/main/java/org/gradle/samples/test/runner/SamplesRunner.java#L155-L157

Context

Steps to Reproduce (for bugs)

Execute pretty much any gradle/gradle sample with version 0.8.0 and you will get an exception of cannot find directory like .../groovy/kotlin.

Your Environment

• Build scan URL:

JUnit runners don't handle @Before nor @After

Using @RunWith(GradleSamplesRunner::class) or @RunWith(SamplesRunner::class)

Expected Behavior

@Before and @After functions to be invoked properly

Current Behavior

@Before and @After functions silently ignored

Context

I wanted to skip some samples under certain conditions in a @Before function using JUnit Assume. I didn't find a satisfying way so far.

@BeforeClass and @AfterClass are already handled by the parent ParentRunner.

Prevent duplication of gradle command between docs and sample config

There is no longer a way for the user manual to get the command (e.g. gradle help --quiet) to be run for each sample, and it will be duplicated.

Possible approaches:

• Use an Asciidoctor extension that reads a given sample config file and extracts the command
• Expose the command through a yet-unwritten sample-expo library that reports on available samples

Relevant docs:

• https://asciidoctor.org/docs/asciidoctorj/#extension-api
• Failed attempt at IncludeProcessor — This would try to include the sample command and output by reading from Classpath resources, however, because the asciidoctor extension was run within a different classloader, addressing these resources is difficult/impossible.

Allow repository mirrors in GradleSamplesRunner

During the course of solving https://github.com/gradle/gradle-private/issues/1375, we want to tweak the samples before execution, i.e. the sample seen by users is:

repositories {
     jcenter()
}

But the sample actually executed by GradleSamplesRunner is:

repositories {
     maven {
             <our mirror of jcenter>
     }
}

Can we have some hooks in GradleSamplesRunner lifecycle?

Provide an API for listing samples

It would be useful for other tools to be able to report discovered samples and their metadata (URL, title, etc)

Expected Behavior

sample-discovery should provide an API that lists samples and their metadata.

Current Behavior

No such API exists

Context

A couple Gradle projects have requested the ability to have indexed samples. Listing samples in a report is the first step toward this.

Allow normalization on both sides

Expected Behavior

It should be possible to have expected output normalization, just like we have actual output normalization.

Current Behavior

Currently, only the actual output is normalized.

Context

In dependency management, we often have quite large outputs of dependency graphs, and often from one version to another, we have insignificant chances like number of spaces, due to the fact that this attribute has been renamed or whatever. So often tests are failing because this number of spaces in the expected output has changed (or additional blank lines, or whatever). It would be nice if we could just apply normalization on both sides, so that when we "search and replace" in a sample, we don't have to bother aligning text ourselves.

The idea would be that if norm(output)==norm(expected output), then we're good, and not only if norm(output)==expected manually normalized output.

GradleOutputNormalizer doesn't support build time like `508ms`

Expected Behavior

Support build time with multiple time unit chars.

Current Behavior

A line like BUILD SUCCESSFUL in 574ms is not normalized because the pattern doesn't match ms. See https://github.com/gradle/exemplar/blob/master/sample-check/src/test/java/org/gradle/samples/test/normalizer/GradleOutputNormalizer.java#L28.

Context

Steps to Reproduce (for bugs)

Your Environment

• Build scan URL:

Allow specifying custom Gradle version/distro/install

For testing purposes it would be useful to be able to use a custom Gradle version, distribution or installation.

The Gradle version or path to distro/install could be set as a test annotation or in the HOCON configuration file leveraging environment variable substitution or including a generated HOCON file.

Release GradleOutputNormalizer as part `sample-check`

Expected Behavior

Users can use GradleOutputNormalizer (https://github.com/gradle/exemplar/blob/master/sample-check/src/test/java/org/gradle/samples/test/normalizer/GradleOutputNormalizer.java) within their tests as shown in the README.

Current Behavior

The GradleOutputNormalizer class is inside the test source set and not released with the sample-check project.

Context

Steps to Reproduce (for bugs)

Your Environment

• Build scan URL:

Allow IDE autocomplete for Exemplar config files

Expected Behavior

As a user, I want to more easily author my sample.conf files by leveraging autocomplete

Current Behavior

No completion available

Context

It's not obvious what properties and values are available. Better to use something like JSON Schema Store or a similar mechanism to allow completion through IDEs. Though there are not very many Exemplar users yet, all of them are likely to want this feature.

Setting the temporary folder for Sample doesn't work

Expected Behavior

A temporary folder can be provided to Sample as root directory.

Current Behavior

The temporary isn't initialized yet when the Rule for the Sample kicks in. You will receive the following exception:

java.lang.IllegalStateException: the temporary folder has not yet been created
	at org.junit.rules.TemporaryFolder.getRoot(TemporaryFolder.java:145)
	at org.junit.rules.TemporaryFolder.newFolder(TemporaryFolder.java:94)
	at org.junit.rules.TemporaryFolder.newFolder(TemporaryFolder.java:86)
	at org.gradle.samples.test.rule.Sample$2.getDir(Sample.java:74)
	at org.gradle.samples.test.rule.Sample.computeSampleDir(Sample.java:165)
	at org.gradle.samples.test.rule.Sample.getDir(Sample.java:158)
	at org.gradle.samples.test.rule.Sample$3.evaluate(Sample.java:138)
	at org.spockframework.runtime.extension.builtin.TestRuleInterceptor.intercept(TestRuleInterceptor.java:39)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:97)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:748)

Context

Setting a temporary folder with any of the into methods.

@Rule
TemporaryFolder temporaryFolder = new TemporaryFolder()

@Rule
Sample sample = Sample.from("src/docs/samples").into(temporaryFolder)

Steps to Reproduce (for bugs)

See example above.

Replace 'allow-disordered-output' with a normalizer that reorders task output groups

"allow-disordered-output" is a very blunt tool to verify the output of a build that does stuff in parallel (which will more and more be the default) and was really only used because the plain console did not group output. Now that it does, a better strategy might be to use a normalizer that reorders task output groups but does not reorder the text within a group.

For example, reorder the groups alphabetically:

Normalize:

> Task :b
some output

> Task :a
other output

To:

> Task :a
other output

> Task :b:
some output

The configuration file should provide a way to define a human-readable test name

Expected Behavior

The sample.conf defines a key for defining a test name. For example: description: "plugin should create the task named x". Upon test execution, the test name uses the description instead of auto-generated description.

Current Behavior

The test name is derived of the directory hierarchy and the prefix of the sample.conf file. The test name is hard to read and doesn't really express its intent.

Context

Auto-discovering tests with one of the Runner implementations produces cryptic test names.

Discover and verify samples embedded in Markdown

remark seems like it may be capable of extracting what we need from Markdown documents and executing samples.

Given that markdown is more widely used than Asciidoctor, this may greatly increase the possible user base of this library.

Document @Sample and @UsesSample in README

Allow optional title, description, and tags in sample config

This gives samples metadata for indexing purposes, should we want to expand this project to foster discoverability of samples.

Sample-check brings in slf4j-simple dependency

https://github.com/gradle/exemplar/blob/master/sample-check/build.gradle.kts#L15-L17

The problem with this is that it collides with Gradle's own SLF4J implementation, producing the following output whenever an integration test is executed:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [file:/Users/lptr/Workspace/gradle/gradle/out/production/logging/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lptr/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.16/f0cacc3d21e1027c82006a7a6f6cf2cad86d2a4f/slf4j-simple-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext]

I guess it should be up to the user of this library to decide what SLF4J bindings they want to use.

Feature: Gradle Plugin samples runner

As a Gradle plugin author, I'd like to inject the Gradle plugin under test into the samples test.

Expected Behavior

Provide a GradlePluginSamplesRunner that configures withPluginClasspath() on TestKit.

Current Behavior

Context

There's now 2 plugins that want this. Be cool to provide this.

Make sample check configurable to use repository mirrors

When testing samples, there is no mirroring of repositories possible AFAICT. This is causing noise in CI due to the Bintray availability issues.
sample-check should be enhanced so that it can run with an option similar to GradleExecuter.withRepositoryMirrors() to reduce flakiness in execution.

Improve allow-additional-output to allow output before and not only after

Preserve sample directory name when running checks

Currently the temp directory used when running sample-check has a different name. This cases Gradle project names to differ when one is not explicitly set in settings.gradle. This should be preserved.

samples-check should not export Groovy dependency

IIUC, Groovy is only used to test the project, so there should be no production dependency on Groovy at all.

Allow configuring the target name for a sample directory

After #6 is merged, there is still one use-case missing for replacing the Sample Rule in gradle/gradle: The Sample rule allows configuring testSampleDirName, the target directory name for the sample.

I tried removing that functionality in the gradle/gradle code base, but I ran into long path issues on Windows: gradle/gradle#5713.

So there should be a possibility to configure the target directory where a sample is copied to, either by specifying a fixed relative path to the target base dir or a target directory directly.

I think using a target directory supplied as a File would suffice. This field could then be wired in the method computeSampleDir.

Publish library to JCenter

At the moment you have to define an additional repository just for this dependency.

repositories {
    jcenter()
    maven {
        url = uri("https://repo.gradle.org/gradle/libs")
    }
}

The dependency should be available in JCenter.

Use exemplar to test the samples in its own documentation

Exemplar should be able to test its own samples. For example, there are some samples embedded in the README that should be discoverable and testable.

Allow user to register executor for given executable name

From #22 (comment):

Allow exemplar/user to register some extension if one wants to customize how they run "gradle" or "groovy" or "bash". For example, they could run "ruby" with specific RVM config.

Publish to Maven Central

Context

We should resolve this library from Maven Central so consumers do not have to access repo.gradle.org. This will be faster and more stable.

@marcphilipp has recommended that we use the group org.gradle.exemplar. I agree.

Allow samples to configure environment variables

This used to be a part of this project, but was removed because TestKit doesn't support it.

We should allow this at least for CLI Samples, as there are other samples that need this.