dlang/ci

dlang CI testing pipelines

ci's Issues

webhook signature validation spuriously fails

For quite a few PRs github shows that it got a 400 response with a signature mismatch.
Provided signature [e6a7a2d4f9d5c65873795035a9bfcda0b90c14d5] did not match to calculated

Fatal: bad object -> show at least a "waiting for job executors" message"

First time build. Skipping changelog.
fatal: bad object fdc308de96c15cf1a93d94970968b0b0a5ad968c

Seen e.g. here: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fci/detail/PR-166/1/pipeline

Otherwise people don't know what's going on.

Restart individual nodes in the pipeline

#36 works great for what it's intended to do, but I think over 90% of the spurious failures are due to only dub or vibe.d, so an option restart a leaf node (basically a third-party project a.t.m.) would be very handy.

For example, I had to restart Jenkins because of dub here:
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-5648/3/pipeline/173
The whole run lasted 09:15 min, while the dub node took about 1:30 mins, meaning this option could save around 7-8 mins.

Avoid period rebuild of all open PRs

My guess is that this happens because Build fork PRs (merged with base branch) is selected and thus Jenkins considers the PR builds stale when master changed (similar to auto-tester atm.).
As initially decided we don't to be pedantically correct and avoid the quadratic retesting behavior.

Seems like that leaves us with 2 choices, do the merge ourselves (and ask Jenkins to only build PR heads) or disable the periodic organization scans.
Not clear if env.CHANGE_TARGET still get's set with disabled merges, but looks OKish https://github.com/jenkinsci/branch-api-plugin/blob/a5e6362d03fdf31f6d7d7905885b38b67464a532/src/main/java/jenkins/branch/BranchNameContributor.java#L63.

setup server

Executors should be sandboxed away from Jenkins.
Should be scalable to more executors and VMs for Win/OSX.
At best maintain (cleanup up) working dirs to save on download/clone time and reduce connectivity issues.

Already have a decent machine (Dedibox® LT 2016 - Online.net) to start with.

dead lock with parallel PRs

Just happened b/c Jenkins scheduled all open druntime PRs after merging the Jenkinsfile into master.
Turns out that multiple jenkins jobs can starve waiting for an executor, taking up all executor slots.
Any node step requires to allocate an executor.

Reconsider usage of stash vs artifacts upon integration of release scripts

allunittests: ocean.task.util.Timer_test: testing ...Unhandled exception in epoll/scheduler

allunittests: ocean.task.util.Timer_test: testing ...Unhandled exception in epoll/scheduler
	unit test has failed (./src/ocean/task/util/Timer_test.d:124)
submodules/makd/Makd.mak:577: recipe for target '/var/lib/jenkins/dlang_projects/sociomantic-tsunami/ocean/build/production/tmp/allunittests.stamp' failed
make: *** [/var/lib/jenkins/dlang_projects/sociomantic-tsunami/ocean/build/production/tmp/allunittests.stamp] Aborted (core dumped)

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2083/17/pipeline

Cancel builds of updated branches / PRs

There was some option on the old project-tester ci.dawg.eu, maybe in GitHub pull request builder plugin - Jenkins - Jenkins Wiki.

Checklist for enabling the CI on dlang organization

• Move this repository to new "dlang/ci" repository
• Adjust URLs to refer to "real" endpoints
• Add Jenkinsfiles with trigger to all main dlang repos

Jenkins fails to clone/setup druntime

https://ci.dlang.io/job/dlang-org/job/tools/job/PR-302/1/console

[dmd] Running shell script
+ make -f posix.mak AUTO_BOOTSTRAP=1 --jobs=4
make -C src -f posix.mak all
make[1]: Entering directory '/var/lib/jenkins/dlang_ci/dmd/src'
no cpu specified, assuming X86
make[1]: /tmp/.host_dmd-2.072.2/dmd2/linux/bin64/dmd: Command not found
c++ -Wno-deprecated -Wstrict-aliasing -Werror -fno-exceptions -fno-rtti -D__pascal= -DMARS=1 -DTARGET_LINUX=1 -DDM_TARGET_CPU_X86=1 -m64 -fPIC -std=gnu++98 -Idmd/tk dmd/backend/optabgen.c -o ../generated/linux/release/64/optabgen
  (CC)  ROOT_OBJS  dmd/root/newdelete.c
c++ -c -o../generated/linux/release/64/newdelete.o -Wno-deprecated -Wstrict-aliasing -Werror -fno-exceptions -fno-rtti -D__pascal= -DMARS=1 -DTARGET_LINUX=1 -DDM_TARGET_CPU_X86=1 -m64 -fPIC -std=gnu++98 -Idmd/root -MMD -MF ../generated/linux/release/64/newdelete.deps dmd/root/newdelete.c
CC="c++" /tmp/.host_dmd-2.072.2/dmd2/linux/bin64/dmd -conf=/tmp/.host_dmd-2.072.2/dmd2/linux/bin64/dmd.conf -lib -of../generated/linux/release/64/lexer.a -m64 -J../generated/linux/release/64 -L-lstdc++ -version=MARS -fPIC -w -de dmd/console.d dmd/entity.d dmd/errors.d dmd/globals.d dmd/id.d dmd/identifier.d dmd/lexer.d dmd/tokens.d dmd/utf.d dmd/root/array.d dmd/root/ctfloat.d dmd/root/file.d dmd/root/filename.d dmd/root/outbuffer.d dmd/root/port.d dmd/root/rmem.d dmd/root/rootobject.d dmd/root/stringtable.d dmd/root/hash.d
echo "$DEFAULT_DMD_CONF" > ../generated/linux/release/64/dmd.conf
Error: cannot find source code for runtime library file 'object.d'
       dmd might not be correctly installed. Run 'dmd -man' for installation instructions.
       config file: /tmp/.host_dmd-2.072.2/dmd2/linux/bin64/dmd.conf
import path[0] = /tmp/.host_dmd-2.072.2/dmd2/linux/bin64/../../src/phobos
import path[1] = /tmp/.host_dmd-2.072.2/dmd2/linux/bin64/../../src/druntime/import
posix.mak:447: recipe for target '../generated/linux/release/64/lexer.a' failed
make[1]: *** [../generated/linux/release/64/lexer.a] Error 1
make[1]: *** Waiting for unfinished jobs....
../generated/linux/release/64/optabgen
OPTABGEN... generating files
mv debtab.c optab.c cdxxx.c elxxx.c fltables.c tytab.c ../generated/linux/release/64
make[1]: Leaving directory '/var/lib/jenkins/dlang_ci/dmd/src'
posix.mak:8: recipe for target 'all' failed
make: *** [all] Error 2

Also this while probably not related, this

[dmd] First time build. Skipping changelog.
[dmd] fatal: bad object fdc308de96c15cf1a93d94970968b0b0a5ad968c

because it looks like druntime was properly cloned:

[druntime]  > git rev-parse --is-inside-work-tree # timeout=10
[druntime] Fetching changes from the remote Git repository
[druntime]  > git config remote.origin.url https://github.com/dlang/druntime.git # timeout=10
[druntime] Cleaning workspace
[druntime]  > git rev-parse --verify HEAD # timeout=10
[druntime] Resetting working tree
[druntime]  > git reset --hard # timeout=10

Always checks out upstream/master (instead of the PR branch)

tl;dr: dlang/dmd#7688 should have shown an CI error.
It's only displayed on the builds now

src/zmqd.d(462,40): Error: `data.ptr` cannot be used in `@safe` code, use `&data[0]` instead
src/zmqd.d(477,40): Error: `data.ptr` cannot be used in `@safe` code, use `&data[0]` instead
src/zmqd.d(567,46): Error: `data.ptr` cannot be used in `@safe` code, use `&data[0]` instead
src/zmqd.d(582,46): Error: `data.ptr` cannot be used in `@safe` code, use `&data[0]` instead
src/zmqd.d(624,52): Error: `data.ptr` cannot be used in `@safe` code, use `&data[0]` instead
src/zmqd.d(636,52): Error: `data.ptr` cannot be used in `@safe` code, use `&data[0]` instead
src/zmqd.d(1662,54): Error: `value.ptr` cannot be used in `@safe` code, use `&value[0]` instead
src/zmqd.d(840,66): Error: template instance zmqd.Socket.setArrayOption!() error instantiating
src/zmqd.d(1892,9): Error: `items.ptr` cannot be used in `@safe` code, use `&items[0]` instead

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2018/7/pipeline

On dlang/dmd#7688 the log is:

> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/dlang/dmd.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Fetching upstream changes from https://github.com/dlang/dmd.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/dlang/dmd.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b1f6aec796286a08395d689b54dacdd01d141f9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b1f6aec796286a08395d689b54dacdd01d141f9e
Commit message: "Merge pull request #7695 from rainers/default_platform_toolset"
 > git rev-list --no-walk b1f6aec796286a08395d689b54dacdd01d141f9e # timeout=10

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7688/8/

Updates sometimes don't propogate

PR: dlang/dmd#7553

was fixed in #108 and even though it was repeatedly restarted afterwards, e.g.

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7553/7/pipeline/183

It still showed;

Checking out Revision 2427af70404bbc8878da7295d0e662a458a6a93e (v4.0.0-alpha.2)

Found by @JinShil

PR build history (jobs) vanish after merge

Clicking on details of dlang-test/dmd#4 leads to a no longer existing PR-4 page.

Tried to disable "Discard old items" in the configuration, let's see whether that fixes it.

seems like the unified workspace `ws: 'dlangci'` is no longer working

Might be a bug due to a plugin update :/.

Vibe.d: linker errors

+ cd tests/vibe.web.rest.auth
+ dub --compiler=dmd --override-config=vibe-d:core/libevent --build-mode=separate
Performing "debug" build using dmd for x86_64.
diet-ng 1.4.3: target for configuration "library" is up to date.
stdx-allocator 2.77.0: target for configuration "library" is up to date.
vibe-d:utils 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:data 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:core 0.8.3-alpha.1: target for configuration "libevent" is up to date.
vibe-d:crypto 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:stream 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:textfilter 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:inet 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:tls 0.8.3-alpha.1: target for configuration "openssl" is up to date.
vibe-d:http 0.8.3-alpha.1: target for configuration "library" is up to date.
vibe-d:web 0.8.3-alpha.1: target for configuration "library" is up to date.
tests ~master: building configuration "application"...
Non-@safe methods are deprecated in REST interfaces - Mark Service.getPublic as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getAny as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getAnyA as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe role evaluator methods are deprecated - annotate Service.getAdmin() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getAdmin as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe role evaluator methods are deprecated - annotate Service.getAdminA() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getAdminA as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe role evaluator methods are deprecated - annotate Service.getMember() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getMember as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe role evaluator methods are deprecated - annotate Service.getAdminMember() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getAdminMember as @safe.
Non-@safe .authenticate() methods are deprecated - annotate Service.authenticate() with @safe or @trusted.
Non-@safe role evaluator methods are deprecated - annotate Service.getNobody() with @safe or @trusted.
Non-@safe methods are deprecated in REST interfaces - Mark Service.getNobody as @safe.
Linking...
collect2: error: ld returned 1 exit status
Error: linker exited with status 1

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7420/22/pipeline

Incorrectly linked PRs/jobs?

Looking at the Jenkins failure at dlang/druntime#1810 PR. It brings me to the page https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-1810/3/changes
where commit (Commit: 751aa09 on Jenkins UI,dlang/druntime@751aa09) but the content of the job looks very different.

Test building DMD with itself?

Might be a good idea to build DMD with itself to ensure it can build itself fine. Can't seem to build a valid release version with the latest DMD.

Vibe.d: failed to connect to host

object.Exception@../../core/vibe/core/drivers/libevent2.d(365): Failed to connect to host 79.140.41.234:80: 7F63A166E62C
Program exited with code 1
script returned exit code 2

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7723/2/pipeline

ERROR: Issue with creating launcher for agent runner-01-gce. The agent has not been fully initialized yet

ERROR: Issue with creating launcher for agent runner-01-gce. The agent has not been fully initialized yet
ERROR: Issue with creating launcher for agent runner-01-gce. The agent has not been fully initialized yet
remote file operation failed: /var/lib/jenkins/dlang_projects@5 at hudson.remoting.Channel@1d0a274f:JNLP4-connect connection from 16.251.189.35.bc.googleusercontent.com/35.189.251.16:44424: hudson.remoting.ChannelClosedException: Remote call on JNLP4-connect connection from 16.251.189.35.bc.googleusercontent.com/35.189.251.16:44424 failed. The channel is closing down or has closed down

From: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6139/5/pipeline/219

Integrate coverage testing

Generate Coverage report

This should nearly at the end as it messes up the build repositories. It should be something similar to:

stage ('Test coverage') {
   def repo_name = /python3 -c "import sys, json; print(json.load(sys.stdin)['repository']['name']).toupper()"/
   switch (repo_name) {
      case 'dmd':
          sh 'mv src/dmd src/host_dmd'
          sh 'make -j$N -C dmd/src -f posix.mak MODEL=$MODEL HOST_DMD=./host_dmd clean'
          sh 'make -j$N -C dmd/src -f posix.mak MODEL=$MODEL HOST_DMD=./host_dmd dmd.conf'
          sh 'make -j$N -C dmd/src -f posix.mak MODEL=$MODEL HOST_DMD=./host_dmd ENABLE_COVERAGE=1'
          sh 'make -j$N -C dmd/test MODEL=$MODEL ARGS="-O -inline -release" DMD_TEST_COVERAGE=1'
          sh 'mv src/host dmd src/dmd'
          break;
      case 'druntime':
          sh 'TEST_COVERAGE="1" make -j$N -C . -f posix.mak MODEL=$MODEL unittest-debug'
          break;
      case 'phobos':
          sh 'ENABLE_COVERAGE="1" make -f posix.mak MODEL=$MODEL unittest-debug'
          // tests needs to be run separately due to https://issues.dlang.org/show_bug.cgi?id=16397
          sh 'make -f posix.mak $(find std etc -name "*.d" | sed "s/[.]d$/.test")'
          break;
      default:
   }

CodeCov

Assuming the CodeCov tokens (can be fetched at e.g. https://codecov.io/gh/dlang/phobos/settings) are loaded in the environment something as CODECOV_TOKEN_PHOBOS (and similarly for dmd and druntime) like this should work:

stage ("Upload coverage report") {
    def repo_name = /python3 -c "import sys, json; print(json.load(sys.stdin)['repository']['name']).toupper()"/
    sh 'curl -s https://codecov.io/bash | bash -s - -t CODECOV_TOKEN_${DMD}'
}

CodeCov uptime doesn't seem to be perfect, we might want to provide a fallback URL as well - see e.g.
dlang/phobos#5140

(though I am not sure what's the proper behavior as the CodeCov API is then probably down as well)

Support badges and add them to the README

If we don't go with #138 (Only build on PRs and don't build on commits to stable or master), it probably makes sense to increase the visibility of the status and add a badge for it to all repos (or maybe start to create a dedicated CI overview page).

There's already a plugin which could probably be easily installed as a start:

https://plugins.jenkins.io/embeddable-build-status

GH API rate limit exceeded

CI is passing:

But it doesn't report back a success message:

e.g. #132 (but appeared on more PRs)

Vibe.d - Memory allocation failed

Linking...
To force a rebuild of up-to-date targets, run again with --force.
Running ./tests 
Watchdog receiver 1
Watchdog receiver 2
Watchdog receiver 3
send Hello World
Watchdog receiver 4
Received string message: Hello, World!
Watchdog receiver 5
Watchdog receiver 6
Watchdog receiver 7
send int 1
Watchdog receiver 8
Received double: 1
Watchdog receiver 9
Watchdog receiver 10
Watchdog receiver 11
send double 1.2
Watchdog receiver 12
Watchdog receiver 13
Watchdog receiver 14
send int 2
Received int message: 2
Received double: 1.2
Watchdog receiver 15
Watchdog receiver 16
Watchdog receiver 17
send 3xint 1 2 3
Watchdog receiver 18
Watchdog receiver 19
Watchdog receiver 20
Watchdog receiver 21
send string Bye bye
Watchdog receiver 22
Received string message: Bye bye
Received iii: 1 2 3
Receive loop finished.
Receiver watchdog failure.
uncaught exception
core.exception.OutOfMemoryError@src/core/exception.d(696): Memory allocation failed

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-5641/1/pipeline

FYI @s-ludwig

Maybe look at splitTests for balancing parallelism

https://jenkins.io/blog/2016/06/16/parallel-test-executor-plugin/
https://jenkins.io/doc/pipeline/steps/parallel-test-executor/

Looks a bit complicated but might be a useful option to keep in mind when parallel oversaturates the server.

Use EC2 Spot instances

Given the big amount of idle time, and the not so urgent workload, using EC2 spot instances might heavily reduce server cost, while at the same time providing more capacity when needed.
jenkinsci/ec2-fleet-plugin: The EC2 Spot Jenkins plugin launches EC2 Spot instances as worker nodes for Jenkins CI server, automatically scaling the capacity with the load.
For example c4.xlarge instances in US EAST (Ohio) cost about 0.03$/hr. most of the time.

shallow checkout for Pipeline from repo?

Currently does a fully checkout of the PR repo in order to get the Jenkinsfile, as each PR is a new job, they have a separate workspace each.
Would be nice to at least only do a shallow clone.

Load ci repo as "shared library"

Might work out better in the long run than to use clone+load.
Extending with Shared Libraries

fetch latest ci runPipeline script on replay

Fairly confusing for people that replay doesn't fix anything if the ci repo was updated in the meantime.

GCE workers sometimes die

Cannot contact runner-01-gce: java.io.IOException: remote file operation failed: /var/lib/jenkins/dlang_projects@5/BlackEdder/ggplotd at hudson.remoting.Channel@2ad430c7:JNLP4-connect connection from 30.0.195.35.bc.googleusercontent.com/35.195.0.30:58308: hudson.remoting.ChannelClosedException: Remote call on JNLP4-connect connection from 30.0.195.35.bc.googleusercontent.com/35.195.0.30:58308 failed. The channel is closing down or has closed down

script returned exit code -1

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7661/3/pipeline

Don't follow the log by default

When looking at a build which is still running, Jenkins follows the log automatically.
It's quite tricky to jump up and un-expand the log tab of the still running component, s.t. one can browse the other components.

java.nio.channels.ClosedChannelException

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6141/1/pipeline

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 16.251.189.35.bc.googleusercontent.com/35.189.251.16:53316
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.Request.call(Request.java:192)
		at hudson.remoting.Channel.call(Channel.java:907)
		at hudson.FilePath.act(FilePath.java:986)
		at hudson.FilePath.act(FilePath.java:975)
		at hudson.FilePath.archive(FilePath.java:457)
		at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:107)
		at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:113)
		at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:101)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
		at hudson.security.ACL.impersonate(ACL.java:260)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)

shouldn't we try to restart such builds?

Compare various parallelization model and chose best performance

After experimental organization project is setup via http://github.com/dlang-test to use full pipeline, we need to make few stress tests that compare how different parallelization rations affect overall behavior.

Cannot contact runner-01-gce: java.lang.InterruptedException

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fci/detail/PR-98/18/pipeline/

That one is a bit different to the typical IOChannelException when a worker dies.

Jenkins doesn't spawn builds

Jenkins seems to be losing PR builds, e.g. it simply doesn't start the build and doesn't send any notification to GH.
This has happened about five times this week already. Closing and re-opening the PR solves the problem, but that's not an ideal solution, because

• it requires manual work
• only contributors with write access can do so
• people need to know about it
• it creates more noise on the PR

dlang/dmd#7752

Jenkins's fails to write files (locking problem?)

[rejectedsoftware/vibe.d] vibe-d:textfilter 0.8.3-alpha.1: building configuration "library"...
[rejectedsoftware/vibe.d] ../../core/vibe/core/log.d(565,21): Deprecation: Built-in hex string literals are deprecated, use `std.conv.hexString` instead.
[rejectedsoftware/vibe.d] vibe-d:inet 0.8.3-alpha.1: building configuration "library"...
[rejectedsoftware/vibe.d] ../../core/vibe/core/log.d(565,21): Deprecation: Built-in hex string literals are deprecated, use `std.conv.hexString` instead.
[rejectedsoftware/vibe.d] vibe-d:tls 0.8.3-alpha.1: building configuration "openssl"...
[rejectedsoftware/vibe.d] ../../core/vibe/core/log.d(565,21): Deprecation: Built-in hex string literals are deprecated, use `std.conv.hexString` instead.
[rejectedsoftware/vibe.d] vibe-d:http 0.8.3-alpha.1: building configuration "library"...
[rejectedsoftware/vibe.d] ../../core/vibe/core/log.d(565,21): Deprecation: Built-in hex string literals are deprecated, use `std.conv.hexString` instead.
[rejectedsoftware/vibe.d] immutable(ubyte)[]
[rejectedsoftware/vibe.d] ../../http/vibe/http/server.d(2026,56): Deprecation: variable vibe.http.server.HTTPServerOption.distribute is deprecated - Use runWorkerTaskDist or start threads separately. It will be removed in 0.9.
[rejectedsoftware/vibe.d] Error: Error writing file '../../http/.dub/build/library-debug-linux.posix-x86_64-dmd_2078-BD83D681FC3A568855FE06D6DCD0FFE1/libvibe-d_http.a'
[rejectedsoftware/vibe.d] dmd failed with exit code 1.

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2061/1/pipeline

Use a custom context name

This is an enhancement request.

Apparently the default context is set to continuous-integration/jenkins/pr-merge which looks quite long and could something more expressive like dlang/projects

According to the docs their DSL can be used for this:

  wrappers {
        downstreamCommitStatus {
            context('CONTEXT NAME')
        }
  }

integrate style checks from CircleCI tests

Simply reducing the number CIs, see https://github.com/dlang/phobos/blob/395ae88ac75ee0c03f6475ad4a5b073cfaf4e084/circleci.sh#L133-L135.

Don't try to checkout matching branches for case of reverts?

In case GitHub's revert functionality is used, GitHub will create a branch in the blessed repo with the revert commit in it. Since it's a separate branch, Jenkins will try to find matching branches with the same name in dmd/phobos/dub/utils repositories, and usually it will fail: https://ci.dlang.io/job/dlang-org/job/druntime/job/revert-1879-soreusse/

Perhaps if the branch name starts with revert-* we should still use masters? I don't have high hopes that's configurable in Jenkins though, but who knows.

ggplotd hangs often

+ ./travis.sh
Generating test runner configuration 'ggplotd-test-default' for 'default' (library).
Performing "unittest" build using dmd for x86_64.
derelict-util 2.1.0: building configuration "library"...
derelict-ft 1.0.2: building configuration "library"...
x11 1.0.19: building configuration "library"...
../../.dub/packages/x11-1.0.19/x11/source/x11/extensions/XI.d(3,8): Deprecation: Symbol `string` is not visible because it is privately imported
cairod 0.0.1-alpha.3+1.10.2: building configuration "static-custom-posix"...
dstats 1.0.5: building configuration "library"...
dunit 1.0.14: building configuration "library"...
painlesstraits 0.2.0: building configuration "library"...
ggplotd 1.1.4: building configuration "ggplotd-test-default"...
../../.dub/packages/color-0.0.5/color/std/experimental/normint.d(55,29): Deprecation: integral promotion not done for `-cast(ubyte)255u`, use '-transition=intpromote' switch or `-cast(int)(cast(ubyte)255u)`
../../.dub/packages/color-0.0.5/color/std/experimental/normint.d(55,29): Deprecation: integral promotion not done for `-cast(byte)127`, use '-transition=intpromote' switch or `-cast(int)(cast(byte)127)`
source/ggplotd/guide.d(439,12): Deprecation: module ggplotd.colour member `RGBA` is not accessible
source/ggplotd/geom.d(50,12): Deprecation: module ggplotd.guide member `GuideStore` is not accessible
source/ggplotd/ggplotd.d(520,12): Deprecation: module ggplotd.legend member `Legend` is not accessible

It usually times out after one hour or so, i.e. when Google terminate the GCE runners.

Seen here: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-6676/11/pipeline (but on many other PRs too)

leftover .git/index.lock from aborted build or preempted worker

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fci/detail/PR-155/3/pipeline#step-706-log-27

Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
stdout:
stderr: fatal: Unable to create '/var/lib/jenkins/dlang_projects@6/dlang-community/dfmt/.git/index.lock': File exists.

Requires to automatically remove git locks in pipeline finally handler and after machine boot.

java.io.IOException: Failed to extract dlang-build.tar.gz - Text file busy

remote file operation failed: /var/lib/jenkins/dlang_projects@6 at hudson.remoting.Channel@f787666:JNLP4-connect connection from 201.62.205.35.bc.googleusercontent.com/35.205.62.201:45376: java.io.IOException: Failed to extract dlang-build.tar.gz

Seen here:

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7831/1/pipeline

Full exception:

java.nio.file.FileSystemException: /var/lib/jenkins/dlang_projects@6/distribution/bin/dmd: Text file busy
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
	at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
	at java.nio.file.Files.newOutputStream(Files.java:216)
	at hudson.util.IOUtils.copy(IOUtils.java:39)
	at hudson.FilePath.readFromTar(FilePath.java:2307)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 201.62.205.35.bc.googleusercontent.com/35.205.62.201:45376
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.FilePath.act(FilePath.java:986)
		at hudson.FilePath.act(FilePath.java:975)
		at hudson.FilePath.untar(FilePath.java:528)
		at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129)
		at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74)
		at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
		at hudson.security.ACL.impersonate(ACL.java:260)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused: java.io.IOException: Failed to extract dlang-build.tar.gz
	at hudson.FilePath.readFromTar(FilePath.java:2317)
	at hudson.FilePath.access$400(FilePath.java:196)
	at hudson.FilePath$4.invoke(FilePath.java:530)
	at hudson.FilePath$4.invoke(FilePath.java:528)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2760)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1$1.run(Engine.java:98)
Caused: java.io.IOException: remote file operation failed: /var/lib/jenkins/dlang_projects@6 at hudson.remoting.Channel@f787666:JNLP4-connect connection from 201.62.205.35.bc.googleusercontent.com/35.205.62.201:45376
	at hudson.FilePath.act(FilePath.java:993)
	at hudson.FilePath.act(FilePath.java:975)
	at hudson.FilePath.untar(FilePath.java:528)

Is this due to unstash name: 'dlang-build' happening in parallel?

dub - feat663-search.sh

[INFO] Running /var/lib/jenkins/dlang_projects@6/dlang/dub/test/feat663-search.sh...
[ERROR] /var/lib/jenkins/dlang_projects@6/dlang/dub/test/feat663-search.sh:11 `dub search dub` failed
        Using dub registry url 'https://code.dlang.org/'
        Refreshing local packages (refresh existing: true)...
        Looking for local package map at /var/lib/dub/packages/local-packages.json
        Looking for local package map at /var/lib/jenkins/dlang_projects@6/.dub/packages/local-packages.json
        Try to load local package map at /var/lib/jenkins/dlang_projects@6/.dub/packages/local-packages.json
        Determined package version using GIT: dub 1.7.0
        ==== registry at https://code.dlang.org/ (fallback ["registry at http://code.dlang.org/", "registry at https://code-mirror.dlang.io/", "registry at https://code-mirror2.dlang.io/", "registry at https://dub-registry.herokuapp.com/"]) ====
        dub (1.7.1-beta.1)              Package manager for D packages
        gitcompatibledubpackage (1.0.4) Example of a DUB package also usable as git submodule. For DUB test suite.
        arsd-official (1.2.2)           A container of various subpackages that do lots of different things. You should use one of the subpackages instead of the main package in most cases, but you can try the complete package if you get duplicated dependency issues.
        dub-registry (2.2.0)            Online registry for dub packages
        dfl2 (1.0.7)                    dfl2 is a GUI library for Windows,now it can be built by dco, DUB or Visual Studio .NET(such as VS 2010).
        dco (0.1.2)                     dco is a build tool, it can be built by DUB and Visual Studio .NET(such as VS 2010).
        hmod-dub (0.1.1)                Generates DDoc documentation for any package in the DUB registry
        dwtlib (3.1.1)                  The dub package for dwt
        dubimport (1.1.3)               A command line tool to extract import paths on dub project.
        rdub (3.0.1)                    RDUB is a front end for DUB, a D language build tool.
        dubbio (0.5.0)                  Dub wrapper
        dub-data-mod (0.9.23-mod.1)     Temporary fork of dub that includes 'dub describe --data=' feature (until it's merged and released in mainline dub).
        gdub (1.1.0)                    GDUB is a DWT GUI front end for DUB, a D language build tool
[ERROR] Script failure.

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6036/1/pipeline

Build important projects with -de

When a deprecation is triggered, it should be ensured that all important repositories don't use the deprecated feature anymore. In other words, it should be the job of the person introducing the deprecation to ensure that (1) the ecosystem is ready for it and (2) suitable replacements exists.

Some projects are less active than others, so we might only want to do this for projects to which we have merge access, i.e. projects at dlang-community

Extend Cookie expiration period

The cookies are currently only saved for the current session:

Any chance we could set the expiration to sth. more convenient like 30 days?

Use a fancy domain: how about ci.dlang.io or jenkins.dlang.io?

Feel free to use a subdomain like with @ dlang.io

@MartinNowak I sent you an invitation for the Namecheap domain, s.t. you can add DNS entries yourself.

dub: fetchzip tests fails sometimes

Sometimes the new fetchzip test fails:

Linking...
[main(----) INF] Listening for requests on http://[::]:18791/
Failed to listen on 0.0.0.0:18791
Trying to download gitcompatibledubpackage (1.0.4)
Fetching gitcompatibledubpackage 1.0.4...
[ERROR] :17 command failed
[main(----) INF] Received signal 15. Shutting down.
[ERROR] Script failure.

From: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7777/11/pipeline

https://github.com/dlang/dub/blob/master/test/fetchzip.sh

Line 17 is:

timeout 1s "$DUB" fetch gitcompatibledubpackage --version=1.0.4 --skip-registry=all --registry=http://localhost:$PORT

Name shell scripts

This might be quite confusing in the beginning:

However, it seems that this hasn't been addressed yet:

https://issues.jenkins-ci.org/browse/JENKINS-36933

As possible workaround, "labelled stages" could be used:

http://stackoverflow.com/questions/39414921/jenkins-pipeline-sh-display-name-label

Only build on PRs and don't build on commits to stable or master

• we don't display the status of Jenkins anywhere
• no one would notice that master broke - except it appears at a PR
• the PR already gets merged with its target branch
• it's a waste of resources
• we already do the same for AppVeyor and SemaphoreCI at dlang/dmd

Implement support for non-master base branches