openzfs/openzfs-build

EOL: This is not being maintained, please see: https://github.com/openzfs/openzfs-ci

This repo is archived. Alternate processes for contributing ZFS changes include following the illumos procedures, or opening a PR against ZFSonLinux.

The reasoning behind this are outlined in an email to the [email protected] mailing list, which is reproduced below:

The OpenZFS code repository on github (http://github.com/openzfs/openzfs) is a clone of the illumos repo, with basically identical code. The OpenZFS repo made it easier to contribute ZFS code to illumos, by leveraging the github pull request and code review processes, and by automatically building illumos and running the ZFS Test Suite.

Unfortunately, the automated systems have atrophied and we lack the effort and interest to maintain it. Meanwhile, the illumos code review and contribution process has been working well for a lot of ZFS changes (notably including ports from Linux).

Since the utility of this repo has decreased, and the volunteer workforce isn't available to maintain it, we will be archiving http://github.com/openzfs/openzfs in the coming week. Thank you to everyone who helped maintain this infrastructure, and to those who leveraged it to contribute over 500 commits to ZFS on illumos! Alternate processes for contributing ZFS changes (including those in open PR's) include following the illumos procedures, or opening a PR against ZFSonLinux.

openzfs-build's Issues

Disable the "can one of the admins verify this patch" comment of zettabot

For each pull request not submitted by a user of the openzfs organization, zettabot will post a comment that asks: "can one of the admins verify this patch?". This is unnecessary and should be removed.

Disable the 'synchronize-jenkins-mirror' job for development instances

When standing up a new Jenkins master instance for testing or development, the 'synchronize-jenkins-mirror' will be enabled by default which is not intended. That job should only be run from the production master instance, and disabled everywhere else.

Disable zettabot integration on public Jenkins mirror

Currently when a pull request is opened on the openzfs/openzfs repository, two comments will be posted by the zettabot user asking "Can one of the admins verify this patch?" .. This occurs because the internal Jenkins instance will post the question (which is intended), but also the public mirror instance asks the question too (which is not intended).

We need to disable the GitHub integration logic in the public Jenkins mirror to prevent this.

The "openzfs-run-zfs-test" Jenkins job fails

Currently the openzfs-run-zfs-test Jenkins job doesn't run successfully on the openzfs/master branch. The root cause for the failure is not yet understood, and we need this to work in order for it to be useful for testing incoming pull requests and used for the illumos RTI process.

Jenkins jobs should have meaningful titles

The jenkins jobs should have titles that help you correlate with the pull requests. Right now they are either blank, or some HTML that looks like it embeds the useful info.

Useful info would be: pull request # and the pull request summary (which looks like it's just the first line of the last commit? - whatever shows up in github)

Split up commit status results into more sections

Right now, when a build and test cycle is triggered, the pull request will have a single "OpenZFS Regression Tests" status that is displayed which encapsulates the build, ztest, and zfs test suite sections of the entire cycle. It might be useful to split this up, such that a status is reported back to the pull request of each individual section.. e.g. there could be a section for the build, another for ztest, and one more section for the zfs test suite. This way, if only the ztest section fails, it's obvious that the build and zfs test suite sections passed without having to click into the Jenkins UI.

Make "destroy-build-slave" job more resilient when the clone step fails

If the clone-build-slave job fails, it's likely to cause the destroy-build-slave job to fail, potentially leaking VMs. For example, see: http://psurya-jenkins.dcenter.delphix.com:8080/job/destroy-build-slave/55/console

TASK: [destroy-dc-instance | Unregister DCenter instance] ********************* 
<dcenter.delphix.com> ESTABLISH CONNECTION FOR USER: blackbox
<dcenter.delphix.com> REMOTE_MODULE command /opt/dcenter/bin/dc unregister --expires 2 openzfs-9a98bee0-A #USE_SHELL
<dcenter.delphix.com> EXEC sshpass -d11 ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=blackbox -o ConnectTimeout=10 dcenter.delphix.com /bin/sh -c 'mkdir -p /tmp/ansible-tmp-1446188082.15-224873084695844 && chmod a+rx /tmp/ansible-tmp-1446188082.15-224873084695844 && echo /tmp/ansible-tmp-1446188082.15-224873084695844'
<dcenter.delphix.com> PUT /tmp/tmpDvp7O2 TO /tmp/ansible-tmp-1446188082.15-224873084695844/command
<dcenter.delphix.com> EXEC sshpass -d11 ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=blackbox -o ConnectTimeout=10 dcenter.delphix.com /bin/sh -c 'LANG=C LC_CTYPE=C /usr/bin/python /tmp/ansible-tmp-1446188082.15-224873084695844/command; rm -rf /tmp/ansible-tmp-1446188082.15-224873084695844/ >/dev/null 2>&1'
failed: [dcenter.delphix.com] => {"changed": true, "cmd": "/opt/dcenter/bin/dc unregister --expires 2 openzfs-9a98bee0-A", "delta": "0:00:00.348058", "end": "2015-10-30 06:54:46.851365", "rc": 1, "start": "2015-10-30 06:54:46.503307", "warnings": []}
stderr: there is no VM named 'openzfs-9a98bee0-A'

FATAL: all hosts have already failed -- aborting

In this case, the openzfs-9a98bee0 still existed, but it was not destroyed because this failure aborted the job before openzfs-9a98bee0 could be destroyed. We should ignore cases where the VM does not exist, so any other VMs can still be processed.

in-progress jenkins jobs appear at bottom of list

On the internal jenkins instance (http://psurya-jenkins.dcenter:8080/job/openzfs-regression-tests/), in-progress jobs appear at the bottom of the list. If you have enough old jobs, the only way to see the in-progress ones is to use the bizarre jenkins paging buttons that appear when you mouseover the build history.

Low priority for now, but once the "real" jenkins instance is made public, this issue will be more widely seen.

Investigate using the GitHub OpenZFS organization to automatically white-list users

The Jenkins Pull Request Builder plugin has support for automatically white-listing users and adding them as admins via the orgWhitelist and allowMembersOfWhitelistedOrgsAsAdmin methods. If possible, we should adopt this convention. See this link for more information: https://jenkinsci.github.io/job-dsl-plugin/#method/javaposse.jobdsl.dsl.helpers.triggers.TriggerContext.pullRequest

The "openzfs-run-ztest" Jenkins job fails

Currently the openzfs-run-ztest Jenkins job doesn't run successfully on the openzfs/master branch. The root cause for the failure is not yet understood, and we need this to work in order for it to be useful for testing incoming pull requests and used for the illumos RTI process.

Set up proper public Jenkins master and DNS entry

Right now we're rsync-ing the private Jenkins master to build.prakashsurya.com on a personal EC2 instance of mine, and using my personal domain. It'd be great to use a more "official" EC2 instance and domain name for this infrastructure. E.g. I'm thinking of something like build.open-zfs.org as the domain name.

Run regression tests against each commit to "master"

Currently we only build and test pull requests. It'd be great (and probably not much work) to also add a tigger to the regression job to also test each new commit to the master branch, and also have the commit statuses appended to each of those commits. Obviously we shouldn't introduce breakage to the master branch, but sometimes processes fail, so this would be a good way to double check ourselves and quickly get feedback in case something did break on master.

openzfs-regression-tests job has java.io.IOException

See e.g. http://psurya-jenkins.dcenter:8080/job/openzfs-regression-tests/36/console

Could not update commit status of the Pull Request on GitHub.
java.io.IOException: {"message":"Validation Failed","errors":[{"resource":"Status","code":"custom","field":"target_url","message":"target_url must use http(s) scheme"}],"documentation_url":"https://developer.github.com/v3/repos/statuses/#create-a-status"}
    at org.kohsuke.github.Requester.handleApiError(Requester.java:503)
    at org.kohsuke.github.Requester._to(Requester.java:248)
    at org.kohsuke.github.Requester.to(Requester.java:194)
    at org.kohsuke.github.GHRepository.createCommitStatus(GHRepository.java:829)
    at org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus.createCommitStatus(GhprbSimpleStatus.java:213)
    at org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus.onBuildStart(GhprbSimpleStatus.java:149)
    at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(GhprbBuilds.java:122)
    at org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(GhprbBuildListener.java:24)
    at org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(GhprbBuildListener.java:17)
    at hudson.model.listeners.RunListener.fireStarted(RunListener.java:215)
    at hudson.model.Run.execute(Run.java:1737)
    at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:73)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.IOException: Server returned HTTP response code: 422 for URL: https://api.github.com/repos/openzfs/openzfs/statuses/8a0b8077093812ab9274f8255b9aa7bdad1a4e31
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at org.kohsuke.github.Requester.parse(Requester.java:458)
    at org.kohsuke.github.Requester._to(Requester.java:227)
    ... 12 more
Caused by: java.io.IOException: Server returned HTTP response code: 422 for URL: https://api.github.com/repos/openzfs/openzfs/statuses/8a0b8077093812ab9274f8255b9aa7bdad1a4e31
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at org.kohsuke.github.Requester.parse(Requester.java:454)
    ... 13 more

Make "clone-build-slave" job more resilient to DNS failures

There's been a failure in the clone-build-slave job where dcenter.delphix.com failed to resolve: http://build.prakashsurya.com:8080/job/clone-build-slave/39/console

PLAY [dcenter.delphix.com] **************************************************** 

GATHERING FACTS *************************************************************** 
<dcenter.delphix.com> ESTABLISH CONNECTION FOR USER: blackbox
<dcenter.delphix.com> REMOTE_MODULE setup
<dcenter.delphix.com> EXEC sshpass -d4 ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=blackbox -o ConnectTimeout=10 dcenter.delphix.com /bin/sh -c 'mkdir -p /tmp/ansible-tmp-1446188076.55-78171385027248 && chmod a+rx /tmp/ansible-tmp-1446188076.55-78171385027248 && echo /tmp/ansible-tmp-1446188076.55-78171385027248'
fatal: [dcenter.delphix.com] => SSH Error: ssh: Could not resolve hostname dcenter.delphix.com: No address associated with hostname
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

TASK: [snapshot-dc-instance | Check for snapshot_name variable] *************** 
FATAL: no hosts matched or all hosts have already failed -- aborting

In order to mitigate this issue, the IP for dcenter.delphix.com should be added to the /etc/hosts on the Jenkins master.

Remove "psurya-jenkins" links

There's a number of places where my internal Jenkins domain name is being leaked via hyperlinks in the public Jenkins UI, as well as in the GitHub pull request status messages that are posted. These should all get updated to link to the public Jenkins instance instead.

"gnutls_handshake() failed" when the create-build-slave job is trying to fetch the openzfs-build repository

We've see the following strange failure at least one, here: http://psurya-jenkins.dcenter:8080/job/create-build-slave/144/console

Started by upstream project "openzfs-regression-tests" build number 142
originally caused by:
 GitHub pull request #56 of commit 477dddd52b8ea3016c2eefa1947ad6476b393a01, no merge conflicts.
[EnvInject] - Loading node environment variables.
Building in workspace /var/lib/jenkins/jobs/create-build-slave/workspace@2
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/openzfs/openzfs-build.git # timeout=10
Fetching upstream changes from https://github.com/openzfs/openzfs-build.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress https://github.com/openzfs/openzfs-build.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/openzfs/openzfs-build.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:763)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1012)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1043)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/openzfs/openzfs-build.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: unable to access 'https://github.com/openzfs/openzfs-build.git/': gnutls_handshake() failed: A TLS packet with unexpected length was received.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1388)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:62)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:313)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:761)
    ... 11 more
ERROR: null
Finished: FAILURE

I haven't done any investigation into this issue, but it appears as though the job had trouble connecting to the GitHub API when fetching the openzfs-build repository.

Use Amazon EC2 for builds and testing

When setting up this repository, I tried testing illumos on EC2 and the system failed to boot after an onu of stock illumos-gate. This is a blocker to moving the entire infrastructure to AWS EC2.

User pull request trigger phrase that's more directed at zettabot

The current trigger phrase of "this this please" can easily be confused by a contributor to be directed at them, instead of zettabot. We should change the phrase to be something like "@zettabot please test this" to avoid confusion about who the comment was directed to.

Add `-l` to the `NIGHTLY_OPTIONS` of the build

The README.md states the following:

NOTE: The -l option has been temporarily removed from from NIGHTLY_OPTIONS due to lint errors being reported and failing otherwise good builds. The failures are not yet understood, so they are being suppressed for the time being. This is not intended to be a long term fix; once the failures are investigated and/or fixed, this option will be re-enabled.

This must be fixed prior to the automated results actually being used for the illumos RTI process.

Update build and test slaves to OmniOS r151016

With the release of OmniOS r151016, @danmcd has made the process of building and testing illumos on OmniOS more streamlined. He's detailed these changes in this post to the OpenZFS mailing list here: http://lists.open-zfs.org/pipermail/developer/2015-November/001980.html

We should update the current build and test slave, that are based on OmniOS r151014, to use the new r151016 image and use the new tools that Dan has built. At the very least, moving the builders to using the new tools and process will ensure it is regularly tested, and make sure it continually works as expected.