devhub-tud/build-server

A private docker-based build server.

build-server's Issues

Update /version API endpoint to include Maven version and commit ref

Upgrade build server to use Java 8

Some students for the TI2206 labwork wish to use Java 8 functionality.

Release on Devhub Nexus server

For the TI1706 course we want students to be able to write tests for devhub. For this it would be easiest if the git-server client and build-server client would be released on the Devhub Nexus maven repository. This is also useful for re-enabling the CI without having to clone all repos, checkout the right branch, and to a maven install.

Grading build instruction.

A GradeBuild instruction with a corresponding GradeBuild instruction interpreter.

We should add a plugin to this interpreter to place the needed grading jar into the staging directory.
Then we can safely run the grading.

The generated json output from the grading should be reported back from a post plugin processor and throw it on the message bus

Unknown build failure

java.util.concurrent.ExecutionException: java.io.IOException: org.eclipse.jgit.api.errors.TransportException: ssh://[email protected]/courses/ti1706/1617/group-35: verify: false
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.get(FutureTask.java:206) [na:1.8.0_131]
	at nl.tudelft.ewi.build.builds.BuildManager$Build.run(BuildManager.java:154) ~[build-server.jar:1.1.1-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.io.IOException: org.eclipse.jgit.api.errors.TransportException: ssh://[email protected]/courses/ti1706/1617/group-35: verify: false
	at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.cloneRepository(GitStagingDirectoryPreparer.java:33) ~[build-server.jar:1.1.1-SNAPSHOT]
	at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.prepareStagingDirectory(GitStagingDirectoryPreparer.java:17) ~[build-server.jar:1.1.1-SNAPSHOT]
	at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.prepareStagingDirectory(GitStagingDirectoryPreparer.java:13) ~[build-server.jar:1.1.1-SNAPSHOT]
	at nl.tudelft.ewi.build.builds.BuildManager$BuildRunner.prepareStagingDirectory(BuildManager.java:376) ~[build-server.jar:1.1.1-SNAPSHOT]
	at nl.tudelft.ewi.build.builds.BuildManager$BuildRunner.call(BuildManager.java:262) ~[build-server.jar:1.1.1-SNAPSHOT]
	at nl.tudelft.ewi.build.builds.BuildManager$BuildRunner.call(BuildManager.java:237) ~[build-server.jar:1.1.1-SNAPSHOT]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	... 7 common frames omitted
Caused by: org.eclipse.jgit.api.errors.TransportException: ssh://[email protected]/courses/ti1706/1617/group-35: verify: false
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:137) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:178) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:125) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.cloneRepository(GitStagingDirectoryPreparer.java:29) ~[build-server.jar:1.1.1-SNAPSHOT]
	... 13 common frames omitted
Caused by: org.eclipse.jgit.errors.TransportException: ssh://[email protected]/courses/ti1706/1617/group-35: verify: false
	at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:142) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:121) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:248) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:147) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1111) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:128) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	... 16 common frames omitted
Caused by: com.jcraft.jsch.JSchException: verify: false
	at com.jcraft.jsch.Session.connect(Session.java:307) ~[jsch-0.1.46.jar:na]
	at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:116) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
	... 23 common frames omitted

Design build plugins that can be added to a build instruction

For example: add the checkstyle:checkstyle phase to the Maven build, and push checkstyle result XML to Devhub.

Build server failed to clone

java.util.concurrent.ExecutionException: java.io.IOException: org.eclipse.jgit.api.errors.TransportException: ssh://[email protected]/courses/ti1706/1516/group-60: verify: false
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.7.0_101]
    at java.util.concurrent.FutureTask.get(FutureTask.java:202) [na:1.7.0_101]
    at nl.tudelft.ewi.build.builds.BuildManager$Build.run(BuildManager.java:153) ~[build-server.jar:1.1.1-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_101]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_101]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_101]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
Caused by: java.io.IOException: org.eclipse.jgit.api.errors.TransportException: ssh://[email protected]/courses/ti1706/1516/group-60: verify: false
    at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.cloneRepository(GitStagingDirectoryPreparer.java:33) ~[build-server.jar:1.1.1-SNAPSHOT]
    at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.prepareStagingDirectory(GitStagingDirectoryPreparer.java:17) ~[build-server.jar:1.1.1-SNAPSHOT]
    at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.prepareStagingDirectory(GitStagingDirectoryPreparer.java:13) ~[build-server.jar:1.1.1-SNAPSHOT]
    at nl.tudelft.ewi.build.builds.BuildManager$BuildRunner.prepareStagingDirectory(BuildManager.java:376) ~[build-server.jar:1.1.1-SNAPSHOT]
    at nl.tudelft.ewi.build.builds.BuildManager$BuildRunner.call(BuildManager.java:262) ~[build-server.jar:1.1.1-SNAPSHOT]
    at nl.tudelft.ewi.build.builds.BuildManager$BuildRunner.call(BuildManager.java:237) ~[build-server.jar:1.1.1-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_101]
    ... 7 common frames omitted
Caused by: org.eclipse.jgit.api.errors.TransportException: ssh://[email protected]/courses/ti1706/1516/group-60: verify: false
    at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:137) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:178) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:125) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at nl.tudelft.ewi.build.extensions.staging.GitStagingDirectoryPreparer.cloneRepository(GitStagingDirectoryPreparer.java:29) ~[build-server.jar:1.1.1-SNAPSHOT]
    ... 13 common frames omitted
Caused by: org.eclipse.jgit.errors.TransportException: ssh://[email protected]/courses/ti1706/1516/group-60: verify: false
    at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:142) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:121) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:248) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:147) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1111) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:128) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    ... 16 common frames omitted
Caused by: com.jcraft.jsch.JSchException: verify: false
    at com.jcraft.jsch.Session.connect(Session.java:307) ~[jsch-0.1.46.jar:na]
    at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:116) ~[org.eclipse.jgit-3.2.0.201312181205-r.jar:3.2.0.201312181205-r]
    ... 23 common frames omitted

Migrate docker-client, resteasy and jackson to newer versions

Provide a Vagrant file as an alternative for Boot2Docker in development

In the Helios project I found a Docker host Vagrant file (without TLS). This can be a nice starting point for our own Docker host vagrant file.

# -*- mode: ruby -*-
# vi: set ft=ruby :

BOX_NAME = ENV['BOX_NAME'] || "phusion/ubuntu-14.04-amd64"
FORWARD_DOCKER_PORTS = ENV['FORWARD_DOCKER_PORTS']

Vagrant.require_version ">= 1.6.2"

Vagrant.configure("2") do |config|
  # Setup virtual machine box. This VM configuration code is always executed.
  config.vm.box = BOX_NAME

  config.ssh.forward_agent = true
  config.vm.network :forwarded_port, guest: 2375, host: 2375
  config.vm.network :forwarded_port, guest: 5801, host: 5801
  config.vm.network :private_network, ip: "192.168.33.10"

  # sync the maven folder
  config.vm.synced_folder "~/.m2", "/home/vagrant/.m2"

  pkg_cmd = "export DEBIAN_FRONTEND=noninteractive; "
  pkg_cmd << "set -x; "

  # install docker
  pkg_cmd << "curl -s https://get.docker.io/gpg | apt-key add -; "
  pkg_cmd << "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list; "
  pkg_cmd << "apt-get update && apt-get -y install lxc-docker; "

  # Set up docker to listen on 0.0.0.0:2375
  pkg_cmd << "echo 'DOCKER_OPTS=\"--restart=false -D=true -H=tcp://0.0.0.0:2375 -H=unix:///var/run/docker.sock --dns=192.168.33.10\"' > /etc/default/docker; "
  # make docker usable by vagrant user w/o sudo
  pkg_cmd << "groupadd docker; gpasswd -a vagrant docker; service docker restart;"

  # install other helios dependencies and development tools
  pkg_cmd << "apt-get install -y default-jdk maven zookeeperd=3.4.5+dfsg-1 git vim curl golang mercurial; "

  # make sure zk is running
  pkg_cmd << "initctl start zookeeper ;"

  # install helios conf files
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    echo '
    ENABLED=true

    HELIOS_AGENT_OPTS="--state-dir=/var/lib/helios-agent --name=ubuntu-14.skydns.local --zk localhost:2181 --service-registry http://127.0.0.1:4001 --service-registrar-plugin /usr/share/helios/lib/plugins/helios-skydns-0.1.jar --domain skydns.local"
    ' > /etc/default/helios-agent ;
    END
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    echo '
    ENABLED=true

    HELIOS_MASTER_OPTS="--zk localhost:2181"
    ' > /etc/default/helios-master ;
    END

  # Download and install skydns registrar plugin version 0.1 release from github
  # If you change which version, update the conf file bit above
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    curl -L https://github.com/spotify/helios-skydns/releases/download/0.1/helios-skydns_0.1_all.deb -o helios-skydns_0.1_all.deb && \
    dpkg -i helios-skydns_0.1_all.deb ;
    END

  pkg_cmd << "mkdir -p /home/vagrant/.helios;"
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    echo '{"masterEndpoints":["http://localhost:5801"]}' > /home/vagrant/.helios/config;
    END

  # build and install Helios
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    cd /vagrant && sudo -u vagrant mvn -B -DskipTests package && \
        dpkg --force-confdef --force-confold -i \
            /vagrant/helios-tools/target/*.deb \
            /vagrant/helios-services/target/*.deb ;
    END

  #build skydns version c83f12 to be at least consistent
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    export GOPATH=/home/vagrant/gopath && \
    mkdir -p $GOPATH/src/github.com/skynetservices && \
    cd $GOPATH/src/github.com/skynetservices && \
    git clone https://github.com/skynetservices/skydns.git && \
    cd skydns && \
    git checkout c83f12b96ff3331c0412d7d100ba3e0724f8aa84 && \
    go get -d -v ./... && go build -v ./... && \
    cp skydns /usr/bin/skydns ;
    END

  # build etcd v0.4.5
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    cd /home/vagrant && \
    git clone https://github.com/coreos/etcd && \
    cd etcd && \
    git checkout v0.4.5 && \
    git fetch origin pull/899/head:pull-899 && \
    git merge pull-899 && \
    ./build && \
    cp bin/etcd /usr/bin ;
    END

  # put in upstart config for etcd
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    echo '
    description "etcd"

    start on runlevel [2345]
    stop on runlevel [!2345]

    respawn
    respawn limit unlimited

    script
        [ -r /etc/default/etcd ] && . /etc/default/etcd
        /usr/bin/etcd $ETCD_OPTS
    end script

    # prevent respawning more than once every second
    post-stop exec sleep 1
    ' > /etc/init/etcd.conf && \
    initctl start etcd ;
    END

  # put in config for skydns
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    echo '
    SKYDNS_OPTS="-addr=0.0.0.0:53 -nameservers=8.8.8.8:53,8.8.4.4:53"
    ' > /etc/default/skydns ;
    END

  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    echo '
    description "skydns"

    start on runlevel [2345]
    stop on runlevel [!2345]

    respawn
    respawn limit unlimited

    script
        [ -r /etc/default/skydns ] && . /etc/default/skydns
        /usr/bin/skydns $SKYDNS_OPTS
    end script

    # prevent respawning more than once every second
    post-stop exec sleep 1
    ' > /etc/init/skydns.conf && \
    initctl start skydns ;
    END

  # make sure we can look up ubuntu-14.skydns.local
  pkg_cmd << <<-END.gsub(/^ {4}/, '')
    curl -XPUT http://127.0.0.1:4001/v2/keys/skydns/local/skydns/ubuntu-14 \
         -d value='{"host":"192.168.33.10"}' && \
    mv /sbin/resolvconf /sbin/resolvconf.no && \
    echo 'nameserver 127.0.0.1
    search skydns.local
    ' > /etc/resolv.conf;
    END

  config.vm.provision :shell, :inline => pkg_cmd
end


# Providers were added on Vagrant >= 1.1.0
Vagrant::VERSION >= "1.1.0" and Vagrant.configure("2") do |config|
  config.vm.provider :virtualbox do |vb, override|
    config.vm.box = BOX_NAME
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
    vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
  end
end

if !FORWARD_DOCKER_PORTS.nil?
  Vagrant::VERSION < "1.1.0" and Vagrant::Config.run do |config|
    (49000..49900).each do |port|
      config.vm.forward_port port, port
    end
  end

  Vagrant::VERSION >= "1.1.0" and Vagrant.configure("2") do |config|
    (49000..49900).each do |port|
      config.vm.network :forwarded_port, :host => port, :guest => port
    end
  end
end

Support additional build plugins

• sbt
• npm and bower
• make
  etc..