h2oai/h2o-jenkins-pipeline-lib

Library of different Jenkins pipeline building blocks.

h2o-jenkins-pipeline-lib's Issues

Report start time in stage description

It would be helpful to report stage start time here:

h2o-jenkins-pipeline-lib/src/ai/h2o/ci/buildsummary/StagesSummary.groovy

Line 250 in 3c45559

return 'In progress...'

Instead of "In progress" i would like to see "In progress from 12:33am"

CC: @michal-raska

s3up is unable to use parent job's env

Looks like the s3up is not able to use the parent job's env variable on jenkins and reads it as null, which forces us to extract values from env before using the s3up.

Failing example:

s3up {
...
version = "${env.BUILD_ID}"
...
}

Healthcheck for scripted pipelines.

Add a health check mechanism. It should check the health of given node (based on configurable metrics) and if the node is healthy, the proceed with the stage (build, test, etc.). If not, then mark this node as unhealthy and move to next healthy node. Retry this up to n (something like n=5) times.

By default it'll check two things:

1. whether a specified docker container can be started
2. if there is more than 8% of free space in $HOME (because we need to store pipeline data somewhere) and in / (because of docker, etc.)

If either of the checks fails, then the node will be marked as unhealthy (internally by HealthChecker, nothing is changed in Jenkins).

The API should look something like:

// Instantiate.
HealthChecker healthChecker = HealthChecker.getDefaultInstance()
// Set base label, will be used as base for finding nodes. In practice, the label will look like <BASE_LABEL> && !<UNHEALTHY_1> && !<UNHEALTHY_2>...
healthChecker.setBaseLabel('docker && linux && !micro')
// Set number of max retries to 5.
healthChecker.setMaxRetries(5)
// Add custom check.
// CustomCheck should be Closure, which returns boolean value and takes no parameters.
def customCheck = {
  return new File('/path/to/required/file').exists()
}
// Add the check to healthChecker under 'required-file-check' key. If such key already exists, it'll be overridden.
healthChecker.addCheck('required-file-check', customCheck)
// Remove space requirement check (for whatever reason).
healthChecker.removeCheck(HealthChecker.SPACE_REQUIREMENT_CHECK_ID)
// Main pipeline code.
// HealthyNode ensures the stage will be executed on healthy node.
healthyNode {
  stage('Build') {
    // build steps
  }
}
healthyNode {
  stage('Test') {
    // test steps
  }
}

Issues are following:

• AFAIK, this cannot be used by declarative pipelines
• it doesn't contain any validation that we have node which satisfies the current label (BASE_LABEL + unhealthy nodes)
  • in practice, this is not much of an issue
  • it's mostly relevant when you want to execute your stage on a small set of nodes or on a single node only -> in this case, just don't use the health check (you probably need to run benchmarks or release or something very node specific)
  • if there is no node satisfying the current label, the job will just hang (executor on master is not consumed)

We are using something very similar (a bit simplified version) in H2O-3, and it works well. For example this might be helpful for Datatable and possibly Sparkling Water as well (@jakubhava is planning to move to dockerized pipelines). Both projects use scripted pipelines.

WDYT @mmalohlava and @jakubhava ?

Add queue time on stage summary