librecores/librecores-ci-jenkins-server

LibreCores Continuous Integration

librecores-ci-jenkins-server's Issues

Deploy the EDA tool infrastructure

We have some heavy tools, which we cannot easily pack into Docker. We discussed it and decided we want to have a old-style Custom Tool Infrastructure. I had a success story with Custom Tools Plugin, si I consider just reimplementing it in LibreCores CI

Create the Librecores Pipeline Library

Currently Pipeline scripts are being stored across PoC projects and my local PC. It would be great to integrate the stuff into a single Pipeline library using the newest features. If the stuff does not work, fallback to Pipeline remote loader.

TODOs:

• - Fusesoc-Docker Packaging Lib
• - FuseSoC Commons Lib - for modification of FuseSoC project registry via scripts
• - FuseSoC CI Lib - CI flows specific to LibreCores CI

Setup Goss library on LibreCores CI

https://github.com/mschuchard/jenkins-devops-libs/blob/master/docs/Goss.md

It is needed for librecores/docker-images#7
CC @Nancy-Chauhan

• @Nancy-Chauhan will try the library and report results in librecores/docker-images#7
• @oleg-nenashev will install the library on the current instance so that we can verify it
• if everything works well, we could add it to the Configuration-as-Code definition in this repo

Rename the repository to avoid confusion with Docker images

Docker images become somewhat as important as the Jenkins server in the project, and it would be great to rename the repo to avoid confusion. E.g. "librecores-ci-jenkins" or whatever

TAP Plugin needs update to a version with Fixed Pipeline snapshot

Currently we use a SNAPSHOT build of jenkinsci/tap-plugin#14 ( jenkinsci/tap-plugin@e609a9b ), the new version should be picked after the release

Streamline packaging of the Hosted LibreCores CI and Docker images

Currently Docker images differ from the LibreCores CI Web instance. We should unify the Pipelines for it. Suggested approach:

• Jenkins Custom WAR Packager is used to define the Docker and Production environments
  ** All configurations except credentials are included into the bundle so that they are immutable on the server;s side
• Docker and Librecores CI dev images are packaged using the same scripts
• Plugins are updated to recent versions so that we start from a modern setup

Add Librecores CI status reporting to LibreCores Web

• Minimal-valuable feature - Reporting of the Master branch status via Embedded build status
• Ideally: Reporting of a subset of branches with some additional info

Provide testing scripts for OPTIMSoC

@imphil is going to provide testing scripts, which we can use for demo purposes.

Move LibreCores CI OAuth Application to the librecores org

Now it is being managed as personal application of @oleg-nenashev. Not good

DockerHub build fails on the master branch

See https://cloud.docker.com/u/librecores/repository/registry-1.docker.io/librecores/librecores-ci-dev/builds/b0f6dd1c-5cda-43ff-bace-77a3743ce7ee .

Jun 17, 2019 6:46:39 AM io.jenkins.tools.warpackager.lib.util.MavenHelper listDependenciesFromPom
INFO: Listing dependencies from POM file: pom.xml
[INFO] Scanning for projects...
[WARNING] The POM for org.jenkins-ci.tools:maven-hpi-plugin:jar:3.5 is missing, no dependency i
nformation available
[WARNING] Failed to build parent project for org.librecores.librecores-ci:librecores-ci:pom:1.0-SNAPSH
OT
[I
N

Exception in thread "main" java.io.IOException: Invalid update center url : http://updates.jenkins.io/update-center.json

Maybe just a network glitch on the DockerHub build machine, needs investigation

Move shared code of OpenRISC Pipelines to LibreCores Pipeline Library

OpenRISC projects i.e mor1kx and or1k-marocchino CI involves multi-branch-pipeline structure which also uses librecores-ci and a librecores-ci-openrisc docker image .
This issue involves Detaching reusable logic of mor1kx and or1k-marochhino to librecores-pipeline-lib. This will simplify the pipeline libraries

Switch the LCCI server to Java 11

Currently Jenkins runs with Java 8. We could switch it to Java 11 to get some extra performance benefits

Workaround/replace/remove unmaintained Jenkins plugins

Abandoned extremely old plugins are potential security and integration hazards.
If there's a possibility to do so, dependency on them should be removed or effort should be made to maintain them again.

List of the least maintained plugins used in LCCI as of 7.11.17:

filesystem_scm:
(WARNING!) Extremely old (6y since last release) plugin.
Tracks changes in the filesystem like it was an SCM repo.
Oleg Nenashev got control over plugin and currently tries to modernize it.
Github repo

saferestart:
(WARNING!!!) Extremely old plugin (5y since last release). Allows restarting Jenkins safely: it's waiting till all builds in progress finish before launching restart.

simple-theme-plugin:
(WARNING!!!) Extremely old plugin (5y since release).
Custom CSS styling and JS on the page.
(WARNING!!!) Potentially hazardous due to its nature and age.

description-setter:
(WARNING!) Old plugin (3y since last release).
Sets build description based on build logs.

log-parser:
(WARNING!) Old plugin (2y since release).
Parses Jenkins CI build logs and collects warnings and errors.

Add support of the email notifications from the instance

Use-cases:

• Notifications to infra admins when something goes wrong
• Notifications to project owners when dependent projects fails (maybe overkill)

Setup reporting of build statuses to IRC

It's a follow-up to the comment from @olofk
#13 (comment)

Setup demo CI flow for mor1kx

The flow should include:

• Building of reference ELF files
• Running of ELF file simulation with FuseSoC
• Reporting of results

@wallento is also going to provide TAP reporting in FuseSoC. If it's done, we will need to write converter for xUnit plugin

Plugins fail to load when running Docker image

When running docker run --rm --name ci-jenkins-io-dev -v maven-repo:/root/.m2 -e DEV_HOST=${CURRENT_HOST} -p 8080:8080 -p 50000:50000 -v ${MY_SECRETS_DIR}:/var/jenkins_home/imported_secrets:ro librecores/librecores-ci-dev, the Jenkins instance fails with the following message on the web page:

In the terminal, it seems like the issue is due to several plugins failing to load:

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Jenkins Parameterized Trigger plugin v2.33 (parameterized-trigger)
java.io.IOException: Jenkins Parameterized Trigger plugin version 2.33 failed to load.
 - Jenkins Subversion Plug-in version 1.54 is older than required. To fix, install version 2.5.7 or later.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Docker Commons Plugin v1.15 (docker-commons)
java.io.IOException: Docker Commons Plugin version 1.15 failed to load.
 - Pipeline: Declarative Extension Points API version 1.3.2 is older than required. To fix, install version 1.3.8 or later.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Docker Pipeline v1.14 (docker-workflow)
java.io.IOException: Docker Pipeline version 1.14 failed to load.
 - Docker Commons Plugin version 1.15 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Pipeline: Declarative v1.3.2 (pipeline-model-definition)
java.io.IOException: Pipeline: Declarative version 1.3.2 failed to load.
 - Docker Pipeline version 1.14 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Pipeline implementation for Blue Ocean v1.16.0 (blueocean-pipeline-api-impl)
java.io.IOException: Pipeline implementation for Blue Ocean version 1.16.0 failed to load.
 - Pipeline: Declarative version 1.3.2 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Bitbucket Pipeline for Blue Ocean v1.16.0 (blueocean-bitbucket-pipeline)
java.io.IOException: Bitbucket Pipeline for Blue Ocean version 1.16.0 failed to load.
 - Pipeline implementation for Blue Ocean version 1.16.0 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Git Pipeline for Blue Ocean v1.16.0 (blueocean-git-pipeline)
java.io.IOException: Git Pipeline for Blue Ocean version 1.16.0 failed to load.
 - Pipeline implementation for Blue Ocean version 1.16.0 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Blue Ocean Pipeline Editor v1.16.0 (blueocean-pipeline-editor)
java.io.IOException: Blue Ocean Pipeline Editor version 1.16.0 failed to load.
 - Pipeline implementation for Blue Ocean version 1.16.0 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Jenkins Cobertura Plugin v1.13 (cobertura)
java.io.IOException: Jenkins Cobertura Plugin version 1.13 failed to load.
 - Maven Integration plugin version 2.14 is older than required. To fix, install version 3.1.2 or later.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Events API for Blue Ocean v1.16.0 (blueocean-events)
java.io.IOException: Events API for Blue Ocean version 1.16.0 failed to load.
 - Pipeline implementation for Blue Ocean version 1.16.0 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin GitHub Pipeline for Blue Ocean v1.16.0 (blueocean-github-pipeline)
java.io.IOException: GitHub Pipeline for Blue Ocean version 1.16.0 failed to load.
 - Pipeline implementation for Blue Ocean version 1.16.0 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Blue Ocean v1.16.0 (blueocean)
java.io.IOException: Blue Ocean version 1.16.0 failed to load.
 - Bitbucket Pipeline for Blue Ocean version 1.16.0 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Parallel Test Executor Plugin v1.12 (parallel-test-executor)
java.io.IOException: Parallel Test Executor Plugin version 1.12 failed to load.
 - Jenkins Parameterized Trigger plugin version 2.33 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Pipeline v2.6 (workflow-aggregator)
java.io.IOException: Pipeline version 2.6 failed to load.
 - Pipeline: Declarative version 1.3.2 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 21, 2020 7:19:53 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Yet Another Docker Plugin v0.1.0-rc52 (yet-another-docker-plugin)
java.io.IOException: Yet Another Docker Plugin version 0.1.0-rc52 failed to load.
 - Docker Commons Plugin version 1.15 failed to load. Fix this plugin first.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:921)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:544)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Get Vivado available in the environment

https://github.com/librecores/librecores-ci/wiki/OPTIMSoC-Story

Create LibreCores Bot account on GitHub

Use-cases:

• - Polling of repos with better API requests limit
• - Commit/PR status reporting to GitHub

Adopt Configuration-as-Code in LibreCores CI

Last September Jenkins projects has released the Configuration-as-Code plugin, which allows simplifying the configuration logic. It would be great to adopt JCasC in this repository.

Demo: https://github.com/oleg-nenashev/demo-jenkins-config-as-code

Setup Shared Tool Infrastructure for internal LibreCores nodes

Requirements:

• Availability via network (shared storage on NFS)
• Support of multiple tool versions with automatic version listing
• Nice2Have: Integration with Custom Tools (environment setup, etc.)
  • Still needs Pipeline integration :(

Open questions:

• Samba/NFS or Amazon-specific options?

Librecores CI failing for mor1kx

The librecores server fails to run jobs for mor1kx. It seems there is a filesystem error on the server.

See issue:
openrisc/mor1kx#125

Cloning repository https://github.com/openrisc/mor1kx.git
ERROR: Failed to clean the workspace
jenkins.util.io.CompositeIOException: Unable to delete '/var/lib/jenkins/workspace/Projects_OpenRISC_mor1kx_master'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 54 additional exceptions)
	at jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:90)
	at hudson.Util.deleteContentsRecursive(Util.java:261)
	at hudson.Util.deleteContentsRecursive(Util.java:250)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:743)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1132)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1177)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/Projects_OpenRISC_mor1kx_master/build/mor1kx_5.0-r3/synth-icestorm/mor1kx_5.0-r3.eda.yml: Operation not permitted
		at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
		at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238)
		at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260)
		at java.nio.file.Files.setPosixFilePermissions(Files.java:2045)
		at jenkins.util.io.PathRemover.makeWritable(PathRemover.java:286)
		at jenkins.util.io.PathRemover.makeRemovable(PathRemover.java:259)
		at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:239)
		at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
		at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
		at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
		at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
		at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
		at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
		at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
		at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
		at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
		at jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:87)
		... 14 more

Restrict execution of jobs on Master

Librecores project maintainers may be allowed to write their own Jenkinsfiles. Due to the security reasons, they must not be allowed to run any job on master in such case.

Solution: Install and configure Job Restrictions plugin

Add a Plot plugin for publishing PnR and Synthesis stats

We will need a plugin to publish synthesis and PnR stats in Jenkins. It could be done via https://wiki.jenkins.io/display/JENKINS/Performance+Plugin or https://plugins.jenkins.io/perfpublisher .

Once @Nancy-Chauhan provides information which plugin should be used, we need to get it installed on ci.librecores.org and added to the Docker image

Add support of packaging LibreCores CI as a WAR/Docker file using Custom WAR Packager

Currently the hosted instance of LibreCores CI runs without Docker due to the Docker-in-Docker limitations I wanted to avoid at that point. Nowadays there is a Custom WAR Packager, which can package both WAR files and Docker images.

It would be interesting to adopt the Custom WAR Packager build flow in the repository so that we can deliver the patches to ci.librecores.org from this repo.

Acceptance criteria:

• There is a Jenkinsfile added to the repository
• ci.librecores.org is configured to build a job using the Jenkinsfile
• The self-build flow is offered as one of the demos in this repository

Migrate prototype flows from the PoC CI instance

We need to migrate existing automation from http://ec2-52-209-86-82.eu-west-1.compute.amazonaws.com/job/Projects/ .

• FuseSoC
• sockit
• wb_sdram_ctrl

Setup External Workspace Manager for CI builds

External WS manager allows passing workspaces between different VMs, which may be useful for virtualized environments, where each Docker image provides only a couple of tools.

• Create a Docker volume for shared workspaces
• Configure Docker Cloud to map this volume and to use it as a workspace
• Adjust several jobs to exws() calls

https://github.com/jenkinsci/external-workspace-manager-plugin/

Add resource utilization restrictions to Docker stuff

There is a risk that particular Docker images may kill the instance.

Enable Dependabot for the repository

In #56 the repository has been migrated to the Custom WAR Packager build flow. Now we can use Dependabot to automatically suggest plugin updates.

Bonus points for better CI in this repository before it is implemented

Create Kubernetes specs for LibreCores CI

Currently LibreCores CI runs in Docker, and it causes some problems in agent management. It would be cool to also have a K8s mode so that we avoid issues like Docker-in-Docker. Later it can be also used to automate deployments of LibreCores CI on a hosted instance.

Acceptance criteria:

• LibreCores CI can still run in Docker
• There are K8s POD definitions for LibreCores CI

Upgrade LibreCores CI to 2.19.3 and Disable CLI

Just an ongoing maintenance work to address SECURITY-360 flaw. Previously it has been worked around in a different way

Update LCCI Docker Image to 2.73.2 + bump plugins

The current snapshot is outdated, and misses important things like security patches (the public LCCI instance is updated). Ideally the configuration needs to be bumped.

Setup Deployment flow for LibreCores CI

Once #47 is done, we may create an automated job which builds and deploys LibreCores CI WAR on the ci.librecores.org

Acceptance Criteria:

• There is an external or internal flow which interacts with ci.librecores.org and deploys new WAR/Docker versions there
• Flow:
  • Configuration Management tool takes a new build from a flow in #47
  • Make a backup of the instance
  • Replace the configuration by a new one
  • Restart the instance
  • Verify that Jenkins starts up, rollback otherwise

Implementation could be similar to https://github.com/librecores/librecores-web:

• Ansible scripts/definitions to manage the deployment

An alternative would be to move ci.librecores.org into a Docker container and to manage it using Docker and DockerCompose/Kubernetes features. Kubernetes would be preferable

Add a secondary CI/CD engine to LCCI

Currently the repo is being vuilt by ci.librecores.org. While it is good to build CI systems by themselves, it is nice to have a backup plan if something goes wrong. I suggest we have a build on Travis or other CI system as Plan B

Setup Log Parser plugin and provide reference parsers from the repo

• - Define several default log parsers ("default" and specific parsers for tools)
• - Parsers should be stored in this repo
• - Instance should automatically pick parsers

Setup ownership-based Security Strategy

This strategy is required for Fine-grain permission management in Project folders.

Concept:

• Each project has one primary and several secondary owners
• The owners are being defined during project registration in LibreCores Web (GitHub accounts)
• Owners automatically get permissions on CI using backend scripts or CLI

See https://wiki.jenkins-ci.org/display/JENKINS/Ownership-Based+security

LCCI Groovy script tries to load scripts as folders... and fails

GroovyInitBootstrap should not load folders as Groovy scripts

INFO: Executing bundled script: file:/var/jenkins_home/war/WEB-INF/init.groovy.d/org/
Jun 15, 2019 7:00:00 PM jenkins.model.Jenkins$5 runTask
WARNING: GroovyInitScript.init failed perhaps due to plugin dependency issues
java.lang.Error: java.lang.reflect.InvocationTargetException
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1091)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        ... 8 more
Caused by: java.lang.ClassFormatError: Illegal class name "" in class file
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at groovy.lang.GroovyClassLoader.access$400(GroovyClassLoader.java:62)
        at groovy.lang.GroovyClassLoader$ClassCollector.createClass(GroovyClassLoader.java:500)
        at groovy.lang.GroovyClassLoader$ClassCollector.onClassNode(GroovyClassLoader.java:517)
        at groovy.lang.GroovyClassLoader$ClassCollector.call(GroovyClassLoader.java:521)
        at org.codehaus.groovy.control.CompilationUnit$17.call(CompilationUnit.java:834)
        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
        at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584)
        at jenkins.util.groovy.GroovyHookScript.execute(GroovyHookScript.java:136)
        at jenkins.util.groovy.GroovyHookScript.execute(GroovyHookScript.java:119)
        at jenkins.util.groovy.GroovyHookScript.run(GroovyHookScript.java:89)
        at hudson.init.impl.GroovyInitScript.init(GroovyInitScript.java:41)
        ... 13 more

Split Jenkins server and LibreCores CI SaaS repositories

Right now we have everything in a single repository, and it is hard to maintain it. My plan is to split out the SaaS repo to the separate repo. Likely I will start there from Helm charts so that we can have a more modern deployment.

• Create a new repo for SaaS: https://github.com/librecores/ci.librecores.org
• Move the top-level description
• Move the GitHub bot documentation to the new repo

Each item can be handled in separate PRs by different contributors

Shrink down the instance size

We do not scale Jenkins master much right now, so we could reduce the instance size for a while. Other option is to convert the instance to a permanent one

Update FuseSoC demos to the recent versions of FuseSOC and tools

All demos were created in 2017, and they are quite outdated now. It would be great to update Docker images and the LibreCores Pipeline Library so that we run demos with recent versions of the tools.

• https://github.com/librecores/librecores-pipeline-lib
• https://github.com/librecores/docker-images
• https://github.com/olofk/fusesoc/releases/tag/1.9 . Other tools also may need an update

Nice2Have: Connect a bare-metal node with FPGA to the instance (for OPTiMSoC)

TODO:

• - Define the scope
• - Define labeling policy and desired utilization restrictions
• ...

Add TAP and Yosys code parsing & publishing steps to LibreCores Pipeline library

Currently, in the OpenRISC project, CI is in progress :

1. It involves Yosys Synthesis (openrisc/mor1kx#87 ) which requires performance plugin later to visualize the results graphically.
2. Currently, the Jenkins pipeline checks for the or1k-tests and gives result as success or failure, that can be represented in tap format and later tap plugin of Jenkins can be used to visualize the results

After these steps, we want to add these steps to Librecores pipeline library

Create a simple Jenkinsfile to have a pull request builder for the repository

It would be great to have some basic CI for the LibreCores CI repository. We can set it up on .ci.librecores.org. In order to do that, we need...

• A simple Jenkinsfile, which builds the repo on a host with Docker
• A job should be created on ci.librecores.org . All credentials should be already in place

Easier LibreCores CI Slave Setup

Overall Goal

For LibreCores CI, we want to give users the ability to easily hook their machines into the CI system to run CI on their own project.

Why?

• Users have the required tools setup (that we cannot provide for legal, license and money reasons)
• FOSSi does not need to pay for all CI instances

Workflow: Set up a new slave node (rough first idea)

prerequisite: User "activates CI" for his project on librecores: essentially a web page on LibreCores (CI) where CI is enabled for a given project.

1. User downloads a installable package from the LibreCores CI page
2. User installs this package on his/her machine. The installation results in a daemon being installed.
3. As part of the installation process, the user registers this node to build "CI for project XYZ on LibreCores"
4. Jenkins master registers required build triggers with Git or librecores (details to be figured out)

Now whenever a build trigger happens, the node builds & tests the code, and uploads the results (most likely no build artifacts for license reasons) to the Jenkins master.

Constraints

• Slaves should not be required to be "always on". Users may turn off their slave nodes, run them only once a day, etc.
• Many organizations have complex firewall or proxy setups. The only communication that reliably works in such cases are HTTPS requests, initiated from the slave itself. For us, this means:
  • all communication between slave and master runs over a encrypted HTTP (HTTPS) API
  • all communication is slave-initiated: polling for new build/test jobs, status reporting, uploading of build results.

Related work:

• GitLab CI Runner: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner

Move view definitions to jenkins.yaml

Once https://issues.jenkins-ci.org/browse/JENKINS-52311 is fixed, we can move view definitions from Groovy hooks to jenkins.yaml

Prepare a short demo (up to ten minutes)

• Runtime modification of project
• PR builder with status update
• "Successful onboarding path"
• (?) Hosted CI for hardware projects and maybe EDA
  • EDA is not interesting for us unless there runs on hardware && we use it in the ecosystem

Setup Docker on the instance

We need to setup Docker or Docker Swarm in order to run on-demand CI builds in https://github.com/oleg-nenashev/fusesoc-docker containers

Define License for the repository

According to @olofk , MIT is fine for such infrastructure code. It does not have IP in any case, hence it should be safe

LibreCores CI Styling

• - Create a custom theme Simple Theme plugin
• - Deploy this theme to the repo and then setup automatic deployment