rcsb/devops-cicd-github-actions

Repo to test CI/CD pipelines and store reusable workflows

devops-cicd-github-actions's Issues

Build and push docker images on every branch tagged with branch name

subj

Implement integration tests

subj

War bundle needs a different handling than tgz bundle for buildlocker upload

Our war files are single all-in-one bundles. That means for the war case, we just need to upload the war file already produced by maven (see any of the old gitlab CI scripts).

I guess we are going to need an if condition somewhere here:

https://github.com/rcsb/devops-cicd-github-actions/blob/master/.github/workflows/deploy-to-buildloker.yaml#L27-L29

Give a try to GitHub Action's cache

https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

A few warnings when running push2master

When running push-to-master in rcsb-shape a few warnings came up, see https://github.com/rcsb/rcsb-shape/actions/runs/4199106580

@Ingvord could you have a look at resolving them?

Move callable actions to a dedicated repo

Perhaps devops-runner-images?

Check if we can dispatch to actions outside this repo, hence move those actions

Consider multi module maven projects

Turns out we do have a few multi module maven projects. There is a couple of ways to deal with them:

1. get rid off them
2. alter GitHub actions workflow to be aware of those

I would personally go with the first option, as frankly speaking maven multi module add very little value (if any), but introduces a lot of maintenance pain, for instance when one project produces mutiple artifacts. That simply violates the General rule of thumb - single artifact per repo

Second option requires some research...

Builds pushed to build-locker without dependency jars included

The current step when archiving a .jar build for BuildLocker only includes the jar file: https://github.com/rcsb/devops-cicd-github-actions/blob/master/.github/workflows/deploy-to-buildloker.yaml#L27

In comparison, the old GitLab CI/CD pipeline will create an archive file that also includes the dependency-jars/ directory: https://github.com/rcsb/rcsb-everglades/blob/master/.gitlab-ci.yml#L22

Without that dependency directory, the weekly update workflow tasks which call the everglades Java library will fail due to missing dependencies:

A task failed when running. Most likely run() raised an exception.
name	etl.create_indexes.CreateTermsAutocompleteIndex
contentType	computational
Command line
/usr/local/bin/luigi --module weekly_update.etl.end RunETLWorkflow --ETLWorkflowConfig-appendCoastAndChannelToConfigLoc --MasterConfig-whatCoast east --MasterConfig-appendCoastToStateFilesBasePath --MasterConfig-appendCoastToFtpFileBasePath --scheduler-host etl-east.rcsb.org --workers 1
Traceback
Runtime error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 133, in _run_get_new_deps
    task_gen = self.task.run()
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/etl/create_indexes.py", line 141, in run
    run_everglades_java_command("org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete", self.contentType)
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/etl/redwood_functions.py", line 102, in run_everglades_java_command
    return run_loader_java_command(ETLWorkflowConfig().indexerProjectName,
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/etl/redwood_functions.py", line 83, in run_loader_java_command
    run_command(cmd)
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/utils/general_utils.py", line 309, in run_command
    raise ValueError("Command '%s' failed with return %d. The stderr follows:\n%s"
ValueError: Command '/usr/bin/java -Xmx32g -DrcsbConfigProfile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/ -Dlog4j.configurationFile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/log4j2.everglades.xml -DlogFileName=/var/log/rcsb/everglades.CreateTermsAutocomplete-computational.log -cp /tmp/wuw_CreateTermsAutocomplete_n8pdddqh/rcsb-everglades-1.11.10-SNAPSHOT.jar org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete -ct=computational' failed with return 1. The stderr follows:
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
	at org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete.(CreateTermsAutocomplete.java:28)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 1 more

The solution should be to include that directory as part of the deploy-build-locker-jar job step, but I'm not sure if this would impact other repos that call this workflow or not. FYI @Ingvord

Enable `concurrency`

Following @JonStargaryen advice.

It would be nice to automatically cancel currently running jobs, if a new one has to start due to e.g. multiple fast sequential commits/pushes.

See GHA concurrency

Integrate surefire reports action to display tests result in PR/Action

subj

Define manual workflow RELEASE that will trigger release process

currently .github/workflows/commit-to-branch.yaml