Testing changes before merging

[openshift-eng]

Testing changes before merging

[stevekuznetsov]

I assume you are talking about the sjb/ jobs. The current process is manual and described in #201. When we move to a Jenkinsfile process (eta: one or two sprints) we have aim to introduce a process. This is not a simple problem to solve but it is clearly on our radar and I don't think you'll find anyone who disagrees that a process would be helpful.

[rhcarvalho]

@stevekuznetsov thanks. We're on the same team/boat/whatever-we-call it. Understanding that yes this is already in the radar, with the best open source intentions, I'd like to bring the discussion upstream. This way we have more transparency and allow for contribution from a broader audience.

I meant the repo as a whole, not only jobs that affect openshift-ansible.

From #201:

In order to test a job, it is necessary to copy a configuation file under sjb/config/ to a new YAML file with a different name, then re-generate XML and use sjb/push-update.sh to push only the test job up to the server. Cleanup of these jobs post-test is still manual.

• While other PRs merge quite fast, I wonder why is this still open? It's hard for someone landing on the repo homepage to figure out there will be some docs coming up in a PR :)
• Assuming those steps are being followed (though I haven't seen e.g. a link to a successful job execution in the PRs I've seen here), have anyone done a post-mortem analysis to understand why the recent bugs were introduced?
• Is there any initial plan for the 'inception', automating the verification of changes to this repo itself?

[rhcarvalho]

While other PRs merge quite fast, I wonder why is this still open? It's hard for someone landing on the repo homepage to figure out there will be some docs coming up in a PR :)

Sorry, now I see it was created only 38 min ago.

[stevekuznetsov]

While other PRs merge quite fast, I wonder why is this still open? It's hard for someone landing on the repo homepage to figure out there will be some docs coming up in a PR :)

I made this in the morning and I don't think Justin has been in yet to look at it.

Assuming those steps are being followed (though I haven't seen e.g. a link to a successful job execution in the PRs I've seen here), have anyone done a post-mortem analysis to understand why the recent bugs were introduced?

Yes, it's a mix of the process not being followed, being followed poorly, differences between "test" jobs and "real" jobs (when something has been tested in isolation but broke in the middle of a larger job) or simple mistakes or oversight.

Is there any initial plan for the 'inception', automating the verification of changes to this repo itself?

It is a tricky question. From my understanding, no current system supports testing for the CI underpinnings. Historically Origin and AOS have not had a suite of tests for many of the tools that underpin the CI and definitely not for the jobs. I was actually the first to add or attempt to add automated or semi-automated testing to tools like test-pull-requests and vagrant-openshift.

From what I can tell, there is not really a first-class concept of testing in the JJB sphere either, other than simply running verification steps to ensure that the syntax has been followed. Even in the Groovy Jenkins pipeline/library ecosystem on large projects there is no CI.

It's just not a very common thing to do, testing the libraries that underpin CI.

I don't think it will be reasonable to have per-PR CI on the changes that make it into the job definitions, therefore. Instead, we will feed two servers with our jobs. Changes land in master branch and go live immediately on a testing server that mirrors some small representative portion of the workload hitting the production service and we promote changes into other branches (versioned with the branch of origin/openshift-ansible they support, so like integration, staging, enterprise-3.5, etc) once they have proven themselves in the test master.

[thiagoalessio]

@rhcarvalho @stevekuznetsov would you guys be ok if I close this issue? Is so old, a lot has changed since then (i guess)