cburgmer / buildviz Goto Github PK
View Code? Open in Web Editor NEWTransparency for your build pipeline's results and runtime
Home Page: https://buildviz.cburgmer.space/
License: BSD 2-Clause "Simplified" License
Transparency for your build pipeline's results and runtime
Home Page: https://buildviz.cburgmer.space/
License: BSD 2-Clause "Simplified" License
Working with outdated sync can be tedious and unintuitive as the build data slowly vanishes from the graphs, as newer dates are just empty if no new sync has been undertaken. In the case of the examples inside the repo, they just plainly don't show anything for historical builds, if they are e.g. a year back.
Rather let's just show the last x days/months offset from the latest build synched.
The GoCD sync should not fail on build artifact files that are not JUnit XML. The current heuristic of checking the filename is not good enough.
A simple magic check on top could suffice to weed out other XML files.
Workaround for 1575 will leave the job with a nil
value for start
, and will probably fail at the schema validation, now that start is required.
Only solution is probably to wholly ignore past builds from renamed items.
See e.g. hack in 4eeccf3
Currently a 400 error fails the script
If a job is removed or just renamed (id changes) and the last build failed, the fail phase goes on indefinitely as that job is never turned back into green.
Possible solution:
Fuzzy logic by which a certain interval without new builds means the job has been decommissioned.
In Go:
Given two failing jobs in a stage, when rerunning one of the failed one, subsequently turning into green, then the result of the stage will be incorrectly reported as passed.
Although http://llg.cubic.org/docs/junit/ mentions that testcase
time
is optional:
At least that might be the culprit why curl http://localhost:3000/testsuites | sort -rnt, -k5
returns a ghost entry.
Grouping by package structure will make it easier to find out where time is spent.
Go.cd (and other's maybe as well) first schedule a job after an external event (e.g. source change), and when a build node gets available assigns a job.
Including the delay between both in the job runtime in statistics is misleading, as an increased timing there indicates that build system might be overloaded, rather than the actual step taking more time. However monitoring the delay is important when planning to reduce cycle time.
As teams will likely end up with multiple independent sets of jobs we should probably support some kind of separation of those sets. This could be a complex setup of multiple pipelines or many simple pipelines of microservices.
Color coding is one important visual aspect, as jobs of one group should by identifiable.
The "fail phases" graph follows a pipeline model, aggregating multiple pipelines in one overview will conflate what's happening below.
Ideas:
/builds/:ns/:job/:build
What we don't want:
buildviz does not want to hold information for multiple teams, rather it should be easy to deploy multiple instances of it.
Pipeline triggers are modelled on stage level. Modelling this on job level doesn't seem to be straight forward.
transform_tests.clj:7 disallows colons in test names. We should be more lenient, considering that we are guessing anyways.
source_id
?job
(in responses) vs. job-name
in triggered-by
informationSimilar to the TeamCity sync process, don't print the password inside the URL to stdout/logs.
Although http://llg.cubic.org/docs/junit/ describes classname
as required, the JSON schema does not require a classname.
Right now multiple ones don't accumulate runtime but actually outcome is lower due to avg.
Gosync currently will ask buildviz for the newest job and start from there. However it will look only on a stage level, and compare the stage scheduled-time
, which is a few seconds earlier than the actual job building-time
.
Given gosync was started at time T, it would pick up all stages scheduled until that time. If this included only jobs building at T+2, then stages scheduled at T+1 would not be included on a subsequent sync. Instead gosync would start at T+2.
Offer CSV as alternate output so the statistics can be processed in other tools.
When builds are stacking up, the following job might be triggered by multiple builds:
"causes" : [
{
"shortDescription" : "Started by upstream project \"Deploy\" build number 161",
"upstreamBuild" : 161,
"upstreamProject" : "Deploy",
"upstreamUrl" : "job/Deploy/"
},
{
"shortDescription" : "Started by upstream project \"Deploy\" build number 162",
"upstreamBuild" : 162,
"upstreamProject" : "Deploy",
"upstreamUrl" : "job/Deploy/"
}
]
Currently the first cause is selected, any further ignored. This then will show shorter, partial pipeline runs in the pipeline graph.
The upcoming wait time graph will also not calculate the consequently longer wait times.
Subproject names in TeamCity include colons (e.g. Project :: Subproject
) which is invalid for file names under Windows.
There are two fields we can extract that information from:
"snapshot-dependencies": {
"build": [
{
"buildTypeId": "SimpleSetup_Test",
"href": "/httpAuth/app/rest/builds/id:46",
"id": 46,
"number": "14",
"state": "finished",
"status": "SUCCESS",
"webUrl": "http://localhost:8111/viewLog.html?buildId=46&buildTypeId=SimpleSetup_Test"
}
],
"count": 1
}
and
"triggered": {
"date": "20160508T071736+0000",
"details": "##triggeredByBuildType='bt3' triggeredByBuild='14'",
"type": "unknown"
}
snapshot-dependencies
seems to have the data in the format we need, but seems to have a bunch of problems:
triggered
seems to have its own set of issues:
/app/rest/buildTypes/bt3
for example.triggered
value will only call out the user action (type: user).Wiremock supports recording and replaying previous request patterns. This might enable end2end testing of the sync implementations without starting up the Vagrant boxes. This could also close the gap left when removing the previous sync through Wiremock in 9e70d51.
On going phase (whether red/green) is not shown. As the end of the phase is unknown, we could take the timestamp of the last synced build as a current value.
Show
Investigate how flaky tests are currently found.
Technically the algorithm does not need to rely on the build outcome, but just needs to look at all build pairs with same input, and find tests with different outcome.
This would make sure to find flaky tests even if the build still fails for other reasons.
Currently dots seem to trigger a 400 from buildviz server.
cctray.xml for Go.cd exports a "build" like pipeline :: stage :: job
, we should use the same for consistency. However that would be a breaking change.
For several inputs the ordering of the revision with source_id is important. Two same inputs in different order will not me recognised as the overall same build input.
Null values are not caught for required start
field. Fixed upstream probably, but not released: bigmlcom/closchema#35
The XML generated by JUnit 5 can be rejected by buildviz with a 400. The offending content looks like this:
<testcase name="my_test" classname="com.example.something" time="14,029.255">
</testcase>
The buildviz.jenkins.sync job fails with a String format exception if a percentage sign shows up in the base URL (say due to basic auth).
Provide less items in an overview, then only provide all for a "zooming in" (what ever that will mean).
http://cburgmer.github.io/buildviz/ci.jenkins-ci.org/#graph_averageTestRuntime has one burst per test class, and we can't shine with the hierarchical structure that we normally offer.
When storing job information, fail early for unknown parameters as to provide early feedback.
.. in Chrome
Given package com.example.a
and class a
in package com.example
, both accumulated test runtimes will collide when rendered in the 'Average test runtime' graph.
Showing older test names (or even a temporary rename which will result in a single datapoint and so be prone to outliers) does not provide value. Filtering those out might be a trivial and effective solution.
Let the sync jobs be identifiable, to give API providers some way of understanding their traffic.
List of user feedback
Sync should stop for first ongoing build so we can resume later.
Current example has a flaky build with count 1 and a test with > 100 flaky failures.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.