Create AQAvit certification request template and workflow

[adoptium]

Create AQAvit certification request template and workflow

[llxia]

I would suggest the vendor calls an API to register first. We can collect the basic information, such as vendor info, contact info, etc. A token will be sent to the vendor (i.e., JWT). The vendor should use this token when requesting certification. Only vendors with valid token can request certification. This process will help us verify the vendor, allow the vendor to update if needed, keep track of who requested the certification and avoid malicious access.

[smlambert]

While the template / workflow will actually live in the aqa-tests repository, the ultimate goal is to make this a very automated process (even if near-term is a somewhat manual process).

A high-level view of the automated process will be:

• vendor pushes a request to an API (likely to TRSS) for a certification request. This request contains at least one but possibly multiple sections, each containing:
  • URL to the binary that the request is for
  • platform for the binary under request (while we can autodetect this, knowing it upfront allows us to direct audit tests to the right machines without having to parse non-standard named files or loop through machines searching for where autodetect succeeds)
  • URL to the source code repo(s) (and SHA/tag) or source.zip used to create the binary
  • URL to CI server where tests run or URL to where test results artifacts are stored
• request triggers the creation of a certification request issue in the aqa-tests repository (or some repository specifically created for certification requests and auditing)
• link to certification request issue is returned to vendor
• certification request contents are parsed
  • test result artifacts are downloaded from supplied URL
  • test result artifacts are reviewed (these artifacts include the TAP summary of what test targets were executed, which passed, failed, were skipped and which were excluded)
  • check: SHA.txt artifact matches expected AQAvit SHAs
  • check: number of test targets executed matches expected number (100% of expected test targets to have been executed)
  • check: 98% pass rate or higher expected (with failing targets passing on retry)
  • check: test exclusions each are linked to known trackable non-blocking open issues
  • depending on passed/failed/excluded report, (optional) audit/rerun of subset of test targets are run against the binary
  • check audit/rerun results
• push result of checks listed above to comment in the certification request issue
• if all checks pass, certification is granted (cert request issue is updated with comment and closed)
• if some checks fail, manual intervention and checking occurs (AQAvit project team updates the cert request issue on next steps)
• automated certification request results can be overruled by manual intervention and verification of binaries by AQAvit project team (should there appear to be issues with automated results)

[smlambert]

re: #21 (comment) yes, was going to build off of the jwt PR and & prototype that you and Renfei worked on for TRSS.

[smlambert]

See: https://github.com/adoptium/website-v2/blob/main/src/asciidoc-pages/docs/aqavit-verification.adoc