jenkins-infra/crawler

tools crawler

Infra Backend Crawlers

These scripts generate machine readable tool installer metadata. You need Groovy 2.0 or later to run them. You can add new one by doing monkey-see monkey-do. Any one of these scripts are individually runnable, for example:

groovy -Dgrape.config=./grapeConfig.xml ./lib/runner.groovy mongodb.groovy

Signing Files

If you need to "only" sign the JSON files (after an update center certificate renewal for instance):

mkdir -p ./target/
rsync -av "${source}/*.json" ./target/
export JENKINS_SIGNER="-key path/to/uc-cert.key -certificate path/to/uc-cert.cert -root-certificate path/to/uc-root-ca.crt"
groovy -Dgrape.config=./grapeConfig.xml ./lib/runner.groovy ./lib/signer.groovy
# You can find the signed files in ./target/*.html

crawler's Issues

groovy.groovy retrieve only versions with single digit part

The regexp used to filter versions for Groovy accept only one digit in each version part x.y.z:

Not able to run the the selenium Test cases on chrome driver version121

Reproduction steps

Currently jenkins supports only 114 version (https://updates.jenkins.io/updates/org.jenkins-ci.plugins.chromedriver.ChromeDriver.json)

so latest version is not getting picked

Expected Results

It has to pick the latest version of chrome driver while executing .

https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json

Actual Results

https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json

Anything else?

No response

scriptler.groovy fails since 2022-03-12

Noticed in jenkins-infra/helpdesk#2950

Caught: java.io.FileNotFoundException: /home/jenkins/workspace/crawler/./jenkins-scripts/scriptler
java.io.FileNotFoundException: /home/jenkins/workspace/crawler/./jenkins-scripts/scriptler
at scriptler.run(scriptler.groovy:18)
at runner$_run_closure1.doCall(runner.groovy:13)
at runner.run(runner.groovy:10)

Use `main` as primary branch

Suggestion

We should use main instead of master as primary branch.

This will need updating not only this repo but also all references to master in Jenkinsfile(s) and job on trusted.ci.jenkins.io

Links

• jenkins-infra/helpdesk#2671

groovy.groovy fails

While working on jenkins-infra/helpdesk#2950 we noticed the script https://github.com/jenkins-infra/crawler/blob/master/groovy.groovy to retrieve last groovy versions fails with the following error, preventing it to be signed and updated on Update Center.

12:00:44 + export JENKINS_SIGNER=-key ****/update-center.key -certificate ****/update-center.cert -root-certificate ****/jenkins-update-center-root-ca.crt
12:00:44 + groovy -Dgrape.config=./grapeConfig.xml ./lib/runner.groovy groovy.groovy
12:02:22 loading dependencies...done
12:02:22 Caught: com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: "fetch" is not defined. (https://groovy.jfrog.io/ui/externals/systemjs/dist/s.min.js#1)
12:02:22 ======= EXCEPTION START ========
12:02:22 EcmaError: lineNumber=[1] column=[0] lineSource=[] name=[ReferenceError] sourceName=[https://groovy.jfrog.io/ui/externals/systemjs/dist/s.min.js] message=[ReferenceError: "fetch" is not defined. (https://groovy.jfrog.io/ui/externals/systemjs/dist/s.min.js#1)]
12:02:22 com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: "fetch" is not defined. (https://groovy.jfrog.io/ui/externals/systemjs/dist/s.min.js#1)
12:02:22 at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:883)

Ref: jenkins-infra/helpdesk#2950 (comment)

Fix the JDK crawler and get rid of the hack from INFRA-2345

Reproduction steps

Notice the use of a hack for the JDK crawler:

crawler/jdk.groovy

Lines 29 to 31 in 5b0906a

	// HACK HACK HACK INFRA-2345
	// Temporary bandaid to give up scraping and use a snapshot
	return JSONObject.fromObject(new File("jdk.hack.json").text)

jdk.hack.json is a snapshot of the last successful run dated from November 2019.

Expected Results

No hack, JDK retrieved from Oracle website or other source of truth.

Actual Results

The returned JDK list dates from November 2019.

Anything else?

Source of the hack:

• jenkins-infra/helpdesk#1964 (comment)

Add paging or sensible page size for GitHub release crawlers

Reproduction steps

• Use a Jenkins plugin that utilizes tools and provides and "Install from GitHub" option
• The tool has more than 30 released versions returned from GitHub API

---

• execute crawler script locally that uses github API to list releases
• if the github repo has more than 30 releases, only last 30 will be listed in the result

Expected Results

Seing all releases of the said tool.

Actual Results

Only maximum of 30 releases are shown for selection

Anything else?

After checking all crawlers, the following might be affected:

• codeql.groovy
• dependencycheck.groovy (seemingly aware of this issue, only one to mention the "per_page" parameter)
• leiningen.groovy
• sonarqubescannermsbuild.groovy (already submitted PR #137 )

codeql.groovy fails

Noticed in jenkins-infra/helpdesk#2950

Caught: java.lang.NumberFormatException: For input string: "v2.6.0-beta.1"
java.lang.NumberFormatException: For input string: "v2.6.0-beta.1"