jfrog/artifactory-scripts

Scripts for Artifactory (Usually, for REST API), community driven.

artifactory-scripts's Issues

Cleanup aborts when maven metadata is missing.

When running dryRun=true, the cleanup lists the work it would do. When running without dryRun, if a project is missing a maven-metadata.xml (lets say from a failed deploy), the script will abort.

I can generate the metadata for an individual artifact by POSTing to /api/maven/calculateMetadata/..repoId../..groupId../..artifactId.. However, I wind up needing to do this for every artifact in a multimodule project to get the cleanup to work.

docker_fetch_markers returns 404

I've tried to run the script but it returned 404 codes. To see more details, I've added set +x to the script and runned it again:

❯ ./docker_marker.sh
+ echo 'Enter your Artifactory URL: '
Enter your Artifactory URL:
+ read Source_ART_URL
https://[MASKED]/artifactory
+ SOURCE_ART=https://[MASKED]/artifactory
+ echo 'Enter your docker remote repository name without the '\''-cache'\'' suffix: '
Enter your docker remote repository name without the '-cache' suffix:
+ read Source_repo_name
registry-gitlab-com
+ echo 'Enter username with Read and Deploy permissions to the above repository in Artifactory: '
Enter username with Read and Deploy permissions to the above repository in Artifactory:
+ read source_username
[MASKED]
+ echo 'Password for Artifactory: '
Password for Artifactory:
+ read -s source_password
+ echo

+ REMOTE_REPO=registry-gitlab-com-cache
++ curl -X POST -sS -u[MASKED] --write-out '%{http_code}' --silent --output /dev/null https://[MASKED]/artifactory/api/search/aql -d 'items.find({"$and": [{"repo" : "registry-gitlab-com-cache"}, {"name" : {"$match" : "*.marker"}}]})' -H 'Content-Type: text/plain'
+ status_code=200
+ [[ 200 -eq 401 ]]
+ [[ 200 -eq 000 ]]
+ [[ 200 -eq 404 ]]
+ [[ 200 -eq 400 ]]
+ [[ 200 -ne 200 ]]
+ curl -X POST -sS -u[MASKED] https://[MASKED]/artifactory/api/search/aql -d 'items.find({"$and": [{"repo" : "registry-gitlab-com-cache"}, {"name" : {"$match" : "*.marker"}}]})' -H 'Content-Type: text/plain'
+ jq -M -r '.results[] | "\(.path)/blobs/\(.name)"' marker_layers.txt
+ sed 's/[“,]//g' marker_paths.txt
+ sed 's|library/||g'
+ sed s/.marker//g
+ sed s/__/:/g
+ awk 'sub("[/][^,;/]+[/]blobs/", "/blobs/", $0)'
+ echo 'Here are the number of marker layers in this repository'
Here are the number of marker layers in this repository
+ echo

+ cat download_markers.txt
+ wc -l
      12
+ echo

+ echo 'Do you want to download these marker layers?(yes/no)'
Do you want to download these marker layers?(yes/no)
+ read input
yes
+ [[ yes =~ [yY](es)* ]]
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:28d445cfca767cb2a0dbdc152ed5eeed364db2b6353f8dedf8f917562c116b9b
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:2b7e0e3b754519331113e409b44b7401dda685cba747e0fbb619b907b85e65a6
+ read p
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:28d445cfca767cb2a0dbdc152ed5eeed364db2b6353f8dedf8f917562c116b9b -o /dev/null
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:2baf61d06a27504c4b8421899469ed523b29ea095f17de8c8a15248304ae4133
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:2b7e0e3b754519331113e409b44b7401dda685cba747e0fbb619b907b85e65a6 -o /dev/null
+ read p
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:2baf61d06a27504c4b8421899469ed523b29ea095f17de8c8a15248304ae4133 -o /dev/null
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:32eb6dfb922720a338efac9871565ad3ca1a3ac69b540f214e1224bc55191c69
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:493333e9491fba57ff1ea5b8058546fc5a3e124d325f8720c018568f852e385b
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:32eb6dfb922720a338efac9871565ad3ca1a3ac69b540f214e1224bc55191c69 -o /dev/null
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:493333e9491fba57ff1ea5b8058546fc5a3e124d325f8720c018568f852e385b -o /dev/null
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:4db90bcd796b9a31f9366d08b929131bdc58ba070f57d3195df477af51d0c898
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:4db90bcd796b9a31f9366d08b929131bdc58ba070f57d3195df477af51d0c898 -o /dev/null
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:5a38b3726f4b24fa93b80450be63ad67fd3239c2f3b83695118d7b1a88447d84
+ read p
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba -o /dev/null
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:7732b60e038bc1bd67df0ba16b60246242be428b5342f88f38a1a2a28cea09b6
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:5a38b3726f4b24fa93b80450be63ad67fd3239c2f3b83695118d7b1a88447d84 -o /dev/null
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:962e97023e1ad99b2f9ef422b8965d95386a547a3c39ca8d42504054e846b2cb
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:7732b60e038bc1bd67df0ba16b60246242be428b5342f88f38a1a2a28cea09b6 -o /dev/null
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:e5fa5deb334027202841b051d10e7c7137fa3b63e97734309cedf6b48804df5f
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:962e97023e1ad99b2f9ef422b8965d95386a547a3c39ca8d42504054e846b2cb -o /dev/null
+ read p
+ prefix=https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:f75ff45c06b697172607d2d8278f8200f3d0a803e1153ab02b2751061b834175
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:e5fa5deb334027202841b051d10e7c7137fa3b63e97734309cedf6b48804df5f -o /dev/null
+ read p
+ wait
+ curl -sS -u[MASKED] -w 'HTTP/1.1 %{http_code} | %{time_total} seconds | %{size_download} bytes\n' https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:f75ff45c06b697172607d2d8278f8200f3d0a803e1153ab02b2751061b834175 -o /dev/null
HTTP/1.1 404 | 0.073892 seconds | 87 bytes
HTTP/1.1 404 | 0.074858 seconds | 87 bytes
HTTP/1.1 404 | 0.091974 seconds | 87 bytes
HTTP/1.1 404 | 0.082049 seconds | 87 bytes
HTTP/1.1 404 | 0.073985 seconds | 87 bytes
HTTP/1.1 404 | 0.081994 seconds | 87 bytes
HTTP/1.1 404 | 0.081283 seconds | 87 bytes
HTTP/1.1 404 | 0.084249 seconds | 87 bytes
HTTP/1.1 404 | 0.078050 seconds | 87 bytes
HTTP/1.1 404 | 0.083284 seconds | 87 bytes
HTTP/1.1 404 | 0.087073 seconds | 87 bytes
HTTP/1.1 404 | 0.087608 seconds | 87 bytes
+ rm marker_layers.txt marker_paths.txt download_markers.txt
+ [[ yes =~ [nN](o)* ]]

When I try to call one of the URLs from the terminal, I receive the following response:

❯ curl -u[MASKED] https://[MASKED]/artifactory/api/docker/registry-gitlab-com/v2/gitlab-org/cluster-integration/auto-build-image/blobs/sha256:f75ff45c06b697172607d2d8278f8200f3d0a803e1153ab02b2751061b834175 
{
  "errors" : [ {
    "status" : 404,
    "message" : "authenticationrequired"
  } ]
}%

[ERROR] java.lang.NoClassDefFoundError: org/apache/ivy/core/report/ResolveReport

Hi,

I wanted to try this groovy script on my Jenkins but I got this error.
Error:
java.lang.NoClassDefFoundError: org/apache/ivy/core/report/ResolveReport
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
at java.lang.Class.getDeclaredMethods(Class.java:1855)
at org.codehaus.groovy.reflection.CachedClass$3$1.run(CachedClass.java:84)
at java.security.AccessController.doPrivileged(Native Method)
at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:81)
at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:79)
at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:46)
at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:33)

Do you have any idea what the problem is here? Or missing something in Jenkins?

Thanks and Regards,
Gabor

_ElementInterface instance has no attribute 'iter' error

When running the packageCheck.py script with Python 2.6.9, I've been running into the following error message:

$ ./packageCheck.py http://localhost:8080/artifactory
Getting repository list
Authorization required
Username: admin
Password: 
Traceback (most recent call last):
  File "./packageCheck.py", line 178, in <module>
    try: sys.exit(PackageCheck(getargs()).status)
  File "./packageCheck.py", line 33, in __init__
    for key, types in self.getRepoList(args, auth):
  File "./packageCheck.py", line 124, in getRepoList
    for repo in root.iter(ns + name + "Repository"):
AttributeError: _ElementInterface instance has no attribute 'iter'

The fix for me was to change line 124 from:

            for repo in root.iter(ns + name + "Repository"):

to:

            for repo in root.getiterator(ns + name + "Repository"):

[filestoreIntegrity] Add option to specify repo

Currently the script fetches and runs against all repos. Being able to run it one repo at a time, or just against specific repos, would be a nice feature.

AQL query pattern for Docker images

Hi Shay,

I hope you are doing great.

I wanted to use your following script in my Artifactory system for cleaning up the docker images.

https://github.com/JFrogDev/artifactory-scripts/blob/master/cleanup/aqlCleanup.groovy

Basically I wanted to retain only last 3 Docker images and delete rest of those.

However I am not able to find what pattern to use for Dockerimages, specifically in this line from your code:

def query = 'items.find({"type":"file","name":{"$match":"*.jar"}})' // replace this with your AQL query

Anything else you can suggest regarding cleaning up Dockerimages will be really helpful.

Thanks,
Giriraj

Move artifactory-cleanup.sh to cleanup directory

filestoreIntegrity.py changes the download date and time of the artifacts

Using the filestoreIntegrity.py script changes the last downloaded date and time of the artifacts which in turn would impact using the artifactCleanup plugin as this plugin is based on the last downloaded time.

filestoreIntegrity.py fails when encountering an artifact with a space in its name.

Script aborts and fails if it encounters an artifact with a space in its name:

File "/usr/lib64/python3.6/http/client.py", line 1127, in putrequest
raise InvalidURL(f"URL can't contain control characters. {url!r} "
http.client.InvalidURL: URL can't contain control characters. '/artifactory/test-repo-local/space file.txt' (found at least ' ')

Script requestToUsage.sh skips lines ending with zero, leading to wrong calculations

artifactory-scripts/requestToUsage/requestToUsage.sh

Line 10 in 732dc1c

awk '!/0$/' $FILE > $OUTPUT

I assume the intention was to skip lines with zero transfer bytes when log format was keeping that data at the end of log line.
Even then, this would remove valid lines if byte number will just end with 0 but would be greater than 0, like 1234567890.

Log format changed since then and now request.log entry has user agent at the end. That field may still end with 0.
Examples: curl/7.29.0, Artifactory/7.12.6 71206900.

Easiest workaround is to replace awk command with:
cat $FILE > $OUTPUT
It will copy all the lines from request.log into CSV file without cutting anything and script remains functional.

Use newer HAProxy config

In the Configuring haproxy load balancer in front of ha artifactory cluster question was requested how to migrate to a newer HAProxy version.

The original answer from JFrog Support is from 2018-02-05 How to configure HAProxy with Artifactory? which looks wrong because there are some missing characters.

# wrong
reqirep ^([^ :]*) /v2(.*$) 1 /artifactory/api/docker/docker/v22
# right
reqirep ^([^ :]*) /v2(.*$) \1 /artifactory/api/docker/docker/v22\1

I assume that the HAProxy config for the knowledge-base article is this file https://github.com/jfrog/artifactory-scripts/blob/master/proxyexamples/haproxy/haproxy-ports.cfg

In general should the config be adopted to the new HAProxy versions.
This means.

• replace reqadd with http-request set-header
• replace reqirep with http-request replace*

aqlCleanup.groovy build error: unable to resolve class

I downloaded the aqlCleanup.groovy in Github, and modified artifactoryURL ,then run this script show error :" unable to resolve class groovy.json.internal.LazyMap". Do I need to add other library?
Look forward to your reply.

unable to resolve class groovy.json.internal.LazyMap

Hi When I ran this script, I am seeing following error:

aqlCleanup.groovy: 51: unable to resolve class groovy.json.internal.LazyMap
@ line 51, column 22.
public constructPath(groovy.json.internal.LazyMap item) {
^

1 error

replicationDiff.sh is reporting differences which are not valid

Executed command:
./replicationDiff.sh -source_adminuser {user} -target_adminuser {user} -source_art http://{source-hostname}:8081/artifactory-target_art http://{target-hostname}:8081/artifactory-source_repo lib-release-local -target_repo lib-release-local-source_password *********** -target_password ************-download_missingfiles no

Result:

Here is the count of files sorted according to the file extension that arepresent in the source repository and are missing in the target repository.Please note that if there are SHA files in these repositories which will have no extension, then the entire URL will be seen in the output. The SHA files will be seen for docker repositories whose layers are named as per their SHA value.


  2 .zip
 16 .xml
 46 .jar

· And below is the output diff file contents:

o cat filepaths_uri.txt

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.0.0-v0492/{product}-inttest-cluster-10.0.0-v0492-javadoc.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.0.0-v0492/{product}-inttest-cluster-10.0.0-v0492-sources.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.0.0-v0492/{product}-inttest-cluster-10.0.0-v0492.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.0.0-v0492/ivy-10.0.0-v0492.xml

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.1.0-v0508/{product}-inttest-cluster-10.1.0-v0508-javadoc.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.1.0-v0508/{product}-inttest-cluster-10.1.0-v0508-sources.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.1.0-v0508/{product}-inttest-cluster-10.1.0-v0508.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-inttest-cluster/10.1.0-v0508/ivy-10.1.0-v0508.xml

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-webApp/9.12.0-v0226/{product}-webApp-9.12.0-v0226-javadoc.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-webApp/9.12.0-v0226/{product}-webApp-9.12.0-v0226-p2docmeta.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-webApp/9.12.0-v0226/{product}-webApp-9.12.0-v0226-sources.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-webApp/9.12.0-v0226/{product}-webApp-9.12.0-v0226.jar

http://{source-hostname}:8081/artifactory/lib-release-local/com/{company}/{product}/{product}-webApp/9.12.0-v0226/ivy-9.12.0-v0226.xml

... etc ...

However, comparing couple of them manually didn't indicate any actual differences.