aquasecurity / harbor-scanner-trivy Goto Github PK
View Code? Open in Web Editor NEWUse Trivy as a plug-in vulnerability scanner in the Harbor registry
Home Page: https://goharbor.io
License: Apache License 2.0
Use Trivy as a plug-in vulnerability scanner in the Harbor registry
Home Page: https://goharbor.io
License: Apache License 2.0
Trivy requires read/write access to the locations configured via SCANNER_TRIVY_CACHE_DIR
and SCANNER_TRIVY_REPORTS_DIR
environment variables.
The adapter service should do a sanity check on init and fail fast, if it dosn't have enough permissions.
$ helm repo add aqua-helm https://helm.aquasec.com
$ helm search repo aqua-helm/harbor
NAME CHART VERSION APP VERSION DESCRIPTION
aqua-helm/harbor-scanner-aqua 0.10.0 0.10.0 Harbor scanner adapter for Aqua CSP scanner
aqua-helm/harbor-scanner-trivy 0.11.1 0.14.1 Harbor scanner adapter for Trivy
So far Trivy 0.1.6 is used by default. We should be able to easily upgrade to newer version.
Version should be also reflected in the /api/v1/matadata response. As of now it's hardcoded.
Currently this Scanner Adapter exposes the REST API for Harbor and uses Trivy standalone (mode) to perform scans.
We could extend it and expose also Trivy server's twirp endpoint to allow Trivy clients to connect to the harbor-scanner-trivy deployment as depicted below. We could actually rename harbor-scanner-trivy
to a more generic scanner-trivy
deployment suitable for integrations with Harbor and other clients.
(Twirp request)
Harbor core ---→ scanner-trivy -----------------> trivy-server
(HTTP REST Endpoint) (Twirp Endpoint)
↑ ↑
↑ ↑
(deployed by Helm) ↑ ↑
---------------------------------------------------------------------------
(prepared by a user) ↑ ↑
↑ (HTTP request) ↑ (Twirp request)
SDK/CLI ↑
for Pluggable Scanner API trivy-client
We only set TTL for Finished or Failed scan jobs. However, it might happen that a scan job fails and is never retried. Therefore we'll have stale scan jobs stored in Redis.
Hi,
After updating to harbor-scanner-trivy v0.10.0, we are seeing a sharp difference in the CVE report and count. Earlier we were running on v0.9.0.
Does this scanner internally use trivy?
From the commit history i could make out 0.9.0 was using trivy 0.6.0 and now it's using 0.7.0.
Some tests i ran:
trivy 0.6.0:
As you can see there is a difference in the individual counts, although vuln DB is the same.
CC: @svisagan83
New version of Trivy has some cool features, including support for more linux distributions. We should have it. @knqyf263
This is mainly to avoid GitHub rate limit errors in Harbor CI/CD environments
The handler should validate the incoming ScanRequest, enqueue it for processing, and return an identifier for fetching the corresponding ScanReport.
Do not bother with private registries. We have a dedicated issue for that #12
It could be nice if we can customized some parameters of the trivy
scan :
My use case is to provide
CRITICAL
and HIGH
severities and only with fixed vulnsAs scanner wraps trivy
under the hood, we could simply use trivy
env variables to do this :
TRIVY_SEVERITY
to set severitiesTRIVY_IGNORE_UNFIXED
to display fixed vulnerabilitiesOr introducing new env variables for scanner like for example ?
SCANNER_TRIVY_SEVERITY
SCANNER_TRIVY_IGNORE_UNFIXED
What do you think about this proposition ?
Currently harbor-scanner-trivy scans only for os vulnerabilities, even Trivy support for some application dependency checking.
When Trivy fails for some reason, the logs accessible from the Harbor web console sometimes do not contain all the information required for troubleshooting. Typically the logs of the scanner adapter have more info. For example, when we do misconfigure GITHUB_TOKEN, the following meaningless error is shown in Harbor
2020-03-13T15:54:22Z [INFO] [/pkg/scan/job.go:126]: Report mime types: [application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0]
2020-03-13T15:54:22Z [INFO] [/pkg/scan/job.go:153]: Get report for mime type: application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0
2020-03-13T15:54:24Z [ERROR] [/pkg/scan/job.go:235]: check scan report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0: running trivy wrapper: running trivy: exit status 1: : general response handler: unexpected status code: 500, expected: 200
Whereas in the adapter's logs we have:
{"exit_code":1,"image_ref":"core:8080/library/alpine@sha256:acd3ca9941a85e8ed16515bfc5328e4e2f8c128caa72959a58a127b7801ee01f","level":"error","msg":"Running trivy failed","std_err":"","std_out":"2020-03-13T15:56:36.134Z\t\u001b[34mINFO\u001b[0m\tUse your github token\n2020-03-13T15:56:36.134Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n2020-03-13T15:56:36.134Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n2020-03-13T15:56:36.392Z\t\u001b[31mFATAL\u001b[0m\tfailed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: GET https://api.github.com/repos/aquasecurity/trivy-db/releases: 401 Bad credentials []\n","time":"2020-03-13T15:56:36Z"}
I'd love to have linter configured that can be run locally withmake lint
as well as integrated with our CI pipeline. Even though our CI is using Travis CI, the configuration should be pipeline agnostic in case we want to switch to something else.
This would streamline release management and upgrades to Trivy. @knqyf263
For Clair Harbor was able to display the upstream vulnerabilities DB update timestamp. If possible we should support the same, i.e. add the harbor.scanner-adapter/vulnerability-database-updated-at
property to the metadata response.
curl -k https://harbor-scanner-trivy:8443/api/v1/metadata
{
"scanner": {
"name": "Trivy",
"vendor": "Aqua Security",
"version": "0.1.7",
},
"capabilities": [
{
"consumes_mime_types": [
"application/vnd.oci.image.manifest.v1+json",
"application/vnd.docker.distribution.manifest.v2+json"
],
"produces_mime_types": [
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0",
"application/vnd.scanner.adapter.vuln.report.raw"
]
}
],
"properties": {
"harbor.scanner-adapter/scanner-type": "os-package-vulnerability",
"harbor.scanner-adapter/vulnerability-database-updated-at": "2019-08-13T08:16:33.345Z",
"harbor.scanner-adapter/vulnerability-database-next-update-at": "2019-08-14T08:16:33.345Z"
}
}
Harbor will send robot account name and token as the value of registry authorization
property of ScanRequest. Those values should be set as TRIVY_USERNAME
and TRIVY_PASSWORD
envs.
Based on kube/harbor-scanner-trivy.yaml we should provide service and deployment templates and define configuration values to install adapter with helm install ./charts/harbor-scanner-trivy
skip_update
/data/trivy-adapter/trivy/db/trivy.db
error in vulnerability DB initialize: failed to open db: open /home/scanner/.cache/trivy/db/trivy.db: permission denied
but the trivy.db permission is -rwxr-xr-x 1 root root 128770048 Apr 10 01:00 trivy.db
With Harbor 2.0 things are much easier when it comes to deploying Trivy scanner adapter.
For online/offline installer it's just:
$ ./install.sh --with-trivy
For HA deployment with Harbor Helm chart, Trivy is enabled by default
I noticed that the API sever returns status code 302 in case the scan is not ready yet but I don’t think 302 is a good status code for that. For one, 302 means the requested resource resides temporarily under a different URI
which isn’t true by all means (the scan definitely resides under this URI). Yet, the bigger problem is that when the server returns 302, the http client automatically redirects to the url in the Location header and in this case, it is exactly the same URL. What I observed is that the HTTP client makes 10-20 redirects (depending on what I am using) at which point it returns something like maximum redirections reached
. Does the 302 status code come from some kind of specification or you chose it specifically for some reason? I think a 2xx status code can be better suited for such kind of response. Maybe 204 (no content)?
P.S. I'm talking about this part of the code.
I keep getting this from a fresh harbor installation I don't unfortunately know what its complaining about though.. is it pulling the image or ?
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:287]: {
"uuid": "348a8354-7a20-11ea-ad51-7aa0e75b9f43",
"name": "Trivy",
"description": "The Trivy scanner adapter",
"url": "http://harbor-harbor-trivy:8080",
"disabled": false,
"is_default": true,
"health": "healthy",
"auth": "",
"skip_certVerify": false,
"use_internal_addr": true,
"adapter": "Trivy",
"vendor": "Aqua Security",
"version": "v0.5.3",
"create_time": "2020-04-09T05:08:52.72567Z",
"update_time": "2020-04-09T05:08:52.725672Z"
}
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:276]: scanRequest:
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:287]: {
"registry": {
"url": "http://harbor-harbor-core",
"authorization": "[HIDDEN]"
},
"artifact": {
"namespace_id": 2,
"repository": "devops/alert",
"tag": "",
"digest": "sha256:7717b452552995583e2eb8563d8d40a310bbf6ec50344709c15d767b206196ac",
"mime_type": "application/vnd.docker.distribution.manifest.v2+json"
}
}
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:126]: Report mime types: [application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0]
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:153]: Get report for mime type: application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0
2020-04-09T05:15:32Z [INFO] [/pkg/scan/job.go:170]: Report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 is not ready yet, retry after 5 seconds
2020-04-09T05:15:37Z [ERROR] [/pkg/scan/job.go:235]: check scan report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0: running trivy wrapper: running trivy: exit status 1: 2020-04-09T05:15:34.272Z �[34mINFO�[0m Use your github token
2020-04-09T05:15:34.272Z �[34mINFO�[0m Need to update DB
2020-04-09T05:15:34.272Z �[34mINFO�[0m Downloading DB...
2020-04-09T05:15:36.122Z �[34mINFO�[0m Reopening DB...
2020-04-09T05:15:36.207Z �[31mFATAL�[0m unable to initialize the docker scanner: unable to initialize a image struct: failed to initialize source: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password
: general response handler: unexpected status code: 500, expected: 200
Hi,
We are seeing below duplications in the scan results. And also a discrepancy between scans done short intervals apart. These two were done less than a minute apart.
First scan:
scanner version 0.10.0
CC: @svisagan83
Regards.
Hi.
I am attempting to start up the trivy container.
It doesn't seem to fail, but has many error messages.
Snippet from pod logs
ERROR: requeuer.process - EOF
ERROR: worker.fetch - EOF
ERROR: requeuer.process - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
Debug and Trace Level Logs switched on
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: trivy
labels:
app: trivy
spec:
replicas: 1
serviceName: trivy
selector:
matchLabels:
app: trivy
template:
metadata:
labels:
app: trivy
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10000
fsGroup: 10000
automountServiceAccountToken: false
containers:
- name: trivy
image: goharbor/trivy-adapter-photon:v2.1.2
imagePullPolicy: IfNotPresent
securityContext:
privileged: false
allowPrivilegeEscalation: false
env:
- name: "SCANNER_LOG_LEVEL"
value: "trace"
- name: "SCANNER_TRIVY_CACHE_DIR"
value: "/home/scanner/.cache/trivy"
- name: "SCANNER_TRIVY_REPORTS_DIR"
value: "/home/scanner/.cache/reports"
- name: "SCANNER_TRIVY_DEBUG_MODE"
value: "true"
- name: "SCANNER_TRIVY_VULN_TYPE"
value: "os,library"
- name: "SCANNER_TRIVY_GITHUB_TOKEN"
valueFrom:
secretKeyRef:
name: trivy
key: gitHub.token
- name: "SCANNER_TRIVY_SEVERITY"
value: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
- name: "SCANNER_TRIVY_IGNORE_UNFIXED"
value: "false"
- name: "SCANNER_TRIVY_SKIP_UPDATE"
value: "false"
- name: "SCANNER_TRIVY_INSECURE"
value: "false"
- name: SCANNER_API_SERVER_ADDR
value: ":8080"
- name: "SCANNER_REDIS_URL"
valueFrom:
secretKeyRef:
name: trivy
key: redis.url
- name: "SCANNER_STORE_REDIS_URL"
valueFrom:
secretKeyRef:
name: trivy
key: redis.url
- name: "SCANNER_JOB_QUEUE_REDIS_URL"
valueFrom:
secretKeyRef:
name: trivy
key: redis.url
ports:
- name: api-server
containerPort: 8080
volumeMounts:
- name: data
mountPath: /home/scanner/.cache
subPath:
readOnly: false
livenessProbe:
httpGet:
scheme: HTTP
path: /probe/healthy
port: api-server
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 10
readinessProbe:
httpGet:
scheme: HTTP
path: /probe/ready
port: api-server
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 200m
memory: 512Mi
volumeClaimTemplates:
- metadata:
name: data
labels:
app: trivy
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: "5Gi"
Implement logic for processing enqueued ScanJobs, i.e. exec trivy, parse the output report, map it to Harbor vulnerability report, and finally persist.
I just deployed Trivy with my harbor environment and everything seems to be working fine in the portal, except for one thing.
I'm trying to scan a local image using the trivy server that was deployed in my harbor server.
# trivy client --remote https://harbor-trivy.mydomain:8443 image:tag
2020-02-19T18:00:00.291Z FATAL error in image scan: failed to scan the image:
failed to detect vulnerabilities: failed to detect vulnerabilities via RPC:
twirp error bad_route: Error from intermediary with HTTP status code 404 "Not Found"
The address harbor-trivy.mydomain is a cname I created that points to the load balancer helm created.
I installed with the following params:
helm install harbor-scanner-trivy ./helm/harbor-scanner-trivy \
--namespace harbor-test \
--set service.port=8443 \
--set scanner.api.tlsEnabled=true \
--set scanner.api.tlsCertificate="`cat tls.crt`" \
--set scanner.api.tlsKey="`cat tls.key`"
Can you pls tell me if I am doing something wrong or if this function (trivy client --remote) is not available for this kind of deployment?
We're using Cilium with default-closed egress, so we have to explicitly allow all egress traffic. A quick way to get started is usually by looking at logs and seeing what gets rejected. Trivy first politely complains about not being able to talk to api.github.com (via kubectl -n harbor logs pod/harbor-harbor-trivy-0 trivy
and some re-formatting):
{
"error": "running trivy wrapper: running trivy: exit status 1:
2020-06-09T10:02:18.244Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n
2020-06-09T10:02:18.245Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n
2020-06-09T10:02:48.245Z\t\u001b[31mFATAL\u001b[0m\tfailed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: Get https://api.github.com/repos/aquasecurity/trivy-db/releases: dial tcp 140.82.118.6:443: i/o timeout\n",
"level": "error",
"msg": "Scan job failed",
"scan_job_id": "4f0dd8b163f6ea192e88d3b0",
"time": "2020-06-09T10:02:48Z"
}
But after opening for api.github.com
it still doesn't work, and the complaints are getting way less informative:
{
"error": "running trivy wrapper: running trivy: exit status 1:
2020-06-09T10:23:31.853Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n
2020-06-09T10:23:31.853Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n
2020-06-09T10:25:02.422Z\t\u001b[31mFATAL\u001b[0m\tfailed to download vulnerability DB: failed to download vulnerability DB: DB file not found\n",
"level": "error",
"msg": "Scan failed",
"time": "2020-06-09T10:25:02Z"
}
From looking at package-drops in cilium (kubectl -n kube-system -it exec cilium-qbhbq -- cilium monitor -t drop
) and some reverse-IP'ing, I'm stranded somewhere in S3:
xx drop (Policy denied) flow 0x285d9398 to endpoint 0, identity 25037->2: 10.217.5.243:40494 -> 52.216.160.91:443 tcp SYN
xx drop (Policy denied) flow 0x543eaa25 to endpoint 0, identity 25037->2: 10.217.5.243:40494 -> 52.216.160.91:443 tcp SYN
...
(10.217.5.243 is my trivy-container; opening http://52.216.160.91 redirects you to a S3-page)
Practically speaking, I guess downloads from api.github.com are probably re-directed to come from S3, which you obviously can't do anything about.
But getting a better error message would definitely be helpful here.
Currently in value.yaml what I see is the below,
persistence:
enabled: true
storageClass: ""
accessMode: ReadWriteOnce
size: 5Gi
Is there a way to add the existing pvc? I tried the below, but it doesn't seem to be working.
persistence:
enabled: false
resourcePolicy: "keep"
persistentVolumeClaim:
existingClaim: "trivy-claim"
storageClass: ""
accessMode: ReadWriteOnce
size: 2Gi
Current container only support redis://
protocol.
When connecting to redis using SSL/TLS, the rediss:// protocol is used.
Would it be worthwhile adding support for the rediss://
protocol from Trivy ?
The handler should check whether the corresponding ScanJob completed and return appropriate HTTP status codes. If the ScanJob is completed, a ScanReport should be returned based on the Accept MIME type request by the client.
Hello,
At OVH we plan to deploy Trivy for our Managed private reigstry service based on Harbor.
Context I would like to deploy Trivy for hundreds of customers.
Environment Harbor on Kubernetes clusters thanks to Harbor Operator.
/home/scanner/.cache
for a single customer?/home/scanner/.cache
? Is there personal data?/home/scanner/.cache
directory?/home/scanner/.cache/trivy/db/trivy.db
file?Any help appreciated.
Cheers
https://redis.io/topics/sentinel
Harbor can be deployed in HA environment where Redis is also available in Sentinel mode. We should be able to support such setup.
The task here is to check what are the requirements to support sentinel mode and whether the https://github.com/gomodule/redigo lib is suitable. Then, implement the changes to the connection pool and config.
This is relevant for the integration with https://github.com/goharbor/harbor-operator and https://github.com/goharbor/harbor-cluster-operator
When deploying in Kubernetes using the Helm chart, there are some issues with the configuration of persistent volumes.
If you're using persistent volumes, the .cache directory is not writable by uid 1000. If you disable persistent volumes, it's also not writable due to it not being a separate volume and the root filesystem being read only.
It would also be nice to use the default storage class by default like most Helm charts do.
Hi @danielpacak
For issue #5
I think you can consider using something like below command to get the latest release on github:
curl -sL -o /dev/null -w %{url_effective} https://github.com/aquasecurity/harbor-scanner-trivy/releases/latest
This is actually referenced from goreleaser script
Send request:
{"registry_url":"10.16*.*.*","registry_token":"<MY-TOKEN>","repository":"scanners/mongo","tag":"3.4-xenial","digest":"sha256:762277c0de87e5499e418c40cfe7352f7f976802b019e7ac0948d39e5f4a7413"}
The following error occurred:
2019/08/13 11:27:57 Starting harbor-scanner-trivy with config &{:8080 }
2019/08/13 11:29:46 RegistryURL: http://10.16*.*.*
2019/08/13 11:29:46 Repository: scanners/mongo
2019/08/13 11:29:46 Tag: 3.4-xenial
2019/08/13 11:29:46 Digest: sha256:762277c0de87e5499e418c40cfe7352f7f976802b019e7ac0948d39e5f4a7413
2019/08/13 11:29:46 Scan request: 46b3e41a-b096-44ad-b8c0-c014e10eb8c0
2019/08/13 11:29:46 Started scanning http://10.16*.*.*/scanners/mongo:3.4-xenial ...
2019-08-13T11:29:46.612Z DEBUG cache dir: /root/.cache/trivy
2019-08-13T11:29:46.613Z DEBUG db path: /root/.cache/trivy/db/trivy.db
2019-08-13T11:29:46.623Z INFO Updating vulnerability database...
2019-08-13T11:29:46.623Z DEBUG git pull
2019-08-13T11:29:47.444Z DEBUG total updated files: 1
2019-08-13T11:29:47.475Z FATAL invalid image:
github.com/knqyf263/trivy/pkg.Run
/root/project/pkg/run.go:153
- parsing image "http://10.16*.*.*/scanners/mongo:3.4-xenial" failed: invalid reference format
2019/08/13 11:29:47 ERROR: running trivy: exit status 1
Currently I am building a test instance of Harbor and wanting to test the use of harbor-scanner-trivy, but I don't (yet) want to set up the whole kittten caboodle as a HA cluster (using Helm). I'm able to get a simple container set up and can access the metadata (see below) but nothing else appears to be working.
I've tried various environment variables to be passed to the scanner-trivy container but there is no log output to work with.
Suggestions would be appreciated. The dump from api/v1/metadata is:
{
"scanner":{
"name":"Trivy",
"vendor":"Aqua Security",
"version":"0.4.2"
},
"capabilities":[
{
"consumes_mime_types":[
"application/vnd.oci.image.manifest.v1+json",
"application/vnd.docker.distribution.manifest.v2+json"
],
"produces_mime_types":[
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"
]
}
],
"properties":{
"harbor.scanner-adapter/scanner-type":"os-package-vulnerability",
"org.label-schema.build-date":"unknown",
"org.label-schema.vcs":"https://github.com/aquasecurity/harbor-scanner-trivy",
"org.label-schema.vcs-ref":"none",
"org.label-schema.version":"dev"
}
}
So far I have to run docker compose, wait for all service to become ready, and only then execute the out of process component tests. We'd like to have a single make goal that does all.
Before:
docker-compose -f test/component/docker-compose.yaml up -d
make test-component
docker-compose -f test/component/docker-compose.yaml down
After:
make container && make test-component
When the harbor installed by helm in k8s cluster, the interal url is like http://harbor-harbor-core
. Then scan the imagelibrary/redis@sha256:86edeb058fb97b7e9441515a5f1bb57f1547432fedbaf3513117ba521d99ef9
by trivy, the trivy adapter will process the imags as harbor-harborr-core/library/redis@sha256:86edeb058fb97b7e9441515a5f1bb57f1547432fedbaf3513117ba521d99ef9
, the trivy command will recognite the registry is index.docker.io
and the scanning failed.
Trivy adapter should append port (80 for http, 443 for https) to the image before call the trivy command.
This is the issue in harbor-helm goharbor/harbor-helm#547 , harbor-helm will include port in core url as workaround to fix this issue, but we think that fix this issue in trivy adapter will be better.
So far we have a persistent keys for scan jobs, i.e. harbor.scanner.trivy:data-store:scan-job:<scan_job_id>
. However it's not required by Scanner Adapter API to store scan reports forever. The clients, in particular Harbor, are supposed to cache the latest scan report.
We could use volatile key instead with configurable TTL. See https://redis.io/commands/expire
When we enable policy check in Harbor Portal Trivy cannot exchange robot account credentials to JWT access token.
We don't have much details in the logs as well
{"exit_code":1,"image_ref":"core.harbor.domain/library/jenkins/jenkins@sha256:c3b6fa9216336022067be94adb9d4696459903bf3e67e01a8d0009994f5ab78d","level":"error","msg":"Running trivy failed","std_err":"","std_out":"","time":"2019-11-07T09:38:29Z"}
{"error":"running trivy wrapper: running trivy: exit status 1: ","level":"error","msg":"Scan failed","time":"2019-11-07T09:38:29Z"}
{"error":"running trivy wrapper: running trivy: exit status 1: ","level":"error","msg":"Scan job failed","scan_job_id":"b53336ec9e8028250ee78d75","time":"2019-11-07T09:38:31Z"}
It might happen that the adapter cannot connect to redis, even if accroding to redis logs it's up and running:
component_test.go:130: adatper logs
[{"built_at":"unknown","commit":"none","level":"info","msg":"Starting harbor-scanner-trivy","time":"2020-03-25T14:03:00Z","version":"dev"}
ERROR: write_concurrency_controls_max_concurrency - dial tcp: lookup redis on 192.168.65.1:53: no such host
{"addr":":8080","level":"warning","msg":"Starting API server without TLS","time":"2020-03-25T14:03:00Z"}
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: heartbeat - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: write_known_jobs - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: periodic_enqueuer.should_enqueue - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: periodic_enqueuer.loop.enqueue - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
/cc @simar7
The goal here is to:
.travis.yaml
if it's not absolutely necessaryA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.