marcinguy/betterscan-ce

Code Scanning/SAST/Static Analysis/Linting using many tools/Scanners + OpenAI GPT with One Report (Code, IaC) - Betterscan Community Edition (CE)

Notice: For commercial use PRO version is recommended (all features available). The CE (Community Edition) WILL NOT show the findings location(s). Serves as a DEMO to check Betterscan capabilities. If you work on Open Source project or you are a Nonprofit, free licenses are available for PRO version.

Open DevSecOps Orchestration Toolchain

Scan your source code and infra IaC against top security risks

Betterscan is a orchestration toolchain that uses state of the art tools to scan your source code and infrastructure IaC and analyzes your security and compliance risks.

Currently supports: PHP, Java, Scala, Python, PERL, Ruby, .NET Full Framework, C#, C, C++, Swift, Kotlin, Apex (Salesforce), Javascript, Typescript, GO, Infrastructure as a Code (IaC) Security and Best Practices (Docker, Kubernetes (k8s), Terraform AWS, GCP, Azure), Secret Scanning (166+ secret types), Dependency Confusion, Trojan Source,

Open Source and Proprietary Checks (total ca. 6,000+ checks).

Checks for misconfigurations across all major (and some minor) cloud providers (AWS Checks, Azure Checks, GCP Checks, CloudStack Checks, DigitalOcean Checks, GitHub Checks, Kubernetes Checks, OpenStack Checks, Oracle Checks)

Open and Developer friendly DevSecOps toolchain

Betterscan uses many tools for Code, Cloud, secrets, dependencies - SCA (software composition analysis) and Supply Chain Risks, and also precise Graph-based SAST analysis for Code and AI/OpenAI GPT. All the best Tools, researched, setup, ran together, unifed and de-duplicated results, so you don't have to do it. Added our own checkers also. Continuous Security. Fit for purpose and results. For commercial use PRO version is recommended (all features available)

Above is sample engine (Binary runtime) run powering everything (CLI, Web Platform, CI/CD Actions, GitHub App, DefectDojo, Reviewdog)

Above is Web Interface.

Even more screenshots and integrations in Wiki

OpenAI GPT plugin is available only in PRO version and requires paid OpenAI plan (billed per usage)

If you want to scan your Code and Infrastructure (including Secrets, SBOMs, and dependencies)

Below setup is for Linux (Ubuntu), you can also run it on MacOS/Docker and Windows via WSL/Docker setup (see here)

Install Docker Engine (Instructions for Ubuntu or on Ubuntu via one command via snap sudo snap install docker), if you don't already have it, and run this in your Git code directory

Quickstart

Sigstore cosign images are available.

2 options are available:

1. Binary runtime

CLI output

Run in command prompt in your Git repository folder:

sh <(curl https://dl.betterscan.io/cli.sh)

HTML, JSON, SARIF output

The result will be in the current directory in "report.html", "report.json" and "report.sarif" file

Run in command prompt in your Git repository folder:

sh <(curl https://dl.betterscan.io/cli-html.sh)

2. Platform with Webinterface and workers

Docker

If you need CI/CD and Web Interface, you need Docker-Compose (Instructions for Ubuntu) installed as well, if you don't already have it.

Run in command prompt (or docker-compose up or docker compose up ):

git clone https://github.com/marcinguy/betterscan-ce.git
cd betterscan-ce/dockerhub
./start.sh

Open up the Browser to:

http://localhost:5000

Sign up locally (and login in when needed)

Kubernetes

For Kubernetes Platform (also minikube)

It is available via Helm package manager

helm repo add betterscan-repo https://marcinguy.github.io/betterscan-chart
helm repo update
helm install betterscan betterscan-repo/betterscan

That's it.

Read more in the Wiki, also for GitHub/GitLab/Azure DevOps Server integration, PR scanning, GitHub Action, GitHub App, DefectDojo, Reviewdog

Sample integrations for BitBucket Pipelines, GitLab CI, Google CloudBuild, CircleCI, Jenkins, TravisCI are also provided.

betterscan-ce's Issues

How can SBOM in cyclonedx format be integrated?

We are using syft/gryft for sbom and vulnerability scanning, outputing in XML or JSON SBOM format. How can we integrate this with betterscan?

Ability to configure issues at or after run time?

I am testing this against a code base comprising >7k files (Java and JS mainly). The ensemble approach is great, but I have not been able to determine if it is possible to exclude issues from analysis and thereby speed up the overall analysis time.

The UI seems to have this functionality, but 1) the filtering API/current database schema does not seem to support selection or exclusion of e.g., "readbility" issues en masse (see my PR for a bug in this feature btw), and 2) manually deselecting via the on/off switch does not appear to affect scans currently in progress.

Any advise on how I can exclude issues from consideration for a scan, either before or after scan is started?

Adding custom rules?

Couldn't find any documentation to this effect (PR's welcome?)

How would one add custom semgrep rules?

Ability to see percentage progress of scan

The web UI on http://localhost:5000 shows "Analysis in progress..." and a spinning symbol. Meanwhile the docker process started with start.sh is emitting hundreds of messages, although they seem to repeat a variation of themselves periodically.
This is a relatively small project although it has node_modules dependencies included in the repository so this may make it seem like more.
The PC running docker has plenty of RAM and cores.
The point is, will it take 10 minutes, 10 hours or 10 days, I have no idea?

CLI documentation for checkmate tool

I have no idea what commands are available for the checkmate tool, other than what I found in the sh scripts. Where are the commands and arguments documented? checkmate help returns:

Unknown command: help

"checkmate git init" step is unresponsive

Thanks for creating and sharing this tool. Appreciate your idea and efforts.

I'm facing issues while running the pre-built docker images on MacOS with M1 chip.
"checkmate git init" step is stuck for hours.
Is there a debug option to run the command to troubleshoot further?

Please find below further details about this blocker:

❯ uname -a Darwin EELPD01407 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000 arm64

`❯ docker run -ti scanmycode/scanmycode3-ce:worker-cli-arm64 checkmate
/root
/root
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: gosec
Loading plugin: confused
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: semgrepdefi
Loading plugin: semgrepjs
Loading plugin: checkov
Usage: checkmate [command] [command] [...] [args]

Type "checkmate help" for help`

`❯ docker run -v /Users/karthik/Downloads/java-project-master:/Users/karthik/Downloads/java-project-master -ti scanmycode/scanmycode3-ce:worker-cli-arm64 bash
root@68484e859bd2:~# cd /Users/karthik/Downloads/java-project-master

root@68484e859bd2:/Users/karthik/Downloads/java-project-master#
root@68484e859bd2:/Users/karthik/Downloads/java-project-master# checkmate init
/root
/Users/karthik/Downloads/java-project-master
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: gosec
Loading plugin: confused
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: semgrepdefi
Loading plugin: semgrepjs
Loading plugin: checkov
Initializing new project in the current directory.
root@68484e859bd2:/Users/karthik/Downloads/java-project-master# checkmate git init
/root
/Users/karthik/Downloads/java-project-master
/Users/karthik/Downloads/java-project-master
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: gosec
Loading plugin: confused
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: semgrepdefi
Loading plugin: semgrepjs
Loading plugin: checkov`

Also tried on Linux machine and the result is same.
# uname -a Linux test-server 4.19.56-coreos-r1 #1 SMP Tue Jul 30 06:40:10 -00 2019 x86_64 Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz GenuineIntel GNU/Linux

`# docker run -ti scanmycode/scanmycode3-ce:worker-cli bash
root@1421087bdf21:#
root@1421087bdf21:# checkmate init
/root
/root
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: gosec
Loading plugin: confused
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: semgrepdefi
Loading plugin: semgrepjs
Loading plugin: checkov
Initializing new project in the current directory.
root@1421087bdf21:~# checkmate git init
/root
/root
/root
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: gosec
Loading plugin: confused
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: semgrepdefi
Loading plugin: semgrepjs
Loading plugin: checkov

client_loop: send disconnect: Broken pipe
❯`

No space left on device during build

Hello,
as per README instructions, I tried to clone the repo on macOS 13.6.1 and start it, this is the output:

$ ./start.sh
[+] Running 142/4
 ⠼ server 36 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulling                                                             292.4s
 ✔ postgres 14 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                  289.5s
 ⠼ worker_1 82 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulling             292.4s
 ⠼ rabbitmq3 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulling                                                                                      292.4s
failed to register layer: write /usr/local/lib/python3.8/site-packages/pylint/test/messages/func_e0204.txt: no space left on device

My docker configuration reserves as much as 64GB of disk space for virtual disks, the host has 38GB free: is that not enough for installation?

setup fails to create tasks directory and blocks worker from working on tasks

When I run docker-compose up in the dockerhub directory for this repo, it creates two folders, data1 and data2. The worker then attempts to create log files in data2/tasks, but that directory doesn't exist, so the worker fails to run the task. Simply creating the tasks directory solves the issue.

The setup script should run mkdir on that directory so that the worker can write its logs.

How to add my own private/public key to access my private git repo via Betterscan

Currently I installed betterscan-ce via Docker but when I try to run scan on a Private Git Repo I am not able to connect due to incorrect SSH Key. The tool is expecting us to add the SSH Public Key that it is sharing into the GIT Repo which is not allowed based on our Security Policies. Following are some questions that I need answers for:

• I do have both private /public key for to access the code, can these private and public keys be used in the code to access the Git Repo as opposed to what Tool is Providing currently.
• If it is not possible with the current code, please help/suggest me where can I make the changes in the code locally to satisfy my requirement
  Below is the Screenshot of the Issue for reference:
  

Thanks
Srikanth

Getting error in checkmate issue

Hi,
I trying to add betterscan in Azure devops pipeline using cli image. But getting error at last step
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate issues'

Traceback (most recent call last):
File "/usr/local/bin/checkmate", line 33, in
sys.exit(load_entry_point('checkmate==0.2.0', 'console_scripts', 'checkmate')())
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/scripts/manage.py", line 114, in main
result = command.run()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/management/commands/issues.py", line 65, in run
if issue["line"]==1:
File "/usr/local/lib/python3.8/site-packages/blitzdb3_ce-4.0.0-py3.8.egg/blitzdb/document.py", line 191, in getitem
return self.attributes[key]
KeyError: 'line'

Could you please help

rules documentation

Hi, I am trying to use SAST tools on java projects, including betterscan-ce. And I want to find out what concrete rules better-ce uses to detect vuls. So is there some documentation about it like something in sonarqube, like this Java static code analysis?

Thanks a lot if you can point it out.

Mark

Dependency files should always be scanned

Issue

I ran osv-scan directly against my project, and it detected 4 vulnerable packages, but betterscan didn't find any. I expected the scan reports to match.

npm vulnerability scans.zip

I also included results for OWASP depedency-check in the zipfile since it provided similar results to osv-scan, but osv-scan has been more thorough in my testing against node.js and nuget projects.

Theory

I'm wondering if the issue I'm seeing is that the package-lock.json was updated in a commit that had been scanned before BetterScan added the osv-scan tool. It takes hours to run BetterScan against all commits in the repo, so I have not attempted to verify this yet.

Version Info

I used the latest version of all of these tools as of today, 1/10/2023.

Private github repo fails to fetch despite adding public ssh key to github account?

Am I missing something?

I have seen the instructions here #45

but I am able to add public ssh keys to my github account, so I believe that thread isn't relevant to me.

I have tried cloning from within dockerhub_worker_1_1 but I get Permission denied (publickey)

# git clone [email protected]:jopfre/xxxx.git
Cloning into 'xxxx'...
The authenticity of host 'github.com (140.82.113.3)' can't be established.
ECDSA key fingerprint is XXXXXXXX
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,140.82.113.3' (ECDSA) to the list of known hosts.
[email protected]: Permission denied (publickey).

I added the public ssh key from betterscan dashboard > settings > git settings to https://github.com/settings/ssh/new

Am I missing a step perhaps?

Full log output:

[INFO / 2023-03-05 00:02:20] Running pre-analysis hooks for project aird (e55c2c0f7273437880aa58a4ea51d307).
[INFO / 2023-03-05 00:02:20] Fetching data for project aird (e55c2c0f7273437880aa58a4ea51d307).
[ERROR / 2023-03-05 00:02:21] Fetching data for project aird (e55c2c0f7273437880aa58a4ea51d307) failed!
[ERROR / 2023-03-05 00:02:21] Analysis of project aird (e55c2c0f7273437880aa58a4ea51d307) failed!
[ERROR / 2023-03-05 00:02:21] Traceback (most recent call last):
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
    settings.hooks.call(hook, project)
  File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
    hook(*args, **kwargs)
  File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 100, in fetch_remote
    raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
    _analyze_project(project)
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
    hook_step(project, "project.analyze.fetch", "Fetching data")
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
    logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'

[INFO / 2023-03-05 00:02:20] Running pre-analysis hooks for project aird (e55c2c0f7273437880aa58a4ea51d307).
[INFO / 2023-03-05 00:02:20] Fetching data for project aird (e55c2c0f7273437880aa58a4ea51d307).
[ERROR / 2023-03-05 00:02:21] Fetching data for project aird (e55c2c0f7273437880aa58a4ea51d307) failed!
[ERROR / 2023-03-05 00:02:21] Analysis of project aird (e55c2c0f7273437880aa58a4ea51d307) failed!
[ERROR / 2023-03-05 00:02:21] Traceback (most recent call last):
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 42, in hook_step
    settings.hooks.call(hook, project)
  File "/srv/scanmycode/quantifiedcode/helpers/hooks.py", line 42, in call
    hook(*args, **kwargs)
  File "/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py", line 100, in fetch_remote
    raise IOError("Cannot fetch git repository!")
OSError: Cannot fetch git repository!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 76, in analyze_project
    _analyze_project(project)
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 92, in _analyze_project
    hook_step(project, "project.analyze.fetch", "Fetching data")
  File "/srv/scanmycode/quantifiedcode/backend/tasks/project/analyze.py", line 45, in hook_step
    logger.error("Exception {} {}.".format(e.__class__.__name__, e.message))
AttributeError: 'OSError' object has no attribute 'message'

Purchased Pro License in September haven't recieved it yet

Hello:

I purchased the pro licence via the betterscan.io webpage and haven't recieved it yet, how can I get it?

Error during analysis

On a private git repository, it has access, connects then gives:

dockerhub-server-1 | <blitzdb.backends.sql.queryset.QuerySet object at 0x7f58ae8c73d0>
dockerhub-server-1 | local variable 'branch' referenced before assignment
dockerhub-server-1 | Traceback (most recent call last):
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/backend/api/resource.py", line 129, in handle
dockerhub-server-1 | handler_response = handler(*args, **kwargs)
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/backend/decorators.py", line 381, in decorated_function
dockerhub-server-1 | return f(*args, **kwargs)
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/backend/decorators.py", line 58, in decorated
dockerhub-server-1 | return func(*args, **kwargs)
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/backend/decorators.py", line 124, in decorated_function
dockerhub-server-1 | return f(*args, **kwargs)
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/backend/api/v1/badge.py", line 35, in get
dockerhub-server-1 | snapshot = get_snapshot(project, snapshot_id, raw=False, include=('project',))
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/backend/helpers/snapshot.py", line 31, in get_snapshot
dockerhub-server-1 | snapshot = params['provider'](project, snapshot_id, raw=raw, only=only, include=include)
dockerhub-server-1 | File "/srv/scanmycode/quantifiedcode/plugins/git/backend/providers/snapshot.py", line 64, in resolve
dockerhub-server-1 | snapshot['branch'] = branch
dockerhub-server-1 | UnboundLocalError: local variable 'branch' referenced before assignment

CE edition default run requires PRO to view files?

Using this default command to run after installing docker and docker-compose

sh <(curl https://dl.betterscan.io/cli.sh)

Result

                                                                             Scan Report
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━┓
┃ Description                                                                                                        ┃ Severity ┃                  File ┃ Line ┃    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━┩
│ powershell                                                                                                         │  Warning │ Please upgrade to PRO │    1 │ ❌ │
│ Big_Numbers3                                                                                                       │  Warning │ Please upgrade to PRO │    1 │ ❌ │
│ Big_Numbers1                                                                                                       │  Warning │ Please upgrade to PRO │    1 │ ❌ │
│ Big_Numbers1                                                                                                       │  Warning │ Please upgrade to PRO │    1 │ ❌ │
│ Big_Numbers3                                                                                                       │  Warning │ Please upgrade to PRO │    1 │ ❌ │
│ Big_Numbers1                                                                                                       │  Warning │ Please upgrade to PRO │    1 │ ❌ │
│ Big_Numbers1                                                                                                       │  Warning │ Please upgrade to PRO │    1 │ ❌ │

While the documentation shown this image instead :

Is that normal? Isn't kinda misleading?

The documentation on the CE sample told us to use that so as a new user we will try the easiest tries first and see what it contains.

Experience on M1 Mac

Tried to run scanmycode locally on my M1 Macbook with docker-for-mac and stumbled onto some problems.

1. Port 5000 is used by the Mac Control Center by default - and changing it to 5001 via the exposed port in docker-compose.yaml is not carried through - every generated link still points to localhost:5000.
2. There is no pre-build arm64 image, and building it fails. When I change gosec from amd64 to arm64 it builds the images, and I can run the stack. But at runtime some errors about glibc are thrown.

documentation - staticgen

Hi,

What about creating a betterscan-doc repo to start a documentation using a staticgen to replace https://github.com/marcinguy/betterscan-ce/wiki.

Because we can't contribute/PR to this embedded wiki.

Docker is failing during building

I am using M1
=> ERROR [19/75] RUN ./go_installer 0.2s

[19/75] RUN ./go_installer:
#0 0.198 fatal: morestack on g0
#0 0.199 SIGTRAP: trace trap
#0 0.200 PC=0x8092f51 m=4 sigcode=128
#0 0.200
#0 0.200 goroutine 0 [idle]:
#0 0.201 runtime.morestack()
#0 0.201 /usr/local/google/home/cbro/go/src/runtime/asm_386.s:434 +0x21
#0 0.201
#0 0.201 goroutine 19 [syscall]:
#0 0.201 syscall.Syscall(0x3, 0x7, 0x186f0000, 0x8000, 0x0, 0x8000, 0x0)
#0 0.201 /usr/local/google/home/cbro/go/src/syscall/asm_linux_386.s:20 +0x5 fp=0x18621674 sp=0x18621670 pc=0x80aef95
#0 0.202 syscall.read(0x7, 0x186f0000, 0x8000, 0x8000, 0x18621601, 0x0, 0x0)
#0 0.202 /usr/local/google/home/cbro/go/src/syscall/zsyscall_linux_386.go:756 +0x45 fp=0x1862169c sp=0x18621674 pc=0x80ae345
#0 0.202 syscall.Read(0x7, 0x186f0000, 0x8000, 0x8000, 0x80563a8, 0x1, 0x0)

Is analyzers\yara\WShell_THOR_Webshells.yar an active exploit?

Windows Defender automatically removes this file after I clone the repo, stating that it's a high risk backdoor exploit. I did not expect files in this repo to have active exploits in them. I was expecting heuristics for finding exploitable code.

The file is: analyzers\yara\WShell_THOR_Webshells.yar

Submodules causing issues

For my project "we" include some submodules with:

git submodule update --init --recursive

When importing the "main" project

[WARNING / 2022-02-14 13:16:50] Cannot read source file: test/test_helper/bats-assert [ERROR / 2022-02-14 13:16:50] Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/lib/code/environment.py", line 546, in analyze_file_revision analyzer_results = analyzer.analyze(file_revision) File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/contrib/plugins/all/semgrep/analyzer.py", line 46, in analyze f.write(file_revision.get_file_content()) File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/lib/models.py", line 155, in get_file_content return self._file_content() File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/contrib/plugins/git/models.py", line 109, in <lambda> file_revision._file_content = lambda commit_sha = commit_sha, file_revision = file_revision: self.repository.get_file_content(commit_sha,file_revision.path) File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 517, in get_file_content raise IOError IOError

Example:
https://github.com/hestiacp/hestiacp/tree/main/test/test_helper

find_unicode_control2.py incorrectly flags UTF-8 BOM as a Trojan Source exploit

Visual Studio automatically adds a unicode control character sequence at the beginning of every file, known as a BOM, to indicate the file's encoding. This sequence is being incorrectly identified as a Trojan Source exploit by the find_unicode_control2.py analyzer.

See this SO Answer for more details, including the exact byte sequence:

0xEF, 0xBB, 0xBF

Service 'server' failed to build

$ ./start.sh
Building server
Sending build context to Docker daemon  14.85kB
Step 1/30 : FROM python:2.7
 ---> 68e7be49c28c
Step 2/30 : WORKDIR /
 ---> Using cache
 ---> d6858cc88a37
Step 3/30 : RUN mkdir -p /srv
 ---> Using cache
 ---> b70ac21e5cb7
Step 4/30 : RUN git clone https://github.com/marcinguy/scanmycode-ce.git /srv/scanmycode
 ---> Using cache
 ---> d98392912441
Step 5/30 : WORKDIR /srv/scanmycode
 ---> Using cache
 ---> 2e54818b8ffa
Step 6/30 : RUN git pull
 ---> Using cache
 ---> 6619cf1ee554
Step 7/30 : RUN apt update && apt install -y libcurl4-nss-dev libssl-dev tree sudo git ssh rsync npm ruby-sass
 ---> Using cache
 ---> bc668975da08
Step 8/30 : RUN tree
 ---> Using cache
 ---> 6a8da75efb63
Step 9/30 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> b1caa06fb2d0
Step 10/30 : RUN pip install pylint===1.9.2
 ---> Using cache
 ---> cda9ca606748
Step 11/30 : RUN pip install stripe
 ---> Using cache
 ---> fb63ce6873aa
Step 12/30 : RUN git clone https://github.com/marcinguy/checkmate-ce /checkmate
 ---> Using cache
 ---> 7d2b41952c77
Step 13/30 : WORKDIR /checkmate
 ---> Using cache
 ---> fd32c6a1e7b9
Step 14/30 : RUN tree /checkmate
 ---> Using cache
 ---> c4f585f63486
Step 15/30 : RUN python setup.py install
 ---> Using cache
 ---> 91e20749b951
Step 16/30 : RUN ln -s /srv/scanmycode/quantifiedcode/settings/default.yml /srv/scanmycode/settings.yml
 ---> Using cache
 ---> 69a72977d5fe
Step 17/30 : RUN pip install psycopg2 --upgrade
 ---> Using cache
 ---> 488aca4eaeab
Step 18/30 : WORKDIR /srv/scanmycode/quantifiedcode/frontend
 ---> Using cache
 ---> b7783c525c4f
Step 19/30 : RUN npm install -g bower
 ---> Using cache
 ---> bd1aa019f444
Step 20/30 : RUN npm install --save-dev @babel/core @babel/cli
 ---> Running in bc39259dec57
npm WARN npm npm does not support Node.js v10.24.0
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm WARN notice [SECURITY] clean-css has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=clean-css&version=3.2.8 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.npm WARN tar write after end
npm WARN notice [SECURITY] bower has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=bower&version=1.4.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN tar write after end
npm WARN notice [SECURITY] uglify-js has the following vulnerabilities: 2 low. Go here for more details: https://www.npmjs.com/advisories?search=uglify-js&version=2.3.6 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN tar write after end
npm WARN tar write after end
npm WARN tar write after end
npm WARN notice [SECURITY] ua-parser-js has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=ua-parser-js&version=0.7.21 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN tar write after end
npm WARN tar write after end
npm WARN tar write after end
npm WARN notice [SECURITY] trim-newlines has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=trim-newlines&version=1.0.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] decompress-zip has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=decompress-zip&version=0.1.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] bl has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=bl&version=1.2.2 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN tar write after end
npm WARN tar write after end
npm WARN notice [SECURITY] request has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=request&version=2.53.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] handlebars has the following vulnerabilities: 2 critical, 4 high, 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=handlebars&version=2.0.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] set-value has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=set-value&version=2.0.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] hawk has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=hawk&version=1.1.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] node-fetch has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=node-fetch&version=1.7.3 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] braces has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=braces&version=1.8.5 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] tough-cookie has the following vulnerabilities: 1 high, 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=tough-cookie&version=0.12.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] path-parse has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=path-parse&version=1.0.6 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] minimist has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=minimist&version=0.0.10 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] lodash has the following vulnerabilities: 2 high. Go here for more details: https://www.npmjs.com/advisories?search=lodash&version=4.17.19 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] json-schema has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=json-schema&version=0.2.3 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] glob-parent has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=glob-parent&version=2.0.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] bl has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=bl&version=0.9.5 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] lodash has the following vulnerabilities: 4 high, 2 low. Go here for more details: https://www.npmjs.com/advisories?search=lodash&version=2.4.2 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] ini has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=ini&version=1.3.5 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] cryptiles has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=cryptiles&version=0.2.2 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] hosted-git-info has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=hosted-git-info&version=2.8.8 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN tar write after end
npm WARN notice [SECURITY] hoek has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=hoek&version=0.9.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] deep-extend has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=deep-extend&version=0.2.11 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] minimist has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=minimist&version=0.0.8 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] hawk has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=hawk&version=2.3.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] tunnel-agent has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=tunnel-agent&version=0.4.3 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] qs has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=qs&version=2.3.3 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] semver has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=semver&version=2.3.2 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] hoek has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=hoek&version=2.16.3 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] minimatch has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=minimatch&version=2.0.10 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] cryptiles has the following vulnerability: 1 high. Go here for more details: https://www.npmjs.com/advisories?search=cryptiles&version=2.0.5 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-02-07T04_48_57_242Z-debug.log
The command '/bin/sh -c npm install --save-dev @babel/core @babel/cli' returned a non-zero code: 1
ERROR: Service 'server' failed to build : Build failed

LICENSE

Please add the LICENSE on this repository. Thanks :).

Checkmate error(?)

Hi there

I was using the cli-html option in a few repositories and it was working without issues, then last night the scan started to show this error:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/lib/code/environment.py", line 561, in analyze_file_revision
analyzer_results = analyzer.analyze(file_revision)
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/all/fluidattacksscanner/analyzer.py", line 72, in analyze
next(reader)
StopIteration

And now I'm stucked, if I try to run the scan again it says that it's already created and abort but I don't have the HTML report

Thanks in advance

Update osv-scan to v1.1.0 to include support for NuGet and PIP

I think you just need to run the Docker build to pull in v1.1.0

It adds support for NuGet (C#) and PIP (Python).

only enable specified plugins

hi, can you give some samples on how to enable\disable plugins ?

azure devops integration failing

Hi @marcinguy I'm trying to integrate betterscan in azure devops pipeline as per documentation but while running SAST task I'm betting below error message

Starting: Static Application Security Test (SAST)

Task : Command line
Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version : 2.212.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line

Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /__w/_temp/78c91d4f-67b0-43bc-8ab3-6df1b970aab7.sh
Switched to a new branch 'master'
/root
/__w/1/s
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: gosec
Loading plugin: confused
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: semgrepdefi
Loading plugin: semgrepjs
Loading plugin: checkov
Loading plugin: kubescape
Loading plugin: insidersecswift
Loading plugin: insiderseckotlin
Loading plugin: insiderseccsharp
Loading plugin: pmdapex
Loading plugin: semgrepccpp
Loading plugin: semgrepjava
Loading plugin: semgrepeslint
Loading plugin: graudit
Loading plugin: text4shell
Loading plugin: yara
Cannot find a checkmate project in the current directory tree, aborting.
##[error]Bash exited with code '255'.
Finishing: Static Application Security Test (SAST)