michelin/chopchop

ChopChop is a CLI to help developers scanning endpoints and identifying exposition of sensitive services/files/folders.

ChopChop

ChopChop is a command-line tool for dynamic application security testing on web applications, initially written by the Michelin CERT.

Its goal is to scan several endpoints and identify exposition of services/files/folders through the webroot. Checks/Signatures are declared in a config file (by default: chopchop.yml), fully configurable, and especially by developers.

"Chop chop" is a phrase rooted in Cantonese. "Chop chop" means "hurry" and suggests that something should be done now and without delay.

---

Table of Contents

• Building
• Usage
  • Available flags
  • Advanced usage
• Creating a new check/signature
• External Libraries
• Talks
• Licence
• Authors

Building

We tried to make the build process painless and hopefully, it should be as easy as:

$ go mod download
$ go build .

There should be a resulting gochopchop binary in the folder.

Using Docker

Thanks to Github Container Registry, we are able to provide you some freshly-build Docker images!

docker run ghcr.io/michelin/gochopchop scan https://foobar.com -v debug

But if you prefer, you can also build it locally, see below:

Build locally

docker build -t gochopchop .

Usage

We are continuously trying to make goChopChop as easy as possible. Scanning a host with this utility is as simple as :

$ ./gochopchop scan https://foobar.com

Using Docker

docker run gochopchop scan https://foobar.com

Custom configuration file

docker run -v ./:/app chopchop scan -c /app/chopchop.yml https://foobar.com

What's next

The Golang rewrite took place a couple of months ago but there's so much to do, still. Here are some features we are planning to integrate : [x] Threading for better performance [x] Ability to specify the number of concurrent threads [x] Colors and better formatting [x] Ability to filter checks/signatures to search for [x] Mock and unit tests [x] Github CI And much more!

Testing

To quickly end-to-end test chopchop, we provided a web-server in tests/server.go. To try it, please run go run tests/server.go then run chopchop with the following command ./gochopchop scan http://localhost:8000 --verbosity Debug. ChopChop should print "no vulnerabilities found".

There are also unit test that you can launch with go test -v ./.... These tests are integrated in the github CI workflow.

Available flags

You can find the available flags available for the scan command :

Flag	Full flag	Description
-h	--help	Help wizard
-v	--verbosity	Verbose level of logging
-c	--signature	Path of custom signature file
-k	--insecure	Disable SSL Verification
-u	--url-file	Path to a specified file containing urls to test
-b	--max-severity	Block the CI pipeline if severity is over or equal specified flag
-e	--export	Export type of the output (csv and/or json)
	--export-filename	Specify the filename for the export file(s)
-t	--timeout	Timeout for the HTTP requests
	--severity-filter	Filter Plugins by severity
	--plugin-filter	Filter Plugins by name of plugin
	--threads	Number of concurrent threads

Advanced usage

Here is a list of advanced usage that you might be interested in. Note: Redirectors like > for post processing can be used.

• Ability to scan and disable SSL verification

$ ./gochopchop scan https://foobar.com --insecure

• Ability to scan with a custom configuration file (including custom plugins)

$ ./gochopchop scan https://foobar.com --insecure --signature test_config.yml

• Ability to list all the plugins or by severity : plugins or plugins --severity High

$ ./gochopchop plugins --severity High

• Ability to specify number of concurrent threads : --threads 4 for 4 workers

$ ./gochopchop plugins --threads 4

• Ability to block the CI pipeline by severity level (equal or over specified severity) : --max-severity Medium

$ ./gochopchop scan https://foobar.com --max-severity Medium

• Ability to specify specific signatures to be checked

./gochopchop scan https://foobar.com --timeout 1 --verbosity --export=csv,json --export-filename boo --plugin-filters=Git,Zimbra,Jenkins

• Ability to list all the plugins

$ ./gochopchop plugins

• List High severity plugins

$ ./gochopchop plugins --severity High

• Set a list or URLs located in a file

$ ./gochopchop scan --url-file url_file.txt

• Export GoChopChop results in CSV and JSON format

$ ./gochopchop scan https://foobar.com  --export=csv,json --export-filename results

Creating a new check

Writing a new check is as simple as :

  - endpoint: "/.git/config"
    checks:
      - name: Git exposed
        match:
          - "[branch"
        remediation: Do not deploy .git folder on production servers
        description: Verifies that the GIT repository is accessible from the site
        severity: "High"

An endpoint (eg. /.git/config) is mapped to multiple checks which avoids sending X requests for X checks. Multiple checks can be done through a single HTTP request. Each check needs those fields:

Attribute	Type	Description	Optional ?	Example
name	string	Name of the check	No	Git exposed
description	string	A small description for the check	No	Ensure .git repository is not accessible from the webroot
remediation	string	Give a remediation for this specific "issue"	No	Do not deploy .git folder on production servers
severity	Enum("High", "Medium", "Low", "Informational")	Rate the criticity if it triggers in your environment	No	High
status_code	integer	The HTTP status code that should be returned	Yes	200
headers	List of string	List of headers there should be in the HTTP response	Yes	N/A
no_headers	List of string	List of headers there should NOT be in the HTTP response	Yes	N/A
match	List of string	List the strings there should be in the HTTP response	Yes	"[branch"
no_match	List of string	List the strings there should NOT be in the HTTP response	Yes	N/A
query_string	GET parameters that have to be passed to the endpoint	String	Yes	query_string: "id=FOO-chopchoptest"

External Libraries

Library Name	Link	License
Viper	https://github.com/spf13/viper	MIT License
Go-pretty	https://github.com/jedib0t/go-pretty	MIT License
Cobra	https://github.com/spf13/cobra	Apache License 2.0
strfmt	https://github.com/go-openapi/strfmt	Apache License 2.0
Go-homedir	https://github.com/mitchellh/go-homedir	MIT License
pkg-errors	https://github.com/pkg/errors	BSD 2 (Simplified License)
Go-runewidth	https://github.com/mattn/go-runewidth	MIT License

Please, refer to the third-party.txt file for further information.

Talks

• PyCon FR 2019 (The tool was initially developed in Python) - https://docs.google.com/presentation/d/1uVXGUpt7tC7zQ1HWegoBbEg2LHamABIqfDfiD9MWsD8/edit
• DEFCON AppSec Village 2020 "Turning offsec mindset to developer's toolset" - https://drive.google.com/file/d/15P8eSarIohwCVW-tR3FN78KJPGbpAtR1/view

License

ChopChop has been released under Apache License 2.0. Please, refer to the LICENSE file for further information.

Authors

• Paul A.
• David R. (For the Python version)
• Stanislas M. (For the Golang version)

chopchop's Issues

Set up an automated Docker build

Currently, there is no automated Docker build nor a published image on Docker Hub (hub.docker.com).

It could be great to set up an automated Docker build to build & push the last Docker image to the Docker Hub.

The usage with Docker would then be straightforward:

docker run michelin/chopchop scan --url https://foobar.com

This simple process is well explained here: https://docs.docker.com/docker-hub/builds/.
I can't set up this build myself as I am not owner of this repository.

You can see here a sample on the forked repository:

Output to file ?

run first time - Error: Path of signatures file is not valid

I downloaded the tar.gz release for Linux. When I run it I get the following error:

$ ./ChopChop_linux_amd64 plugins
Error: Path of signatures file is not valid
{"level":"warning","msg":"Path of signatures file is not valid","time":"2022-11-15T10:43:19+02:00"}

Where do i get the signatures file as it is not in the tar.gz?

Query String attribute for checks

It'd be useful to be able to specify a querystring as part of a check rather than having to include it in the URI.

I've been playing around with ChopChop by creating unit tests for some WAF rules, one set of which is QS arg type enforcement, so I end up with something like

• uri: "/?id=FOO-chopchoptest"
  checks:

  • name: ID type enforcement
    match:
    • "Homepage"
      remediation: "Check the WAF dynamic rules are active"
      description: "Verifies the WAF is enforcing type for QS arg 'id'"
      status_code: 200
      severity: "Medium"

• uri: "/?catid=FOO-chopchoptest"
  checks:

  • name: catid type enforcement
    match:
    • "Homepage"
      remediation: "Check the WAF dynamic rules are active"
      description: "Verifies the WAF is enforcing type for QS arg 'catid'"
      status_code: 200
      severity: "Medium"

Which is fine, but it's be awesome to be able to do something like

• uri: "/"
  checks:

  • name: ID type enforcement
    qs: "id=FOO-chopchoptest"
    match:

    • "Homepage"
      remediation: "Check the WAF dynamic rules are active"
      description: "Verifies the WAF is enforcing type for QS arg 'id'"
      status_code: 200
      severity: "Medium"

  • name: catid type enforcement
    qs: "catid=FOO-chopchoptest"
    match:

    • "Homepage"
      remediation: "Check the WAF dynamic rules are active"
      description: "Verifies the WAF is enforcing type for QS arg 'catid'"
      status_code: 200
      severity: "Medium"

So that if the path needs to be updated for some reason, it only needs doing in one place

Specify specific signatures to be checked by severity

Hi,

It would be great to being able to perform scanning by specifying signatures based on their rating {Informational, Low, Medium, High}.

For example if a user wants to scan URLs with only "High" signatures, it would be :

$ ./chopchop scan -u http://foobar.com --severity High

Confusion: Azure VS Tomcat

Regarding this:

ChopChop/chopchop.yml

Lines 45 to 50 in 8feacb5

	- name: Azure
	match:
	- "catalina.base"
	remediation: Check that the application has been deployed and delete the default pages
	description: Detects the presence of Azure installation by default
	severity: "Informational"

I might be mistaken as I don't know what would be a "Azure installation by default", but usually "catalina" refers to Tomcat so there's maybe a confusion here :)

Generate a better documentation (and a website)

Hi,

I think it would be great to work on a better documentation, by using MkDocs for example.
Other options should be checked too, such as the project's wiki, ...

Link: https://www.mkdocs.org/

Ability to control request method

It'd be useful to be able to control the request method, up to and including being able to specify some POST data.

It'd also mean where you're only interested in headers, you could place a HEAD instead of a GET (particularly useful if the asset you're requesting against is sizeable)

Match binary

Tried this rule:

plugins:
  - checks:
      - name: Database file
        match:
          - "\x1F\x8B\x08"
        remediation: Delete this file
        description: Verifies a database dump is accessible.
        severity: "High"
    uri: /db.sqlite3.tar.gz

On my test server, it correctly hits the file:

[verbose] Testing URL: http://0.0.0.0:8000/db.sqlite3.tar.gz

And I see it as 200 in the logs:

127.0.0.1 - - [24/Oct/2020 16:54:38] "GET /db.sqlite3.tar.gz HTTP/1.1" 200 -

Returning the expected bytes:

$ curl 0:8000/db.sqlite3.tar.gz | hexdump -C | head -n 1
00000000  1f 8b 08 00 00 00 00 00  00 03 ec 9d 79 6c 1c d7  |............yl..|

But the match may fail, as ChopChop does not report it:

$ ./gochopchop scan --url http://0.0.0.0:8000 
No vulnerabilities found.

So either I'm tired, either ChopChop don't want to match arbitrary bytes sequences?

Colors handling

Hi,

It would be great that we handle colors within ChopChop results within the ChopChop results table.
For example, something like :

• High : red
• Medium : orange
• Low : yellow or something quite neutral
• Informational : blue

Specify specific signatures to be checked by signature name

Hi,

It would be great to being able to perform scanning by specifying signatures based on their names.

For example if a user wants to scan URLs with only signatures containing "git" keyword, it would be :

$ ./chopchop scan -u http://foobar.com --signature "git*"

socks5 option

Hello! I want to know if is possible to add a option to pass a socks5 or proxy address. I'm trying to make a forwarding but that's not working cause I can't find a option to pass the proxy address.

Best regards,
oppsec.

ChopChop not exiting properly

If you do not specify the url and/or a url-file, the binary exists and displays "No vulnerabilities found." which is wrong.

See:

/tmp # ./gochopchop  scan
No vulnerabilities found.
/tmp # 

It would be better to fail saying that no url and/or url-file has been specified.

Integrate new signatures from nmap

Hi,

It would be interesting to have those new rules integrated in ChopChop, see : https://github.com/nnposter/nndefaccts/blob/master/http-default-accounts-fingerprints-nndefaccts.lua

filepath of url file is not valid

Hi and gg for le bon travail.

I was playing with it after seeing your video on YouTube (OSSIR), and the following command :

docker run ghcr.io/michelin/gochopchop scan --url-file file.txt --json-file "resultats.json"

gives me the following error :
Error: filepath of url file is not valid filepath of url file is not valid

I tried with -f, --url-file, "./file.txt", ./file.txt, file.txt, "file.txt", [absolute_path] and "[asbolute_path] with the same result.

Any idea why ?

Regards,

trickbot false positive?

hi,
thx a lot to the clermont ferrand dream team for this tool, i just used it on one of our website, and i am surprised with the response.
i get logs for files that are not present on the server:

| https://xxxxx.xxxxxxx.fr/ | /images/imgpaper.png | High | Possible Trickbot Trojan Payload hosting imgpaper.png | Make sure your system is'nt compromised |
| https:/xxxxxx-xxxxxxxxxx.fr/ | /images/cursor.png | High | Possible Trickbot Trojan Payload hosting cursor.png | Make sure your system is'nt compromised |
| https://xxxxxxxxxx.xxxxxxxxx/ | /images/redcar.png | High | Possible Trickbot Trojan Payload hosting redcar.png | Make sure your system is'nt compromised |
| https://xxxxxxxxx.xxxxxxxxx/ | /ico/VidT6cErs | High | Possible Trickbot Trojan Payload hosting VidT6cErs | Make sure your system is'nt compromised |

any idea what it means?
thxx for your time

Detect the non-presence of a HTTP header

Hello everyone, thanks for sharing ChopChop.

According to the documentation, there is no way to detect the lack of a HTTP header.

It can be a nice feature to detect the absence of some security headers like "Strict-Transport-Security" or "Content-Security-Policy".

Accept lists of URIs

Thanks for opening ChopChop!

Looked at chopchop.yml and though « I'll gladly add some... », but wanted to do it like so:

- uri: ["/db.sql", "/db.sql.gz", "/db.sqlite", "/db.sqlite.gz", "/db.sqlite3", "/db.sqlite3.gz", "/data.sql", "/data.sql.gz", "/users.sql", "/users.sql.gz", "/dump.sql", "/dump.sql.gz", "/mysqldump.sql", "/mysqldump.sql.gz", "/backup.sql", "/backup.sql.gz", "/db.backup", "/db.backup.gz", "/database.sql", "/database.sql.gz", "/db-data.sql", "/db-data.sql.gz", "/db_test.sql", "/db_test.sql.gz", "/db-test.sql", "/db-test.sql.gz"]
  checks:
    - name: Database file
      status_code: 200
      remediation: Delete this file
      description: Verifies a database dump is accessible.
      severity: "High"

(and I bet we could continue for hours adding to this list)

As you imagine, I don't want to copy/paste the name, status_code, remediation, description, and severity 26 times.

Doodling around the idea, it would be great to be able to express those as a « genex », or something similar, something like:

/(db|database|backup|mysqldump|dump|data).(sql|sqlite|sqlite3)(\.gz)?