cisagov/ioc-scanner

Search a filesystem for indicators of compromise (IoC).

ioc-scanner 🔎🆖

The ioc-scanner can search a filesystem for indicators of compromise (IoC). Indicators are defined by their md5, sha-1, or sha-256 hashes. The tool is very flexible about how it receives the IoC hashes. It will search blobs of input for strings that look like md5, sha-1, and sha-256 hashes.

Command line usage

Usage:
  ioc-scan [--log-level=LEVEL] [--stdin | --file=hashfile] [--target=root]
  ioc-scan (-h | --help)

Options:
  -h --help              Show this message.
  -f --file=hashfile     Search for hashes in specified file.
  -L --log-level=LEVEL   If specified, then the log level will be set to
                         the specified value.  Valid values are "debug", "info",
                         "warning", "error", and "critical". [default: warning]
  -s --stdin             Search for hashes on stdin.
  -t --target=root       Scan target root directory. [default: /]

Example output

❱ ioc-scan --target /bin
0313fd399b143fc40cd52a1679018305 /bin/bash

Scan elapsed time: 0:00:00.176262
Hit count by indicators:
70a6058952ed3212217105ec7865ba21    0
dff4b51907018f5cf325120aec2caf45    0
2d7a648ebe64e536944c011c8dcbb375    0
132646a2ad9deac1944be4264da30b01    0
8c109784750142b158a1459751ae5faf    0
fff485a90ef0a86fb2813eb64fd3442c    0
2d7a648ebe64e536944c011c8dcbb375    0
2a2410cef5497cbd3f6c13eaff9619da    0
3e7eb6abcce304de0822a618de756fd2    0
350cba65e28c723cbf0724c19bd7ee69    0
0313fd399b143fc40cd52a1679018305    1
ac56f4b8fac5739ccdb45777d313becf    0
69630e4574ec6798239b091cda43dca0    0
50973a3fc57d70c7911f7a952356188b9939e56b    0
b509f8545501588ecd828f970d91afc7c4aa6e238e838bd6a08ee2cd920fbe98    0

Stand-alone usage

The ioc_scanner.py file was designed to be dependency-free. This allows it to be deployed anywhere python3 is available without a full install process. There is an embedded hash list in this file that can be easily edited. This makes it possible to run this tool with automation tools like Ansible.

Here is an example of running the script remotely using the Ansible script module :

ansible --inventory=hosts-file cool-servers \
        --module-name=ansible.builtin.script \
        --args="cmd=src/ioc_scan/ioc_scanner.py executable=python3" \
        --become --ask-become-pass --user="ian.kilmister"

Optionally you can use the --file option to use a file on the remote host as a source for hashes.

ansible --inventory=hosts-file cool-servers \
        --module-name=ansible.builtin.script \
        --args="'cmd=src/ioc_scan/ioc_scanner.py --file hash_file.txt] \
               executable=python3" \
        --become --ask-become-pass --user="ian.kilmister"

To scan for indicator strings on AWS instances that are accessible via SSM, the ioc_scan_by_host.sh shell script has been provided in the extras directory:

$ ./extras/ioc_scan_by_host.sh my-ioc-strings.txt i-0123456789abcdef0

IOC List is: "192.168.1[.]1 sketchy-site[.]org IOC_STRING_1 IOC_STRING_2"
Instances are: "i-0123456789abcdef0"

Searching i-0123456789abcdef0:

Starting session with SessionId: iam.username-0123456789abcdef0
0 found for 192.168.1[.]1
0 found for sketchy-site[.]org
0 found for IOC_STRING_1
0 found for IOC_STRING_2

Exiting session with sessionId: iam.username-0123456789abcdef0.

Search of i-0123456789abcdef0 is complete.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

ioc-scanner's Issues

Improve usage information

🚀 Feature Proposal

Improve the usage information for --file and --stdin.

Motivation

As @mcdonnnj pointed out in #7:

I do think the usage for --file and --stdin could be improved, because reading just the usage it was unclear to me that you could use those options to provide hashes.

Example

It would reduce the sort of confusion that generated #7.

Pitch

Eschew obfuscation!

FIPS preventing remote connection

🐛 Bug Report

Tagging @felddy and @dav3r

yavin | FAILED! => {

    "changed": true,

    "msg": "non-zero return code",

    "rc": 1,

    "stderr": "Shared connection to yavin closed.\r\n",

    "stderr_lines": [

        "Shared connection to yavin closed."

    ],

    "stdout": "Traceback (most recent call last):\r\n  File \"/home/gollam/.ansible/tmp/ansible-tmp-1593119061.2345998-36805-301601859236/ioc_scanner.py\", line 130, in <module>\r\n    sys.exit(main())\r\n  File \"/home/gollam/.ansible/tmp/ansible-tmp-1593119061.2345998-36805-301601859236/ioc_scanner.py\", line 102, in main\r\n    hashes = hash_file(file)\r\n  File \"/home/gollam/.ansible/tmp/ansible-tmp-1593119061.2345998-36805-301601859236/ioc_scanner.py\", line 48, in hash_file\r\n    hash_md5 = hashlib.md5()  # nosec\r\nValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips\r\n",

    "stdout_lines": [

        "Traceback (most recent call last):",

        "  File \"/home/gollam/.ansible/tmp/ansible-tmp-1593119061.2345998-36805-301601859236/ioc_scanner.py\", line 130, in <module>",

        "    sys.exit(main())",

        "  File \"/home/gollam/.ansible/tmp/ansible-tmp-1593119061.2345998-36805-301601859236/ioc_scanner.py\", line 102, in main",

        "    hashes = hash_file(file)",

        "  File \"/home/gollam/.ansible/tmp/ansible-tmp-1593119061.2345998-36805-301601859236/ioc_scanner.py\", line 48, in hash_file",

        "    hash_md5 = hashlib.md5()  # nosec",

        "ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips"

    ]

}

To Reproduce

Steps to reproduce the behavior:

Run the following:
ansible -i hosts-file all -m script -e "--file md5sums.txt" -a ./src/ioc_scan/ioc_scanner.py --become

Expected behavior

Expect no errors

Any helpful log output

Workaround:

Edit this line in ioc_scanner.py

hash_md5 = hashlib.md5(usedforsecurity=False) # nosec

Support hashes besides MD5

🚀 Feature Proposal

Add the ability to scan for other hash types besides MD5, such as SHA-1 and SHA-256.

Motivation

When we are asked to scan for Indicators of Compromise (IOCs), we occasionally are given SHA-1 and SHA-256 hashes, in addition to MD5 hashes.

Example

Sample IOC hashes:

SHA256: b509f8545501588ecd828f970d91afc7c4aa6e238e838bd6a08ee2cd920fbe98
SHA-1:  31B54AEBDAF5FBC73A66AC41CCB35943CC9B7F72
SHA-1:  50973A3FC57D70C7911F7A952356188B9939E56B
SHA-1:  244EB62B9AC30934098CA4204447440D6FC4E259
SHA-1:  5C8F83CC4FF57E7C67925DF4D9DAABE5D0CC07E2

Pitch

It will give us more comprehensive scanning capabilities.

Support Hashes in external file

🚀 Feature Proposal

Instead of having the list of hashes located in ioc_scanner.py block, we would ideally be able to point to a text file (as is done with the hosts-file).

Motivation

Don't have to edit a python file each time you want to look for a new hash

Pitch

Because it currently is somewhat cumbersome. and could be improved :)

Fix failing tests

💥 Regression Report

Our APB runs on this repository keep erroring out because of some failed tests. See this APB run, for example.

Last working version

Looks like this was the last build that succeeded.

To Reproduce

• Check out the repo from scratch
• Install Python dependencies
• Run tests

Expected behavior

All tests should pass.

Fix Instructions to Run ioc_scanner.py with Ansible

🐛 Bug Report

The given instructions in the README do not correctly run ioc_scanner.py using Ansible.

To Reproduce

Attempt to run ioc_scanner.py against a target with Ansible per the README's instructions.

Expected behavior

Running Ansible based on the example command works.

Any helpful log output

Traceback (most recent call last):
  File "/Users/redmind/.pyenv/versions/ioc-scanner/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/Users/redmind/.pyenv/versions/ioc-scanner/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 663, in _execute
    result = self._handler.run(task_vars=variables)
  File "/Users/redmind/.pyenv/versions/ioc-scanner/lib/python3.8/site-packages/ansible/plugins/action/script.py", line 80, in run
    source = parts[0]
IndexError: list index out of range
i-062eeafa99a01da3c | FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

Add a --file Option to ioc_scanner.py

🚀 Feature Proposal

There was some miscommunication in #7 with respect to command line arguments for this project. The way this library is written has the CLI side in ioc_scan_cli.py provide command line options, and then call the functionality in ioc_scanner.py appropriately. This is fine when using this package as installed with pip. However, when using Ansible to run the script on a remote host as we show in the README, it bypasses this design. If we want to support the idea of using ioc_scanner.py to scan using Ansible, then it would be helpful to use the standard argparse library to add support for a --file option in ioc_scanner.py.

Motivation

If we are going to continue to support the example Ansible method of using a part of this package, then it would beneficial to provide the bare minimum functionality of accepting the --file switch as provided by the library as a whole.

Example

ansible --inventory=hosts-file cool-servers --module-name=script \
--args="src/ioc_scan/ioc_scanner.py" --extra-vars="--file md5sums.txt" \
--become --ask-become-pass --user="ian.kilmister"

Pitch

It aligns with an example usage of a part of this library, and helps alleviate annoyances like that seen in #7