The ClamAV Large Archive Scanner utility is a wrapper around the ClamAV clamd and clamdscan programs that provides a way to scan archives which exceed ClamAV's maximum file size limit. At the time of writing (2024/03/09), ClamAV may not scan any file or archive larger than 2 GiB.
Important: This utility is a workaround to supplement ClamAV until such time as archives larger than 2 GiB can be scanned. This utility is not intended to replace
clamscanorclamdscan. We have no intention of providing feature parity between this utility andclamscanorclamdscan.This utility works around ClamAV's file size limitations for non-archive files. It will not enable you scan large documents, graphics, videos, etc. In case you were wondering if large files could be chunked into smaller files and then scanned... No. That is not an effective solution to scan large files.
The utility has three sub-commands: scan, unpack, and cleanup.
The scan command combines the other two commands to unpack, scan, and clean up.
The unpack command provides the ability extract archives or mount disk images of the supported archive types without scanning them.
The cleanup command is complementary to the unpack command, enabling you to easily un-mount or delete the extracted archive contents.
The ClamAV Large Archive Scanner supports extraction or mounting of the following types of archives:
- TAR
- ZIP
- ISO
- VMDK
- TARGZ
- QCOW2
We provide two options for installation. You may run the utility in your local environment or you may run the utility in a Docker container. The Docker container is easier.
To use the ClamAV Large Archive Scanner in your local environment, you will need to install an assortment of supporting tools and libraries:
-
Install Python 3.9 or newer.
-
Install the required Python packages. We suggest using a
venvvirtual environment. From the project root directory, run the following:python3 -m venv .venv source .venv/bin/activate pip3 install .
If you open a new terminal, you will need to reactivate your Python virtual environment again, using:
source .venv/bin/activate -
Install ClamAV. Both
clamdandclamdscanare required. On some Linux distributions, these are packaged separately. You can verify that they are present by runningwhich clamdandwhich clamdscan. -
Install libmagic which is required to determine file types.
-
Install libguestfs which is needed to unpack VMDK/QCOW2 disk images.
You will need to start the clamd service before you can use the ClamAV Large Archive Scanner. This may require some initial configuration to include using freshclam to download the latest malware detection signatures. See the ClamAV documentation for more information on how to set up ClamAV.
Regarding clamd.conf config options, you must set the LocalSocket option (or TCPSocket option), at a minimum. On some systems, this is preconfigured. For the ClamAV Large Archive Scanner project, the goal is to scan extremely large archives, so you'll also need to add the following settings to max out ClamAV's file size capabilities:
MaxFileSize 0
MaxScanSize 0
MaxScanTime 3600000
MaxFiles 100000
MaxRecursion 20
Finally, you may also wish to raise an alert if the limits have been exceeded. You can do so by adding this option:
AlertExceedsMax yes
Note: Regarding the selected config options...
MaxFileSize 0- This maxes out the file size limit for ClamAV.
MaxScanSize 0- This maxes out the scan size limit. The scan size is the total amount of bytes scanned per file when extracting files from archives, decompressing embedded files, normalizing scripts, or even re-scanning the same data as a different type. The total number of bytes scanned is often much larger than the original file size, even for plain text files.
MaxScanTime 3600000- This increases the scan time limit per file scanned to 1 hour (60 x 60 x 1000 milliseconds). Scanning large archives will take a long time. You may wish to increase or disable this limit.
MaxFiles 100000- This increases the limit for the number of embedded files scanned. You may wish to increase or disable this limit.
MaxRecursion 20- This increases the maximum recursion depth from 17 to 20. Scan recursion is the process of unpacking and scanning embedded files. Files unpacked by the Large Archive Scanner before passing to ClamAV for scanning do not count towards the maximum recursion depth. The maximum recursion depth cannot be disabled.
AlertExceedsMax yes- This option will cause scans to alert when a scan limit was exceeded, with the signature names like:
Heuristics.Limits.Exceeded.MaxFileSizeHeuristics.Limits.Exceeded.MaxScanSizeHeuristics.Limits.Exceeded.MaxFilesHeuristics.Limits.Exceeded.MaxRecursionHeuristics.Limits.Exceeded.MaxScanTimeSee the
clamd.conf.sampleconfig for more details.
After you've installed everything and have started clamd, you may run a scan with the ClamAV Large Archive Scanner. For example:
archive scan /path/to/archiveTip: If you have multiple ClamAV installations outside the normal
$PATH, you may need to add thebindirectory for your preferred version to the$PATHbefore attempting a scan.For example:
PATH=~/clams/1.3.0/bin:$PATH archive scan /path/to/archive
To learn more, run archive --help, or skip to Usage.
The simplest way to use the ClamAV Large Archive Scanner is using a Docker container.
The provided Dockerfile may be used to build an container with the environment and tools necessary to run this utility. This Dockerfile also increases the ClamAV scan limits described in the previous section.
This Docker container is based on the ClamAV project's clamav-debian image. You can find additional instructions for how to customize and use this container here.
Note: Privileged mode will be needed for to mount ISO archives when they are unpacked.
To build the image, run:
docker build . -t clamav-large-archive-scanner --loadTo start the container, run:
docker run \
--interactive \
--tty \
--rm \
--name "clam_container_01" \
clamav-large-archive-scannerTip: You may wish to mount a
/some/pathcontaining the archives you wish to scan as a volume in the container. You can do so when starting the container, like this. Replace/some/pathwith the actual directory you wish to mount. The directory contents will be found within the running container under/target:docker run \ --interactive \ --tty \ --rm \ --mount type=bind,source=/some/path,target=/target \ --name "clam_container_01" \ clamav-large-archive-scanner
After the container is up and running, and after clamd has finished loading, you may execute commands in the running container, or open a shell in the running container to execute commands.
Tip: Within the container, you can use
clamdscan --ping 100to wait up to 100 second forclamdto finish loading. Ifclamdtakes longer than 100 seconds to load, or fails to load, then the command will exit with a non-zero exit code. For example:docker exec --interactive --tty "clam_container_01" clamdscan --ping 100
Suppose you used the --mount option to mount a directory at /target containing some_archive.tgz, you might try scanning it like this:
docker exec --interactive --tty "clam_container_01" archive scan /target/some_archive.tgzOr, to enter a shell in the container, run:
docker exec --interactive --tty "clam_container_01" /bin/bashTo shut down the container, run:
docker kill clam_container_01Usage: archive [OPTIONS] COMMAND [ARGS]...
Options:
-t, --trace Enable trace logging. By default, log all actions to
/tmp/clam_unpacker.log
--trace-file PATH Override the default trace log file
-v, --verbose Enable verbose logging
-q, --quiet Disable all logging
--help Show this message and exit.
Commands:
cleanup
scan
unpack
-
scanThis command is used to scan regular files and directories.
The
scancommand combines the other two commands to unpack, scan, and clean up.Use the following options to customize scan behavior:
Usage: archive scan [OPTIONS] PATH Options: --min-size TEXT Minimum file size to unpack (default: 2.0 GiB). --ignore-size Ignore file size lower limit (equivalent to --min-size=0). --tmp-dir PATH Temporary working directory (default: /tmp). -ff, --fail-fast Stop scanning after the first failure. --allmatch Continue scanning if a signature match occurs. --help Show this message and exit. -
unpackThis command unpacks or mounts supported large archives to a given directory. By default, a "large" archive is a one greater than 2 GiB. This action is recursive.
Archives smaller than 2 GiB will be skipped. You may use
--ignore-sizeor--min-size=0unpack all supported archives, regardless of size.Usage: archive unpack [OPTIONS] PATH Options: -r, --recursive Recursively unpack files. --min-size TEXT Minimum file size to unpack (default: 2.0 GiB). --ignore-size Ignore file size lower limit (equivalent to --min-size=0). --tmp-dir PATH Directory to unpack files to (default: /tmp). --help Show this message and exit. -
cleanupThis command will clean up the temp directories/files created as part of the script to scan input file or directory.
Usage: archive cleanup [OPTIONS] PATH Options: --file Recursively cleanup directories associated with the file. --tmp-dir PATH Directory to search for unpacked files(default: /tmp). --help Show this message and exit.
Using the scan command to scan an archive:
archive -t -v scan /path/to/archiveUsing the unpack command to unpack and archive:
archive -t -v unpack /path/to/archiveThere are many ways to contribute.
This repo includes some tests to verify correct functionality. You can run the tests from your local environment or within the running Docker container.
-
First install ClamAV Large Archive Scanner utility one of the two ways.
-
Then run this to install the test prerequisites:
source .venv/bin/activate
pip3 install -r ./src/clamav_large_archive_scanner/test/requirements.txt- Now run the unit tests:
pytest -vThis project is licensed under the BSD 3-Clause license.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent