jasongi/collectfasta

A fasta collectstatic command.

Collectfasta

A faster collectstatic command. This is a fork of the archived collectfast by @antonagestam and a drop-in replacement - you must not have both installed at the same time.

Features

• Efficiently decide what files to upload using cached checksums
• Two-pass uploads for Manifest storage which can be slow using a single pass - files are hashed/post-processed in Memory/Local filesystem and then the result is copied.
• Parallel file uploads

Supported Storage Backends

• storages.backends.s3boto3.S3Boto3Storage
• storages.backends.s3boto3.S3StaticStorage
• storages.backends.s3boto3.S3ManifestStaticStorage
• storages.backends.gcloud.GoogleCloudStorage
• django.core.files.storage.FileSystemStorage

Running Django's collectstatic command can become painfully slow as more and more files are added to a project, especially when heavy libraries such as jQuery UI are included in source code. Collectfasta customizes the builtin collectstatic command, adding different optimizations to make uploading large amounts of files much faster.

Installation

Install the app using pip:

$ python3 -m pip install Collectfasta

Make sure you have this in your settings file and add 'collectfasta' to your INSTALLED_APPS, before 'django.contrib.staticfiles':

STORAGES = (
    {
        "staticfiles": {
            "BACKEND": "storages.backends.s3.S3Storage",
        },
    },
)
COLLECTFASTA_STRATEGY = "collectfasta.strategies.boto3.Boto3Strategy"
INSTALLED_APPS = (
    # ...
    "collectfasta",
)

Note: 'collectfasta' must come before 'django.contrib.staticfiles' in INSTALLED_APPS.

Upload Strategies

Collectfasta Strategy	Storage Backend
collectfasta.strategies.boto3.Boto3Strategy	storages.backends.s3.S3Storage
collectfasta.strategies.boto3.Boto3Strategy	storages.backends.s3.S3StaticStorage
collectfasta.strategies.boto3.Boto3ManifestMemoryStrategy (recommended)	storages.backends.s3.S3ManifestStaticStorage
collectfasta.strategies.boto3.Boto3ManifestFileSystemStrategy	storages.backends.s3.S3ManifestStaticStorage
collectfasta.strategies.gcloud.GoogleCloudStrategy	storages.backends.gcloud.GoogleCloudStorage
collectfasta.strategies.filesystem.FileSystemStrategy	django.core.files.storage.FileSystemStorage

Custom strategies can also be made for backends not listed above by implementing the collectfasta.strategies.Strategy ABC.

Usage

Collectfasta overrides Django's builtin collectstatic command so just run python manage.py collectstatic as normal.

You can disable Collectfasta by using the --disable-collectfasta option or by setting COLLECTFASTA_ENABLED = False in your settings file.

Setting Up a Dedicated Cache Backend

It's recommended to setup a dedicated cache backend for Collectfasta. Every time Collectfasta does not find a lookup for a file in the cache it will trigger a lookup to the storage backend, so it's recommended to have a fairly high TIMEOUT setting.

Configure your dedicated cache with the COLLECTFASTA_CACHE setting:

CACHES = {
    "default": {
        # Your default cache
    },
    "collectfasta": {
        # Your dedicated Collectfasta cache
    },
}

COLLECTFASTA_CACHE = "collectfasta"

If COLLECTFASTA_CACHE isn't set, the default cache will be used.

Note: Collectfasta will never clean the cache of obsolete files. To clean out the entire cache, use cache.clear(). See docs for Django's cache framework.

Note: We recommend you to set the MAX_ENTRIES setting if you have more than 300 static files, see #47.

Enable Parallel Uploads

The parallelization feature enables parallel file uploads using Python's concurrent.futures module. Enable it by setting the COLLECTFASTA_THREADS setting.

To enable parallel uploads, a dedicated cache backend must be setup and it must use a backend that is thread-safe, i.e. something other than Django's default LocMemCache.

COLLECTFASTA_THREADS = 20

Debugging

By default, Collectfasta will suppress any exceptions that happens when copying and let Django's collectstatic handle it. To debug those suppressed errors you can set COLLECTFASTA_DEBUG = True in your Django settings file.

Contribution

Please feel free to contribute by using issues and pull requests. Discussion is open and welcome.

Testing

The test suite is built to run against localstack / fake-gcs-server OR live S3 and GCloud buckets. To run live tests locally you need to provide API credentials to test against as environment variables.

export AWS_ACCESS_KEY_ID='...'
export AWS_SECRET_ACCESS_KEY='...'
export GCLOUD_API_CREDENTIALS_BASE64='{...}'  # Google Cloud credentials as Base64'd json

Install test dependencies and target Django version:

python3 -m pip install -r test-requirements.txt
python3 -m pip install django==5.0.2

Run test suite:

make test

Run test against localstack/fake-gcs-server:

make test-docker

Code quality tools are broken out from test requirements because some of them only install on Python >= 3.7.

python3 -m pip install -r lint-requirements.txt

Run linters and static type check:

make lint

License

Collectfasta is licensed under the MIT License, see LICENSE file for more information.

collectfasta's Issues

Simplify tests to just use pytest

The tests are currently a confusing mixture of unittest and pytest held together by some decorators in:

https://github.com/jasongi/collectfasta/blob/master/collectfasta/tests/utils.py

It will be much easier to write tests and for others contribute to the package if we use regular pytest constructs instead of these such as Parametrizing fixtures and marks to replace generate_storage_variations, many, make_test_aws_backends etc.

s3 location prefix matching

When using a custom storage class like the following I receive the error "FileNotFoundError: [Errno 2] No such file or directory: 'files.json'" during collectstatic.

class StaticStorage(S3ManifestStaticStorage):
    location = "static"
    manifest_name = "staticfiles.json"

It appears the following code is changing "staticfiles.json" to "files.json" in my project.

collectfasta/collectfasta/strategies/hashing.py

Lines 111 to 119 in 96add58

	class WithoutPrefixMixin(StrategyWithLocationProtocol):
	def copy_args_hook(self, args: Task) -> Task:
	assert isinstance(self.remote_storage, HasLocationProtocol)
	return (
	args[0].replace(self.remote_storage.location, ""),
	args[1].replace(self.remote_storage.location, ""),
	args[2],
	)

I am thinking a possible solution would be to first check if self.remote_storage.location does not end with a "/" and append a trailing slash if needed. Happy to help with the work if you agree the fix is appropriate.

And thank you for the package, I am hoping this will improve my deploy times :)