The fast collectstatic
for Django projects with S3 as storage
backend.
Features
- Comparing and caching of md5 checksums before uploading
- Parallel file uploads using Python's multiprocessing module
Running Django's collectstatic
command can become really slow as
more and more files are added to a project, especially if heavy
libraries such as jQuery UI are included in the source. This is a custom
management command that compares the md5 sum of files with S3 and
completely ignores modified_time
. The results of the hash lookups
are cached locally using your default Django cache. This can make
deploying much faster!
Install the app using pip:
$ pip install Collectfast
Make sure you have this in your settings file and add 'collectfast'
to your INSTALLED_APPS
:
STATICFILES_STORAGE = "storages.backends.s3boto.S3BotoStorage"
AWS_PRELOAD_METADATA = True
INSTALLED_APPS = (
# …
'collectfast',
)
'collectfast'
should come before 'django.contrib.staticfiles'
.
Please note, that failure to do so will cause Django to use
django.contrib.staticfiles
's collectstatic
.
Note: preload_metadata
of the storage class will be overwritten
even if AWS_PRELOAD_METADATA
is not set to True see
#30
Collectfast overrides Django's builtin collectstatic
command so just
run python manage.py collectstatic
as normal. You can disable
Collectfast by using the --disable-collectfast
option.
You can also disable collectfast by setting
COLLECTFAST_ENABLED = False
in your settings file. This is useful
when using a local file storage backend for development.
It's recommended to setup a dedicated cache backend for Collectfast.
Every time Collectfast does not find a lookup for a file in the cache it
will trigger a lookup to the storage backend, so it's recommended to
have a fairly high TIMEOUT
setting.
Set up your dedicated cache in settings.py with the
COLLECTFAST_CACHE
setting:
CACHES = {
'default': {
# Your default cache
},
'collectfast': {
# Your dedicated Collectfast cache
}
}
COLLECTFAST_CACHE = 'collectfast'
By default Collectfast will use the default
cache.
Note: Collectfast will never clean the cache of obsolete files. To
clean out the entire cache, use cache.clear()
. Read more about
Django's cache
framework.
Note: We recommend you to set the MAX_ENTRIES
setting if you
have more than 300 static files, see
#47
The parallelization feature enables parallel file uploads using Python's
multiprocessing module. Enable it by setting the COLLECTFAST_THREADS
setting.
To enable parallelization of file copying, a dedicated cache backend must be setup and it must use a backend that is threadsafe, i.e. something other than Django's default LocMemCache.
COLLECTFAST_THREADS = 20
By default, Collectfast will suppress any exceptions that happens when copying
and let Django's collectstatic
handle it. To debug those suppressed errors
you can set COLLECTFAST_DEBUG = True
in your Django settings file.
Please feel free to contribute by using issues and pull requests. Discussion is open and welcome.
Testing
To run tests with tox, setup a virtualenv and install tox with
pip install tox
then run tox
in the project directory. To only run
tests for a certain environment run e.g. tox -e py35-django110
.
Collectfast is licensed under the MIT License, see LICENSE file for more information. Previous versions of Collectfast was licensed under Creative Commons Attribution-ShareAlike 3.0 Unported License.