Support for S3 and CDN - CloudFront

[BhuviTheDataGuy]

Support for S3 and CDN - CloudFront

[BhuviTheDataGuy]

worked!
Awesome

[horilla-opensource]

Yes, we are also trying to find out the cause of this issue. But couldn't clearly identify the cause of this.

[horilla-opensource]

Hi @BhuviTheDataGuy ,
Can you try out with these below code in the settings.py ?

AWS_ACCESS_KEY_ID = 'your-access-key'
AWS_SECRET_ACCESS_KEY = 'your-secret-access-key'
AWS_STORAGE_BUCKET_NAME = 'your-bucket-name'
AWS_S3_REGION_NAME = 'your-bucket-region' 
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com'% AWS_STORAGE_BUCKET_NAME
AWS_S3_FILE_OVERWRITE = False

STORAGES = {

    # Media file (image) management   
    "default": {
        "BACKEND": "storages.backends.s3boto3.S3StaticStorage",
    },
    
    # CSS and JS file management
    "staticfiles": {
        "BACKEND": "storages.backends.s3boto3.S3StaticStorage",
    },
}

There were some missing elements in the blog. Sorry for the mistake in that.

With Regards,
Team Horilla

[horilla-opensource]

Also can you make sure that the public access to the bucket is not blocked.

[BhuviTheDataGuy]

Getting this error

django.core.exceptions.ImproperlyConfigured: DEFAULT_FILE_STORAGE/STORAGES are mutually exclusive.

[horilla-opensource]

Can you check inside the settings.py if there is a variable defined with name DEFAULT_FILE_STORAGE ?

[BhuviTheDataGuy]

Nope, just confirmed that I didnt use DEFAULT_FILE_STORAGE inside the settings.py

[horilla-opensource]

Can you run the pip freeze in the project environment and share the output?

[BhuviTheDataGuy]

acme==1.21.0
APScheduler==3.10.4
arabic-reshaper==3.0.0
asgiref==3.8.1
asn1crypto==1.5.1
attrs==21.2.0
Automat==20.2.0
Babel==2.8.0
bcrypt==3.2.0
blinker==1.4
boto3==1.34.91
botocore==1.34.91
certbot==1.21.0
certbot-nginx==1.21.0
certifi==2020.6.20
cffi==1.16.0
chardet==4.0.0
charset-normalizer==3.3.2
click==8.1.7
cloud-init==23.4.4
colorama==0.4.4
command-not-found==0.3
ConfigArgParse==1.5.3
configobj==5.0.6
constantly==15.1.0
cryptography==42.0.5
cssselect2==0.7.0
dbus-python==1.2.18
distro==1.7.0
distro-info==1.1+ubuntu0.2
Django==4.2.11
django-admin==2.0.2
django-amazon-ses==4.0.1
django-apscheduler==0.6.2
django-cors-headers==4.3.1
django-environ==0.11.2
django-excel-response2==3.0.6
django-filter==24.2
django-haystack==3.2.1
django-jsonfield==1.4.1
django-mathfilters==1.0.0
django-model-utils==4.5.0
django-s3-storage==0.15.0
django-ses==4.0.0
django-simple-history==3.5.0
django-six==1.0.5
django-storages==1.14.2
django-widget-tweaks==1.5.0
ec2-hibinit-agent==1.0.0
et-xmlfile==1.1.0
excel-base==1.0.4
future==1.0.0
gunicorn==22.0.0
hibagent==1.0.1
html5lib==1.1
httplib2==0.20.2
hyperlink==21.0.0
idna==3.3
importlib-metadata==4.6.4
incremental==21.3.0
isoweek==1.3.3
jeepney==0.7.1
Jinja2==3.0.3
jmespath==1.0.1
josepy==1.10.0
jsonpatch==1.32
jsonpointer==2.0
jsonschema==3.2.0
keyring==23.5.0
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lxml==5.2.1
MarkupSafe==2.0.1
more-itertools==8.10.0
netifaces==0.11.0
numpy==1.26.4
oauthlib==3.2.0
openpyxl==3.1.2
oscrypto==1.3.0
packaging==24.0
pandas==2.2.2
parsedatetime==2.6
pexpect==4.8.0
pillow==10.3.0
psycopg2==2.9.9
psycopg2-binary==2.9.9
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.1
pycparser==2.22
PyGObject==3.42.1
PyHamcrest==2.0.2
pyHanko==0.23.2
pyhanko-certvalidator==0.26.3
PyICU==2.8.1
PyJWT==2.3.0
pyOpenSSL==24.0.0
pyparsing==2.4.7
pypdf==4.2.0
pypng==0.20220715.0
pyRFC3339==1.1
pyrsistent==0.18.1
pyserial==3.5
python-apt==2.4.0+ubuntu3
python-bidi==0.4.2
python-dateutil==2.9.0.post0
python-debian==0.1.43+ubuntu1.1
python-magic==0.4.24
pytz==2022.1
PyYAML==6.0.1
pyzk==0.9
qrcode==7.4.2
reportlab==4.0.9
requests==2.31.0
requests-toolbelt==0.9.1
s3transfer==0.10.1
screen==1.0.1
SecretStorage==3.3.1
service-identity==18.1.0
six==1.16.0
sos==4.5.6
sqlparse==0.5.0
ssh-import-id==5.11
svglib==1.5.1
swapper==1.3.0
systemd-python==234
TimeConvert==3.0.13
tinycss2==1.3.0
Twisted==22.1.0
typing_extensions==4.11.0
tzdata==2024.1
tzlocal==5.2
ubuntu-advantage-tools==8001
ufw==0.36.1
unattended-upgrades==0.1
uritools==4.0.2
urllib3==1.26.5
utils==1.0.2
wadllib==1.3.6
webencodings==0.5.1
whitenoise==6.6.0
Whoosh==2.7.4
xhtml2pdf==0.2.15
XlsxWriter==3.2.0
xlwt==1.3.0
zipp==1.0.0
zope.component==4.3.0
zope.event==4.4
zope.hookable==5.1.0
zope.interface==5.4.0

[horilla-opensource]

Can you update the values of STORAGE variable with:

STORAGES = {

    # Media file (image) management   
    "default": {
        "BACKEND": "storages.backends.s3boto3.S3StaticStorage",
    },
}
#removed the staticfiles location for testing

and check ?

[BhuviTheDataGuy]

Same error

DEBUG logs

root@ip-10-10-28-15:/opt/horilla# python3 manage.py collectstatic
INFO:environ.environ:/opt/horilla/.env not found - if you're not configuring your environment separately, check this.
DEBUG:environ.environ:get 'SECRET_KEY' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'DEBUG' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'ALLOWED_HOSTS' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'DATABASE_URL' casted as 'None' with default 'None'
DEBUG:environ.environ:get 'DB_ENGINE' casted as 'None' with default 'django.db.backends.sqlite3'
DEBUG:environ.environ:get 'DB_NAME' casted as 'None' with default '/opt/horilla/TestDB_Horilla.sqlite3'
DEBUG:environ.environ:get 'DB_USER' casted as 'None' with default ''
DEBUG:environ.environ:get 'DB_PASSWORD' casted as 'None' with default ''
DEBUG:environ.environ:get 'DB_HOST' casted as 'None' with default ''
DEBUG:environ.environ:get 'DB_PORT' casted as 'None' with default ''
DEBUG:environ.environ:get 'CSRF_TRUSTED_ORIGINS' casted as 'None' with default '<NoValue>'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 255, in fetch_command
    app_name = commands[subcommand]
KeyError: 'collectstatic'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/horilla/manage.py", line 22, in <module>
    main()
  File "/opt/horilla/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 262, in fetch_command
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 282, in __init__
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: STATICFILES_STORAGE/STORAGES are mutually exclusive.

[horilla-opensource]

Hi @BhuviTheDataGuy ,
If you are not using the django-environ for setting the environment variables for the project, can you remove the file with name similar to .env or .env.dist and try again?

With Regards,
Team Horilla

[BhuviTheDataGuy]

Removed .env.dist and restarted the app. But still same error.

[horilla-opensource]

Can you revert back all the changes and try again with the previous configurations that you had (with public access to s3 bucket)?

[BhuviTheDataGuy]

I did it, but nothing is uploaded to s3

Found another file with the destination path 'admin/img/inline-delete.svg'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'admin/img/gis/move_vertex_off.svg'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'admin/img/gis/move_vertex_on.svg'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.
Found another file with the destination path 'pipeline/pipeline.js'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure everystatic file has a unique path.
Found another file with the destination path 'recruitment/candidate.js'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.

0 static files copied to '/opt/horilla/staticfiles', 761 unmodified, 608 post-processed.

[horilla-opensource]

From the log I think the files are still getting loaded to the local static directory.
Can you share the complete contents of the settings.py file?

[BhuviTheDataGuy]

I think I figured it out, there was a line in the bottom

STATICFILES_STORAGE = "whitenoise.storage.CompressedStaticFilesStorage"

Removed it, and the files are now uploaded to S3.

[horilla-opensource]

Oh, yes that might have caused the issue because for Django to upload the files to the S3 bucket the STATICFILES_STORAGE should be configured to storages.backends.s3boto3.S3Boto3Storage.
It was replaced with the whitenoise configuration as the value was overwritten by it.

[BhuviTheDataGuy]

One last issue in this.

In the reimbursement, I have uploaded a file. When I click on view, the URL is

https://mydomain.comhttps//s3bucket/prefix/etcetc

How can I change it? Ideally, it should be from my cloudfront.

All the static files are served from the S3 itself instead of cloudfront.

AWS_STORAGE_BUCKET_NAME = 'XXX-XXX-horilla'
#AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_REGION_NAME = "ap-south-1"
AWS_S3_SIGNATURE_VERSION = "s3v4"
AWS_QUERYSTRING_EXPIRE = 604800
CLOUDFRONT_DOMAIN = 'XXX.cloudfront.net'

STATIC_LOCATION = "static"
STATIC_URL = f'{CLOUDFRONT_DOMAIN}/static/'
# Add your path in the STATICFILES_STORAGE
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE =  'storages.backends.s3boto3.S3Boto3Storage'

MEDIA_LOCATION = "media"
MEDIA_URL = f'{CLOUDFRONT_DOMAIN}/media/'

[horilla-opensource]

Can you change the AWS_S3_CUSTOM_DOMAIN to you CLOUDFRONT_DOMAIN and check ?

[BhuviTheDataGuy]

Profile images are not loading also whatever images im uploading to leave type also not updating.

I tried to upload again and again, but its not showing.

NO errors in app

when I click on the hyperlink, in the new tab, the image is visible, but its not rendering inside.

And still the attachment link in reimbursement is still using domain.comhttps//cdnaddress.

[BhuviTheDataGuy]

Tried many URL format options, but still not able to fix these 2 issues.

1. Images are not loading (the profile image and the leave type images)
2. Attachments from reimbursements (URL format is wrong)

AWS_STORAGE_BUCKET_NAME = 'xx-xxx-horilla'
AWS_S3_REGION_NAME = "ap-south-1"
AWS_S3_SIGNATURE_VERSION = "s3v4"
AWS_QUERYSTRING_EXPIRE = 604800
CLOUDFRONT_DOMAIN = 'xxxxx.cloudfront.net'
AWS_S3_CUSTOM_DOMAIN = CLOUDFRONT_DOMAIN

STATIC_LOCATION = "static"
STATIC_URL = f'https://{CLOUDFRONT_DOMAIN}/static/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

MEDIA_LOCATION = "media"
MEDIA_URL = f'https://{CLOUDFRONT_DOMAIN}/media/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

[horilla-opensource]

Hi @BhuviTheDataGuy ,
We are looking into it.
We'll provide and update on it asap.

With Regards,
Team Horilla

[horilla-opensource]

Hi @BhuviTheDataGuy ,
In the settings.py, please comment out the line MEDIA_ROOT = os.path.join(BASE_DIR, "media/") and if you face any error after commenting it out, remove the line

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

and restart the server.
This should upload the media file to the s3 media folder and retrieve the same from there when required.

Please let us know if you face any more issues.

With Regards,
Team Horilla

[BhuviTheDataGuy]

The image is fixed, but still the last one URL on the attachments

the URL should be https://somehting.cloudfront.net/payroll/reimbursment/file.pdf
but what Im getting is,
http://myhorilladomain.comhttps//somehting.cloudfront.net/payroll/reimbursements/file.pdf

[horilla-opensource]

Hi @BhuviTheDataGuy ,
We are checking on that.
Meanwhile can you refer this video for the configuration section
Django with AWS - S3 Buckets and CloudFront Distributions for Media Files

With Regards,
Team Horilla

[BhuviTheDataGuy]

I followed everything, but not sure where is the issue.

[BhuviTheDataGuy]

Ok, let me know once you are done.
And we are live with whatever the customization we did.

Thanks for your efforts. I need to talk to any one of the team members who is managing this project. Any email?

[horilla-opensource]

Thanks.