django-cumulus/django-cumulus

An interface to python-swiftclient and rackspace cloudfiles API from Django.

django-cumulus

django-cumulus provides a set of tools to utilize the python-swiftclient and Rackspace Cloud Files API from Django. It includes a custom file storage class, CumulusFilesStorage.

More documentation about the usage and installation of django-cumulus can be found on django-cumulus.readthedocs.org.

The source code for django-cumulus can be found and contributed to on github.com/django-cumulus/django-cumulus. There you can also file issues.

To find out what's new in this version of django-cumulus, please see the changelog

Core Team

This library is maintained by:

• Ferrix Hovi (@ferrix)
• Thomas Schreiber (@rizumu)

They will be able to answer questions, give architectural guidance and merge pull requests.

Quick Installation Guide

Please visit django-cumulus.readthedocs.org for the full documentation.

• Install Django Cumulus with your favorite Python package manager:

  pip install django-cumulus
  

• To install the in-development version of Django Cumulus:

  pip install django-cumulus==dev
  

• Add 'cumulus' to the INSTALLED_APPS setting in your project's settings.py:

  INSTALLED_APPS = (
      ...
      'cumulus',
      ...
  )
  

• Add the following to your project's settings.py file:

  CUMULUS = {
      'USERNAME': 'YourUsername',
      'API_KEY': 'YourAPIKey',
      'CONTAINER': 'ContainerName',
      'PYRAX_IDENTITY_TYPE': 'rackspace',
  }
  DEFAULT_FILE_STORAGE = 'cumulus.storage.CumulusStorage'
  

The PYRAX_IDENTITY_TYPE parameter can either be rackspace or keystone depending on whether you use Rackspace or OpenStack respectively.

django-cumulus's Issues

Pyrax Support

Rackspace's new API version is necessary to sync with containers outside of your "default" location (chicago vs dallas)

They deprecated python-cloudfiles in favor of pyrax.

Random syncstatic/SSL Errors

I get cloudfiles read timeout errors every few days:

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/django/db/models/fields/files.py", line 70, in _get_url
  return self.storage.url(self.name)

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cumulus/storage.py", line 207, in url
  return '%s/%s' % (self.container_url, name)

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cumulus/storage.py", line 91, in _get_container_url
  self._container_public_uri = self.container.public_uri()

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cumulus/storage.py", line 72, in _get_container
  self.container = self.connection.get_container(

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cumulus/storage.py", line 62, in _get_connection
  **self.connection_kwargs)

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cloudfiles/__init__.py", line 98, in get_connection
  return Connection(*args, **kwargs)

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cloudfiles/connection.py", line 86, in __init__
  self._authenticate()

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cloudfiles/connection.py", line 92, in _authenticate
  (url, self.cdn_url, self.token) = self.auth.authenticate()

File "/home/deploy/.virtualenvs/rizumu/lib/python2.6/site-packages/cloudfiles/authentication.py", line 68, in authenticate
  response = conn.getresponse()

File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
  response.begin()

File "/usr/lib/python2.6/httplib.py", line 391, in begin
  version, status, reason = self._read_status()

File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
  line = self.fp.readline()

File "/usr/lib/python2.6/socket.py", line 397, in readline
  data = recv(1)

File "/usr/lib/python2.6/ssl.py", line 96, in <lambda>
  self.recv = lambda buflen=1024, flags=0: SSLSocket.recv(self, buflen, flags)

File "/usr/lib/python2.6/ssl.py", line 222, in recv
  raise x

SSLError: The read operation timed out

I think a retry loop, something along these lines might be able solve it.
https://launchpadlibrarian.net/62711399/bug-629136.diff

Just not sure where exactly in cumulus this should go. Thoughts?

Related to:
https://bitbucket.org/david/django-storages/issue/65/cloudfiles-and-sslerror
http://stackoverflow.com/questions/4023595/django-ssl-error

Tests fail on Django 1.4.4, 1.5.1, 1.6 alpha with the same 4 errors

Here's the result of running the tests locally:

me$ python manage.py test cumulus
Creating test database for alias 'default'...
EEEE
======================================================================
ERROR: test_custom_content_type (cumulus.tests.CumulusTests)
Ensure content type is set properly when custom content type is supplied.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/tests/__init__.py", line 16, in setUp
    self.thing = Thing.objects.create(image=self.image, document=self.document, custom=self.custom)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 137, in create
    return self.get_query_set().create(**kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 377, in create
    obj.save(force_insert=True, using=self.db)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 911, in execute_sql
    for sql, params in self.as_sql():
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 872, in as_sql
    for obj in self.query.objs
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 249, in pre_save
    file.save(file.name, file, save=False)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 86, in save
    self.name = self.storage.save(name, content)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 44, in save
    name = self.get_available_name(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 70, in get_available_name
    while self.exists(name):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 269, in exists
    self._get_cloud_obj(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 162, in _get_cloud_obj
    return self.container.get_object(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 126, in _get_container
    self.container = self.connection.get_container(self.container_name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 115, in _get_connection
    **self.connection_kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/__init__.py", line 98, in get_connection
    return Connection(*args, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/connection.py", line 84, in __init__
    raise TypeError("Incorrect or invalid arguments supplied")
TypeError: Incorrect or invalid arguments supplied

======================================================================
ERROR: test_file_api (cumulus.tests.CumulusTests)
Make sure we can access file attributes using the Django File API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/tests/__init__.py", line 16, in setUp
    self.thing = Thing.objects.create(image=self.image, document=self.document, custom=self.custom)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 137, in create
    return self.get_query_set().create(**kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 377, in create
    obj.save(force_insert=True, using=self.db)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 911, in execute_sql
    for sql, params in self.as_sql():
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 872, in as_sql
    for obj in self.query.objs
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 249, in pre_save
    file.save(file.name, file, save=False)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 86, in save
    self.name = self.storage.save(name, content)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 44, in save
    name = self.get_available_name(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 70, in get_available_name
    while self.exists(name):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 269, in exists
    self._get_cloud_obj(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 162, in _get_cloud_obj
    return self.container.get_object(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 126, in _get_container
    self.container = self.connection.get_container(self.container_name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 115, in _get_connection
    **self.connection_kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/__init__.py", line 98, in get_connection
    return Connection(*args, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/connection.py", line 84, in __init__
    raise TypeError("Incorrect or invalid arguments supplied")
TypeError: Incorrect or invalid arguments supplied

======================================================================
ERROR: test_image_content_type (cumulus.tests.CumulusTests)
Ensure content type is set properly for the uploaded image.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/tests/__init__.py", line 16, in setUp
    self.thing = Thing.objects.create(image=self.image, document=self.document, custom=self.custom)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 137, in create
    return self.get_query_set().create(**kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 377, in create
    obj.save(force_insert=True, using=self.db)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 911, in execute_sql
    for sql, params in self.as_sql():
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 872, in as_sql
    for obj in self.query.objs
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 249, in pre_save
    file.save(file.name, file, save=False)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 86, in save
    self.name = self.storage.save(name, content)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 44, in save
    name = self.get_available_name(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 70, in get_available_name
    while self.exists(name):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 269, in exists
    self._get_cloud_obj(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 162, in _get_cloud_obj
    return self.container.get_object(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 126, in _get_container
    self.container = self.connection.get_container(self.container_name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 115, in _get_connection
    **self.connection_kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/__init__.py", line 98, in get_connection
    return Connection(*args, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/connection.py", line 84, in __init__
    raise TypeError("Incorrect or invalid arguments supplied")
TypeError: Incorrect or invalid arguments supplied

======================================================================
ERROR: test_text_content_type (cumulus.tests.CumulusTests)
Ensure content type is set properly for the uploaded text.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/tests/__init__.py", line 16, in setUp
    self.thing = Thing.objects.create(image=self.image, document=self.document, custom=self.custom)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 137, in create
    return self.get_query_set().create(**kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 377, in create
    obj.save(force_insert=True, using=self.db)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 911, in execute_sql
    for sql, params in self.as_sql():
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 872, in as_sql
    for obj in self.query.objs
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 249, in pre_save
    file.save(file.name, file, save=False)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/db/models/fields/files.py", line 86, in save
    self.name = self.storage.save(name, content)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 44, in save
    name = self.get_available_name(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/django/core/files/storage.py", line 70, in get_available_name
    while self.exists(name):
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 269, in exists
    self._get_cloud_obj(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 162, in _get_cloud_obj
    return self.container.get_object(name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 126, in _get_container
    self.container = self.connection.get_container(self.container_name)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cumulus/storage.py", line 115, in _get_connection
    **self.connection_kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/__init__.py", line 98, in get_connection
    return Connection(*args, **kwargs)
  File "/Users/me/.virtualenvs/mine/lib/python2.7/site-packages/cloudfiles/connection.py", line 84, in __init__
    raise TypeError("Incorrect or invalid arguments supplied")
TypeError: Incorrect or invalid arguments supplied

----------------------------------------------------------------------
Ran 4 tests in 0.006s

FAILED (errors=4)
Destroying test database for alias 'default'...

May syncstatic command operates like collectstatic?

The syncstatic command complains about the lack of files in my STATIC_ROOT directory, as if I had to collectstatic to the local STATIC_ROOT before sync.

I use to have a settings like:

STATIC_ROOT = '.../site-static'

STATICFILES_DIRS = (
    '.../static',
)

The syncstatic won't get my static of STATICFILES_DIRS or any application specific static (e.g. grappelli).

Is this a desired approach?

Add CDN purging

Version 1.7.9 of python-cloudfiles introduced container- and object-level CDN purging. Come up with a clever way to handle purging through Django. Not sure if the best way is a management command, custom admin interface or something else.

syncstatic command not found

I installed cumulus with - pip install django-cumulus and all the necessary things to my app settings file. But when i do django-admin.py syncstatic I get Unknown command: 'syncstatic'. Can you please point me. what am I doing wrong.

My env details :
Python 2.7.3
Django 1.4.2
django-cumulus 1.0.5
python-cloudfiles 1.7.10

AUTH_URL not passing to Connection in management commands

See https://github.com/richleland/django-cumulus/blob/master/cumulus/management/commands/container_list.py#L32 we need to add authurl=settings.CUMULUS['AUTH_URL'] on line 35

Same on container_create, container_delete and container info.

Does not work with USE_PYRAX = False

Places I ran into issues:

1. ThreadSafeSwiftclientStorage will only connect with pyrax
2. get_container returns None
3. _get_object fails because of get_container (self.connection.get_object(self.container_name, name) could be used instead)
4. Swiftclient's get_object returns a 2-tuple of a metadata dictionary and the file data. This expects an object to be returned with attributes like total_bytes.

I wanted to use without pyrax because it has some pretty awful performance issues with get_object. See pycontribs/pyrax#156 and pycontribs/pyrax#160.

Wheel support

See http://pythonwheels.com/. Simple to setup and publish. We should consider throwing in a Makefile so we can publish easier too.

Acceptance tests

This library does not have a test set that could verify that even the basic functionality works when making a new release. This makes merging pull requests and releasing a dreaded task for me and requires a lot of time and effort to cover some of the use cases that I don't have. We need some tests to verify it did not break.

So anybody willing contribute to this library, hear me out:

Any pull requests adding to the test coverage will be accepted to within a week from committing.

Additionally:

No pull requests adding functionality without introducing tests for that functionality will be merged.

Unicode filenames problem with Rackspace Cloudfiles

Hi Rich,

The python-cloudfiles app has problems with unicode filenames. If I use filename with non-ascii characters, I got an error.

The problem is with the urllib's quote function, as listed on http://bugs.python.org/issue1712522 and http://mail.python.org/pipermail/python-dev/2006-July/067248.html - it doesn't convert unicode, the string must be converted to utf-8 before being used.

I don't know who must convert it: urllib, python-cloudfiles or django-cumulus. Anyway, I think it is easy to accomplish it here ;)

I fixed it and I can send a patch or even make a fork and apply on it. What do you prefer?

SSLError when sending several files

Hi Rich,

We talked about this error on the django-storages bitbucket... I migrated my app to django-cumulus and I still have the exception being rise after some time.

I don't think it is a timeout problem because it happens on a unpredictable intervals. To be honest with you, on Friday, when I migrate about 1,000 files to a cloudfiles container, i got this error several times. Today I'm trying to reproduce the error and it happens only once, much less than on Friday.

To fix it, I'm deleting the connection and the container attributes from the storage instance, it forces the storage to make a new connection.

What do you think about make it happens by default inside CloudFilesStorage?

Migration from default storage class

This is more of a question than an issue.

I'm currently using a CharField to store a path to a file on the local harddrive, and I would like to upgrade to django-cumulus. I have already uploaded all of the files to cloudfiles and I need to create a migration (in South) to move from CharField to FileField with the new storage class in place.

Can you think of a way of doing that? Would you happen to have any tips?

Thanks

Leading '/' in upload_to path leads to infinite-loop condition.

I created a function that returns a dynamic upload_to location for my files. I started this path with a leading '/'. This threw Cumulus into an endless loop as seen in my other issue. I just wanted to post this issue in case anybody else runs into it. It'd be a nice to see a "check" in place for this condition. If I get time, I may fork and create a patch.

Support/document multiple regions

I created a new Cloud Files container in Chicago (ORD, which for me was the default) and django-cumulus could not see it. Had to file a ticket with support, who told me that this binding doesn't support multiple regions, and if I recreated the container in DFW it would work. It did. Still, this is confusing and ought to be documented at least until it is supported.

Remove support for old CUMULUS_* settings

In cumulus.settings we support the old CUMULUS_* style settings. We need to remove this backwards-compatibility in 1.2.

AUTH_URL missing in other management commands

It's still missing in container_create, container_delete and container info.

Sorry for not being clear in the previous issue.

When saving a file with the pointer at the end, its saved with 0 bytes

This issue was observed in the following scenario.

A Django's ContentFile is created, and data is written. At this point the content file's pointer is at the end of the file (after the last appended data).

Using Django's default storage (filesystem), when saving the data to a FileField with its save method, its properly saved, however, when using SwiftclientStorage, the files end up with 0 bytes. Calling seek(0) on the content file before saving solves this issue (SwiftclientStorage properly saves the files correctly).

As this behavior differs from the default storage's behavior it might be a bug in cumulus.

Python3 support

'nuff said.

Pull changes in from forks

Since this mainline has been passive, the development has been going on in forks. Review and merge commits from @ngenworks and @lincolnloop branches. Would it be possible to get pull requests?

Add support for tempurls

This is my story with tempurls:

The rackspace docs are: http://docs.rackspace.com/files/api/v1/cf-devguide/content/TempURL-d1a4450.html

For getting the public URL and Tenant ID use:

curl -X POST https://lon.auth.api.rackspacecloud.com/v2.0/tokens -d 
"{\"auth\":{\"RAX-KSKEY:apiKeyCredentials\":{\"username\": 
\"$username\",\"apiKey\": \"$apiKey\",\"tenantName\": 
\"10013520\"}}}" -H "Content-Type: application/json" -H "Accept: 
application/json"

I added two new settings for cumulus:

CUMULUS = {
    'USERNAME': 'xx',
    'API_KEY': 'xxx',
    'CONTAINER': 'xxx',
    'AUTH_URL': 'uk_authurl',
    'TIMEOUT': 15,
    'TEMPURL_KEY': 'xxx_created_for_this_xxx',
    'CONTAINERS_PUBLIC_URL': 'https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_xxx_your_TenantID_xxx',
}

Then the code used is:

import hmac
from hashlib import sha1
from time import time

seconds = 60

# get the model file field from somewhere
file_field = i.image

container_name, file_name = (file_field.storage.container.name, file_field.name)

key = CUMULUS['TEMPURL_KEY']
public_url = CUMULUS['CONTAINERS_PUBLIC_URL']
method = 'GET'
expires = int(time() + seconds)

url = '%s/%s/%s' % (public_url, container_name, file_name)
base_url, object_path = url.split('/v1/')
object_path = '/v1/' + object_path

hmac_body = '%s\n%s\n%s' % (method, expires, object_path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()
print '%s%s?temp_url_sig=%s&temp_url_expires=%s' % (base_url, object_path, sig, expires)

Something like model_instance.file_field.temp_url would be really nice :) What do you think?

Accessing width/height on an image creates infinite loop

Hello,

I'm getting stuck in an infinite loop when uploading a file through Django admin.

Config:

INSTALLED_APPS += (
    'cumulus',
)

CUMULUS = {
    'USERNAME': '{ my username} ',
    'API_KEY': '{ my api key }',
    'AUTH_URL': "us_authurl",
    'CONTAINER': 'g3rdMediaMedia',
    'STATIC_CONTAINER': 'CementJobStatic',
    'SERVICENET': True,
    'TIMEOUT': 15,
    'USE_SSL': False,
    'REGION': 'ORD',
    'HEADERS': (
        (r'.*\.(eot|otf|woff|ttf)$', {
            'Access-Control-Allow-Origin': '*'
        }),
    ),
    'USE_PYRAX': True,
    'PYRAX_IDENTITY_TYPE': 'rackspace',
    'CNAMES': {
        '{ rackspace url }': 'http://static.domain.com',
        '{ rackspace url }': 'http://mediastatic.domain.com',
    }
}

DEFAULT_FILE_STORAGE = 'cumulus.storage.SwiftclientStorage'

STATICFILES_STORAGE = 'cumulus.storage.SwiftclientStaticStorage'

TEMPLATE_CONTEXT_PROCESSORS += (
    'cumulus.context_processors.cdn_url',
)

In production:

Nginx + Gunicorn + Django

$ pip freeze
Babel==1.3
Django==1.6
Jinja2==2.7
Markdown==2.3.1
MarkupSafe==0.18
MySQL-python==1.2.4b4
PIL==1.1.7
Pygments==1.6
Routes==1.13
Scrapy==0.20.2
South==0.7.6
Sphinx==1.2b1
Twisted==13.2.0
Wand==0.3.0
WebOb==1.2.3
Werkzeug==0.9.4
boxmath==0.1.3
bpython==0.12
coverage==3.6
cssselect==0.9.1
distribute==0.6.28
django-assets==0.1.0
django-audiovideo==0.0.1
django-autocomplete-light==1.4.7
django-braces==1.0.0
django-colorful==0.1.3
django-countdown==0.0.1
django-cumulus==1.0.12
django-debug-toolbar==0.11.0
django-discover-runner==0.4
django-extensions==1.2.5
django-football==0.0.1
django-fronts==0.0.1
django-g3rdmedia-utils==0.0.1
django-htmlblock==0.0.1
django-lister==0.0.1
django-localflavor==1.0
django-location-field==1.1
django-mathfilters==0.1.3
django-model-utils==1.3.1
django-people==0.1.0
django-radioschedule==0.0.1
django-roster==0.0.1
django-schools==0.0.1
django-sports==0.0.1
django-sports-associations==0.0.1
django-taggit==0.10
django-teams==0.0.1
django-timetable==0.1.6
django-typogrify==1.3
django-wrestling==0.0.1
docutils==0.10
gunicorn==17.5
hashids==0.8.3
httplib2==0.8
iso8601==0.1.8
json-schema-validator==2.3
jsonfield==0.9.19
keyring==3.3
logutils==0.3.3
lxml==3.3.0beta1
minimogrify==2.0.3
mock==1.0.1
nanoweb==1.0
os-diskconfig-python-novaclient-ext==0.1.1
os-networksv2-python-novaclient-ext==0.21
os-virtual-interfacesv2-python-novaclient-ext==0.14
pbr==0.5.23
prettytable==0.7.2
psycopg2==2.5.1
pyOpenSSL==0.13.1
pyrax==1.6.2
python-instagram==0.8.0
python-novaclient==2.15.0
python-swiftclient==1.8.0
pytz==2013b
queuelib==1.1.1
rackspace-auth-openstack==1.2
rackspace-novaclient==1.4
raven==3.2.1
rax-default-network-flags-python-novaclient-ext==0.1.3
rax-scheduled-images-python-novaclient-ext==0.2.1
repoze.lru==0.6
requests==2.1.0
sh==1.08
simplejson==3.3.0
six==1.4.1
smartypants==1.8.1
sqlparse==0.1.10
sss==1.2
testfixtures==3.0.0
w3lib==1.5
wsgiref==0.1.2
zope.interface==4.0.5

Model:

class Graphic(models.Model):
    def get_upload_to(self, filename):
        return '/'.join(['graphics', slugify(self.alt_text), filename])

    graphic = models.ImageField(_('Graphic'), upload_to=get_upload_to, width_field='width', height_field='height')
    width = models.IntegerField(_('Width'), default=-1)
    height = models.IntegerField(_('Height'), default=-1)
    alt_text = models.CharField(_('Alt Text'), max_length=140)

Debugging:

I cloned the repo locally, and put a ton of print statements in. (I really need to learn how to properly debug in Python)

just got file from _storage
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
<bound method SwiftclientStorageFile._get_pos of <SwiftclientStorageFile: graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg>>
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
SwiftclientStorageFile._get_pos()
SwiftclientStorageFile.seek()
0
SwiftclientStorageFile.read()
1024
SwiftclientStorageFile._get_size()
size: graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg
_get_object: graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg
_get_container
<Container 'g3rdMediaMedia'>
Ä'Listen-Card-Bedford-at-Chestnut-Ridge.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_1.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_10.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_11.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_12.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_13.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_14.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_2.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_3.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_4.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_5.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_6.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_7.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_8.jpg', 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_9.jpg', 'photos'Ü
_get_container
<Container 'g3rdMediaMedia'>
_get_container
<Container 'g3rdMediaMedia'>
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
1024
SwiftclientStorageFile.read()
2048
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
3072
SwiftclientStorageFile.read()
4096
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
7168
SwiftclientStorageFile.read()
8192
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
15360
SwiftclientStorageFile.read()
16384
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
31744
SwiftclientStorageFile.read()
32768
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
64512
SwiftclientStorageFile.read()
65536
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
130048
SwiftclientStorageFile.read()
131072
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
261120
SwiftclientStorageFile.read()
262144
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
445296
SwiftclientStorageFile.read()
524288
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
new pos
629472
SwiftclientStorageFile.read()
1048576
SwiftclientStorageFile._get_size()
size is
184176
SwiftclientStorageFile._get_file()
<Object 'graphics/wrestling-bedford-at-chestnut-ridge-dec-21-2013-on-wayc/Listen-Card-Bedford-at-Chestnut-Ridge_15.jpg' (image/jpeg)>
^C

Not sure how to fix. Or even how it is getting stuck in the loop.

Help?!?

Thanks,
Chad.

maximum recursion depth exceeded

I'm not sure on the cause of my problem but the storages.py -> _save() method is setup so that it can allow an infinite loop.

Primarily, these lines will run in an endless loop in certain conditions. (109:114)

(path, last) = os.path.split(name)                                                                            
if path:                                                                                                      
    try:                                                                                                      
        self.container.get_object(path)                                                                       
    except NoSuchObject:                                                                                      
        self._save(path, CloudStorageDirectory(path))  

Issues in CloudFilesStorageFile.read()

First of all if you don't specify num_bytes it defaults to None and in line 292 when trying to sum self._pos + num_bytes you get an exception.

And secondly when the file has been finished reading it should return "" and not None as it says in http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects

Django 1.3's "collectstatic" management command

I'm experimenting w/ django 1.3, and the new staticfiles contrib module. The module seems to have its own "collectstatic" management command that collects static media from all apps into a single place. I believe it uses the FileStorage backend to save the files. Will this management command effectively replace the "syncstatic" command that comes with django-cumulus? It'd be great if the docs could cover this.

Thanks!

Error when manually setting the mimetype

Hi Rich,

I have to set the mimetype of all images sent to the cloudfiles as application/octet-stream, this way, I can offer a download button to unexperienced internet users ;)

Cumulus does an automatic mimetype recognition and it works pretty well. The problem is when I need to manually set the mimetype:

if hasattr(content.file, 'content_type'):
    cloud_obj.content_type = content.file.content_type

The content_type attribute isn't on the content.file instance, it is in the content instance:

 if hasattr(content, 'content_type'):
     cloud_obj.content_type = content.content_type

If the content object is an subclass of the django.core.files.uploadedfile.UploadedFile, we can set the mimetype on his constructor:

file = UploadedFile(open(filename), name=name, content_type=content_type)

It is easy to fix it, I can send you a patch or even make a fork and request a pull. What do you prefer?

When using pyrax, content_encoding and gzipped content are not respected.

There is a TODO in the _save method of SwiftclientStorage about custom headers.

I didn't find a way to pass custom headers to Pyrax but it seems to accept another keyword argument called content_encoding.

If you agree, I can fix it and write some tests.

Admin static URLs not respecting STATIC_URL

I've successfully transferred all my static files to Rackspace CloudFiles using the syncstatic command. My STATIC_URL setting is /static/ and the files have all ended up in a static directory in my container. All good.

But when I visit my admin site, none of the static assets load. Their URLs are all like http://...rackcdn.com/path/to/file.css whereas they should be http://...rackcdn.com/static/path/to/file.css. ie, they're missing the value of STATIC_URL.

I haven't seen anyone else mention this, so maybe it's some simple configuration thing I've missed, or messed up, but I'm a bit stumped as to how best to correct those URLs.

Intermittently, I get `NoSuchContainer` on first request

I get NoSuchContainer about half the time when I'm trying to start my Django development server. This only occurs on the first request, and the container name is indeed in my Cloud Files account. I just created the container about an hour ago (not sure if there is some serious lag or something). Is this a known issue?

Upload to a configurable root path instead of STATIC_URL

It seems to me that django-cumulus needs a way to handle absolute urls being specified. It is legal to have a static url setting like:

STATIC_URL = "http://mycdn.com/static/"

However, if you do this and run /manage.py syncstatic -v 2 you can see that it uploads to the container:

$ ./manage.py syncstatic -v 2 -t
Uploaded http://mycdn.com/static/500.html
Uploaded http://mycdn.com/static/favicon.ico
Uploaded http://mycdn.com/static/admin/css/base.css
....

This is really not what you want because it goes into the CDN server's container under my container/http://mycdn.com/static/500.html

When you think about it, you're not uploading to a URL, but to a path. I suggest having a CUMULUS_ROOT setting so that we can specify where to upload to. It can be made backwards compatible by defaulting to /static/ but people can then change it if they specify absolute urls.

IdentityClassNotDefined when importing SwiftclientStorage

At some point in moving from one storage method to another, there seems to be some bits that weren't checked out before the code was committed and made available on PyPi (my advice here is to thoroughly test your code before submitting to PyPi).

When importing cumulus.storage.SwiftclientStorage, the following error is produced:

IdentityClassNotDefined: No identity class has been defined for the current environment.

This occurs if you run manage.py shell or manage.py syncdb as it's in my models.py file (I'm not using it as the default storage mechanism, but rather for one particular field in my model).

Hope this is helpful. Do let me know if you'd like more info, or if you need more help testing large changes in the codeset before they go up to the Python Packaging Index.

Cheers!
-Mark

Clarify documentation around PYRAX_IDENTITY_TYPE

If a user doesn't understand the underlying Pyrax library, they may not set PYRAX_IDENTITY_TYPE or read the docs to find that an env var or pyrax config file are needed. We should just make this point a little more explicit in the documentation.

Tests url validation problem

Hi Rich,

I think there is a bug in the tests, you are asserting that the uploaded file url contains cdn.cloudfiles.rackspacecloud.com, my containers make urls like cnd1.cloudfiles.rackspacecloud.com.

I think we can just compare to cloudfiles.rackspacecloud.com, what do you think about? Can I update it in my fork?

Make configuration flexible

Currently we look explicitly in the settings files for a CUMULUS dict. It would be nice to give users the ability to choose from one of the following:

• Use a pyrax.cfg file
• Set env vars as defined by pyrax
• Use the CUMULUS dict

The idea here is that some users may be using swift for more than just django storage on a system. This would allow them to configure pyrax once using pyrax.cfg and be set for any pyrax-related use cases they have on their machines. We may also want to look at pyrax's support for environments and possibly set up a new setting to take advantage of that.

The syncstatic command don't care about gzip content types config.

It don't use SwiftclientStorageFile and, this way, don't compress gzippable files.

The collectstatic command is buggy.

There is a bug in the collectstatic management command. It imports CloudFilesStorage instead of SwiftClientStorage.

storage.py, line 292: "self._pos + num_bytes" fails when "num_bytes" is None

storage.py, line 292: "self._pos + num_bytes" fails when "num_bytes" is None

Document custom headers

~~>For security reasons Javascript running in a browser is not allowed to make requests to domains other than the one from where it came from. This is referred to as the Same Origin Policy. CORS is specification allowing browsers and application servers work out an agreement whereby these types of requests are allowed.

Swift 1.7.5 introduced CORS support. This means a Javascript application running in a browser and hosted outside of a Swift cluster can still query that cluster’s API. I expect to see lots of Swift based applications over the coming months thanks to this great new feature.

Taken from here. I have a django app that would definitely be making use of CORS if it was easy.~~

EDIT: looks like you've already done this. It would be nice if that was more apparent.

#46

HEADERS option not working

The HEADERS setting doesn't seem to work (they are not being sent at all) as documented with the following settings and the cumulus.storage.SwiftclientStorage:

CUMULUS = {
    'USERNAME': 'foo',
    'API_KEY': 'bar',
    'USE_SSL': False,
    'CONTAINER': 'spam',
    'PYRAX_IDENTITY_TYPE': 'rackspace',
    'SERVICENET': False,
    'HEADERS': (
        (r'.*\.(ttf|otf|eot|woff|svg)$', {
            'Access-Control-Allow-Origin': '*',
            'X-Container-Meta-Access-Control-Allow-Origin': 'http://localhost',
            'X-Container-Meta-Access-Control-Max-Age': 360
        }),
    ),
}

storage.py line 221 contains:

if CUMULUS["USE_PYRAX"]:
    # TODO set headers

Is that related and is this feature currently missing?

101, Network is unreachable

When cloudfiles is unreachable, it raises an Exception that 500's the page, creating a single point of failure.

I believe cumulus is the right place to catch this error and surpress it, because ImageKit is image backend agnostic, and cloudfiles is doing its role by alerting that the file is unreachable.

For the page to render without 500ing, like a HTML file that has an link to a nonexisting image, seems appropiate. Better yet, If the error able to be caught in the application code for those who want to do something fancier, such as falling back to serving from a different location, or using a default image, could be the ideal.

However, maybe you suggest that is something better done in the application code? Interested to hear others thoughts on this.

Here are the relevant bits from the traceback.

Environment:

Request Method: GET
Request URL: http://rizu.mu/

Traceback:
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/django/template/base.py" in _resolve_lookup
  683.                         current = getattr(current, bit)
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/imagekit/models.py" in url
  178.         self.generate()
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/imagekit/models.py" in generate
  191.         if lazy and (getattr(self, '_file', None) or self.storage.exists(self.name)):
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cumulus/storage.py" in exists
  143.             self._get_cloud_obj(name)
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cumulus/storage.py" in _get_cloud_obj
  93.         return self.container.get_object(name)
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cloudfiles/utils.py" in decorator
  45.             return f(*args, **kwargs)
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cloudfiles/container.py" in get_object
  317.         return Object(self, object_name, force_exists=True)
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cloudfiles/storage_object.py" in __init__
  91.             if not self._initialize() and force_exists:
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cloudfiles/storage_object.py" in _initialize
  525.                 'HEAD', [self.container.name, self.name])
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cloudfiles/connection.py" in make_request
  192.             response = retry_request()
File "/home/deploy/.virtualenvs/rizumu/lib/python2.7/site-packages/cloudfiles/connection.py" in retry_request
  185.             self.connection.request(method, path, data, headers)
File "/usr/lib/python2.7/httplib.py" in request
  955.         self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py" in _send_request
  989.         self.endheaders(body)
File "/usr/lib/python2.7/httplib.py" in endheaders
  951.         self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py" in _send_output
  811.         self.send(msg)
File "/usr/lib/python2.7/httplib.py" in send
  773.                 self.connect()
File "/usr/lib/python2.7/httplib.py" in connect
  1154.                                             self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py" in create_connection
  571.         raise err

Exception Type: error at /
Exception Value: (101, 'Network is unreachable')

File like behavior for SwiftclientStorage

Hi,

I use django-cumulus in several projects in pair with python-cloudfiles to fill easy_thumbnails.

Recently, I was updating a project and I noted the efforts on integration with pyrax e novaclient.

The new SwiftclientStorage class defines a read method that demands a chunk_size as argument.

easy_thumbnails don't have set a chunk_size, it assumes the storage class acts like a file (it uses the read method of the storage class, without arguments).

The prior version of cumulus deal with it but the new storage class don't.

It seems that pyrax.cf_wrapper.storage_object.StorageObject accepts a None value for chunk_size and in this case, returns the whole file, so, the fix would be simple set a default for chunk_size on read method.

I'm not an expert in this area :)

Do you have any reason to avoid it?

Gzipped files must be gunzipped when read.

I've added a way to gzip files depending of cumulus configuration. This works well to write files to the cloud and get it from CDN. As the files have a gzip content-encoding, browser will gunzip it transparently.

I forgot that we also may read these gzipped files and it should happen transparently as well.

ImageField error when using the width_field and height_field

When I define width_field and height_field on ImageField that under django-cumuls I get the following error:

Exception Type: AttributeError
Exception Value: 'Object' object has no attribute 'tell'
Exception Location: /home/webuser/public_html/apps/aml/parts/aml/django/core/files/utils.py in , line 22

When I remove the width_field and height_field attributes of the ImageField the upload works just fine.

From the bottom of the stack trace:
/home/webuser/public_html/apps/aml/parts/aml/django/core/files/utils.py in

1. tell = property(lambda self: self.file.tell)

Add option for creation of virtual directories

By default, CloudFiles stores everything flat - there is no concept of directories. However, you can create a directory-like setup using virtual directories with the content type application/directory. We should add a setting, something like CUMULUS['VIRTUAL_DIRS'], that allows the user to enable or disable the creation of the virtual directories.

See http://www.rackspace.com/cloud/blog/2010/01/26/nested-folders-in-rackspace-cloud-files/ and http://docs.rackspace.com/files/api/v1/cf-devguide/content/Pseudo-Hierarchical_Folders_Directories-d1e1580.html for more info.

Django cumulus relies in an specific pyrax version

Hi,

Is there any reason cumulus relies in the pyrax 1.2.6? The current version is 1.4.4 and there is cool improvements since.

Chunks file reading fails

If you call the Django standard method FileField.chunks(), django-cumulus returns an iterator that fetch only the first chunk of the file, repeating multiple times.

The error is in the read method of SwiftclientStorageFile:
https://github.com/richleland/django-cumulus/blob/master/cumulus/storage.py#L357

Error: cumulus.storage doesn't have CloudFilesStaticStorage class

Is it obsolete?

ImproperlyConfigured: Storage module "cumulus.storage" does not define a "OpenStackStaticStorage" class.

mixed storage between StaticFilesStorage and CloudFilesStorage causes 0 files collected

I have a situation where I want upoaded media to be stored in a CDN and staticfiles to be handles normally:

DEFAULT_FILE_STORAGE = 'cumulus.storage.CloudFilesStorage'
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'

having cumulus in my INSTALLED_APPS breaks python manage.py collectstatic by making the assumption that static files are being managed by cumulus. It causes an error where collectstatic collects 0 files.

Commenting out the cumulus app allows me to collectstatic like normal.

Fix tests

The tests need to run in the mainline before we can move on to the milestone 1.2

Creates a new class and connection every time

I'm attempting to use django-cumulus alongside sorl-thumbnail, image and thumbnail uploading works but page loading is very slow. A CloudFilesStorage instance is being created for every image on the page and creating a new connection each time.

Using a Borg style pattern for shared state solves these issues but may not be the best solution.