FileType support

[kvesteri]

FileType support

[Fiveside]

Django's FileField uses a couple different components to implement file storage:

• FileField: the actual column that stores files
• Storage: an object that represents a storage space (eg. Filesystem, Dropbox, etc.)
• File: a file-like object, with an unknown storage place.

Personally I think that File is redundant, and the interface should accept any file-like object, but django's implementation does allow for the istream and ostream distinction, as well as a guaranteed way to access file metadata (filesize, created time, etc.).

The Storage concept makes it persistence layer agnostic, but the interface has a bunch of things that make no sense. Here's my attempt at cleaning it up and making it more sqlalchemy-esque:

class CustomStorage(Storage):

    # Mandatory methods:

    def display_name(self, db_value):
        # Takes the value stored in the sqlalchemy database and returns
        # the user friendly name (usually the filename).
        # This allows for storage via complex systems like CMIS, where it
        # makes more sense to store an identifier in the database rather than
        # the filename.

    def save(self, name, file_like_object):
        # Takes the file like object and persists it to the storage layer.
        # Returns an identifier that can be saved in the database.

    def open(self, db_value):
        # Returns a file like object representing the value in the database.
        # Perhaps this method could take a mode (read/write/binary/append)
        # and return an istream/ostream?

    def delete(self, db_value):
        # Delete the file from the storage

    # Non-mandatory methods:

    def accessed_time(self, db_value):
        # Returns a datetime() object representing the last accessed time.

    def created_time(self, db_value):
        # Returns a datetime() object representing the created time.

    def modified_time(self, db_value):
        # Returns a datetime() object representing the modified time.

    def size(self, db_value):
        # Returns the size of the file in bytes.

It could be used in the construction of a column like so:

class Document(Base):
    # File with a custom storage layer
    custom_file = sa.Column(util.FileType(storage=CustomStorage))
    # File stored locally
    local_file = sa.Column(util.FileType(storage=FileSystemStorage(folder="/srv/media/")))
    # File stored in the database (postgres blob type?)
    db_file = sa.Column(util.DatabaseFileType)

Disclaimer: my usecase involves remote CMIS storage, so that is my focus when creating this.

Thoughts?

[kvesteri]

Looks very good. I think SQLAlchemy-Utils should only provide the basic storage interface. File storaging should be in another package which could provide drivers for different backends just as you suggested.

[mehcode]

This could get pretty big and perhaps deserves its own repo (for the storage layers) that depend on sqlalchemy-utils. Though a big package depending on something named sqlalchemy-utils for the interface just feels weird (but I suppose its fine).

Do you think sqlalchemy would be open to https://github.com/sqlalchemy-contrib/ or https://github.com/sqlalchemy-ext/ ? Mostly to collect all of these things under an org. Further perhaps adding (I'm not sure if its already there) something like flask.ext to sqlalchemy so people could get this project by from sqlalchemy.ext import utils. (Just random various ideas).

[kvesteri]

I haven't yet contacted Mike Bayer about these packages. I will soon though, maybe at that point I could ask about how he feels these extensions should be organized. I like the idea about sqlalchemy.ext, that's what I've been missing too :)

[Fiveside]

If we're going to split them up into different packages, then perhaps the file storaging package should contain the basic storage layer, file types, and common drivers (like the filesystem storage driver). SQLAlchemy-Utils would then provide specialized drivers (such as a CMIS or dropbox driver). That seems to be more in theme with what SQLAlchemy-Utils is providing. It would also remove the strange dependency that @mehcode was referring to.

[kvesteri]

Additional drivers should be extensions of this storage package - not part of utils. SQLAlchemy-Utils doesn't need to have dependency on this storage package and storage package doesn't need to have dependency on SQLAlchemy-Utils.

Let's make things pythonic. SQLAlchemy-Utils should provide an interface for the storage layer merely for documentation purposes. The storage libraries need not to actually subclass this interface. It just needs to provide certain methods.

Remember the python way: If it walks like duck and quacks like a duck, treat it like a duck.
:)

[Fiveside]

That works for me. I'll finish building it and submit a PR later.

[kvesteri]

I've been having discussions with @jpvanhal. This will be implemented using the wonderful silo package: https://github.com/jpvanhal/siilo 🚋

[kvesteri]

The silo package just got its initial release, so we can start implementing this. I thought that maybe its better to have FileModel instead of FileType. We could save a lot meta information about the files in this generic file model.

[kurtmckee]

Closing due to inactivity.