backend doesn't support absolute paths: Support for non local filesystem

[wq]

backend doesn't support absolute paths: Support for non local filesystem

[techdragon]

Because of the way this library leans on itertable, this is trickier than i expected. So far I've got to subclass/replace just as many parts of itertable as data_wizard so I'm probably going to package my final fix up like the separate data_wizard.sources app since otherwise it seems like I'd just be adding a lot of otherwise useless functionality to itertable to support data_wizard in order to get data_wizard to support django-storages.

[sheppard]

I have integrated Django Data Wizard with S3 before, but that was with very large (multi-GB) CSV files and some heavy customization to parallelize the import across several Lambda workers. For the simple case with django-storages, I think all that is needed is to get the file data from S3 into a BytesIO on the file attribute of a custom itertable class. Perhaps something like this:

# myapp/wizard.py

from itertable import ExcelFileIter  # or whichever format
from data_wizard.loaders import FileLoader

class ExcelS3Iter(ExcelFileIter):
    def load(self):
        response = s3.get_object(
            Bucket=self.bucket,
            Key=self.filename,
        )
        self.file = response['Body']  # I think this is already file-like

class S3Loader(FileLoader):
    def load_iter(self):
        return ExcelS3Iter(
            bucket=settings.AWS_STORAGE_BUCKET_NAME,
            filename=self.file.name,
        )

# myproject/settings.py
DATA_WIZARD = {
    'LOADER': 'myapp.wizard.S3Loader',
}

I believe no other customization of itertable should be necessary. That said, itertable exists primarily to support data_wizard - so anything to make that support better is a valid contribution IMO. The only caveat is that itertable shouldn't need to know anything about Django - so it could for example have an optional boto3 integration, but not django-storages specifically.

The goal with having itertable be its own library is to make it easier to test the file loading and parsing code in isolation, without worrying about the complexity introduced by data_wizard's task runner.

[sheppard]

Actually, I think a better approach in this specific case is to update itertable.load_file() to accept arbitrary file-like objects (wq/itertable@5a47f32). Then in data_wizard it's just a matter of passing the file object directly from the storage backend to itertable (4a63066).

I haven't tested this with django-storages specifically, but it should just work. If you would like to try it out before the next release, be sure to update both itertable and data_wizard to the latest development builds.

[sheppard]

This fix has been released.