automatically tarball directories?

[coyotemarin]

automatically tarball directories?

[BrandonHaynes]

I have been running a (somewhat) specialized implementation of this feature that might be generalized for this purpose. Is this still an open issue in MRJob?

[coyotemarin]

(whoops, didn't mean to close that. wooo GitHub keyboard shortcuts)

Yes, this is still an open issue. We didn't do this because we didn't have a good use case. Would love to see your code.

[BrandonHaynes]

I'm currently archiving the specified files on runner instantiation and appending the resulting file(s) onto the python_archives list. Although I'd generally prefer to avoid mucking with the caller's passed-in values, this is an easy shortcut. Thoughts?

[coyotemarin]

Sorry for the slow response. Just to be totally clear, can you show me what your command line looks like and/or some sample code?

[coyotemarin]

Probably better to .zip directories, so they can play well with the py_files option (see #1375).

[coyotemarin]

Finally going to do this!

Just going to create tarballs (since even old versions of Hadoop support them) and not do any filtering of hidden files.