MkdirAll and RemoveAll are missing from filesystem abstraction

[blang]

MkdirAll and RemoveAll are missing from filesystem abstraction

[pbennett]

Gotcha. It sounds like spf13/afero is more like what I'm looking for. Thanks for your time!

[blang]

It's important to keep the filesystem interface as small as possible, therefor both convenience methods are part of vfs and use Remove/Mkdir of the given filesystem:
https://godoc.org/github.com/blang/vfs#MkdirAll
https://godoc.org/github.com/blang/vfs#RemoveAll

[pbennett]

Doesn't this kind of defeat the purpose? If I can't substitute os.xxx calls with an implementer of vfs.Filesystem how does this help? So you think this:
someFS := vfs.OS()
...
someFS.Mkdir(xxx)
...
vfs.MkdirAll(someFS, xxx)

as opposed to:
os.Mkdir(xxx)
...
os.MkdirAll(xx)

Is proper? You've changed the interface (MkdirAll doesn't take a filesystem parameter) and this abstraction meant to be used for testing abstraction and productio use is now directly affecting the code to use something incompatible. I don't see the benefit here.

[blang]

Thanks for this, i thought you missed these methods in the main package.
Yep i'm aware of this issue and i discussed this already with others and it's basically about the balance between a small filesystem interface and compliance. Since the stdlib is not at all designed for creating a good abstract filesystem interface, no replacement like vfs will be a simple drop-in replacement i'm afraid.

The consens to create a good filesystem abstraction is to have an interface as small as possible to support multiple backends like memfs, zipfs and others but also provide the convencience methods like MkdirAll everybody is used to. It's a big problem to decide which methods should be included, for example Link, Symlink, Chmod are not included in the current interface but maybe should. On the other hand, most filesystems (if we are talking about abstract filesystems) do not support/need this features.

The current goal of this project is to provide a stable (but small) interface to the most common filesystem operations and support the most common and useful filesystems.

I'm already working on / thinking about more convenience for the user. One example might be to create a bigger interface in front of interfaces like BaseFilesystem, Linker, Chmoder (example names) to support every possible call through one interface, with any unsupported operation returning an error:

vfs.Create(fs,...)
vfs.OpenFile(fs,...)
vfs.Chmod(fs, ...)
vfs.Link(fs,...)

If you're interested please send further feedback

[pbennett]

If the intent is to abstract Go's filesystem calls (which aren't part of an official interface unfortunately - quite a lot isn't.. :<) then you need to wrap all of the file calls, not just a couple. Otherwise it defeats the purpose. I should be able to have a single filesystem variable that is either a vfs.OS() or vfs/memFS instance depending on whether I'm testing or not. I could define that as a global or pass it around using it as my new 'filesystem' interface.
If I have to change 'some' os.xxxx calls to instead call vfs.xxxx then the library isn't an abstraction at all, it's its own new library with its own interface to use instead of the stdlib.
At that point, it's no longer serving it's intended (IMO) purpose.

[blang]

I don't exactly get your point.
At any point in time you only need one variable, let's call it fs.
fs might be an vfs.OsFS or an vfs.memfs.MemFS ( but never os as in stdlib/os, reason for confusion?)
There's no difference between how you interact with these filesystems. The only difference is, that if you wan't to create recursive directories or use other convenience methods like Create, you can't call fs.MkdirAll() but have to call vfs.MkdirAll(fs, ...). But these calls are the same on every filesystem implementing the interface vfs.Filesystem. The stdlib package os does not implement any interface and therefor you need to use vfs.OsFS.
So you have a single variable containing your filesystem, which can be passed around.

Maybe you think of the advantage to use os as your filesystem variable instead of the import namespace os from stdlib, that's a interesting point but would not work either way since you need os.O_RDWR (e.g) constants from this package.

[pbennett]

If I want to change my code to use vfs, then I have to change 'some' os.xxx calls to use 'someVar.xxxx' with the same method signature as os.xxx - ie os.Mkdir.
(I would expect to do that for ALL os filesystem calls)
I also have to change 'some' os.xxxx calls to instead be vfs.xxxx(someVar, ...), ie: os.MkdirAll
That is not a proper abstraction of the os package filesystem calls.
So given my first message, I expect to translate this:
os.Mkdir(xxx)
os.MkdirAll(xxx)

to this:
someFS := vfs.OS()
...
someFS.Mkdir(xxx)
someFS.MkdirAll(xxx)

NOT to this:
someFS := vfs.OS()
...
someFS.Mkdir(xxx)
vfs.MkdirAll(someFS, xxx)

With that version, I think this is the worst of all worlds. The library just created more work for me, not less and has just created an entirely new api with new method signatures.

[blang]

Sry to hear that.
I honestly don't think that using the 3 calls to vfs.X currently is that much of a dealbreaker. I think more important are missing methods like Chmod but your experience might differ.
As i've written in my previous comment there might be a way around if i will introduce a bigger more complete interface. That would solve your problem only slightly since all calls will follow the format vfs.X(fs,...).
But it's pretty sure that the vfs.Filesystem interface will not include further convenience methods in the future, so someFS.MkdirAll will not exist.
All methods transfered to the vfs package are "convenience" methods which will be the same on every filesystem.

[pbennett]

Yes, your prior comment would be something valuable,
I guess I just don't understand your real goal here with what you currently have.
Trying to have a bare minimum interface is not the same as abstracting an existing interface, Clearly not all implementations will have equivalents but then they're either no-ops or errors when you call them. To instead define a 'new' interface that is the lowest common denominator, I just don't really understand that. I guess I'm coming at it purely from a testing perspective with wanting to test apis that make filesystem calls and not have them use the real filesystem at test time. So basically mocking it out but with the ability to also verify that I can 'read' back what was supposed to be written.

[blang]

It simple, vfs does not abstract the os.X Filesystem but a filesystem in general. Yap, i guess everthing has been discussed so far. You can watch this repo to get notified if something gets changed

[VojtechVitek]

This is very helpful thread for people comparing github.com/blang/vfs and github.com/spf13/afero. Thank you both! 👍