bobvanderlinden/sharpfilesystem

A virtual file system for .NET written in C#

SharpFileSystem

SharpFileSystem is a Virtual File System (VFS) implementation for .NET to allow access to different filesystems in the same way for normal files and directories.

Motivation

After looking a long time for a VFS for .NET, so that I could read files and directories in archives (zip and rar) the same way as any ordinary files and directories. I couldn't find any complete solution for this problem, so I decided to implement a VFS myself. Also what I didn't like in the ordinary filesystems was the path-system. It allowed relative paths (like ..), which can often lead to security issues without explicit checking. It allows referring to directories the same way as referring to files. This often leads to strange behavior, like copying a directory (source) to another directory (destination), here it is often vague whether the destination directory should be overwritten or if it should copy the source-directory inside the destination-directory.

Goals

• Using multiple types of filesystems (ie. zip, rar, ftp, etc) in the same way.
• A way to combine these systems to structure multiple parts of your program resources and configurations.
• A way to restrict or hide parts of your filesystem to parts of your program.
• A robust way of handling paths.

Features

At the moment the following filesystems are implemented:

• PhysicalFileSystem: gives access to the real operating system filesystem (uses System.IO of .NET/Mono).
• MemoryFileSystem: simulates a filesystem in-memory.
• SevenZipFileSystem: gives access to any 7-Zip supported archive.

There are also filesystems that alter the exposed structure of existing filesystems. These allow you to mix different systems together to get the desired file-structure:

• ReadOnlyFileSystem: Only allows read-operations on the wrapped filesystem.
• MergedFileSystem: Merges multiple filesystems into a single file-structure.
• FileSystemMounter: A unix-like mount system, which mounts other filesystems to a specified path.
• SubFileSystem: Allows you to create a filesystem from a directory of another filesystem and therefore restricts access to parent-directories.
• SeamlessSevenZipFileSystem: Allows access to 7-Zip archives as if they're directories.

Implementation

At the heart of the system there is the IFileSystem interface. This describes the operations that should be provided by every filesystem:

public interface IFileSystem : IDisposable
{
	ICollection<FileSystemPath> GetEntities(FileSystemPath path);
	bool Exists(FileSystemPath path);
	Stream CreateFile(FileSystemPath path);
	Stream OpenFile(FileSystemPath path, FileAccess access);
	void CreateDirectory(FileSystemPath path);
	void Delete(FileSystemPath path);
}

Normally in .NET/Mono we refer to files by their path coded as a string. This sometimes leads to inconsistencies, which is why I've created the type FileSystemPath to encapsulate the path in SharpFileSystem. There are operations that help create and alter the path, but the rules of using them are more strict, which creates a much more robust environment for the programmer:

• Independent of the OS, the directory-separator is always /.
• All paths always start with /. This means: there are no relative paths.
• All paths that refer to a directory end with /.
• All paths that refer to a file do not end with /.
• The path cannot be manipulated as a string, only new paths can be created through a small set of operations (like System.IO.Path.Combine for .NET).

Some examples of using FileSystemPath:

Operation	Result
var root = FileSystemPath.Root	/
var mydir = root.AppendDirectory("mydir")	/mydir/
var myfile = mydir.AppendFile("myfile")	/mydir/myfile
myfile.AppendFile("myfile2")	Error: The specified FileSystemPath is not a directory.
mydir.ParentPath	/

sharpfilesystem's Issues

transaction

pls add transaction support

How should I register movers?

I was playing about and came up with this code:

using SharpFileSystem;
using SharpFileSystem.FileSystems;
using System.IO;

class Program
{
    static void Main(string[] args)
    {
        var fs = new PhysicalFileSystem(".");
        var originalPath = FileSystemPath.Root.AppendFile("x");
        using (var x = fs.CreateFile(originalPath))
        using (var xWriter = new StreamWriter(x))
            xWriter.WriteLine("hi");
        fs.Move(originalPath, fs, FileSystemPath.Root.AppendFile("y"));
    }
}

However, it exits like this:

Unhandled Exception: System.ArgumentException: The specified combination of file-systems is not supported.
   at SharpFileSystem.FileSystemExtensions.Move(IFileSystem sourceFileSystem, FileSystemPath sourcePath, IFileSystem destinationFileSystem, FileSystemPath destinationPath) in C:\Users\ohnob\repos\sharpfilesystem\SharpFileSystem\FileSystemExtensions.cs:line 64
   at Program.Main(String[] args) in c:\users\ohnob\onedrive\documents\visual studio 2015\Projects\sharpfilesystemPlay\sharpfilesystemPlay\Program.cs:line 14

I see that there is a PhysicalFileSystemMover and I don’t get this error if I add this line:

        EntityMovers.Registration.AddLast(typeof(PhysicalFileSystem), typeof(PhysicalFileSystem), new PhysicalEntityMover());

But is this really something I should have to orchestrate manually—it feels like I’m writing boilerplate at that point? Is there a quick shortcut to loading all such available plugins, maybe using something like MEF? Could a small snippet be added to the README to demo some simple file operations to get people started in this framework?

On the other hand, I do like that this seems to be quite extensible. The pattern used by Move() is something I could use to easily add a safe, transactional Replace() without needing to alter sharpfilesystem’s code at all.

Async methods

It would be nice if the library supported async/await methods, with cancellation.

EnumerableCollection.ToArray() and .ToList() throw

Because EnumerableCollection.CopyTo() throws NotSupportedException, the standard Linq extensions .ToArray() and .ToList() will also throw because they internally use .CopyTo(). Is there any reason why .CopyTo() isn't supported?

By the way, I was pulling my hair out trying to come up with my own flavour of IFileSystem before I stumbled across this repo -- thanks for saving me lots of work :)

No Documentation

Hey, I was looking into using the VFS for my code but I am not really sure about how to use this code. Can you please send me some sample code usage at my email and I can actually document this so that other people don't face the same issue. Thanks

cloud file api and Projected File system are released with Win10

I think they can help you to improve the implement.

Atomic file operations: Replace, AtomicMove

It is a common pattern when updating a file to:

1. Write out the new version of the file in the same directory with a different name.
2. Rename this new file over the existing file using an atomic rename()/File.Move()/File.Replace().

This pattern allows the program to handle crashes (hardware power loss, process termination) without data corruption. The data in the destination file is either the previous version or the new version, not partially updated.

Atomic operations are often restricted. For example, it is not possible to atomically rename a file across filesystem boundaries. The programmer can generally only assume that a rename can be made atomic if the source and target files share the same directory.

One characteristic of methods representing atomic operations is that they fail without touching data when it is detected that the operation cannot be made atomic. See the documentation for File.Replace() for some examples. (Note that in .net, File.Move() will fallback to a non-atomic operation if the operation cannot be done atomically but it is generally safe to assume that renaming a file within a directory will always use the atomic codepath). So, in the sharpfilesystem universe, this would mean that no such thing as a StandardAtomicEntityMover should exist ;-). To support filesystems which wrap others such as FileSystemMounter, some sort of system for unwrapping before searching for operation implementations as discussed in #4 would help a lot.

Because sharpfilesystem already has a non-atomic Move(), I think the atomic variant should be named AtomicMove() to give us two new calls: AtomicMove() and AtomicReplace(). This would most closely imitate the existing .net APIs which seems to be what sharpfilesystem has done in the past. This way, the programmer can decide to call Move() when atomicity is less of a concern and AtomicMove() when the programmer wants the call to fail if atomicity can’t be guaranteed.

Providing atomic operations support out of the box and implementing them in at least PhysicalFileSytsem would enable existing programs using the transactional file replacement pattern to adopt sharpfilesystem. And then such programs could be more easily testable without needing to give them real filesystem access (my goal ;-)).

Let me know your thoughts and if I should try rigging up a PR for this. I have already messed around a bit with trying to do this sort of thing as an addon to sharpfilesystem rather than changing sharpfilesystem itself, but it’d be much easier/cleaner with internal support for unwrapping and I haven’t finished my original approach yet.

nuget

I'm interested in using this project, but would like a nuget package.
would you accept a pull request for a nuget package?

Any documentation and examples?

Hello,

Are there any documentation and examples?

Remove “off-topic” System.IO classes (SafeNetworkStream is actually dangerous-looking)

SafeNetworkStream seems to be quite unrelated to the core goal of this project. Also, it places itself in System.IO, meaning it will show up in Intellisense in such a way that the unsuspecting developer might thing it is a Microsoft-provided API. And, even worse, it does things like the following to make stuff “safe”:

try
{
    return _stream.Read(buffer, offset, count);
}
catch (IOException)
{
    return 0;
}
catch (ObjectDisposedException)
{
    return 0;
}

Almost always when an IOException would be thrown, the developer should be handling that in some way. For example, it signals that the connection was reset and that the handle needs to be cleaned up. Code written against streams will expect this behavior to mean that the end of file was reached and it will assume that the bytes read so far represents the entirety of the entity being read. Whereas an IOException is a sign that the connection was interrupted without the sender claiming it was done.

Also, encountering an ObjectDisposedException is the sign of a bug in the code which called Read() on the disposed/closed stream. If the caller can’t even figure out if it closed a Stream handle it has, it’s not keeping track of its state properly. That’s something you’d want thrown so that at least it could be logged or, when developing, crash the app so that you can investigate and fix the bug!

Other classes in the IO folder might actually be useful utilities, but they seem to be wrongly polluting the System.IO namespace and not really related to the purpose of sharpfilesystem. If any of them are actually required for sharpfilesystem to work, it would be better for them to either be marked internal or, if they are to support some of the addon archive assemblies, moved to a namespace that clearly marks them as for sharpfilesystem’s own use, such as namespace SharpFileSystem.Internal.IO.

.NET Standard

Is there any plan to migrate this library to .NET Standard so it can be used across all .NET implementations (.NET Core, .NET Desktop)?

Question: does SharpZipLibFileSystem support creating huge files?

I looked over the source code and it appears that SharpZipLibFileSystem stores files in memory when they are created (I'm not positive about that, because I didn't examine everything). Will this prevent files larger than RAM from being created?

Readme confuses

I came to this project page in order to find the way to access the .NET coded file system implementations via Windows Explorer (e.g. like OneDrive disk, network mapped disk, etc.)
This can enable developers to write "general purpose" VFS in .NET and use them in other apps (not only in .NET) just as regular folder.

Now the project is not a OS integrated file system, thus it is not the VFS. It is standardized file/directories interface for .NET and a set of its implementations.
That's why the readme confuses.
It took some time for me to figure out that OS integration bridge is missing, initially I though it is also included =) .

GetEntities: "Path not rooted" exception

When executing the code below with this zip, I get a ParseException ("Path is not rooted") from FileSystemPath.Parse. Am I using it wrong, or is this a bug?

        static void Main(string[] args)
        {
            using (var stream = System.IO.File.OpenRead("sample.zip"))
            using (var fs = SharpFileSystem.SharpZipLib.SharpZipLibFileSystem.Open(stream))
            {
                foreach (var e in fs.GetEntities(FileSystemPath.Root))
                {
                    Console.WriteLine(e.EntityName + " - " + e.Path);
                }
            }
            Console.ReadKey();
        }

SQLite-net?

Do I need to copy the .db3 files somewhere to the physical file system in order to use them with SQLite-net, or rather is it possible to use them directly over a virtual file system?

GetEntitiesRecursive method stack overflow possible?

It looks like one of the methods, GetEntitiesRecursive, uses recursion, which could potentially cause a stack overflow. Maybe it should be turned into an iterative method?