sigurdsvela/layers

Keep all your files in one directory; on multiple drives 0.0

This project is still pre-pre-alpha, so if you somehow stumble onto this page:

• Use at your own discretion
• Feel free to help out by submiting a feature request, pull request, or an issue

Thank you, that was all; I'll now hand it over to the poorly written pre-pre-alpha documentation.

Layers

layers is a CLI tool that allows you create folder on multiple different drives, and have their content mirrored via symlinks, while being able to controll which drive contains the original one.

To me, this is especially usefull for my Plex Media setup. Where I want all media of the same type in the same directory, so they can be kept in the same library, but I dont necesarilly want aaalll the series I've downloaded on the same drive, but I still definetly want high bitrate video that I'm going to watch in the near future on one of the fast drives.

Usage

You create one or more layer to a layer set. Each layer in the has a level number, used to easier reference them in the future.

To create a new layer set with it's first layer run

$ layers new /path/to/first/layer

This will create a new layer, to a new layer set.

To create a second layer for the same layer set you can either run

$ cd /path/to/first/layer
$ layers new /path/to/second/layer

or, if you prefer

$ layers -l /path/to/first/layer new /path/to/second/layer

Every command is run with a layer as target. The layer target is automatically set to the whatever layer the current working directory is in. The new command, however, will create a new layer set if no layer target is found. If a layer target is found, it will create a new layer for that layer set.

Once you have done that, you can start filling the different layers with files you want in each of them. When you want to, you can run sync to create the symlinks between the directories

#cwd: /path/to/one/of/the/layers
$ layers sync

You can easily rename files, and move them between layers with the mv command

#cwd: /path/to/one/of/the/layers

# Move the file one level up
$ layers mv --up ./a/file

# Move the file to level 3
$ layers mv -l 3 ./a/file

# Move the file from a/file to new/path inside the same layer
$ layers mv ./a/file ./new/path

# Move the file from a/file to new/path and move down one layer
$ layers mv --down ./a/file ./new/path

The choises for the options are

• --up,       -u Up one level
• --down,     -d Down one level
• --top,      -t Top level
• --bottom,   -b Bottom level
• --level,  -l N To level n

layers's Issues

Simulate external drives with read/write speed cap in docker

Clean up LocalLayerPath.move function and [r/d]efine behaviour for edge cases

• File
  • Path in target is symlinked
• Dir
  • Path in target is symlinked
  • Directory in path exists. Merge? How to handle conflicts in this case. How to make sure we can roll back in case of an error

Out of sync conditions

• No file conflict in target layer, but conflict in another

What happens if you move a file, that is inside a directory, to a level where that directory itself is a symlink?

It should; Unlink and create the directory in the new level, move the file, and sync

Copy file to temprorary file before finishing

When moving a file, the copy process should copy the file to a temporary file until the copying process is complete. In case of an error, this makes it easier to spot and clean up.

The issue here was just that the LayerMoveProcess didnt know where the temporary file was, making it impossible to track the status of the copy. That should be easily solvable though.

Create much lager test directory structure

We need a much lager structure, with some much larger files, to properly test some of the functionality

Global Config

• LayerSet aliases for easier reference that using path to a layer
• Active Process Tracking

This should also allow us to remove the concept of "base layer", which is just confusing anyway. All layers contain some original files, and symlinks to other files. There is no conceptual "base layer"

Come to think of it, we should just remove the layers list from each layer. This should be in the user config. Each layer can instead list a UUID used to verify the layer in case of corruption.

`sync` should check for directories that are not linked, but can be

If a directory only contains symlinks, it should be linkable. The directory can be removed, and replace with a direct symlink to the same directory in one of the other layers.

This only hold if the filestructure is valid

Move shouldn't copy first if the destination is on the same drive

Kindof an edge case i guess, since i dont excpect anybody to ever have multiple layers on the same drive. But still.

´verify´ and ´repair´ commands

Remove the purge command and replace with a verify and repair command.
Both should verify/repair all files,links, and directories in all linked layers, in all layers (The config of a single layer should not be trusted to list all layers.

It should also verify the config file of all layers, making sure that each layer list all layers, and that no one i missing or listing extraneaus layers.

By defaults, the repair command should be interactive, giving the user a choice for the intervention for each problem, but with options to set those choices before running the command

Run layer changes in mv as background process with status tracking

Moving files from one drive to another can take a lot of time. The progress needs to be displayed.

While where at it, it should also copy the file first, and then remove the original when the copy has been validated.