evanpurkhiser/dots

A portable tool for managing a single set of dotfiles in an organized fashion

⚠️ NOTE

I have rewrote this tool in go, you can still find the old python 2.x version of this tool on the releases/1.x branch. The readme below is still for the 1.x version of this tool!

Dots - A dotfile Management Tool

TODO: Add contents section here

In the desktop *nix world "dotfiles" are the life and blood behind the customization of your environment. Everything from custom bash prompts to window manager configurations, dotfiles define your work flow. Because of this, it has become a very common practice to keep your dotfiles in a git repository and even make the repository available on GitHub for the community to explore.

The Problem

Keeping your dotfiles in a git repository can prove to be rather difficult however. Because dotfiles can be spread all across the users $HOME the process of tracking these files can be tedious. A few strategies have emerged within the community that offers command over dotfiles:

• Keeping $HOME under git version control - This can be difficult and dangerous as every directory under your $HOME now appears to be in version control, so running a git command will never fail.

• Keeping the $XDG_CONFIG_HOME directory under version control and using environment variables and scripts to force programs to read their configurations from the $XDG_CONFIG_HOME directory - A great overall strategy that keeps dotfiles more organized than keeping them in $HOME.

• Keeping dotfiles in a directory under $HOME and writing an 'installation' script to symbolically link the files to their appropriate location in the users $HOME - While this method is rather robust, it can be difficult to manage a installation script specifically for your dot files, and may impose limits in how you organize your dotfiles.

However, none of these methods offer any sort of ability to manage dot files across multiple machines and environments. While you can use any one of these strategies to effectively manage dotfiles for a single machine, using the same dotfiles on another machine will be convoluted and cumbersome.

For example, say you manage a set of dotfiles for your desktop machine that includes configuration for bash, Vim, a window manager, and GTK theme settings. If you would like to use these dotfiles in a headless server environment then you would obviously not need your window manager and GTK configurations. Even worse, changes may need to be made to some of the configuration files for that specific environment.

The Solution

The dots utility aims to help make the installation, revision control, and organization of dot files easy and intuitive. The dots utility solves all of the problems that the solutions mentioned above solve while also facilitating the ability to maintain a single repository of dotfiles that can be installed on any machine you access and would like to have your environment configured in a snap.

dots offers the following features:

• Configuration groups
  When installing your dotfiles onto a new machine, dots offers you the ability to select a specific 'group' of dotfiles that you would like to have installed into that environment. By organizing your dotfiles into logical groups (such as 'machine' groups) it's possible to only install the dotfiles that are required by that environment.

• Cascading file structure
  By selecting multiple configuration groups there is the possibility that two groups both contain a dotfile with matching names. For example, if two configuration groups both contain a bashrc file then the dots utility will automatically merge these two files together. A special syntax is also offered to allow for one file to override another or for the cascading files to be merged into the files at specific points.

• Follows XDG Base Directory Standard
  The XDG Base Directory Standard specifies that all configuration files should be located in the $XDG_CONFIG_HOME directory. By default, this is where all configuration files and directories will be installed. While this does require a little extra work to ensure all programs read their configuration files from here it offers a much more organized view of user dotfiles.

• Post installation scripts per file
  It's possible to include a .install script with any specific dotfile or directory. This installation script will be executed any time the specific dotfile is installed. This is useful for situations where you need to symbolically link a configuration file into $HOME or if something needs to be done after installation (for example executing :BundleInstall for VIM).

These features are expanded on below.

Usage and configuration

NOTE: For a quick and easy way to get going managing your dot files more effectively see the dots-template repository that offers instructions for getting setup.

---

It's recommended to read through the entirety of this README to have a good understanding of how the dots management utility works. Here are a few key points to keep in mind however:

• Configuration files will be installed into $XDG_CONFIG_HOME.
• Configuration group directories are to be located in $HOME/.local/etc.
• A PKGBUILD file is also available for Arch Linux on the AUR.
• The dots binary should be made available in your PATH.
• The dots binary should support Python > 2.7 / Python > 3.2.
• See Evan Purkhisers personal dotfiles for an example configuration.

For details on using the dots tool itself see the dots help USAGE output.

The initialization bootstrap script

A initialization script is included in the contrib directory, providing an easy way to initalize your dotfiles into the proper directory, temporarily setup the PATH for the dots, and temporarily source the bash completion scripts. This way you can quickly setup your dotfiles, activate your configuration groups, and install the dotfiles themselves

For example:

$ cd ~
$ git clone https://github.com/Your/Dotfiles
$ DOTS_CLONE_DIR=~/Dotfiles
$ source dots/contrib/initialize

This will do the following:

1. Move the Dotfiles into $HOME/.local/etc
2. Symbolically link the dots executable into $HOME/.local/bin
3. Add $HOME/.local/bin into the PATH if it's not already
4. Source the contrib/bash_completion script

You can then setup your dotfiles using the dots command:

$ dots groups set base machines/desktop
$ dots install

Bash completion

A bash completion script is included and provides completions for all aspects of the dots command. If you would like to take advantage of this it's recommended that you source this file in your bashrc.

Configuration groups

The primary feature of the dots utility is to allow for dotfiles to be organized into different "Configuration groups". These configuration groups can then be enabled or disabled for the specific environment that the dotfiles are being installed into.

Configuration groups are two-level directories containing configuration files and directories that will be installed when the group is activated. The dots utility also includes a special configuration group that is hard coded into the utility: The base group is a single-level directory, so all files and directories located in the base directory will be installed if the base group is activated.

Extending and Overriding configurations

Configuration groups can override or extend files that are included in configuration groups specified prior to them.

Extending configuration files

If a configuration file in the base group specifies most of what is needed, but for the specific environment you're installing the configuration files into requires a little extra configuration for that file it is possible to append to it.

For example, if you would like to add more options to the bashrc for your machine/desktop group, you can simply include the bash/bashrc file and it will automagically be appended to the base/bash/bashrc file upon installation.

Shebangs will be removed from the first line of the file being appended.

Overriding

You can completely override a configuration file included in a previous group. This is similar to extending a configuration file as described in the previous section, however the file will simply replace the file specified in the environment group.

Enable overriding for a file by appending .override to the filename.

Installation Scripts

For each configuration file you may also include a .install script. This file will be executed when the specific configuration file has been installed. If the destination file has not been changed from the compiled file then the install script will not be executed.

The installation scripts will be executed with the destination directory as the current working directory. In order for the scripts to be executed, they must be executable and include a shebang.

For example: We have a base/vim/vimrc configuration file. We could also include a base/vim/vimrc.install file that executes some commands when the vimrc file is installed. The script will be executed with the $XDG_CONFIG_HOME/vim/ as the working directory.

dots's Issues

Emit a warning when no groups are set

When no groups are set, dots will happily run w/out error leaving the user to figure out why nothing got installed.

dots should emit a warning when there are no groups set for installation.

Move config-groups into ~/.config/dots/

At the moment, config-groups is in ~/.config/. It's not obvious that this file belongs to dots.

Since #6 will require more configuration be stored, config-groups should be moved into ~/.config/dots/.

Make SOURCE_DIR configurable

I don't like keeping my repository in ~/.local/etc/, especially since I have my own conventions for storing repositories that I frequently update.

SOURCE_DIR should be able to be set from both the command-line, and be read from a configuration file. (e.g. ~/.config/dots/source_dir)

V2 Ideas

I've had some ideas bouncing around in my head about a updated version of this tool that's hopefully just as powerful, but even easier to configure and more manageable. Here's some ideas I wanted to write down:

Configurations stored in the repository

Right now when you want to setup your dotfiles on a machine you have to decide all of what groups you want to be installed. If you have more than a few groups this can be annoying to manage. Why not put this part of the configuration in version control?

My thought would be to have a dots.yml (or json possibly) file in the root of your ~/.local/etc/ dotfiles clone that has some configuration for that set of dotfiles along with some other global configurations for the dotfile repository itself. My current configurations would look something like this

# Prefix to use for explicit append points
append_prefix: '!!@@'

# The file suffix used for override files
override_suffix: override

# Compilation installation path.
# Supports environment variable expansions using ${VARIBLE}
install_path: ${HOME}/.config

groups:
  - base
  - machines/desktop
  - machines/work-laptop
  - common/work-system

# Groups to compile for all profiles
base_groups: [ base ]

# Configuration profiles. Each configuration profile should map to a list of
# source paths to compile. Order defines the cascade order.
config_profiles:
  desktop:
    - common/graphical
    - common/pulseaudio
    - machines/desktop
  macbook:
    - common/osx-homebrew
    - machines/macbook
  work-macbook:
    - common/osx-homebrew
    - common/work-system
  work-vm:
    - common/work-system

Something like dots config set desktop would select the desktop configuration.

Configuration lockfile

Right now it can be a little frustrating to test some new configuration files only to remove them from your source later and then find that they're still cluttering your configuration installation directory. This can be especially annoying when they have some continued configuration side effect.

Inspired by library package managers I had the idea of a 'dots/lockfile' which would be installed with the compiled files.

{
    "profile": "desktop",
    "installed_files": [
        "/home/evan/.config/bash/bashrc"
        "/home/evan/.config/git/config",
        "/home/evan/.config/termite/config",
    ]
}

This will simply allow us to know what files were installed previously so that we can know what files are no longer needed and can be removed.

Other things

• Better verbose output. It should say exactly what it's doing.
• TESTING

Python 3

It'd be nice if dots worked w/ both Python 2 and 3.

When I try to run it:

$ python3 ./dots
name 'single_directory_groups' is not defined

Error on directory with the same name as a file + .extension

I have adopted Yan Pritzker's idea of separating vim settings in topical files under settings and a file settings.vim that calls them. Dots does not like this and complains that settings is a directory:

~$ DOTS_DEBUG=true dots install 
Traceback (most recent call last):
  File "/home/ing001/.local/bin/dots", line 602, in <module>
    config.install_tree(args.location, args.file, True)
  File "/home/ing001/.local/bin/dots", line 281, in install_tree
    [f.install(to, exec_install_scripts) for f in self.files(path_filters)]
  File "/home/ing001/.local/bin/dots", line 274, in files
    return [f for f in files if not f.is_named_fragment()]
  File "/home/ing001/.local/bin/dots", line 399, in is_named_fragment
    return match.group(2) in target_file.named_append_points()
  File "/home/ing001/.local/bin/dots", line 375, in named_append_points
    with open(path, 'r') as file:
IOError: [Errno 21] Is a directory: '/home/ing001/.local/etc/base/vim/settings'

~$ ls -l .local/etc/base/vim/
drwxr-xr-x 2  settings
-rw-r--r-- 1  settings.vim
-rwxr-xr-x  vimrc
-rwxr-xr-x  vimrc.install
drwxr-xr-x  vundles
-rw-r--r--  vundles.vim

I have worked around this by calling settings.vim for handle_settings.vim and the same with vundles.vim, but would prefer not to have to do that ;=) Is this a bug or a side effect of the way you handle merging different groups etc.?

python 2.7 support is broken

With the new implemented correct hashing of utf-8 files in commit d252369 the python 2.7 support gets broken, because the python 2.7 open command doesn't have a encoding argument.

Python version requirement

Hi,

Not neccessary an issue with the code, but maybe with the documentation.

When trying to execute the "dots help" or any other dots command I get warning about:

$ dots help
/usr/bin/env: python2.7: No such file or directory

Fixing the python pointer to my actual python version (2.6.6) yields:

$ dots help
Traceback (most recent call last):
File "~/.local/bin/dots", line 83, in
import argparse
ImportError: No module named argparse

So what I can read from this little "test" is that dots is not compatible with python < 2.7?

Maybe the requirements should be stated in the documentation? Or maybe I have misunderstood something and it should be compatible with python2.6 too?

Anyway, seems like a nice tool, Ill try it on some of my other system and see if it works there :) The system I had problem with is RHEL6.5

• Kristian -

Execute .install files after all configs have been installed

Instead of executing .install immediately upon installation of a configuration file, they should be executed after all files have been installed.

md5 checksum hash comparison borken between 2/3

Need to look into this.

Add .install files for directories

If a directory has any files updated in it then we should execute .install files for that directory.

Verbose and "dry-run" mode

Dots should have a verbose mode that tells you all the files it is about to copy and what processing (substitutions, installation scripts, etc) are being performed.

Following that, it'd be nice to have a "dry-run" mode where it tells you what will be copied, but does not actually copied.

Manage removed files

If we assume that the users source configuration directory is under source control ($HOME/.local/etc) then we can keep track of what global version of files have been installed. This would allow us to keep track of what files have been removed since the last configuration install.

So for example when calling the dots install command with no given path we could write a file called config-version to $XDG_CONFIG_HOME. Then during the next install we could look at that version and compare it with the current version of the source configuration files. We could then see what files have been removed and remove those from the installed dotfiles directory.

Ignore missing configuration groups with warning

When an enabled group is missing (the directory doesn't exist) then it should simply be ignored (and a warning should be displayed), instead of raising an error causing program termination.

Explicitly handle error, when .install isn't executable

Currently, when an .install-script that isn't executable is to be executed, the application fails with a cryptic:

[Errno 13]: Permission denied

(which is what subprocess.Popen throws here, when it can't run a script) That's a terrible error message to debug.

Instead I'd suggest to handle it either by a) attempting to chmod +x it manually or by b) exiting with a message that explains what script caused the error.

Force re-running of installation scripts

A '--force' flag for install, or something similar, that forces dots to rerun all installation scripts (*.install files).

At the moment, dots only will execute an installation script if the corresponding configuration file has changed.

Work out bash completion for OSX

Since OSX uses bash 3 by default some of our completion features fail with errors. We should see if we can work this out so it either works with bash 3, or is simply disabled unless the user is using bash 4.

There are also some small inconsistencies in GNU vs BSD utils that the completion script uses (mainly with the find command for caching file and group lists)